tag:status.bubble.io,2005:/historyBubble Status - Incident History2024-03-29T10:39:13-04:00Bubbletag:status.bubble.io,2005:Incident/203974172024-03-29T10:39:13-04:002024-03-29T10:39:13-04:00Issues with Main Bubble Cluster<p><small>Mar <var data-var='date'>29</var>, <var data-var='time'>10:39</var> EDT</small><br><strong>Resolved</strong> - Our systems are functional and we are closing out this incident.</p><p><small>Mar <var data-var='date'>29</var>, <var data-var='time'>09:32</var> EDT</small><br><strong>Investigating</strong> - We are investigating reports of issues with our systems.</p>tag:status.bubble.io,2005:Incident/203843332024-03-27T23:13:01-04:002024-03-27T23:13:01-04:00Issues with Main Bubble Cluster<p><small>Mar <var data-var='date'>27</var>, <var data-var='time'>23:13</var> EDT</small><br><strong>Resolved</strong> - Our systems are functional and we are closing out this incident.</p><p><small>Mar <var data-var='date'>27</var>, <var data-var='time'>23:12</var> EDT</small><br><strong>Investigating</strong> - We are investigating reports of issues with our systems.</p>tag:status.bubble.io,2005:Incident/203815712024-03-27T17:06:32-04:002024-03-27T17:06:34-04:00Dedicated customers are currently having trouble logging in.<p><small>Mar <var data-var='date'>27</var>, <var data-var='time'>17:06</var> EDT</small><br><strong>Resolved</strong> - Our systems are functional and we are closing out this incident.</p><p><small>Mar <var data-var='date'>27</var>, <var data-var='time'>16:27</var> EDT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Mar <var data-var='date'>27</var>, <var data-var='time'>16:20</var> EDT</small><br><strong>Investigating</strong> - Dedicated customers are currently having trouble logging in.</p>tag:status.bubble.io,2005:Incident/203534372024-03-25T15:30:35-04:002024-03-25T15:30:35-04:00Issues with Main Bubble Cluster<p><small>Mar <var data-var='date'>25</var>, <var data-var='time'>15:30</var> EDT</small><br><strong>Resolved</strong> - Our systems are functional and we are closing out this incident.</p><p><small>Mar <var data-var='date'>25</var>, <var data-var='time'>14:51</var> EDT</small><br><strong>Investigating</strong> - We are investigating reports of issues with our systems.</p>tag:status.bubble.io,2005:Incident/203399632024-03-23T20:50:00-04:002024-03-23T20:50:00-04:00Issues with Main Bubble Cluster<p><small>Mar <var data-var='date'>23</var>, <var data-var='time'>20:50</var> EDT</small><br><strong>Resolved</strong> - Our systems are functional and we are closing out this incident.</p><p><small>Mar <var data-var='date'>23</var>, <var data-var='time'>19:37</var> EDT</small><br><strong>Investigating</strong> - We are investigating reports of issues with our systems.</p>tag:status.bubble.io,2005:Incident/203276552024-03-22T10:45:00-04:002024-03-22T10:45:00-04:00Reports of issues with database triggers<p><small>Mar <var data-var='date'>22</var>, <var data-var='time'>10:45</var> EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'>22</var>, <var data-var='time'>08:52</var> EDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Mar <var data-var='date'>22</var>, <var data-var='time'>07:58</var> EDT</small><br><strong>Investigating</strong> - We have received a few reports of database triggers not working in development versions. We are investigating if this is widespread.</p>tag:status.bubble.io,2005:Incident/202720122024-03-15T14:46:26-04:002024-03-15T14:46:26-04:00Issues with Main Bubble Cluster<p><small>Mar <var data-var='date'>15</var>, <var data-var='time'>14:46</var> EDT</small><br><strong>Resolved</strong> - Our systems are functional and we are closing out this incident.</p><p><small>Mar <var data-var='date'>15</var>, <var data-var='time'>14:44</var> EDT</small><br><strong>Investigating</strong> - We are investigating reports of issues with our systems.</p>tag:status.bubble.io,2005:Incident/202719672024-03-15T14:39:50-04:002024-03-15T14:39:50-04:00Issues with Main Bubble Cluster<p><small>Mar <var data-var='date'>15</var>, <var data-var='time'>14:39</var> EDT</small><br><strong>Resolved</strong> - Our systems are functional and we are closing out this incident.</p><p><small>Mar <var data-var='date'>15</var>, <var data-var='time'>14:37</var> EDT</small><br><strong>Investigating</strong> - We are investigating reports of issues with our systems.</p>tag:status.bubble.io,2005:Incident/202719402024-03-15T14:34:16-04:002024-03-15T14:34:16-04:00Issues with Main Bubble Cluster<p><small>Mar <var data-var='date'>15</var>, <var data-var='time'>14:34</var> EDT</small><br><strong>Resolved</strong> - Our systems are functional and we are closing out this incident.</p><p><small>Mar <var data-var='date'>15</var>, <var data-var='time'>14:32</var> EDT</small><br><strong>Investigating</strong> - We are investigating reports of issues with our systems.</p>tag:status.bubble.io,2005:Incident/202284312024-03-12T18:28:54-04:002024-03-12T18:28:55-04:00Issues with Main Bubble Cluster<p><small>Mar <var data-var='date'>12</var>, <var data-var='time'>18:28</var> EDT</small><br><strong>Resolved</strong> - Our automated systems report that they were able to successfully restore service to normal. We are investigating the root cause of the outage.</p><p><small>Mar <var data-var='date'>12</var>, <var data-var='time'>18:19</var> EDT</small><br><strong>Update</strong> - We are still looking into the issue. What we know so far:<br /><br />* We are seeing a higher-than-usual number of 500 error codes using the API Connector to connect to external APIs. The errors seem transient; several apps that have reported problems are currently resolving<br />* Most of the confirmed bug reports involve connecting to Xano's API, and we are looking into the possibility that the issue is on their end, but haven't confirmed that yet</p><p><small>Mar <var data-var='date'>12</var>, <var data-var='time'>17:21</var> EDT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Mar <var data-var='date'>12</var>, <var data-var='time'>17:03</var> EDT</small><br><strong>Investigating</strong> - We are currently investing user reports of issues accessing APIs. This may be related to the brownout, but we are confirming</p>tag:status.bubble.io,2005:Incident/202252462024-03-12T17:22:44-04:002024-03-12T17:22:45-04:00Planned plugin maintenance<p><small>Mar <var data-var='date'>12</var>, <var data-var='time'>17:22</var> EDT</small><br><strong>Resolved</strong> - The brownout is now complete</p><p><small>Mar <var data-var='date'>12</var>, <var data-var='time'>09:59</var> EDT</small><br><strong>Investigating</strong> - “We’re conducting a planned brownout of the outdated plugin API v3. See here [https://forum.bubble.io/t/updating-plugins-in-your-bubble-app-for-2024/296705] for more info</p>tag:status.bubble.io,2005:Incident/202199272024-03-11T21:53:37-04:002024-03-11T21:53:37-04:00Issues with Main Bubble Cluster<p><small>Mar <var data-var='date'>11</var>, <var data-var='time'>21:53</var> EDT</small><br><strong>Resolved</strong> - Our automated systems report that they were able to successfully restore service to normal. We are investigating the root cause of the outage.</p><p><small>Mar <var data-var='date'>11</var>, <var data-var='time'>21:38</var> EDT</small><br><strong>Monitoring</strong> - Some logged out users were unable to log in or access the Bubble editor. We believe we have identified and fixed the problem (which was caused by a change to one of the 3rd party libraries our client-side code loads), and we are monitoring to confirm the situation is stable</p><p><small>Mar <var data-var='date'>11</var>, <var data-var='time'>19:49</var> EDT</small><br><strong>Investigating</strong> - Our automated systems have detected potential issues with the main Bubble cluster. We are currently investigating...</p>tag:status.bubble.io,2005:Incident/201792752024-03-06T15:41:12-05:002024-03-06T15:41:12-05:00Issues with Main Bubble Cluster<p><small>Mar <var data-var='date'> 6</var>, <var data-var='time'>15:41</var> EST</small><br><strong>Resolved</strong> - Our automated systems report that they were able to successfully restore service to normal. We are investigating the root cause of the outage.</p><p><small>Mar <var data-var='date'> 6</var>, <var data-var='time'>15:40</var> EST</small><br><strong>Investigating</strong> - Our automated systems have detected potential issues with the main Bubble cluster. We are currently investigating...</p>tag:status.bubble.io,2005:Incident/201781902024-03-06T13:40:18-05:002024-03-06T13:40:18-05:00Issues with Main Bubble Cluster<p><small>Mar <var data-var='date'> 6</var>, <var data-var='time'>13:40</var> EST</small><br><strong>Resolved</strong> - Our automated systems report that they were able to successfully restore service to normal. We are investigating the root cause of the outage.</p><p><small>Mar <var data-var='date'> 6</var>, <var data-var='time'>13:07</var> EST</small><br><strong>Investigating</strong> - Our automated systems have detected potential issues with the main Bubble cluster. We are currently investigating...</p>tag:status.bubble.io,2005:Incident/201768752024-03-06T10:20:14-05:002024-03-06T10:20:14-05:00Issues with Main Bubble Cluster<p><small>Mar <var data-var='date'> 6</var>, <var data-var='time'>10:20</var> EST</small><br><strong>Resolved</strong> - Our automated systems report that they were able to successfully restore service to normal. We are investigating the root cause of the outage.</p><p><small>Mar <var data-var='date'> 6</var>, <var data-var='time'>10:19</var> EST</small><br><strong>Investigating</strong> - Our automated systems have detected potential issues with the main Bubble cluster. We are currently investigating...</p>tag:status.bubble.io,2005:Incident/201622702024-03-04T21:11:00-05:002024-03-04T21:11:00-05:00Issues with Main Bubble Cluster<p><small>Mar <var data-var='date'> 4</var>, <var data-var='time'>21:11</var> EST</small><br><strong>Resolved</strong> - Our automated systems report that they were able to successfully restore service to normal. We are investigating the root cause of the outage.</p><p><small>Mar <var data-var='date'> 4</var>, <var data-var='time'>21:04</var> EST</small><br><strong>Investigating</strong> - Our automated systems have detected potential issues with the main Bubble cluster. We are currently investigating...</p>tag:status.bubble.io,2005:Incident/201609332024-03-04T20:59:09-05:002024-03-04T20:59:09-05:00Issues with Main Bubble Cluster<p><small>Mar <var data-var='date'> 4</var>, <var data-var='time'>20:59</var> EST</small><br><strong>Resolved</strong> - Our automated systems report that they were able to successfully restore service to normal. We are investigating the root cause of the outage.</p><p><small>Mar <var data-var='date'> 4</var>, <var data-var='time'>17:29</var> EST</small><br><strong>Investigating</strong> - Our automated systems have detected potential issues with the main Bubble cluster. We are currently investigating...</p>tag:status.bubble.io,2005:Incident/201608962024-03-04T18:45:22-05:002024-03-04T18:45:23-05:00Searches not updating in real time<p><small>Mar <var data-var='date'> 4</var>, <var data-var='time'>18:45</var> EST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'> 4</var>, <var data-var='time'>17:22</var> EST</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.bubble.io,2005:Incident/201161222024-02-29T18:44:39-05:002024-02-29T18:44:39-05:00Issues with Main Bubble Cluster<p><small>Feb <var data-var='date'>29</var>, <var data-var='time'>18:44</var> EST</small><br><strong>Resolved</strong> - Our automated systems report that they were able to successfully restore service to normal. We are investigating the root cause of the outage.</p><p><small>Feb <var data-var='date'>29</var>, <var data-var='time'>17:57</var> EST</small><br><strong>Update</strong> - We are currently investigating an issue with images not loading.</p><p><small>Feb <var data-var='date'>29</var>, <var data-var='time'>17:41</var> EST</small><br><strong>Investigating</strong> - Our automated systems have detected potential issues with the main Bubble cluster. We are currently investigating...</p>tag:status.bubble.io,2005:Incident/200728412024-02-25T17:17:58-05:002024-02-25T17:25:31-05:00Issues with Main Bubble Cluster<p><small>Feb <var data-var='date'>25</var>, <var data-var='time'>17:17</var> EST</small><br><strong>Resolved</strong> - We have determined the root cause of the issue and resolved it.</p><p><small>Feb <var data-var='date'>25</var>, <var data-var='time'>16:50</var> EST</small><br><strong>Identified</strong> - We have identified the source of the issue and are working on mitigation.</p><p><small>Feb <var data-var='date'>25</var>, <var data-var='time'>15:36</var> EST</small><br><strong>Investigating</strong> - We are investigating reports of scheduled API workflows not running to completion.</p>tag:status.bubble.io,2005:Incident/199920032024-02-15T09:04:09-05:002024-02-15T09:04:11-05:00Issues with Main Bubble Cluster<p><small>Feb <var data-var='date'>15</var>, <var data-var='time'>09:04</var> EST</small><br><strong>Resolved</strong> - Our automated systems report that they were able to successfully restore service to normal. We are investigating the root cause of the outage.</p><p><small>Feb <var data-var='date'>15</var>, <var data-var='time'>09:03</var> EST</small><br><strong>Investigating</strong> - Our automated systems have detected potential issues with the main Bubble cluster. We are currently investigating...</p>tag:status.bubble.io,2005:Incident/199293392024-02-07T06:41:45-05:002024-02-07T06:54:19-05:00Issues with Main Bubble Cluster<p><small>Feb <var data-var='date'> 7</var>, <var data-var='time'>06:41</var> EST</small><br><strong>Resolved</strong> - Our automated systems report that they were able to successfully restore service to normal. We are investigating the root cause of the outage.</p><p><small>Feb <var data-var='date'> 7</var>, <var data-var='time'>06:30</var> EST</small><br><strong>Identified</strong> - We have identified the root cause as being specific to one shard of our database. User applications belonging on that shard may be inaccessible. We are currently beginning remediation.</p><p><small>Feb <var data-var='date'> 7</var>, <var data-var='time'>05:17</var> EST</small><br><strong>Investigating</strong> - Our automated systems have detected potential issues with the main Bubble cluster. We are currently investigating...</p>tag:status.bubble.io,2005:Incident/199292482024-02-07T05:03:54-05:002024-02-07T05:03:54-05:00Issues with Main Bubble Cluster<p><small>Feb <var data-var='date'> 7</var>, <var data-var='time'>05:03</var> EST</small><br><strong>Resolved</strong> - Our automated systems report that they were able to successfully restore service to normal. We are investigating the root cause of the outage.</p><p><small>Feb <var data-var='date'> 7</var>, <var data-var='time'>05:03</var> EST</small><br><strong>Investigating</strong> - Our automated systems have detected potential issues with the main Bubble cluster. We are currently investigating...</p>tag:status.bubble.io,2005:Incident/198987552024-02-03T00:46:06-05:002024-02-03T00:46:06-05:00Issues with Main Bubble Cluster<p><small>Feb <var data-var='date'> 3</var>, <var data-var='time'>00:46</var> EST</small><br><strong>Resolved</strong> - Our automated systems report that they were able to successfully restore service to normal. We are investigating the root cause of the outage.</p><p><small>Feb <var data-var='date'> 3</var>, <var data-var='time'>00:46</var> EST</small><br><strong>Investigating</strong> - Our automated systems have detected potential issues with the main Bubble cluster. We are currently investigating...</p>tag:status.bubble.io,2005:Incident/198864872024-02-01T14:30:46-05:002024-02-01T14:30:46-05:00Issues with Main Bubble Cluster<p><small>Feb <var data-var='date'> 1</var>, <var data-var='time'>14:30</var> EST</small><br><strong>Resolved</strong> - Our automated systems report that they were able to successfully restore service to normal. We are investigating the root cause of the outage.</p><p><small>Feb <var data-var='date'> 1</var>, <var data-var='time'>13:17</var> EST</small><br><strong>Monitoring</strong> - A temporary fix has been implemented and we are monitoring the results, while we work on a longer-term approach</p><p><small>Feb <var data-var='date'> 1</var>, <var data-var='time'>12:19</var> EST</small><br><strong>Investigating</strong> - Our automated systems have detected potential issues with the main Bubble cluster. We are currently investigating...</p>