All databases have been fixed and are not at risk of recurrence. We are still receiving reports of infinite refresh banners, which should be resolved by a deploy to the site and clearing cache. If this does not resolve, please create a bug report. We are passing this remaining piece to the next shift of engineers and will provide another update soon. In the mean time, we will close this original incident.
Posted Aug 06, 2024 - 07:58 EDT
Update
Editor and runmode should be working properly on the shared environment. We are wrapping up a few fixes on dedicated environments.
Post-upgrade, our metrics for the shared database are not looking healthy. We believe this is due to cache warming etc, but we are monitoring and will validate/invalidate.
Posted Aug 06, 2024 - 06:18 EDT
Update
The current status is that services are up except for the editor on the shared environment, and a few dedicated instances. We currently expect all of these to be up within 1 hour from now, and will provide another update in the next 30 minutes.
The auto-maintenance was forcing our database version up 5 whole versions, which is incompatible with our code. We got ahead of these by manually doing upgrades to 3 versions ahead for all of our databases, which is a database version supported and compatible with our code. Unfortunately we did not get to our AppServer database in time, which is the database that powers the editor. This means that runmode should work fine, but the editor may not load properly. We advise not making any changes in the editor if it can be helped, as the changes may be lost.
Posted Aug 06, 2024 - 05:31 EDT
Identified
Our main services are currently down on the shared environment and at least one dedicated environment. The issue currently appears to be tied to an errant auto-upgrade by Amazon of our databases which is incompatible with our code. We have confirmed auto-update is turned off. We have a SEV1 ticket open with Amazon and are looking to prevent this issue from cascading to further dedicated accounts.
Posted Aug 06, 2024 - 04:32 EDT
Investigating
We are investigating reports of issues with our systems.
Posted Aug 06, 2024 - 04:05 EDT
This incident affected: Bubble Core (Main Bubble Environment).