Degraded performance with Scheduled tasks on a portion of the apps in the main cluster

Incident Report for Bubble

Resolved

During the 4-6AM ET time frame, a fraction of our users in the main cluster experienced degraded performance on Scheduled tasks, and to a lesser extent on general operations.

This manifested as "App too busy" timeout errors and capacity spikes on operations which generally worked nominally.

We are still investigating parts of the root cause, but found behavior in a few key apps that exacerbated the problem, which allowed us to apply some mitigations for the time being, which let the performance return to normal.

Posted May 31, 2023 - 04:00 EDT