We were able to resolve this issue

One of our database shards affecting a subset of user apps on the main cluster got into a bad state, we have bounced it to recover and are monitoring to see if we are in the clear.

We are continuing to work on a fix for this issue.