Resolved
Some content was showing as a white screen
For a period of 20 minutes starting at 4:49am EST, some users saw a white screen when accessing Celero content.
What happened?
We deployed a routine minor version update that includes several fixes. One of the dependencies was not deployed to production. Therefore some of our servers returned an error that resulted in a white screen for some users.
Who was affected?
Only some users were affected. As we run on multiple servers, the code is deployed gradually and immediately verified. As soon as we identified the issue, we started the process to revert to the previously deployed version which took several minutes to complete.
Once completed (20 min after the initial deployment), all of the services were back to normal.
What did we learn?
We followed our internal process of identifying the root causes so that issues like this don't repeat.
As a result, we will be making a few fixes:
1. We have made some changes to our staging deployment process to further ensure that dependencies get deployed together.
2. Unrelated to the issue, we found an opportunity to improve our monitoring and will be adding another monitor for more fine-grained internal alerting.
Resolved
·
24 Nov at 04:49am EST