Postmortem: Broken releases 0.31.64
Incident Summary
On March 28th at 9:30PM, the docker build of NearBeach would crash during boot due to a missing Python library in the Dockerfile.Base.
Leadup
We released the initial vertical build of the NearBeach API in the hopes of getting feedback. This release included several foundation changes.
Fault
- Missed including the required python libraries within the "requirements.txt" files
- Missed included the required python libraries in the "template_pyproject.toml"
- Missed including new configuration within the "settings.py"
Impact
With both the missing libraries and configuration, NearBeach docker would not boot and crashed unexpectely.
Detection
When deploying the latest update to NearBeach production (we use NearBeach to develop NearBeach), we noticed the production environment fell over.
Response
We instantly investigated the reason why NearBeach went down and applied a fix
Recovery
We deployed another NearBeach release that night to fix the issues with 0.31.64
Root Cause
We are currently missing critical tests for the NearBeach Docker deployment. When introducing new functionality that relies on updated libraries or configuration, there are no automated checks in place to ensure a smooth deployment process.
Lessons Learned
We need to implement automated testing specifically around Docker deployments. This will allow us to catch missing dependencies or configuration changes before they reach production and cause downtime.