Postmortem: Broken releases 0.31.64

Incident Summary

On March 28th at 9:30PM, the docker build of NearBeach would crash during boot due to a missing Python library in the Dockerfile.Base.

Leadup

We released the initial vertical build of the NearBeach API in the hopes of getting feedback. This release included several foundation changes. 

Fault

1. Missed including the required python libraries within the "requirements.txt" files
2. Missed included the required python libraries in the "template_pyproject.toml"
3. Missed including new configuration within the "settings.py"

Impact

With both the missing libraries and configuration, NearBeach docker would not boot and crashed unexpectely.

Detection

When deploying the latest update to NearBeach production (we use NearBeach to develop NearBeach), we noticed the production environment fell over.

Response

We instantly investigated the reason why NearBeach went down and applied a fix

Recovery

We deployed another NearBeach release that night to fix the issues with 0.31.64

Root Cause

We are currently missing critical tests for the NearBeach Docker deployment. When introducing new functionality that relies on updated libraries or configuration, there are no automated checks in place to ensure a smooth deployment process.

Lessons Learned

We need to implement automated testing specifically around Docker deployments. This will allow us to catch missing dependencies or configuration changes before they reach production and cause downtime.