Migration Bug: Post-Mortem
Bug Description
During deployment the run_migrations_online() was failing due to a missing revision. This was causing the API service to crash every couple of seconds.
Timeline
During development, some revisions were created (1)
The changes were pushed to remote
Some time passed and the changes were deployed on production (by accident)
After aligning with the team on a different solutions, initial revisions (1) were deleted, and new revisions (2) were created. Changes were pushed to remote and deployed.
API service was crashing with a lot of errors
Root Cause Analysis
The cause of the API crashing was the missing revision. The reason of the missing revision was deploying the wrong change, so it was mainly an issue created by a user.
Lessons Learned
First and foremost, if at any point a revision was pushed to the remote repository, it should never be deleted. Secondly, the current approach of upgrading the databases on every deployment is not an optimal solution as it might cause similar problems in the future.
Bugfix
In order to fix the issue, it was necessary to add all the missing revisions (1) and redeploy with these revisions present.
Closure
The severity of this bug was large, yet the cause of it was somewhat banal. This means that there needs to be something that has to change with the current approach of upgrading the databases. As follow-up a link to the documentation of the new implementation will be added here.