1332000 - test that downgrades actually work

Status: RESOLVED FIXED

(Release Engineering Graveyard :: Applications: Balrog (backend), defect, P3)

Description

[[e-mail] (:bhearsum)]

In theory, we support downgrades of the Balrog database through sqlalchemy-migrate. In practice we've never done one, but it's something that could become important suddenly if we deploy a bad change to production.

We should have unit tests that verify that downgrades actually work, like we do for upgrades. We wouldn't be able to compare db.py to the migrated version, but perhaps we upgrade one database to version X, another to version Y, downgrade it to version X, and make sure they match?

Comment 1

[A.S.]

SqlAlchemy migrate has a feature of testing the change script. Link : http://sqlalchemy-migrate.readthedocs.io/en/v0.7.2/versioning.html#test-the-change-script .
I'll read up on it and try implementing it.

Comment 2

[A.S.]

The sqlachemy-migrate module provides an external api to the versioning system. The *test* API (https://github.com/lugensa/sqlalchemy-migrate/blob/master/migrate/versioning/api.py#L205)  can be invoked to test that upgrades and downgrades work properly in that specific order.  

However, the drawback here is that the database must be exactly one version behind the repository version as explained here (https://code.google.com/archive/p/sqlalchemy-migrate/issues/41). So if repository is at current version say x, then we can only test x+1 version.

Comment 3

[A.S.]

Update: Also the 'test' API makes direct changes to the production DB. So before testing, we should first to create a copy of the production DB and then run tests on the copied DB. Since the prod DB uses mySQL and test DB uses sqllite, it creates a doubt that whether tests running and passing on sqllite DB will also successfully downgrade a mysql relation without any errors or not.

Comment 4

[[github robot]]

Commit pushed to master at https://github.com/mozilla/balrog

https://github.com/mozilla/balrog/commit/d8167ddd70f4dea0cfd246bb6a0167ef74a33400
Bug 1332000 - test that downgrades actually work (#263). r=bhearsum

Comment 5

[[e-mail] (:bhearsum)]

This is in production. This was a very tricky one, and you did a great job Ashish - thank you!