1383271 - [sync] Protect against DB inconsistencies If Heroku dyno restarts during sync

Status: RESOLVED FIXED

(Webtools Graveyard :: Pontoon, enhancement, P3)

Description

[Matjaz Horvat [:mathjazz]]

Heroku dynos restart at least once per day:
https://devcenter.heroku.com/articles/dynos#restarting

If the worker restarts during sync, it can cause inconsistencies between the DB and the repository. We already have some protection against this situation:

Each project sync cycle consists of two tasks:
#1 - task for syncing source strings
#2 - task for syncing translations (triggered by #1)

When #1 completes, we set last synced revision to the project's source repository. If the task is interrupted by the dyno restart, last synced revision still points to the revision before the sync, so the task will be repeated in the next cycle.

#2 also stores last synced revision to the repository. Additionally, we wrap all DB & VCS operations into a transaction, but we do so for each locale separately, since we also push commits per locale.

However, there's still at least one case we don't have a protection against. If a new file (Resource) is added in #1, we pass that information to #2, in which we create the corresponding localized file representation (TranslatedResource):
https://github.com/mozilla/pontoon/blob/master/pontoon/sync/core.py#L149

TranslatedResource enables the file for each locale in Pontoon interface. We cannot create this object during #1, because the information on what locale should the file be enabled for is retrieved from the VCS for some (e.g. dotlang-based) projects. And we only reliably have all locale repositories available in #2.

Now if #2 dies before we create TranslatedResource objects, they won't be recreated in the next sync cycle, because #1 will no longer detect the file as new an pass the information over to #2.

Note that strings and stats should always remain in sync thanks to last_synced_revisions and ChangedEntityLocale objects. TranslatedResource seems to be the only unprotected piece in the puzzle.

--

This is not theory, it can actually happen. A new file was added to Firefox recently:
https://hg.mozilla.org/users/m_owca.info/firefox-central/rev/b7e49dde48f1

And the sync got interrupted by a dyno restart:
https://pontoon.mozilla.org/sync/log/28359/

The source file and the string were correctly stored in the DB during Task #1, but the file wasn't exposed for locales to be translated.

Comment 1

[Vishal Sharma [:vishalcr7]]

Attachment: Link to GitHub pull-request: https://github.com/mozilla/pontoon/pull/1607

Comment 2

[Vishal Sharma [:vishalcr7]]

Fixed as a part of bug 1609870

Now, the sync cycle consists of a single task where we pull the repositories only once. We set the last_synced_revisions close to the end of sync_translations() which ensures that TranslatedResource objects are created before we set last_synced_revisions for source repository.