1178874 - Replace Pontoon load and commit commands with a unified sync command

Status: RESOLVED FIXED

(Webtools Graveyard :: Pontoon, defect)

Description

[Osmose [:osmose, :mkelly]]

(Moving to Bugzilla from https://github.com/mozilla/pontoon/issues/35)

Outside of SVN, Pontoon doesn't currently do any conflict resolution; if someone commits strings directly to version control, Pontoon will overwrite them. This also crops up when two instances of Pontoon are pointing to the same repo (such as the staging and production Heroku instances, which used to both point at mozilla/pontoon-intro). Both instances get in a commit war as each overwrites the other.

# Why?
Pontoon stores the entire state of a project's strings in its database. When Pontoon commits its strings to VCS, it exports the database into language files and then commits them (if there are no changes, the commit for that locale fails or shows no changes).

When someone commits directly to version control, those changes don't hit the database before Pontoon commits its language files, meaning that Pontoon will overwrite the changes made.

# Possible Solution
Instead of exporting the database into language files, whenever someone makes a change that would affect the language files (approving or deleting strings), store that as a `Modification`. When running the automated commit job, start by downloading the latest version of the language files and reading them. Then go through each `Modification` and make the change to the required language file. Once done, save the language file back out and commit them.

This way, external edits are preserved, but a string is both changed externally and changed by Pontoon, Pontoon's change takes precedence. Conflicts in VCS are avoided as we start off by downloading the latest revision, and if we still hit a conflict we can just try again after a short delay (and eventually fail and email an admin if we continually fail to commit).

Comment 1

[Jobava]

Another way to do it would be for Pontoon to create patches from the changes and just patch the repo instead of committing the state of its memory to the repos. Would also be nice it conflicts arising from that are highlighted somwhow.

Comment 2

[Matjaz Horvat [:mathjazz]]

Here's what happened to Artem:
https://github.com/mozilla/pontoon/issues/41

Before the project was even in Pontoon, all English strings were submitted as Ukranian translations:
https://hg.mozilla.org/gaia-l10n/uk/rev/f05390cd0c3d

Then just recently, Artem started adding translations.

On the following commit cycle, translations available by then were committed:
https://hg.mozilla.org/gaia-l10n/uk/rev/2aab07dcc006

Roughly 10 minutes later, updated cycle happened and overwrote changes made after commit.

Because this is how we operate at the moment - VCS wins over Pontoon:
https://github.com/mozilla/pontoon/blob/master/pontoon/base/models.py#L480

When we fix this bug, this should no longer happen.

Comment 3

[Matjaz Horvat [:mathjazz]]

Anoter example, this time French:
http://viewvc.svn.mozilla.org/vc/projects/mozilla.com/trunk/locales/fr/mozorg/plugincheck.lang?sortby=date&r1=142926&r2=142925&pathrev=142926

Comment 4

[Artem Polivanchuk]

Attachment: Overwrite SVN changes

The same problem for Ukrainian. But now with SVN...

Comment 5

[Matjaz Horvat [:mathjazz]]

And another one:
http://viewvc.svn.mozilla.org/vc/projects/l10n-misc/trunk/snippets/ru/jul2015_b.lang

Due to new server infrastructure, we do the sync cycle like this:

1. Update files on the server from VCS (but don't store changes to the DB).
2. Commit DB state to VCS.
3. Update from VCS.

Obviously, this causes problems if translations are submitted to VCS, instead of Pontoon. E.g. what happened for projects in #c3, #c4 and #c5 after translations were submitted to VCS is this:

1. Files on the server are updatded, translations are not in the DB.
2. State from the DB gets committed, removing the translations from VCS.
3. No translations get imported from VCS, because they were erased in 2).

Working on a fix...

Comment 6

[Matjaz Horvat [:mathjazz]]

Issues reported in comments #c3 to #c5 should now be fixed:
https://github.com/mozilla/pontoon/commit/8c68b1b9c4f8290e555f1152de3ef7b54452ca4d

We should probably have a separate bug for them. But OTOH it's good to keep track of those here to check if things work as expected after we make changes to the sync part.

Comment 7

[Théo Chevalier]

Another one in French, in case it helps tracking down the issue http://viewvc.svn.mozilla.org/vc/projects/l10n-misc/trunk/firefoxtiles/fr/tiles_jul2015.lang?r1=143215&r2=143214&pathrev=143215

Comment 8

[[github robot]]

Commit pushed to master at https://github.com/mozilla/pontoon

https://github.com/mozilla/pontoon/commit/58dc759e493a4fefefa5f7ad90b4c31674e1e12b
Bug 1178874: Add sync_projects management command.

Eventually sync_projects will replace commit_projects and
update_projects. For now it only supports langfiles.

Comment 9

[[github robot]]

Commits pushed to master at https://github.com/mozilla/pontoon

https://github.com/mozilla/pontoon/commit/fdd53dd9462e7c1f3fe60c42f4231946de051cd0
Bug 1178874: Fall back to string when entity key is empty.

When constructing a unique key for entities during sync, if Entity.key is blank,
we should use Entity.string as the key value instead.

https://github.com/mozilla/pontoon/commit/db7ef84810fc820d37e221e9e4005d9593aaa9e6
Merge pull request #75 from Osmose/fix-sync-key

Bug 1178874: Fall back to string when entity key is empty.

Comment 10

[[github robot]]

Commits pushed to master at https://github.com/mozilla/pontoon

https://github.com/mozilla/pontoon/commit/dca130e3d453c49a41be22711b505550a2aa05ed
Bug 1178874: Handle paths properly in Project path methods.

A subtle bug in the path methods on the Project model caused paths to be
incorrect for projects that do not have locale directories at the root of their
repo. We fix this bug and also simplify the path methods based on how they're
used.

https://github.com/mozilla/pontoon/commit/ec9cc114d78a99537f6dde48f5a26690a61590b5
Merge pull request #74 from Osmose/fix-sync-bad-paths

Bug 1178874: Handle paths properly in Project path methods.

Comment 11

[[github robot]]

Commit pushed to master at https://github.com/mozilla/pontoon

https://github.com/mozilla/pontoon/commit/7495ab4436452de88665a9a651faef40183ecd02
Bug 1178874: Add tests for sync_projects.py.

Comment 12

[[github robot]]

Commit pushed to master at https://github.com/mozilla/pontoon

https://github.com/mozilla/pontoon/commit/29d7c228b64f0878c0edb30689c846c6784c1f16
Bug 1178874: Add support for pofiles to sync_projects.

Comment 13

[[github robot]]

Commits pushed to master at https://github.com/mozilla/pontoon

https://github.com/mozilla/pontoon/commit/9617cb05fffa1f335dd2d55301a555609730985b
Bug 1178874: Add xliff support to sync_projects.

Also refactors the pontoon.base.formats tests to be reusable between formats.

https://github.com/mozilla/pontoon/commit/cc0c0f0075056d2ff47db3136aaedd9897107314
Merge pull request #110 from Osmose/xliff

Bug 1178874: Add xliff support to sync_projects.

Comment 14

[[github robot]]

Commit pushed to master at https://github.com/mozilla/pontoon

https://github.com/mozilla/pontoon/commit/68026afaa7f86e1909b4ac5b29391571d94e3d17
Bug 1178874: Add silme support for sync_projects.

Adds support for the ini, dtd, and properties formats.

Comment 15

[Matjaz Horvat [:mathjazz]]

Just to let everyone watching the bug know, we are deploying these fixes constantly. For most of the projects (35 out of 40), changes to VCS don't get overwritten by Pontoon anymore.

It can still happen for the following 5 projects:

https://pontoon.mozilla.org/projects/firefox-aurora/
https://pontoon.mozilla.org/projects/firefox-for-android-aurora/
https://pontoon.mozilla.org/projects/lightning-aurora/
https://pontoon.mozilla.org/projects/thunderbird-aurora/
https://pontoon.mozilla.org/projects/firefox-health-report/

But we'll fix those, too. Soon.

Comment 16

[[github robot]]

Commit pushed to master at https://github.com/mozilla/pontoon

https://github.com/mozilla/pontoon/commit/1d2586c0bf49550abd933d6896a8e39bc13192e3
Bug 1178874: Add .inc support to sync_projects.

Comment 17

[Osmose [:osmose, :mkelly]]

mathjazz: Are there any projects still on the old sync process? If not, should we mark this bug as resolved and file a new bug for removing the old sync code?

I'm adding bug 1211271 as a blocker, if there's anything else stopping us from moving a project over to the new sync process we should add it as a blocker as well.

Comment 18

[Matjaz Horvat [:mathjazz]]

No, we only use new sync now! You can close this one. Excellent work!

Comment 19

[Osmose [:osmose, :mkelly]]

Yesssssssssss.