Replace Pontoon load and commit commands with a unified sync command

RESOLVED FIXED

Status

Webtools
Pontoon
--
major
RESOLVED FIXED
3 years ago
3 years ago

People

(Reporter: mkelly, Assigned: mkelly)

Tracking

Dependency tree / graph

Firefox Tracking Flags

(firefox42 affected)

Details

Attachments

(1 attachment)

(Assignee)

Description

3 years ago
(Moving to Bugzilla from https://github.com/mozilla/pontoon/issues/35)

Outside of SVN, Pontoon doesn't currently do any conflict resolution; if someone commits strings directly to version control, Pontoon will overwrite them. This also crops up when two instances of Pontoon are pointing to the same repo (such as the staging and production Heroku instances, which used to both point at mozilla/pontoon-intro). Both instances get in a commit war as each overwrites the other.

# Why?
Pontoon stores the entire state of a project's strings in its database. When Pontoon commits its strings to VCS, it exports the database into language files and then commits them (if there are no changes, the commit for that locale fails or shows no changes).

When someone commits directly to version control, those changes don't hit the database before Pontoon commits its language files, meaning that Pontoon will overwrite the changes made.

# Possible Solution
Instead of exporting the database into language files, whenever someone makes a change that would affect the language files (approving or deleting strings), store that as a `Modification`. When running the automated commit job, start by downloading the latest version of the language files and reading them. Then go through each `Modification` and make the change to the required language file. Once done, save the language file back out and commit them.

This way, external edits are preserved, but a string is both changed externally and changed by Pontoon, Pontoon's change takes precedence. Conflicts in VCS are avoided as we start off by downloading the latest revision, and if we still hit a conflict we can just try again after a short delay (and eventually fail and email an admin if we continually fail to commit).
(Assignee)

Updated

3 years ago
Assignee: nobody → mkelly

Comment 1

3 years ago
Another way to do it would be for Pontoon to create patches from the changes and just patch the repo instead of committing the state of its memory to the repos. Would also be nice it conflicts arising from that are highlighted somwhow.
Here's what happened to Artem:
https://github.com/mozilla/pontoon/issues/41

Before the project was even in Pontoon, all English strings were submitted as Ukranian translations:
https://hg.mozilla.org/gaia-l10n/uk/rev/f05390cd0c3d

Then just recently, Artem started adding translations.

On the following commit cycle, translations available by then were committed:
https://hg.mozilla.org/gaia-l10n/uk/rev/2aab07dcc006

Roughly 10 minutes later, updated cycle happened and overwrote changes made after commit.

Because this is how we operate at the moment - VCS wins over Pontoon:
https://github.com/mozilla/pontoon/blob/master/pontoon/base/models.py#L480

When we fix this bug, this should no longer happen.

Comment 4

3 years ago
Created attachment 8629959 [details]
Overwrite SVN changes

The same problem for Ukrainian. But now with SVN...
And another one:
http://viewvc.svn.mozilla.org/vc/projects/l10n-misc/trunk/snippets/ru/jul2015_b.lang

Due to new server infrastructure, we do the sync cycle like this:

1. Update files on the server from VCS (but don't store changes to the DB).
2. Commit DB state to VCS.
3. Update from VCS.

Obviously, this causes problems if translations are submitted to VCS, instead of Pontoon. E.g. what happened for projects in #c3, #c4 and #c5 after translations were submitted to VCS is this:

1. Files on the server are updatded, translations are not in the DB.
2. State from the DB gets committed, removing the translations from VCS.
3. No translations get imported from VCS, because they were erased in 2).

Working on a fix...
Severity: normal → major
Issues reported in comments #c3 to #c5 should now be fixed:
https://github.com/mozilla/pontoon/commit/8c68b1b9c4f8290e555f1152de3ef7b54452ca4d

We should probably have a separate bug for them. But OTOH it's good to keep track of those here to check if things work as expected after we make changes to the sync part.

Comment 8

3 years ago
Commit pushed to master at https://github.com/mozilla/pontoon

https://github.com/mozilla/pontoon/commit/58dc759e493a4fefefa5f7ad90b4c31674e1e12b
Bug 1178874: Add sync_projects management command.

Eventually sync_projects will replace commit_projects and
update_projects. For now it only supports langfiles.

Comment 9

3 years ago
Commits pushed to master at https://github.com/mozilla/pontoon

https://github.com/mozilla/pontoon/commit/fdd53dd9462e7c1f3fe60c42f4231946de051cd0
Bug 1178874: Fall back to string when entity key is empty.

When constructing a unique key for entities during sync, if Entity.key is blank,
we should use Entity.string as the key value instead.

https://github.com/mozilla/pontoon/commit/db7ef84810fc820d37e221e9e4005d9593aaa9e6
Merge pull request #75 from Osmose/fix-sync-key

Bug 1178874: Fall back to string when entity key is empty.

Comment 10

3 years ago
Commits pushed to master at https://github.com/mozilla/pontoon

https://github.com/mozilla/pontoon/commit/dca130e3d453c49a41be22711b505550a2aa05ed
Bug 1178874: Handle paths properly in Project path methods.

A subtle bug in the path methods on the Project model caused paths to be
incorrect for projects that do not have locale directories at the root of their
repo. We fix this bug and also simplify the path methods based on how they're
used.

https://github.com/mozilla/pontoon/commit/ec9cc114d78a99537f6dde48f5a26690a61590b5
Merge pull request #74 from Osmose/fix-sync-bad-paths

Bug 1178874: Handle paths properly in Project path methods.
Depends on: 1183376

Comment 13

3 years ago
Commits pushed to master at https://github.com/mozilla/pontoon

https://github.com/mozilla/pontoon/commit/9617cb05fffa1f335dd2d55301a555609730985b
Bug 1178874: Add xliff support to sync_projects.

Also refactors the pontoon.base.formats tests to be reusable between formats.

https://github.com/mozilla/pontoon/commit/cc0c0f0075056d2ff47db3136aaedd9897107314
Merge pull request #110 from Osmose/xliff

Bug 1178874: Add xliff support to sync_projects.

Comment 14

3 years ago
Commit pushed to master at https://github.com/mozilla/pontoon

https://github.com/mozilla/pontoon/commit/68026afaa7f86e1909b4ac5b29391571d94e3d17
Bug 1178874: Add silme support for sync_projects.

Adds support for the ini, dtd, and properties formats.
(Assignee)

Updated

3 years ago
Summary: Pontoon overwrites changes made in VCS → Replace Pontoon load and commit commands with a unified sync command
Just to let everyone watching the bug know, we are deploying these fixes constantly. For most of the projects (35 out of 40), changes to VCS don't get overwritten by Pontoon anymore.

It can still happen for the following 5 projects:

https://pontoon.mozilla.org/projects/firefox-aurora/
https://pontoon.mozilla.org/projects/firefox-for-android-aurora/
https://pontoon.mozilla.org/projects/lightning-aurora/
https://pontoon.mozilla.org/projects/thunderbird-aurora/
https://pontoon.mozilla.org/projects/firefox-health-report/

But we'll fix those, too. Soon.
(Assignee)

Updated

3 years ago
Blocks: 1208536
(Assignee)

Comment 17

3 years ago
mathjazz: Are there any projects still on the old sync process? If not, should we mark this bug as resolved and file a new bug for removing the old sync code?

I'm adding bug 1211271 as a blocker, if there's anything else stopping us from moving a project over to the new sync process we should add it as a blocker as well.
Depends on: 1211271
Flags: needinfo?(m)
No, we only use new sync now! You can close this one. Excellent work!
Flags: needinfo?(m)
(Assignee)

Comment 19

3 years ago
Yesssssssssss.
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.