1361017 - [sync] Commit to repository only when there's actual activity in Pontoon

Status: RESOLVED FIXED

(Webtools Graveyard :: Pontoon, enhancement, P2)

Description

[Francesco Lodolo [:flod]]

Here's the current scenario:
1) en-US is updated, e.g. removing some strings
2) Pontoon picks up the change, and commits to all locales just to remove the obsolete strings, e.g. https://hg.mozilla.org/l10n-central/vi/rev/36ee49bd917f

There's no real activity behind 2, it will happen for a dead locale, as long as it's enabled in Pontoon.

A side effect of this behavior is that sign-offs are constantly invalidated: even if a locale hasn't done any work for weeks we'll keep seeing activity in the repository. As a consequence, these days sign-offs are basically impossible to manage.

Is there a good reason to not limit Pontoon's commits to actual activity from localizers? For example: 
1) en-US is updated, Pontoon will commit changes to the file once someone translates a string into it.
2) en-US is updated, Pontoon will commit changes to all the file once someone translates any string.

#1 is what already happens to localized files if they aren't affected by en-US changes.

The only risk is for an old string ID to resurface after months, but it doesn't seem that likely. That risk should also be lower with cross-channel in place, since that would represent a string change unless the string is really old.

Comment 1

[Matjaz Horvat [:mathjazz]]

(In reply to Francesco Lodolo [:flod] from comment #0)

> A side effect of this behavior is that sign-offs are constantly invalidated:
> even if a locale hasn't done any work for weeks we'll keep seeing activity
> in the repository. As a consequence, these days sign-offs are basically
> impossible to manage.

Are sign-offs invalidated for each commit/push?

Even if it's removing an obsolete string or file?

Is there are a reason for this?

> Is there a good reason to not limit Pontoon's commits to actual activity
> from localizers? For example: 
> 1) en-US is updated, Pontoon will commit changes to the file once someone
> translates a string into it.
> 2) en-US is updated, Pontoon will commit changes to all the file once
> someone translates any string.

The reason is performance: When en-US is updated, Pontoon knows which strings and files need to be removed and does that for all locales.

Scenario #1: Should be relatively easy to implement and wouldn't hurt performance, but doesn't cover removing obsolete files.

Scenario #2: On each commit, Pontoon would have to scan all locale files and compare them with en-US files to figure out what has to be removed.

Comment 2

[Francesco Lodolo [:flod]]

(In reply to Matjaz Horvat [:mathjazz] from comment #1)
> Are sign-offs invalidated for each commit/push?
> Even if it's removing an obsolete string or file?
> Is there are a reason for this?

Probably "invalidated" is not the best word to describe it.

1) Let's say I sign off Slovenian, even if incomplete. That's a way for me to state: I've checked up to this point and everything looks good.
https://l10n.mozilla.org/shipping/signoffs/sl/fx55#b398ff597cce

2) Pontoon goes in and commits a bunch of string removals.

3) This commit makes compare-locales run, dashboard tells me there's activity. The previous sign-off is still there, but to make sure I'm not missing actual translations I need to go in, check the diff, and eventually sign-off another time.

Multiply for 44 locales, most of them with two products to sign-off, twice a week. I could do this the last week of the cycle, but that means risking to go to beta1 with completely out-of-date sign-offs.

> Scenario #1: Should be relatively easy to implement and wouldn't hurt
> performance, but doesn't cover removing obsolete files.

I remember going in for a few locales and removing them via mercurial, I always assumed Pontoon doesn't remove files.

Comment 3

[Axel Hecht [:Pike]]

Note, the eager string removals are also an aspect that makes pontoon and plain-text editor localizers hard to work together, as we often have changed strings with old IDs being a good starting point for new strings, and then pontoon just deletes them.

Comment 4

[Matjaz Horvat [:mathjazz]]

(In reply to Francesco Lodolo [:flod] from comment #2)
> Multiply for 44 locales, most of them with two products to sign-off, twice a
> week. I could do this the last week of the cycle, but that means risking to
> go to beta1 with completely out-of-date sign-offs.

Ugh, I see your pain.

Why not make Elmo sign such revisions off automatically? That is, if the previous revision has been signed off and the latest changset didn't introduce any meaningful change.

(In reply to Axel Hecht [out-of-office -May 2nd][:Pike] from comment #3)
> Note, the eager string removals are also an aspect that makes pontoon and
> plain-text editor localizers hard to work together, as we often have changed
> strings with old IDs being a good starting point for new strings, and then
> pontoon just deletes them.

I actually thought about adding an option to opt-out of removing obsolete strings for locales that also use VCS.

The number of teams using both, Pontoon and VCS, seems too low for Pontoon to do that by default.

Comment 5

[Axel Hecht [:Pike]]

(In reply to Matjaz Horvat [:mathjazz] from comment #4)
> (In reply to Francesco Lodolo [:flod] from comment #2)
> Why not make Elmo sign such revisions off automatically? That is, if the
> previous revision has been signed off and the latest changset didn't
> introduce any meaningful change.

elmo doesn't have that data :-/

> (In reply to Axel Hecht [out-of-office -May 2nd][:Pike] from comment #3)
> > Note, the eager string removals are also an aspect that makes pontoon and
> > plain-text editor localizers hard to work together, as we often have changed
> > strings with old IDs being a good starting point for new strings, and then
> > pontoon just deletes them.
> 
> I actually thought about adding an option to opt-out of removing obsolete
> strings for locales that also use VCS.
> 
> The number of teams using both, Pontoon and VCS, seems too low for Pontoon
> to do that by default.

... there would also be the advantage that activity in VCS corresponded to activity in the locale.

I don't think it's a bad thing that strings that can be removed are not removed, if a locale is inactive.

Which is kinda the essense of this bug, pontoon shouldn't pretend for there to be activity if there's none.

Comment 6

[Matjaz Horvat [:mathjazz]]

Let's stick to the Scenario #1: remove obsolete strings only after an activity in Pontoon (approved translation is added or deleted).

The final decision to be made is what to do with obsolete files. Easy thing to do are:
- keep doing what we already do: remove them when they are removed from en-US repository
- stop removing them (and maybe run a cleanup script every now and then)

Comment 7

[Francesco Lodolo [:flod]]

(In reply to Matjaz Horvat [:mathjazz] from comment #6)
> - stop removing them (and maybe run a cleanup script every now and then)

I'd be fine with this. I think with the new project configuration we should also be able to report these.

Comment 8

[Leandro Regueiro]

Is it not possible to do the following?

- Land actual human translations and other changes in separate commits with specific commit messages,
- Have the signoff and Mozilla tools to detect whether actual changes have arrived to hg repos.

Comment 9

[Matjaz Horvat [:mathjazz]]

(In reply to Leandro Regueiro from comment #8)
> Is it not possible to do the following?
> 
> - Land actual human translations and other changes in separate commits with
> specific commit messages,

That's what de facto happens ATM.

> - Have the signoff and Mozilla tools to detect whether actual changes have
> arrived to hg repos.

That was my question in Comment 4, but Elmo doesn't have sufficient data (see Comment 5).

Comment 10

[Axel Hecht [:Pike]]

Further detail, I talked with Matjaz earlier, and it's not as aweful as it sounds:

Instead of going over all files, and syncing them, you just go over the files that you have modifications for to serialize back.

That data should be readily available in the tools themselves. Or should be. It's also a great timesaver on sync if you only work on changes.

Comment 11

[[github robot]]

Commits pushed to master at https://github.com/mozilla/pontoon

https://github.com/mozilla/pontoon/commit/47f20157c7b3b4dab64a4e8e8866cf402242de0b
Bug 1361017: Do not delete obsolete translations

This basically reverts fix for bug 1199489, which added the ability to
delete translations that are missing in the source files.

Note: this only affects asymmetric formats, since obsolete translations
are removed on merge in symmetric ones.

https://github.com/mozilla/pontoon/commit/34dc197ec4eb16265bd6e31ae50f3e1dd114b5ad
Fix bug 1361017: Do not delete obsolete files

This basically reverts fix for bug 1201431, which added the ability to
delete translated files when corresponding source files are deleted.

Note: this only affects asymmetric formats, since obsolete translations
are removed on merge in symmetric ones.