1596667 - Migrate Screenshots content from .properties to Fluent

Status: RESOLVED FIXED

(Mozilla Localizations :: Other, task)

Description

[Francesco Lodolo [:flod]]

Bug 1587544 is going to migrate the Screenshots add-on to Fluent. Once that lands, we need to:

1. Import into l10n-central repositories the existing properties files with their history (webextensions.properties)
   https://github.com/mozilla-services/screenshots/tree/master/locales/en-US

2. Run a migration from the .properties file to .ftl

3. Remove the unused .properties file from l10n-central

Comment 1

[Axel Hecht [:Pike]]

So, I've two WIPs in fluent-migrate:

One is to disable VCS interaction completely. I thought I'd need that for bedrock, but didn't. You could probably just blame the last author, or we can script something that goes through blame and does a pontoon-like commit, primary author in commit, rest in message?
The other is to get blame from git. To make that useful here, we'd need to not just finish that, but also add an option to write to a different l10n repo than to read from.

The other option is to use hg convert? Not only is that going to add a lot of commits, it's also tedious, and even more tedious when we try to linearize history. We'd need a different file map for each locale for sure, and possibly prime the shamap to not add a new root to each l10n-central repo.

flod, any preference?

Comment 2

[Francesco Lodolo [:flod]]

I've written a script to migrate the .properties file in the l10n-central, based on previous similar tasks, and tested it locally on the Italian branch
https://github.com/flodolo/scripts/blob/master/mozilla_l10n/convert_from_github/screenshots-l10n-convert.py

The problem is that, at the end of the migration, .hgtags is completely rewritten because of the versions tagged in the GitHub repository (using the rebase approach)

-b6311141d6de9f302e24dcfe3833c736a3417855 THUNDERBIRD_3_0a3_RELEASE
-b6311141d6de9f302e24dcfe3833c736a3417855 THUNDERBIRD_3_0a3_BUILD2
-76aa831d71fbef3e49016c0d676ec712ef59cca7 THUNDERBIRD_3_0b1_RELEASE
-76aa831d71fbef3e49016c0d676ec712ef59cca7 THUNDERBIRD_3_0b1_BUILD1
-880bf573ea411a55246f5ae5d185014a32b1c627 GECKO_1_9_2_BASE
+0a6a40113bcbfe67ab88cc213ee15532573622b6 18.0.0
+0f0b1d342571a2c97442375da197e75c1031f15f 37.1.0
etc.

Is there a way to avoid touching tags when converting the repository? As ugly as it is, I could also store the content of the file before running the migration, and restore it after.

I also see a 1 other heads for branch "default" message, which I don't remember getting.

$ hg update
2 files updated, 0 files merged, 0 files removed, 0 files unresolved
updated to "2696a9643e71: update tags"
1 other heads for branch "default"

$ hg merge
abort: heads are bookmarked - please merge with an explicit rev
(run 'hg heads' to see all heads)

$ hg heads
changeset:   8083:2696a9643e71
tag:         tip
user:        convert-repo
date:        Fri Nov 15 08:10:44 2019 +0000
summary:     update tags

changeset:   8060:21405bf3e7be
bookmark:    remote/firefox-export-10
tag:         10.11.0
tag:         10.12.0
user:        Francesco Lodolo [:flod] <[e-mail]>
date:        Wed Aug 09 08:11:12 2017 +0200
summary:     Add locales with only 2-3 missing strings

Comment 3

[Axel Hecht [:Pike]]

I don't see a way to do that, unless you un-ref in your git repo. From hg convert -h:

The Git importer converts commits from all reachable branches (refs in
refs/heads) and remotes (refs in refs/remotes) to Mercurial. Branches are
converted to bookmarks with the same name, with the leading 'refs/heads'
stripped. Git submodules are converted to Git subrepos in Mercurial.

I'd also use convert.git.committeractions and convert.git.saverev.

I've had another alternative still, too:

Convert the git repo as one to an hg repo. Fluent-migrate in that repo, replaying blame. Then use your script to convert the commits to the ftl file to each l10n-central repo.

You could also do a bunch of prune and strip on the intermediate hg repo to limit forks.

Comment 4

[Francesco Lodolo [:flod]]

How do I run fluent-migrate on a single repo? I tried a few combinations from the command line, but I always end up with "abort: repository XXX not found).

For example, I thought that this could work, but it doesn't (the ~/screenshots-hg/locales/it exists)

$ migrate-l10n --lang it --reference-dir ~/screenshots-hg/locales/en-US/ --localization-dir ~/screenshots-hg/locales/it bug_1596667_screenshots

Traceback (most recent call last):
  File "/usr/local/bin/migrate-l10n", line 9, in <module>
    load_entry_point('fluent.migrate==0.6.4', 'console_scripts', 'migrate-l10n')()
  File "/Users/flodolo/mozilla/mercurial/fluent-migration/fluent/migrate/tool.py", line 179, in cli
    dry_run=args.dry_run
  File "/Users/flodolo/mozilla/mercurial/fluent-migration/fluent/migrate/tool.py", line 133, in main
    migrator.run(migration)
  File "/Users/flodolo/mozilla/mercurial/fluent-migration/fluent/migrate/tool.py", line 70, in run
    blame = Blame(self.client).attribution(files)
  File "/Users/flodolo/mozilla/mercurial/fluent-migration/fluent/migrate/tool.py", line 38, in client
    self._client = hglib.open(self.localization_dir, 'utf-8')
  File "/Library/Python/2.7/site-packages/hglib/__init__.py", line 11, in open
    return client.hgclient(path, encoding, configs)
  File "/Library/Python/2.7/site-packages/hglib/client.py", line 75, in __init__
    self.open()
  File "/Library/Python/2.7/site-packages/hglib/client.py", line 274, in open
    self._readhello()
  File "/Library/Python/2.7/site-packages/hglib/client.py", line 126, in _readhello
    ch, msg = self._readchannel()
  File "/Library/Python/2.7/site-packages/hglib/client.py", line 153, in _readchannel
    % (ret, serr.strip()))
hglib.error.ServerError: server exited with status 255: abort: repository /Users/flodolo/screenshots-hg/locales/it not found!

Comment 5

[Axel Hecht [:Pike]]

you need to point the --localization-dir to the actual repository, not a directory in the repository.

Comment 6

[Francesco Lodolo [:flod]]

(In reply to Axel Hecht [:Pike] from comment #5)

you need to point the --localization-dir to the actual repository, not a directory in the repository.

And how do you write a migration recipe for that? The paths would change for every single locale. Do I have access to the lang parameter from recipes?

Comment 7

[Axel Hecht [:Pike]]

grmpf. Not sure how much work changing either the repo root or supporting {locale} in paths would be.

Comment 8

[Francesco Lodolo [:flod]]

Not pretty, but I can modify fluent-migrate locally to call migration.migrate() with the locale code (from a quick test it seems to work)

Comment 9

[Francesco Lodolo [:flod]]

I need a bit of help and sanity check.

1. Convert the entire GitHub repository to Mercurial (only the /locales folder)

hg convert --filemap screenshots-filemap screenshots screenshots-hg

Where screenshots-filemap is just

include locales/

Remove all tags (but tip) and bookmarks from the generated repository

hg tags | grep -v 'tip' | awk '{print $1;}' | xargs hg tag --remove $1 
hg bookmarks | awk '{print $1;}' | xargs hg bookmark -d $1

2. Add screenshots.ftl in locales/en-US/.

3. Modify fluent-migrate code to call the migrate function with the locale code

migration.migrate(ctx, self.language)

https://hg.mozilla.org/l10n/fluent-migration/file/tip/fluent/migrate/tool.py#l57

That way I can use the locale code in paths within the recipe.

4. Run the migration in the repository for all locales

#! /usr/bin/env bash

repo_path="/Users/flodolo/github/screenshots-hg/"
export PYTHONPATH="/Users/flodolo/github/fluent-migrations/recipes/"

cd ${repo_path}/locales
locale_list=(*/)
for locale in ${locale_list[@]}
do
    migrate-l10n --lang ${locale%?} --reference-dir ${repo_path} --localization-dir ${repo_path} bug_1596667_screenshots
done

At this point, each locale folder has a screenshots.ftl.

The resulting repository seems to make sense, although I still get the message about multiple heads.
https://bitbucket.org/flod/test-screenshots/src/default/

5. Import screenshots.ftl with its history into l10n-central using the following script. This is a simplified version of the script compared to what we used so far (no option merge/rebase, no git pull).
   https://github.com/flodolo/scripts/blob/master/mozilla_l10n/convert_from_github/screenshots-monorepo-convert.py

Open question: do we still need the shamap part? I'm not sure why it was needed before.

If I run the script, I end up with "1 other heads for branch default", and I need to merge, so something is still not working as expected.

Comment 10

[Axel Hecht [:Pike]]

https://bitbucket.org/flod/test-screenshots/commits/all?page=9 has those heads. Hard to tell where they're coming from. Like, I tried and I end up somewhere on page 40, where I don't see any purpose.

Do you want me to give the conversion a try?

Comment 11

[Francesco Lodolo [:flod]]

(In reply to Axel Hecht [:Pike] from comment #10)

https://bitbucket.org/flod/test-screenshots/commits/all?page=9 has those heads. Hard to tell where they're coming from. Like, I tried and I end up somewhere on page 40, where I don't see any purpose.

Do you want me to give the conversion a try?

Yes, that would definitely help. I don't know when the main bug is going to land, but I'd like to be ready before that happens.

Comment 12

[Francesco Lodolo [:flod]]

I've added the updated steps into the User Story. The last part to verify is the "import into l10n-central".

I've updated my script, adding back the shamap part, and it seems to work correctly on the Italian repo.

Comment 13

[Francesco Lodolo [:flod]]

This is done, I've also added the Screenshots brand, e.g. https://hg.mozilla.org/l10n-central/de/rev/36471dabda7a58fbae1f20f0c28e56889101c7ea

Sadly there are now a ton of "errors", where the "Firefox Screenshots" was translated, so the term replacement didn't work.