1321281 - Should be able to test migrations against en-US

Status: RESOLVED FIXED

(Localization Infrastructure and Tools :: Fluent Migration, defect)

Description

[Axel Hecht [:Pike]]

Reviewing the menubar migration, it took me ages to find out that the FTL.ExternalArgument('sync-brand-short-name') should be a FTL.EntityReference.

We should have a way to test migrations, i.e., run them against en-US, and see if they literally produce the same content.

Comment 1

[Staś Małolepszy :stas]

Axel, can you elaborate on how this would work? Do you mean running a migration in mozilla-central against a snapshot of gecko-strings where the legacy files are still present?

Comment 2

[Axel Hecht [:Pike]]

Back when I filed this bug, gecko-strings didn't exist yet. It could be a crutch, but I was more thinking of actually using m-c. That line of thinking is probably aligned with the hope that the migrations would be part of the original migration patch.

In that case, it's "easy" to assume that the developer writing the migration has the en-US files around.

Our expectations on the authorship and timeline of writing and landing migrations is probably something we need to nail down as part of our migration milestones.

Comment 3

[Axel Hecht [:Pike]]

Noteworthy, there is a change to how we anticipated things to work a year ago:

Back then, we figured we'd have a couple of locales on Nightly, and the bulk of our migration work would happen on the l10n merge on aurora.
Today, we need to have the migrations when we push to gecko-strings, aka, at last a couple of days after the landing of the original string.

Comment 4

[Staś Małolepszy :stas]

Relevant comment from bug 1413514 comment 1:

It's already possible to clone gecko-strings and run a migration against it:

    ./migrate-l10n.py --lang en-US --reference-dir examples --localization-dir ~/moz/gecko-strings examples.about_dialog

This creates browser/about_dialog.ftl in gecko-strings. Its contents are not byte-to-byte the same with the about_dialog.ftl in the examples/ directory because of the lossy-ness of serialization. But running compare locales between mozilla-central and gecko-string works fine. It's a bit clunky because of how it takes the l10n base and a locale code, but it works:

    compare-locales browser/locales/l10n.toml .. gecko-strings

Zibi, perhaps you could try that when writing a migration for bug 1411012?

Comment 6

[Francesco Lodolo [:flod]]

Copying from the duped bug.

While writing migrations for bug 1451992, I realized how much manual work I do for testing migration recipes. In a scenario where other people might start writing them, we need an easier and reliable way to test migrations.

Here's what I currently do.

1) Create a copy of gecko-strings-quarantine. Let's call it "fake-en"
2) Save the migration recipe, run the migration against fake-en using my scripts
3) Compare the resulting FTL file (visual diff) with the one added in the patch

I think we should have a test that runs whenever a patch adds a migration recipe, not sure how it would work though.

Initially I thought: I'm going to create a local script to automate the manual work I'm currently doing, and use that as a base. I think I could build something, but it would be hacky, and hardly reusable as a starting point for such test.

For example, I would expect the test to:
1) Run only if the patch contains changes to migration recipes
2) Identify the FTL files affected by this recipe
3) Run the migration against a temporary clone of gecko-strings-quarantine
4) Parse the migrated file, compare it with the one provided in the patch. All strings existing in both files should be identical, all string referenced in the migration should be available in the generated file.

Additional notes:
* You can't run the migration against mozilla-central. The paths are now incompatible, and more important a patch removes DTD and properties file if it's doing a migration
* You can't test for the files being identical: a patch can add new strings to FTL that are not coming out of a migration.
* It would be nice to check for common errors, like variant names in PLATFORM() expressions.

Comment 7

[Axel Hecht [:Pike]]

Another recipe-test recipe:

1) clone/pull -u gecko-strings-quarantine
2) ast-analyze migration file to find template and target files
3) cross-channel merge the template files to a new directory between current directory and g-s-q
   Might want to use project-config to find the right en-US path.
4) run migration with template files, g-s-q as target
5) run compare-locales diff between template files and g-s-q, ensure that g-s-q is untouched. Possibly run diff.

Comment 8

[Francesco Lodolo [:flod]]

After running a few migrations, I wonder if this could actually be a test.

We have cases where we purposefully change en-US (fix access keys, remove double spacing), and others where we introduce non significant style changes (e.g. multiline strings). If we do this as a test, we should also have a way to signal these.

Comment 9

[Axel Hecht [:Pike]]

Attachment: Bug 1321281, add some validation and inspection of migration recipes, r=stas

Comment 10

[Axel Hecht [:Pike]]

Attachment: Bug 1321281, vendor fluent.migrate, r=flod

This vendors in updates to fluent.migrate, up to
https://hg.mozilla.org/l10n/fluent-migration/rev/650cd8740f091f5127d6f3ab8cc8a9236c341864

Comment 11

[Axel Hecht [:Pike]]

Attachment: Bug 1321281, add test framework for Fluent migration recipes, r=flod,ahal

Basic test infrastructure for Fluent migration recipes, automate creating
a reference from recipe data, and running the recipe dry and wet.

There are no hard failures for diffs, as it might just be OK.

Depends on D40199

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/67d51274556e
vendor fluent.migrate, r=flod
https://hg.mozilla.org/integration/autoland/rev/b9febe64276d
add test framework for Fluent migration recipes, r=flod,ahal

Comment 13

[Razvan Maries]

https://hg.mozilla.org/mozilla-central/rev/67d51274556e
https://hg.mozilla.org/mozilla-central/rev/b9febe64276d