1316640 - Create a module for migrating legacy translations to FTL

Status: RESOLVED FIXED

(Localization Infrastructure and Tools :: Fluent Migration, defect)

Description

[Staś Małolepszy :stas]

In bug 1280685 we want to create a workflow for migrating legacy translations (DTD, properties) to the new FTL format used by L20n.

This PR adds the l20n.migrate module able to run migrations "spec" files written by developers and serialize the resulting FTL files. A tool using this module (e.g. a mach command, see bug 1315005) can then commit the serialized resources, thus preserving the attribution of localizers.

The migration specification was discussed in mozilla-l10n/roadmap#13 (comment). The attribution requirement was discussed in bug 1288141 and bug 1288146.

Comment 1

[Staś Małolepszy :stas]

Attachment: Pull request

Comment 2

[Staś Małolepszy :stas]

Comment on attachment 8809481 [details] [review]
Pull request

I'd like to get input about this from the rest of the team.  I'm flagging Zibi and Pike for review, especially of the changes to l20n/format and the overall design of the transforms.  See tools/migrate/about_dialog.py for an example of a migration spec.

I'd also like to get feedback from flod and Matjaž;  it will be good if more people on the team know how the migrations work so that we can all help developers write them.

Thanks!

Comment 3

[Axel Hecht [:Pike]]

Some general comments outside of the PR.

I did start to file at least one bug to get things out of this PR and on to new bugs, 1317336.

Maybe the same for PLURALS? Generally mutli-line string handling tests?

Comment 4

[Zibi Braniecki [:zbraniecki][:gandalf]]

Comment on attachment 8809481 [details] [review]
Pull request

f+ on parser changes

Comment 5

[Francesco Lodolo [:flod]]

Comment on attachment 8809481 [details] [review]
Pull request

Left a few comments in the PR and did some tests, I guess it's time to clear the flag.

Comment 6

[Staś Małolepszy :stas]

Zibi, what is the best way to generate new JSON fixtures which will reflect the changes I made to the AST?

Comment 7

[Axel Hecht [:Pike]]

He did fix the tools/parse.py command line parser in https://github.com/l20n/python-l20n/pull/6/files#diff-c2a1848348217806f8f50973a68b4bbe, which dumps the AST and errors to stdout. Does that help?

Comment 8

[Staś Małolepszy :stas]

Yeah, thanks.  It looks like the fixture AST were previously generated by l20n.js and now python's JSON serializes them in a completely different property order.  Is that fine?

Comment 9

[Staś Małolepszy :stas]

Attachment: Pull request against l20n.js

We'll need to port the changes to the Python implementation of the parser and the AST to l20n.js.  With the changes ported I should be able to use l20n.js's generate-parser-fixtures.js to generate the fixtures without messing up the JSON too much.

Comment 10

[Axel Hecht [:Pike]]

Comment on attachment 8809481 [details] [review]
Pull request

I think the state of the branch we have now is good to land, and then we'll pound it a bit with some more practical examples :-)

Comment 11

[Staś Małolepszy :stas]

Landed in python-l20n:

https://github.com/l20n/python-l20n/commit/14fd5842c929e9afd328877cc97a9c49a203c5a0

I'll keep this bug open until the l20n.js PR lands.

Comment 12

[Zibi Braniecki [:zbraniecki][:gandalf]]

Comment on attachment 8812319 [details] [review]
Pull request against l20n.js

lgtm

Comment 13

[Staś Małolepszy :stas]

Landed in l20n.js:

https://github.com/l20n/l20n.js/commit/62f39864b3934a213d8e477d55d34efcfcff3048

Marking as fixed \o/.