1426665 - migrate: ne-NP and si break during the annotation phase

Status: RESOLVED WORKSFORME

(Localization Infrastructure and Tools :: Fluent Migration, defect)

Description

[Staś Małolepszy :stas]

I ran the migration from bug 1411012 on all locales I cloned using the following all-locales file:

    https://dxr.mozilla.org/mozilla-central/rev/1624b88874765bf57e9feba176d30149c748d9d2/browser/locales/all-locales

I got errors for ne-NP and si:

Annotating /home/stas/moz/l10n-central/ne-NP
Traceback (most recent call last):
  File "/home/stas/moz/mozilla-central/third_party/python/fluent/tools/migrate/migrate-l10n.py", line 115, in <module>
    blame = Blame(args.localization_dir).main()
  File "/home/stas/moz/mozilla-central/third_party/python/fluent/tools/migrate/blame.py", line 21, in main
    self.handleFile(file_blame)
  File "/home/stas/moz/mozilla-central/third_party/python/fluent/tools/migrate/blame.py", line 46, in handleFile
    line_blame = entity_lines[0]
IndexError: list index out of range

Annotating /home/stas/moz/l10n-central/si
Traceback (most recent call last):
  File "/home/stas/moz/mozilla-central/third_party/python/fluent/tools/migrate/migrate-l10n.py", line 115, in <module>
    blame = Blame(args.localization_dir).main()
  File "/home/stas/moz/mozilla-central/third_party/python/fluent/tools/migrate/blame.py", line 21, in main
    self.handleFile(file_blame)
  File "/home/stas/moz/mozilla-central/third_party/python/fluent/tools/migrate/blame.py", line 46, in handleFile
    line_blame = entity_lines[0]
IndexError: list index out of range

Comment 1

[Staś Małolepszy :stas]

I don't know what's going on but this seems to be a problem in hg.

In ne-NP, the first entity to break the blame is OutputFolder defined in line 119 here:

    https://hg.mozilla.org/l10n-central/ne-NP/file/675d3c598d3f/mail/installer/override.properties#l119

The blame for this file ends at line 118 however:

    https://hg.mozilla.org/l10n-central/ne-NP/annotate/675d3c598d3f/mail/installer/override.properties

Same thing happens in si; the image-width.label is defined on line 6:

    https://hg.mozilla.org/l10n-central/si/file/b8dd9d4daaf9/suite/chrome/browser/metadata.dtd#l6

And yet the blame only has 5 lines:

    https://hg.mozilla.org/l10n-central/si/annotate/b8dd9d4daaf9/suite/chrome/browser/metadata.dtd

Pike, flod - do you have any ideas what might be happening here?

Comment 2

[Staś Małolepszy :stas]

Setting NI on Pike rather than :l10n.

Comment 3

[Francesco Lodolo [:flod]]

ne-NP:
FileError and FileError_NoIgnore are completely messed up (\r\n gone rogue). Since we have those strings from browser, and we don't even have Nepali in Pontoon for TB, I fixed the file.

This makes migration pass, and annotate work
https://hg.mozilla.org/l10n-central/ne-NP/rev/56bc56906bc7290a09f8c480add8dcc0d93ef1b1

I'm leaving "si" alone for now.

Comment 4

[Francesco Lodolo [:flod]]

I fixed also Sinhala
https://hg.mozilla.org/l10n-central/si/rev/edd713c831b67830b6031199127e5674fb18cbd2

The problem in common is that they all had lines ending with just <CR>. For example, the entire metadata.dtd was one line.

Comment 5

[Francesco Lodolo [:flod]]

Leaving NI open for Axel.

I reported the issue in #vcs, still waiting for some feedback. Not sure if we should do something else.

broken: https://hg.mozilla.org/l10n-central/si/annotate/aa77a4ff1663/suite/chrome/browser/metadata.dtd
working: https://hg.mozilla.org/l10n-central/si/annotate/edd713c831b6/suite/chrome/browser/metadata.dtd

Comment 6

[Staś Małolepszy :stas]

Thanks for fixing both of them, flod! I can confirm that the migration runs well now for all locales.

Comment 7

[Axel Hecht [:Pike]]

So the culprit is that both hg and git don't consider mac newlines (only CR) to be newlines, and only support windows and linux newlines.

Our fileformats, though, support mac newlines, and so does compare-locales, via loading the files with universal-newlines.

I wonder if we could reasonably tweak the compare-locales parser (or rather the IO layer) to optionally ignore mac newlines for the conversion use-case.

Comment 8

[Axel Hecht [:Pike]]

Marking as WORKSFORME, we haven't hit this in ages.