1557217 - Cross-channel generation is broken

Status: RESOLVED FIXED

(Localization Infrastructure and Tools :: General, defect)

Description

[Francesco Lodolo [:flod]]

Can it be that bug 1468240 broke it?
https://hg.mozilla.org/comm-central/diff/33551f4516d03c52f67cf624ee8ece12352342df/mail/locales/l10n.toml

Not sure why the error would be about utf8 code though.

Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/home/flodolo/src/cross-channel-experimental/initial/cli/create.py", line 55, in <module>
    i.commit_revisions(src_repo, target, revisions)
  File "initial/__init__.py", line 562, in commit_revisions
    cw.walkGraph()
  File "/home/flodolo/src/version-control-tools/cross-channel-l10n/mozxchannel/graph.py", line 182, in walkGraph
    self.handlerev(src_rev)
  File "initial/__init__.py", line 404, in handlerev
    create_contents(filemap, src_ctxs, project_files)
  File "content.py", line 44, in create_contents
    content = merge.merge_channels(ref_path, contents)
  File "/home/flodolo/src/compare-locales/compare_locales/merge.py", line 39, in merge_channels
    entities = merge_resources(parser, resources)
  File "/home/flodolo/src/compare-locales/compare_locales/merge.py", line 79, in merge_resources
    map(parse_resource, resources))
  File "/home/flodolo/src/compare-locales/compare_locales/merge.py", line 57, in parse_resource
    parser.readContents(resource)
  File "/home/flodolo/src/compare-locales/compare_locales/parser/base.py", line 356, in readContents
    (contents, _) = codecs.getdecoder(self.encoding)(contents)
  File "/home/flodolo/src/xenv/lib/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0x92 in position 938: invalid start byte
Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/home/flodolo/src/cross-channel-experimental/initial/cli/automate.py", line 145, in <module>
    subprocess.check_call(cmd)
  File "/usr/lib/python2.7/subprocess.py", line 190, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['python', '-m', 'initial.cli.create', '-R/home/flodolo/src/comm-central', u'--repo-name=comm-central', '-R/home/flodolo/src/comm-central', u'--repo-name=releases/comm-beta', '-R/home/flodolo/src/comm-central', u'--repo-name=releases/comm-release', '../output', u'comm-central', '0e71e7bac1888f0d0871fc6542dec4b32fb5b532', 'calendar/locales/l10n.toml', 'mail/locales/l10n.toml', 'suite/locales/l10n.toml']' returned non-zero exit status 1

Comment 1

[Axel Hecht [:Pike]]

This is https://hg.mozilla.org/comm-central/rev/0e71e7bac1888f0d0871fc6542dec4b32fb5b532, invalid unicode.

I need to think more about how to fix that.

Comment 2

[Axel Hecht [:Pike]]

I guess there are two ways to fix this:

One, decode content with mochibakes, and add a check for mochibakes. This is somewhat tricky, as we can have mochibakes intentionally. Could only be a warning, I guess.

The other would be to ignore decoding errors in cross channel, and drop the file altogether. At least for Firefox, that's not what the code does, though.

Comment 3

[Richard Marti (:Paenglab)]

I could also fix the string in c-c.

Comment 4

[Axel Hecht [:Pike]]

As we're converting changeset by changeset, that won't help, sadly.

Comment 5

[Francesco Lodolo [:flod]]

(In reply to Axel Hecht [:Pike] from comment #2)

One, decode content with mochibakes, and add a check for mochibakes. This is somewhat tricky, as we can have mochibakes intentionally. Could only be a warning, I guess.

I don't expect the warning to be useful in practice. I wonder if there are actual cases where we want a mojibake in a string.

Comment 6

[Axel Hecht [:Pike]]

Attachment: Bug 1557217, create mochibake on encoding errors, and warn. r=flod

There's a bit of a refactor in the base checker, which is now
good to use standalone. I use it for defines and include files, now.
That's because previously, the encoding errors would have been
reported. Sadly, only in stdout, and not in any structured data.

I've went for a warning instead of an error for two reasons:

• We might want to have an actual mochibake in localized content
• We shall run linters with fatal warnings in most en-US
  scenarios, which is as good as an error.

I doubt this will ever really be triggered by localizers anymore,
as they're basically all in Pontoon, where they can't screw up
encodings.

Comment 7

[Axel Hecht [:Pike]]

https://hg.mozilla.org/l10n/compare-locales/rev/390d6cc6e7e6d418cf74fb67f1e9ab217eebb24e landed, and gecko-strings updated, marking FIXED.