1857512 - `--push-to-lando` throws UnicodeDecodeError on Windows

Status: RESOLVED FIXED

(Developer Infrastructure :: Try, defect)

Description

[Kagami Rosylight [:saschanaz] (they/them)]

> ./mach try --push-to-lando
Auth0 token validated.
Using 461a9c98a535b9896e9d394bdfee72ce90cf7afd as the hg base commit.
Submitting stack of 2 nodes and the try commit.
Error running mach:

    mach try --push-to-lando

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.
You can invoke ``./mach busted`` to check if this issue is already on file. If it
isn't, please use ``./mach busted file try`` to report it. If ``./mach busted`` is
misbehaving, you can also inspect the dependencies of bug 1543241.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

UnicodeDecodeError: 'charmap' codec can't decode byte 0x8d in position 19018: character maps to <undefined>

  File "C:\Users\sasch\Documents\GitHub\gecko-dev\tools\tryselect\mach_commands.py", line 232, in try_default
    return command_context._mach_context.commands.dispatch(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\sasch\Documents\GitHub\gecko-dev\python\mach\mach\registrar.py", line 202, in dispatch
    return self._run_command_handler(handler, context, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\sasch\Documents\GitHub\gecko-dev\python\mach\mach\registrar.py", line 125, in _run_command_handler
    result = fn(instance, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\sasch\Documents\GitHub\gecko-dev\tools\tryselect\mach_commands.py", line 375, in try_auto
    return run(command_context, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\sasch\Documents\GitHub\gecko-dev\tools\tryselect\mach_commands.py", line 196, in run
    return mod.run(**kwargs)
           ^^^^^^^^^^^^^^^^^
  File "c:\users\sasch\documents\github\gecko-dev\tools\tryselect\selectors\auto.py", line 109, in run
    return push_to_try(
           ^^^^^^^^^^^^
  File "c:\users\sasch\documents\github\gecko-dev\tools\tryselect\push.py", line 231, in push_to_try
    push_to_lando_try(vcs, commit_message)
  File "c:\users\sasch\documents\github\gecko-dev\tools\tryselect\lando.py", line 389, in push_to_lando_try
    base_commit, patches = get_stack_info(vcs)
                           ^^^^^^^^^^^^^^^^^^^
  File "c:\users\sasch\documents\github\gecko-dev\tools\tryselect\lando.py", line 95, in get_stack_info
    patches = vcs.get_commit_patches(nodes)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\users\sasch\documents\github\gecko-dev\python\mozversioncontrol\mozversioncontrol\__init__.py", line 861, in get_commit_patches
    return [
           ^
  File "c:\users\sasch\documents\github\gecko-dev\python\mozversioncontrol\mozversioncontrol\__init__.py", line 862, in <listcomp>
    self._run("format-patch", node, "-1", "--stdout").encode("utf-8")
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\users\sasch\documents\github\gecko-dev\python\mozversioncontrol\mozversioncontrol\__init__.py", line 110, in _run
    return subprocess.check_output(
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\sasch\AppData\Local\Programs\Python\Python311\Lib\subprocess.py", line 466, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\sasch\AppData\Local\Programs\Python\Python311\Lib\subprocess.py", line 550, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\users\sasch\documents\github\gecko-dev\third_party\python\sentry_sdk\sentry_sdk\integrations\stdlib.py", line 223, in sentry_patched_popen_communicate
    return old_popen_communicate(self, *a, **kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\sasch\AppData\Local\Programs\Python\Python311\Lib\subprocess.py", line 1196, in communicate
    stdout = self.stdout.read()
             ^^^^^^^^^^^^^^^^^^
  File "C:\Users\sasch\AppData\Local\Programs\Python\Python311\Lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

But it worked a few hours ago, so I guess there is a real error that is somehow shadowed by this?

Comment 1

[:glob ✱]

Commits:

https://hg.mozilla.org/try/rev/b863fc901c7df573c42d3f4180906e3e498c4b04
https://hg.mozilla.org/try/rev/305af738c16a9100a423bfbb1580fbce3e05de52

Comment 2

[Masatoshi Kimura [:emk]]

Attachment: replacement_char

I got this error when I tried landing the attached patch with ./mach try --push-to-lando:

$ ./mach try --push-to-lando
Auth0 token validated.
Using 9b362770f30bc360df921ba24ace5de8c387e631 as the hg base commit.
Submitting stack of 2 nodes and the try commit.
Error running mach:

    mach try --push-to-lando

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.
You can invoke ``./mach busted`` to check if this issue is already on file. If it
isn't, please use ``./mach busted file try`` to report it. If ``./mach busted`` is
misbehaving, you can also inspect the dependencies of bug 1543241.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

UnicodeDecodeError: 'cp932' codec can't decode byte 0xef in position 2257: illegal multibyte sequence

  File "F:\m\mozilla-central\tools\tryselect\mach_commands.py", line 232, in try_default
    return command_context._mach_context.commands.dispatch(
  File "F:\m\mozilla-central\python\mach\mach\registrar.py", line 202, in dispatch
    return self._run_command_handler(handler, context, **kwargs)
  File "F:\m\mozilla-central\python\mach\mach\registrar.py", line 125, in _run_command_handler
    result = fn(instance, **kwargs)
  File "F:\m\mozilla-central\tools\tryselect\mach_commands.py", line 375, in try_auto
    return run(command_context, **kwargs)
  File "F:\m\mozilla-central\tools\tryselect\mach_commands.py", line 196, in run
    return mod.run(**kwargs)
  File "f:\m\mozilla-central\tools\tryselect\selectors\auto.py", line 109, in run
    return push_to_try(
  File "f:\m\mozilla-central\tools\tryselect\push.py", line 231, in push_to_try
    push_to_lando_try(vcs, commit_message)
  File "f:\m\mozilla-central\tools\tryselect\lando.py", line 389, in push_to_lando_try
    base_commit, patches = get_stack_info(vcs)
  File "f:\m\mozilla-central\tools\tryselect\lando.py", line 95, in get_stack_info
    patches = vcs.get_commit_patches(nodes)
  File "f:\m\mozilla-central\python\mozversioncontrol\mozversioncontrol\__init__.py", line 634, in get_commit_patches
    output = self._run(*args).encode("utf-8")
  File "f:\m\mozilla-central\python\mozversioncontrol\mozversioncontrol\__init__.py", line 409, in _run
    return super(HgRepository, self)._run(*args, **runargs)
  File "f:\m\mozilla-central\python\mozversioncontrol\mozversioncontrol\__init__.py", line 110, in _run
    return subprocess.check_output(
  File "C:\mozilla-build\python3\lib\subprocess.py", line 424, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "C:\mozilla-build\python3\lib\subprocess.py", line 507, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "C:\mozilla-build\python3\lib\subprocess.py", line 1121, in communicate
    stdout = self.stdout.read()

Comment 3

[Masatoshi Kimura [:emk]]

Sorry, I did not mean to take this bug.

Comment 4

[:glob ✱]

Both diffs contain the same character:

-        # Python 2.7 compat for: replacement_char = ''
-        replacement_char = b'\xef\xbf\xbd'.decode('utf-8')
+        replacement_char = ''

(let's see if Bugzilla doesn't like that character)

edit: it does not like that character either; view attachment 9357101 [details] directly to see it.
It's U+FFFD REPLACEMENT CHARACTER – used to replace an unknown, unrecognised, or unrepresentable character

Comment 5

[Connor Sheehan [:sheehan]]

Attachment: Bug 1857512: use `utf-8` encoding for mozversioncontrol VCS commands r?#mach-reviewers,zeid

mozversioncontrol runs currently use universal_newlines, which
is an alias for text. This forces the command output file object
to be opened in text mode using the platform's default encoding. On
Windows the default encoding is usually a single-byte encoding such
as cp-1252, which can cause issues when parsing patches that include
multi-byte sequences. Add an encoding option to _run with utf-8
as the default option and change the check_output call to use it.
Callers can revert to the previous behaviour quite easily by setting
encoding=None if necessary.

Comment 6

[Connor Sheehan [:sheehan]]

:saschanaz, :emk, could you apply https://phabricator.services.mozilla.com/D190710, rebase your patches on top and try re-submitting with --push-to-lando? Please let me know if this fixes the issues you're seeing.

Comment 7

[Masatoshi Kimura [:emk]]

It fixes the issue for me, thank you!
https://treeherder.mozilla.org/jobs?repo=try&revision=3c437fc0af78cfb2113b5776095ba5cbc360ba14

Comment 8

[Connor Sheehan [:sheehan]]

Thanks, :emk!

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/dfe9eba97576
use `utf-8` encoding for mozversioncontrol VCS commands r=zeid

Comment 10

[Norisz Fay [:noriszfay]]

https://hg.mozilla.org/mozilla-central/rev/dfe9eba97576