1575804 - mach build fails with UnicodeEncodeError: 'ascii' codec can't encode character u'\xe3' in position 9007: ordinal not in range(128)

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Dão Gottwald [:dao]]

<dao> anyone know what's going on here?
    0:02.35 Traceback (most recent call last):
    0:02.35   File "/home/dao/mozilla/central/configure.py", line 164, in <module>
    0:02.35     sys.exit(main(sys.argv))
    0:02.35   File "/home/dao/mozilla/central/configure.py", line 40, in main
    0:02.35     sandbox.run(os.path.join(os.path.dirname(__file__), 'moz.configure'))
    0:02.35   File "/home/dao/mozilla/central/python/mozbuild/mozbuild/configure/__init__.py", line 498, in run
    0:02.35     func(*args)
    0:02.35   File "/home/dao/mozilla/central/python/mozbuild/mozbuild/configure/__init__.py", line 542, in _value_for
    0:02.35     return self._value_for_depends(obj)
    0:02.35   File "/home/dao/mozilla/central/python/mozbuild/mozbuild/util.py", line 963, in method_call
    0:02.35     cache[args] = self.func(instance, *args)
    0:02.35   File "/home/dao/mozilla/central/python/mozbuild/mozbuild/configure/__init__.py", line 551, in _value_for_depends
    0:02.35     value = obj.result()
    0:02.35   File "/home/dao/mozilla/central/python/mozbuild/mozbuild/util.py", line 963, in method_call
    0:02.35     cache[args] = self.func(instance, *args)
    0:02.35   File "/home/dao/mozilla/central/python/mozbuild/mozbuild/configure/__init__.py", line 157, in result
    0:02.35     return self._func(*resolved_args)
    0:02.35   File "/home/dao/mozilla/central/python/mozbuild/mozbuild/configure/__init__.py", line 1149, in wrapped
    0:02.35     return new_func(*args, **kwargs)
    0:02.35   File "/home/dao/mozilla/central/build/moz.configure/util.configure", line 434, in wrapper
    0:02.35     return func(*args)
    0:02.35   File "/home/dao/mozilla/central/python/mozbuild/mozbuild/configure/__init__.py", line 1149, in wrapped
    0:02.35     return new_func(*args, **kwargs)
    0:02.35   File "/home/dao/mozilla/central/build/moz.configure/init.configure", line 518, in hg_config
    0:02.35     out = out.decode('utf-8', 'replace')
    0:02.35   File "/home/dao/mozilla/central/objdir-frontend/_virtualenvs/init/lib/python2.7/encodings/utf_8.py", line 16, in decode
    0:02.35     return codecs.utf_8_decode(input, errors, True)
    0:02.35 UnicodeEncodeError: 'ascii' codec can't encode character u'\xe3' in position 9007: ordinal not in range(128)
    0:02.39 *** Fix above errors and then restart with\
    0:02.39                "./mach build"
    0:02.39 make: *** [client.mk:115: configure] Error 1
<arai> looks like there's some non-ASCII (or non-UTF-8) character in your `hg config` output?
    or there's some encoding mismatch
    maybe "ã" in your name
<•glandium> dao: please file a bug ; in the meanwhile, try removing that out.decode line
    please indicate whether that works in the bug

Commenting out that line does work, i.e. mach build succeeds.

Comment 1

[Mike Hommey [:glandium]]

Attachment: Bug 1575804 - Don't decode the result from check_cmd_output.

Bug 1575135 changed check_cmd_output to return unicode strings, but a
couple places were already trying to do their own decoding, which now
can fail. Remove those.

Interesting the decoding was previously broken on Windows, this
actually fixes it (the output of hg config is not actually utf-8 on
Windows).

Comment 2

[Frédéric Wang (:fredw)]

I'm getting similar failure with build/moz.configure/toolchain.configure :

    env['LD_PRINT_OPTIONS'] = '1'
    retcode, stdout, stderr = get_cmd_output(*cmd, env=env)

ld returns a string with a "é":

GNU ld (GNU Binutils for Debian) 2.32.51.20190707
Copyright (C) 2019 Free Software Foundation, Inc.
Ce logiciel est libre; si vous le redistribuez, vous devez le faire selon les termes
de la licence GNU General Public License version 3 ou postérieure selon votre besoin.
Ce logiciel n'est couvert par aucune GARANTIE.

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/4451580bbd3b
Don't decode the result from check_cmd_output. r=froydnj

Comment 4

[Alexandru Michis [:malexandru]]

https://hg.mozilla.org/mozilla-central/rev/4451580bbd3b

Comment 5

[Mike Hommey [:glandium]]

Reopening per comment 2. There actually are some others I missed.

Comment 6

[Mike Hommey [:glandium]]

Attachment: Bug 1575804 - Don't decode the result from get_cmd_output.

Also don't pass universal_newlines to check_cmd_output. That was added
to make python 3 happier when check_cmd_output still returned bytes, to
hint subprocess to return a unicode string, but now that
check_cmd_output does the decoding, that's not needed anymore.

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/56248ad24092
Don't decode the result from get_cmd_output. r=chmanchester

Comment 8

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/56248ad24092