1629797 - Mach build fails with an exception while checking for YASM version 1.3.0.

Status: VERIFIED FIXED

(Firefox Build System :: General, defect, P3)

Description

[Marco Zehe (:MarcoZ)]

Neither reinstalling MozillaBuild, nor running Mach bootstrap, nor mach clobber, helped.

Output from a run:

 Config object not found by mach.
 0:01.59 Clobber not needed.
 0:01.59 Adding make options from c:\devel\mozilla-central\mozconfig
    AUTOCLOBBER=1
    MOZ_OBJDIR=c:/devel/mozilla-central/obj-x86_64-pc-mingw32
    OBJDIR=c:/devel/mozilla-central/obj-x86_64-pc-mingw32
    FOUND_MOZCONFIG=c:/devel/mozilla-central/mozconfig
    export FOUND_MOZCONFIG
 0:01.64 c:\mozilla-build\bin\mozmake.EXE -f client.mk -s configure
 0:01.73 cd c:/devel/mozilla-central/obj-x86_64-pc-mingw32
 0:01.75 c:/devel/mozilla-central/configure
 0:02.24 Re-executing in the virtualenv
 0:02.85 Adding configure options from c:\devel\mozilla-central\mozconfig
 0:02.85   --target=x86_64-pc-mingw32
 0:02.85   --host=x86_64-pc-mingw32
 0:02.85   --enable-debug-symbols
 0:02.86 checking for vcs source checkout... hg
 0:02.92 checking for a shell... C:/mozilla-build/msys/bin/sh.exe
 0:03.12 checking for host system type... x86_64-pc-mingw32
 0:03.33 checking for target system type... x86_64-pc-mingw32
 0:03.87 checking whether cross compiling... no
 0:04.04 checking for Python 3... c:/devel/mozilla-central/obj-x86_64-pc-mingw32/_virtualenvs/init_py3/Scripts/python.exe (3.7.4)
 0:04.16 checking for Python 2... c:/devel/mozilla-central/obj-x86_64-pc-mingw32/_virtualenvs/init/Scripts/python.exe (2.7.16)
 0:04.16 checking for hg... c:/mozilla-build/python/Scripts/hg.exe
 0:04.39 checking for Mercurial version... 5.1.2
 0:04.69 checking for sparse checkout... no
 0:04.70 checking for yasm... c:/mozilla-build/bin/yasm.exe
 0:04.70 checking yasm version... 1.3.0
 0:04.74 Exception in thread Thread-11:
 0:04.74 Traceback (most recent call last):
 0:04.74   File "C:\mozilla-build\python3\Lib\threading.py", line 926, in _bootstrap_inner
 0:04.74     self.run()
 0:04.74   File "C:\mozilla-build\python3\Lib\threading.py", line 870, in run
 0:04.74     self._target(*self._args, **self._kwargs)
 0:04.74   File "C:\mozilla-build\python3\Lib\subprocess.py", line 1238, in _readerthread
 0:04.74     buffer.append(fh.read())
 0:04.75   File "c:\devel\mozilla-central\obj-x86_64-pc-mingw32\_virtualenvs\init_py3\lib\encodings\cp1252.py", line 23, in decode
 0:04.75     return codecs.charmap_decode(input,self.errors,decoding_table)[0]
 0:04.75 UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 700: character maps to <undefined>
 0:04.75 DEBUG: Executing: `'C:\Program Files (x86)\Microsoft Visual Studio\Installer\vswhere.exe' -format json -products '*' -requires Microsoft.VisualStudio.Component.VC.Tools.x86.x64`
 0:04.75 Traceback (most recent call last):
 0:04.75   File "c:/devel/mozilla-central/configure.py", line 181, in <module>
 0:04.75     sys.exit(main(sys.argv))
 0:04.75   File "c:/devel/mozilla-central/configure.py", line 52, in main
 0:04.75     sandbox.run(os.path.join(os.path.dirname(__file__), 'moz.configure'))
 0:04.75   File "c:\devel\mozilla-central\python\mozbuild\mozbuild\configure\__init__.py", line 498, in run
 0:04.75     func(*args)
 0:04.75   File "c:\devel\mozilla-central\python\mozbuild\mozbuild\configure\__init__.py", line 542, in _value_for
 0:04.75     return self._value_for_depends(obj)
 0:04.75   File "c:\devel\mozilla-central\python\mozbuild\mozbuild\util.py", line 1018, in method_call
 0:04.75     cache[args] = self.func(instance, *args)
 0:04.75   File "c:\devel\mozilla-central\python\mozbuild\mozbuild\configure\__init__.py", line 551, in _value_for_depends
 0:04.75     value = obj.result()
 0:04.75   File "c:\devel\mozilla-central\python\mozbuild\mozbuild\util.py", line 1018, in method_call
 0:04.75     cache[args] = self.func(instance, *args)
 0:04.75   File "c:\devel\mozilla-central\python\mozbuild\mozbuild\configure\__init__.py", line 155, in result
 0:04.75     return self._func(*resolved_args)
 0:04.75   File "c:\devel\mozilla-central\python\mozbuild\mozbuild\configure\__init__.py", line 1161, in wrapped
 0:04.75     return new_func(*args, **kwargs)
 0:04.75   File "c:/devel/mozilla-central/build/moz.configure/toolchain.configure", line 635, in vc_compiler_paths_for_version
 0:04.75     all_versions = sorted(get_vc_paths(env.topsrcdir), key=itemgetter(0))
 0:04.75   File "c:/devel/mozilla-central/build/moz.configure/toolchain.configure", line 594, in get_vc_paths
 0:04.75     for install in vswhere(['-products', '*', '-requires', 'Microsoft.VisualStudio.Component.VC.Tools.x86.x64']):
 0:04.75   File "c:/devel/mozilla-central/build/moz.configure/toolchain.configure", line 592, in vswhere
 0:04.75     return json.loads(check_cmd_output(vswhere, '-format', 'json', *args))
 0:04.75   File "c:\devel\mozilla-central\python\mozbuild\mozbuild\configure\__init__.py", line 1161, in wrapped
 0:04.75     return new_func(*args, **kwargs)
 0:04.75   File "c:/devel/mozilla-central/build/moz.configure/util.configure", line 54, in check_cmd_output
 0:04.75     retcode, stdout, stderr = get_cmd_output(*args, **kwargs)
 0:04.75   File "c:\devel\mozilla-central\python\mozbuild\mozbuild\configure\__init__.py", line 1161, in wrapped
 0:04.75     return new_func(*args, **kwargs)
 0:04.75   File "c:/devel/mozilla-central/build/moz.configure/util.configure", line 38, in get_cmd_output
 0:04.75     stdout, stderr = proc.communicate()
 0:04.75   File "C:\mozilla-build\python3\Lib\subprocess.py", line 939, in communicate
 0:04.75     stdout, stderr = self._communicate(input, endtime, timeout)
 0:04.75   File "C:\mozilla-build\python3\Lib\subprocess.py", line 1288, in _communicate
 0:04.75     stdout = stdout[0]
 0:04.75 IndexError: list index out of range
 0:04.85 *** Fix above errors and then restart with\
 0:04.85                "./mach build"
 0:04.87 mozmake.EXE: *** [client.mk;115: configure] Error 1

Comment 1

[Ricky Stewart]

Can you provide a bit more information please?

1. Specifically what steps are you taking to reproduce? From your comment I assume you're building with MozillaBuild. What's the full set of commands you're running? Are you simply doing ./mach build?

2. What is the output of python3 -c "import sys; print(sys.getdefaultencoding())"? If it's not utf-8, you should be able to fix it by updating some environment variables, but I'm confused why that should be necessary at all (since this is the first and only time I've ever heard of this issue popping up on someone's dev machine).

Comment 2

[Marco Zehe (:MarcoZ)]

1. I am just running ./mach build. It is the same unchanged, I just did an hg pull -u before, and then it started breaking like reported.
2. The output is UTF-8.

Comment 3

[Marco Zehe (:MarcoZ)]

Attachment: Log from output of a mach build command after mach clobber

This is a full output from a mach build run after first running mach clobber with current (as of Thursday morning CEST) m-c checkout.

Comment 4

[Ricky Stewart]

Can you patch in the following diff and verify it fixes your issue? To my knowledge this should make the code run identically to pre-last week. Once you confirm it works I'll send the change out for review.

diff --git a/build/moz.configure/util.configure b/build/moz.configure/util.configure
index 519081784b3c..69450766b523 100644
--- a/build/moz.configure/util.configure
+++ b/build/moz.configure/util.configure
@@ -34,7 +34,7 @@ def get_cmd_output(*args, **kwargs):
         # On Python 2 on Windows, close_fds prevents the process from inheriting
         # stdout/stderr. Elsewhere, it simply prevents it from inheriting extra
         # file descriptors, which is what we want.
-        close_fds=os.name != 'nt', universal_newlines=True, **kwargs)
+        close_fds=os.name != 'nt', **kwargs)
     stdout, stderr = proc.communicate()
     stdout = six.ensure_text(stdout, encoding=system_encoding, errors='replace')
     stderr = six.ensure_text(stderr, encoding=system_encoding, errors='replace')

Comment 5

[Marco Zehe (:MarcoZ)]

That diff fixes the problem.

Comment 6

[Ricky Stewart]

Attachment: Bug 1629797 - Don't rely on subprocess's default encoding in configure, and instead interpret the binary output ourselves.

This matches up to what we were doing before the Python 3 switch. Subprocess.Popen's default encoding doesn't necessarily match up to the system encoding as reported by mozbuild.util.system_encoding, and pre-Python 3.6 the constructor doesn't have encoding or errors parameters for no apparent reason. In the meantime, do the decoding manually.

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/2033db20b441
Don't rely on subprocess's default encoding in configure, and instead interpret the binary output ourselves. r=glandium

Comment 8

[Arthur Iakab [arthur_iakab]]

https://hg.mozilla.org/mozilla-central/rev/2033db20b441

Comment 9

[Marco Zehe (:MarcoZ)]

Verified fixed. My local builds now work again without any modifications.