1362872 - Cannot build mozilla-inbound 32bit on Windows10 x64 since bug 1318143 if VS2017 is installed

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Alice0775 White]

On windows10 64bit + VS2015 and VS2017 installed.
Using start-shell-msvc2015.bat of mozilla-build2.2.0

mozilla-inbound 32bit build fails with the following error.
Before landing of Bug 1318143, I can build mozilla-inbound 32bit successfully.


$ ./mach build
 0:01.60 c:\mozilla-build2_2_0\mozmake\mozmake.EXE -f client.mk MOZ_PARALLEL_BUILD=3 -s configure
 0:06.90 cd d:/srcinbound/ff-dbg
 0:06.94 d:/srcinbound/configure
 0:07.88 Reexecuting in the virtualenv
 0:08.69 Adding configure options from d:\srcinbound\.mozconfig
 0:08.69   --enable-application=browser
 0:08.69   --enable-update-channel=nightly
 0:08.69   --enable-jemalloc
 0:08.69   --enable-crashreporter
 0:08.70   --enable-release
 0:08.70   --with-branding=browser/branding/nightly
 0:08.70   --disable-warnings-as-errors
 0:08.70   --enable-debug-symbols
 0:08.70   --disable-tests
 0:08.70   --enable-optimize
 0:09.09 checking for a shell... C:/mozilla-build2_2_0/msys/bin/sh.exe
 0:10.17 checking for host system type... i686-pc-mingw32
 0:10.17 checking for target system type... i686-pc-mingw32
 0:10.27 checking for a shell... C:/mozilla-build2_2_0/msys/bin/sh.exe
 0:11.36 checking for host system type... i686-pc-mingw32
 0:11.36 checking for target system type... i686-pc-mingw32
 0:11.44 checking whether cross compiling... no
 0:11.60 Traceback (most recent call last):
 0:11.60   File "d:/srcinbound/configure.py", line 124, in <module>
 0:11.60     sys.exit(main(sys.argv))
 0:11.60   File "d:/srcinbound/configure.py", line 29, in main
 0:11.61     sandbox.run(os.path.join(os.path.dirname(__file__), 'moz.configure'))
 0:11.61   File "d:\srcinbound\python\mozbuild\mozbuild\configure\__init__.py", line 362, in run
 0:11.61     self._value_for(option)
 0:11.61   File "d:\srcinbound\python\mozbuild\mozbuild\configure\__init__.py", line 430, in _value_for
 0:11.61     return self._value_for_option(obj)
 0:11.61   File "d:\srcinbound\python\mozbuild\mozbuild\util.py", line 925, in method_call
 0:11.61     cache[args] = self.func(instance, *args)
 0:11.62   File "d:\srcinbound\python\mozbuild\mozbuild\configure\__init__.py", line 447, in _value_for_option
 0:11.62     not self._value_for(implied_option.when)):
 0:11.62   File "d:\srcinbound\python\mozbuild\mozbuild\configure\__init__.py", line 427, in _value_for
 0:11.62     return self._value_for_depends(obj, need_help_dependency)
 0:11.62   File "d:\srcinbound\python\mozbuild\mozbuild\util.py", line 925, in method_call
 0:11.62     cache[args] = self.func(instance, *args)
 0:11.62   File "d:\srcinbound\python\mozbuild\mozbuild\configure\__init__.py", line 436, in _value_for_depends
 0:11.62     return obj.result(need_help_dependency)
 0:11.63   File "d:\srcinbound\python\mozbuild\mozbuild\util.py", line 925, in method_call
 0:11.63     cache[args] = self.func(instance, *args)
 0:11.63   File "d:\srcinbound\python\mozbuild\mozbuild\configure\__init__.py", line 153, in result
 0:11.63     return self._func(resolved_args)
 0:11.63   File "d:\srcinbound\python\mozbuild\mozbuild\configure\__init__.py", line 152, in <genexpr>
 0:11.63     for d in self.dependencies)
 0:11.64   File "d:\srcinbound\python\mozbuild\mozbuild\configure\__init__.py", line 427, in _value_for
 0:11.64     return self._value_for_depends(obj, need_help_dependency)
 0:11.64   File "d:\srcinbound\python\mozbuild\mozbuild\util.py", line 925, in method_call
 0:11.64     cache[args] = self.func(instance, *args)
 0:11.64   File "d:\srcinbound\python\mozbuild\mozbuild\configure\__init__.py", line 436, in _value_for_depends
 0:11.64     return obj.result(need_help_dependency)
 0:11.65   File "d:\srcinbound\python\mozbuild\mozbuild\util.py", line 925, in method_call
 0:11.65     cache[args] = self.func(instance, *args)
 0:11.65   File "d:\srcinbound\python\mozbuild\mozbuild\configure\__init__.py", line 106, in result
 0:11.65     for d in self.dependencies]
 0:11.65   File "d:\srcinbound\python\mozbuild\mozbuild\configure\__init__.py", line 427, in _value_for
 0:11.65     return self._value_for_depends(obj, need_help_dependency)
 0:11.66   File "d:\srcinbound\python\mozbuild\mozbuild\util.py", line 925, in method_call
 0:11.66     cache[args] = self.func(instance, *args)
 0:11.66   File "d:\srcinbound\python\mozbuild\mozbuild\configure\__init__.py", line 436, in _value_for_depends
 0:11.66     return obj.result(need_help_dependency)
 0:11.66   File "d:\srcinbound\python\mozbuild\mozbuild\util.py", line 925, in method_call
 0:11.67     cache[args] = self.func(instance, *args)
 0:11.67   File "d:\srcinbound\python\mozbuild\mozbuild\configure\__init__.py", line 106, in result
 0:11.67     for d in self.dependencies]
 0:11.67   File "d:\srcinbound\python\mozbuild\mozbuild\configure\__init__.py", line 427, in _value_for
 0:11.67     return self._value_for_depends(obj, need_help_dependency)
 0:11.67   File "d:\srcinbound\python\mozbuild\mozbuild\util.py", line 925, in method_call
 0:11.68     cache[args] = self.func(instance, *args)
 0:11.68   File "d:\srcinbound\python\mozbuild\mozbuild\configure\__init__.py", line 436, in _value_for_depends
 0:11.68     return obj.result(need_help_dependency)
 0:11.68   File "d:\srcinbound\python\mozbuild\mozbuild\util.py", line 925, in method_call
 0:11.68     cache[args] = self.func(instance, *args)
 0:11.69   File "d:\srcinbound\python\mozbuild\mozbuild\configure\__init__.py", line 106, in result
 0:11.69     for d in self.dependencies]
 0:11.69   File "d:\srcinbound\python\mozbuild\mozbuild\configure\__init__.py", line 427, in _value_for
 0:11.70     return self._value_for_depends(obj, need_help_dependency)
 0:11.70   File "d:\srcinbound\python\mozbuild\mozbuild\util.py", line 925, in method_call
 0:11.70     cache[args] = self.func(instance, *args)
 0:11.70   File "d:\srcinbound\python\mozbuild\mozbuild\configure\__init__.py", line 436, in _value_for_depends
 0:11.71     return obj.result(need_help_dependency)
 0:11.71   File "d:\srcinbound\python\mozbuild\mozbuild\util.py", line 925, in method_call
 0:11.71     cache[args] = self.func(instance, *args)
 0:11.71   File "d:\srcinbound\python\mozbuild\mozbuild\configure\__init__.py", line 106, in result
 0:11.71     for d in self.dependencies]
 0:11.71   File "d:\srcinbound\python\mozbuild\mozbuild\configure\__init__.py", line 427, in _value_for
 0:11.72     return self._value_for_depends(obj, need_help_dependency)
 0:11.72   File "d:\srcinbound\python\mozbuild\mozbuild\util.py", line 925, in method_call
 0:11.72     cache[args] = self.func(instance, *args)
 0:11.72   File "d:\srcinbound\python\mozbuild\mozbuild\configure\__init__.py", line 436, in _value_for_depends
 0:11.72     return obj.result(need_help_dependency)
 0:11.73   File "d:\srcinbound\python\mozbuild\mozbuild\util.py", line 925, in method_call
 0:11.73     cache[args] = self.func(instance, *args)
 0:11.73   File "d:\srcinbound\python\mozbuild\mozbuild\configure\__init__.py", line 107, in result
 0:11.73     return self._func(*resolved_args)
 0:11.73   File "d:\srcinbound\python\mozbuild\mozbuild\configure\__init__.py", line 955, in wrapped
 0:11.73     return new_func(*args, **kwargs)
 0:11.74   File "d:/srcinbound/build/moz.configure/toolchain.configure", line 469, in vc_compiler_path
 0:11.74     all_versions = sorted(get_vc_paths(env.topsrcdir), key=itemgetter(0))
 0:11.74   File "d:/srcinbound/build/moz.configure/toolchain.configure", line 434, in get_vc_paths
 0:11.74     for install in vswhere(['-requires', 'Microsoft.VisualStudio.Component.VC.Tools.x86.x64']):
 0:11.75   File "d:/srcinbound/build/moz.configure/toolchain.configure", line 418, in vswhere
 0:11.75     return json.loads(subprocess.check_output([os.path.join(topsrcdir, 'build/win32/vswhere.exe'), '-format', 'json'] + args))
 0:11.75   File "c:\mozilla-build2_2_0\python\Lib\json\__init__.py", line 339, in loads
 0:11.75     return _default_decoder.decode(s)
 0:11.75   File "c:\mozilla-build2_2_0\python\Lib\json\decoder.py", line 364, in decode
 0:11.76     obj, end = self.raw_decode(s, idx=_w(s, 0).end())
 0:11.76   File "c:\mozilla-build2_2_0\python\Lib\json\decoder.py", line 380, in raw_decode
 0:11.76     obj, end = self.scan_once(s, idx)
 0:11.76 UnicodeDecodeError: 'utf8' codec can't decode byte 0x8a in position 0: invalid start byte
 0:11.76 *** Fix above errors and then restart with\
 0:11.76                "c:/mozilla-build2_2_0/mozmake/mozmake.EXE -f client.mk build"
 0:11.76 client.mk:379: recipe for target 'configure' failed
 0:11.77 mozmake.EXE: *** [configure] Error 1


Regression window:
https://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?fromchange=43ccab8eba5e&tochange=93d41c6f2a14

Triggered by: bug 1318143

Comment 1

[Alice0775 White]

FYI,
I uninstalled VS2017. Then the build performed successfully without error.

Comment 3

[Makoto Kato [:m_kato]]

root cause is bug 1362923 comment #0

Comment 4

[Makoto Kato [:m_kato]]

Attachment: Bug 1362872 - Use mbcs encoding for vswhere output.

We add vswhere.exe to support VS2017.  But the description output of this command is localized, so it causes UnicodeDecodeError.

So we should use 'mbcs' encoding for this output. And for minor languages, we should also use replace to avoid decode error.

Review commit: https://reviewboard.mozilla.org/r/136980/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/136980/

Comment 5

[Gregory Szorc [:gps]]

Comment on attachment 8865330 [details]
Bug 1362872 - Use mbcs encoding for vswhere output.

https://reviewboard.mozilla.org/r/136980/#review140288

I hate that Python 2's JSON parser insists JSON is Unicode (as opposed to a series of bytes) and offers no way to treat it as if it were bytes [and defer encoding to post parsing].

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/042c7c020a3b
Use mbcs encoding for vswhere output. r=gps

Comment 7

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

Linux builds failed like https://treeherder.mozilla.org/logviewer.html#?job_id=97430904&repo=autoland after this landed.

Backed out in https://hg.mozilla.org/integration/autoland/rev/afe4400d803d

Comment 8

[Mike Hommey [:glandium]]

(In reply to Gregory Szorc [:gps] from comment #5)
> Comment on attachment 8865330 [details]
> Bug 1362872 - Use mbcs encoding for vswhere output.
> 
> https://reviewboard.mozilla.org/r/136980/#review140288
> 
> I hate that Python 2's JSON parser insists JSON is Unicode (as opposed to a
> series of bytes) and offers no way to treat it as if it were bytes [and
> defer encoding to post parsing].

well... JSON *is* supposed to be Unicode: https://tools.ietf.org/html/rfc7159#section-8.1 (although it's SHALL, not MUST)

Comment 9

[Makoto Kato [:m_kato]]

(In reply to Wes Kocher (:KWierso) from comment #7)
> Linux builds failed like
> https://treeherder.mozilla.org/logviewer.html#?job_id=97430904&repo=autoland
> after this landed.
> 
> Backed out in https://hg.mozilla.org/integration/autoland/rev/afe4400d803d

Why does test_toolchain_configure.py checks VS INSTALLATION on Linux host.....

Comment 10

[Makoto Kato [:m_kato]]

Comment on attachment 8865330 [details]
Bug 1362872 - Use mbcs encoding for vswhere output.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/136980/diff/1-2/

Comment 11

[Gregory Szorc [:gps]]

Comment on attachment 8865330 [details]
Bug 1362872 - Use mbcs encoding for vswhere output.

https://reviewboard.mozilla.org/r/136980/#review141372

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/baad8a03819f
Use mbcs encoding for vswhere output. r=gps

Comment 13

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/mozilla-central/rev/baad8a03819f