1614146 - `mach bootstrap` fails with `TypeError: environment can only contain strings`

Status: RESOLVED FIXED

(Firefox Build System :: Bootstrap Configuration, defect, P1)

Description

[Jonathan Watt [:jwatt]]

I'm using MozillaBuild 3.3, I've rerun ./mach vcs-setup --update-only, I've clobbered, and I have current tip (cb56699431a0) checked out. On trying to run bootstrap I get the error below:

$ ./mach bootstrap
mach bootstrap is not fully implemented in MozillaBuild

Note on Artifact Mode:

Artifact builds download prebuilt C++ components rather than building
them locally. Artifact builds are faster!

Artifact builds are recommended for people working on Firefox or
Firefox for Android frontends, or the GeckoView Java API. They are unsuitable
for those working on C++ code. For more information see:
https://developer.mozilla.org/en-US/docs/Artifact_builds.

Please choose the version of Firefox you want to build:
1. Firefox for Desktop Artifact Mode
2. Firefox for Desktop
3. GeckoView/Firefox for Android Artifact Mode
4. GeckoView/Firefox for Android
Your choice: 2
Error running mach:

['bootstrap']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.
You can invoke |./mach busted| to check if this issue is already on file. If it
isn't, please use |./mach busted file| to report it. If |./mach busted| is
misbehaving, you can also inspect the dependencies of bug 1543241.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

TypeError: environment can only contain strings

File "c:\Users\jwatt\mozilla\trees\u\python/mozboot/mozboot/mach_commands.py", line 44, in bootstrap
bootstrapper.bootstrap()
File "c:\Users\jwatt\mozilla\trees\u\python/mozboot\mozboot\bootstrap.py", line 444, in bootstrap
hg_installed, hg_modern = self.instance.ensure_mercurial_modern()
File "c:\Users\jwatt\mozilla\trees\u\python/mozboot\mozboot\base.py", line 553, in ensure_mercurial_modern
installed, modern, version = self.is_mercurial_modern()
File "c:\Users\jwatt\mozilla\trees\u\python/mozboot\mozboot\base.py", line 546, in is_mercurial_modern
our = self._parse_version(hg, 'version', self._hg_cleanenv())
File "c:\Users\jwatt\mozilla\trees\u\python/mozboot\mozboot\base.py", line 518, in _parse_version
return self._parse_version_impl(path, name, env, "--version")
File "c:\Users\jwatt\mozilla\trees\u\python/mozboot\mozboot\base.py", line 509, in _parse_version_impl
universal_newlines=True)
File "c:\Users\jwatt\mozilla\trees\u\python/mozboot\mozboot\base.py", line 437, in check_output
return subprocess.check_output(*args, **kwargs)
File "c:\mozilla-build\python3\lib\subprocess.py", line 395, in check_output
**kwargs).stdout
File "c:\mozilla-build\python3\lib\subprocess.py", line 472, in run
with Popen(*popenargs, **kwargs) as process:
File "c:\mozilla-build\python3\lib\subprocess.py", line 775, in __init__
restore_signals, start_new_session)
File "c:\mozilla-build\python3\lib\subprocess.py", line 1178, in _execute_child
startupinfo)

I presume this isn't part of the expected incompleteness inferred by the "mach bootstrap is not fully implemented in MozillaBuild" message, since this seems like an uncaught exception.

Comment 1

[Jonathan Watt [:jwatt]]

I should say I'm trying to run bootstrap since I get the following cbindgen related error when trying to build:

1:04.78 checking for cbindgen...
1:04.78 DEBUG: trying cbindgen: c:/Users/jwatt/.mozbuild/cbindgen/cbindgen.exe
1:04.78 DEBUG: Executing: `c:/Users/jwatt/.mozbuild/cbindgen/cbindgen.exe --version`
1:04.78 DEBUG: c:/Users/jwatt/.mozbuild/cbindgen/cbindgen.exe has version 0.12.0
1:04.78 DEBUG: trying cbindgen: c:/Users/jwatt/.mozbuild/cbindgen/cbindgen.exe
1:04.78 DEBUG: Executing: `c:/Users/jwatt/.mozbuild/cbindgen/cbindgen.exe --version`
1:04.78 DEBUG: c:/Users/jwatt/.mozbuild/cbindgen/cbindgen.exe has version 0.12.0
1:04.78 ERROR: cbindgen version 0.12.0 is too old. At least version 0.13.0 is required.
1:04.78 Please update using 'cargo install cbindgen --force' or running
1:04.78 './mach bootstrap', after removing the existing executable located at
1:04.78 c:/Users/jwatt/.mozbuild/cbindgen/cbindgen.exe.
1:04.86 *** Fix above errors and then restart with\
1:04.86 "./mach build"
1:04.87 mozmake.EXE: *** [client.mk;115: configure] Error 1

Comment 2

[Joaquín Serna]

I have the same exact error when trying to do ./mach bootstrap on Windows 10 after following all the instructions in here.

Comment 3

[Masatoshi Kimura [:emk]]

https://searchfox.org/mozilla-central/rev/b1e51ad5613ad3d911cf42e3e525885ce2278915/python/mozboot/mozboot/base.py#507
env should be sanitized by mozbuild.util.ensure_subprocess_env.

Comment 5

[Michael Froman [:mjf]]

I'm having the same issue. Bug 1614399 may also be related.

Comment 7

[Masatoshi Kimura [:emk]]

This is regressed by bug 1569059.
Python 2 subprocess used to require that env does not contain unicode strings. Python 3 subprocess requires that env does not contain byte strings.
As written in the duped bug, _hg_cleanenv should stop using bytes literals. If we know we will never run ./mach bootstrap with Python 2 anymore, ensure_subprocess_env is redundant.

Comment 8

[Masatoshi Kimura [:emk]]

Attachment: Bug 1614146 - Stop using byte literals in _hg_cleanenv. r?#build

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/b497898c5844
Stop using byte literals in _hg_cleanenv. r=firefox-build-system-reviewers,glandium

Comment 10

[Michelle Goossens [:masterwayz]]

Same issue here on Windows 10.
Applying the above patch manually fixes the issue for me and things work again.

Comment 11

[Bogdan Tara[:bogdan_tara | bogdant]]

https://hg.mozilla.org/mozilla-central/rev/b497898c5844

Comment 12

[Masatoshi Kimura [:emk]]

Comment on attachment 9125740 [details]
Bug 1614146 - Stop using byte literals in _hg_cleanenv. r?#build

Beta/Release Uplift Approval Request

• User impact if declined: Cannot setup build environment on Windows. If this patch is declined, bug 1569059 should be backed out from 74 branch.
• Is this code covered by automated tests?: No
• Has the fix been verified in Nightly?: Yes
• Needs manual test from QE?: No
• If yes, steps to reproduce:
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): Build configuration issue, no end-user impact.
• String changes made/needed:

Comment 13

[Pascal Chevrel:pascalc]

Comment on attachment 9125740 [details]
Bug 1614146 - Stop using byte literals in _hg_cleanenv. r?#build

Low risk, approved for 74.0b3

Comment 14

[Dorel Luca [:dluca]]

https://hg.mozilla.org/releases/mozilla-beta/rev/f39fb51a62a6