1659602 - mach bootstrap failing on Windows with "TypeError: a bytes-like object is required, not 'str'"

Status: RESOLVED FIXED

(Firefox Build System :: Toolchains, defect)

Description

[(No longer employed by Mozilla) Aaron Klotz]

Error running mach:

    ['artifact', 'toolchain', '--bootstrap', '--from-build', 'win64-node']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.
You can invoke |./mach busted| to check if this issue is already on file. If it
isn't, please use |./mach busted file artifact| to report it. If |./mach busted| is
misbehaving, you can also inspect the dependencies of bug 1543241.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

TypeError: a bytes-like object is required, not 'str'

  File "c:\Users\dblohm7\src\mc2\python/mozbuild/mozbuild/artifact_commands.py", line 311, in artifact_toolchain
    toolchains = load_tasks_for_kind(params, 'toolchain', root_dir=root_dir)
  File "c:\Users\dblohm7\src\mc2\taskcluster\taskgraph\generator.py", line 431, in load_tasks_for_kind
    for task in tgg.full_task_set
  File "c:\Users\dblohm7\src\mc2\taskcluster\taskgraph\generator.py", line 153, in full_task_set
    return self._run_until('full_task_set')
  File "c:\Users\dblohm7\src\mc2\taskcluster\taskgraph\generator.py", line 375, in _run_until
    k, v = next(self._run)
  File "c:\Users\dblohm7\src\mc2\taskcluster\taskgraph\generator.py", line 279, in _run
    parameters, list(all_tasks.values()), self._write_artifacts,
  File "c:\Users\dblohm7\src\mc2\taskcluster\taskgraph\generator.py", line 82, in load_tasks
    for task_dict in transforms(trans_config, inputs)]
  File "c:\Users\dblohm7\src\mc2\taskcluster\taskgraph\generator.py", line 72, in <listcomp>
    tasks = [Task(self.name,
  File "c:\Users\dblohm7\src\mc2\taskcluster\taskgraph\transforms\task.py", line 1971, in check_run_task_caches
    for task in tasks:
  File "c:\Users\dblohm7\src\mc2\taskcluster\taskgraph\transforms\task.py", line 1889, in check_task_dependencies
    for task in tasks:
  File "c:\Users\dblohm7\src\mc2\taskcluster\taskgraph\transforms\task.py", line 1877, in check_task_identifiers
    for task in tasks:
  File "c:\Users\dblohm7\src\mc2\taskcluster\taskgraph\transforms\task.py", line 1860, in chain_of_trust
    for task in tasks:
  File "c:\Users\dblohm7\src\mc2\taskcluster\taskgraph\transforms\task.py", line 1694, in build_task
    for task in tasks:
  File "c:\Users\dblohm7\src\mc2\taskcluster\taskgraph\transforms\task.py", line 1685, in try_task_config_routes
    for task in tasks:
  File "c:\Users\dblohm7\src\mc2\taskcluster\taskgraph\transforms\task.py", line 1675, in try_task_config_chemspill_prio
    for task in tasks:
  File "c:\Users\dblohm7\src\mc2\taskcluster\taskgraph\transforms\task.py", line 1665, in try_task_config_env
    for task in tasks:
  File "c:\Users\dblohm7\src\mc2\taskcluster\taskgraph\transforms\task.py", line
dblohm7@aklotz-P52s ~/src/mc2 ?
$   for task in tasks:
  File "c:\Users\dblohm7\src\mc2\taskcluster\taskgraph\transforms\task.py", line 1435, in validate
    for task in tasks:
  File "c:\Users\dblohm7\src\mc2\taskcluster\taskgraph\transforms\task.py", line 1412, in task_name_from_label
    for task in tasks:
  File "c:\Users\dblohm7\src\mc2\taskcluster\taskgraph\transforms\task.py", line 1372, in set_defaults
    for task in tasks:
  File "c:\Users\dblohm7\src\mc2\taskcluster\taskgraph\transforms\task.py", line 1351, in set_implementation
    for task in tasks:
  File "c:\Users\dblohm7\src\mc2\taskcluster\taskgraph\transforms\cached_tasks.py", line 61, in cache_task
    for task in order_tasks(config, tasks):
  File "c:\Users\dblohm7\src\mc2\taskcluster\taskgraph\transforms\cached_tasks.py", line 22, in order_tasks
    pending = deque(tasks)
  File "c:\Users\dblohm7\src\mc2\taskcluster\taskgraph\transforms\job\__init__.py", line 369, in make_task_description
    configure_taskdesc_for_run(config, job, taskdesc, job['worker']['implementation'])
  File "c:\Users\dblohm7\src\mc2\taskcluster\taskgraph\transforms\job\__init__.py", line 429, in configure_taskdesc_for_run
    func(config, job, taskdesc)
  File "c:\Users\dblohm7\src\mc2\taskcluster\taskgraph\transforms\job\toolchain.py", line 155, in docker_worker_toolchain
    'digest-data': get_digest_data(config, run, taskdesc),
  File "c:\Users\dblohm7\src\mc2\taskcluster\taskgraph\transforms\job\toolchain.py", line 85, in get_digest_data
    data = [hash_paths(GECKO, files)]
  File "c:\Users\dblohm7\src\mc2\taskcluster\taskgraph\util\hash.py", line 38, in hash_paths
    finder = get_file_finder(base_path)
  File "c:\Users\dblohm7\src\mc2\python/mozbuild\mozbuild\util.py", line 1009, in __call__
    self[args] = self.func(*args)
  File "c:\Users\dblohm7\src\mc2\taskcluster\taskgraph\util\hash.py", line 26, in get_file_finder
    return get_repository_object(base_path).get_tracked_files_finder()
  File "c:\Users\dblohm7\src\mc2\python/mozversioncontrol\mozversioncontrol\__init__.py", line 432, in get_tracked_files_finder
    self._run(b'files', b'-0').split('\0') if p)
  File "c:\Users\dblohm7\src\mc2\python/mozversioncontrol\mozversioncontrol\__init__.py", line 332, in _run
    return super(HgRepository, self)._run(*args, **runargs)
  File "c:\Users\dblohm7\src\mc2\python/mozversioncontrol\mozversioncontrol\__init__.py", line 116, in _run
    universal_newlines=True)
  File "c:\Users\dblohm7\mozilla-build\python3\lib\subprocess.py", line 336, in check_output
    **kwargs).stdout
  File "c:\Users\dblohm7\mozilla-build\python3\lib\subprocess.py", line 403, in run
    with Popen(*popenargs, **kwargs) as process:
  File "c:\Users\dblohm7\src\mc2\third_party/python/sentry-sdk\sentry_sdk\integrations\stdlib.py", line 190, in sentry_patched_popen_init
    rv = old_popen_init(self, *a, **kw)  # type: ignore
  File "c:\Users\dblohm7\mozilla-build\python3\lib\subprocess.py", line 709, in __init__
    restore_signals, start_new_session)
  File "c:\Users\dblohm7\mozilla-build\python3\lib\subprocess.py", line 971, in _execute_child
    args = list2cmdline(args)
  File "c:\Users\dblohm7\mozilla-build\python3\lib\subprocess.py", line 461, in list2cmdline
    needquote = (" " in arg) or ("\t" in arg) or not arg
Sentry is attempting to send 0 pending error messages
Waiting up to 2 seconds
Press Ctrl-Break to quit
Error running mach:

    ['bootstrap']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.
You can invoke |./mach busted| to check if this issue is already on file. If it
isn't, please use |./mach busted file bootstrap| to report it. If |./mach busted| is
misbehaving, you can also inspect the dependencies of bug 1543241.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

subprocess.CalledProcessError: Command '['c:\\Users\\dblohm7\\mozilla-build\\python3\\python3.exe', 'c:\\Users\\dblohm7\\src\\mc2\\mach', 'artifact', 'toolchain', '--bootstrap', '--from-build', 'win64-node']' returned non-zero exit status 1.

  File "c:\Users\dblohm7\src\mc2\python/mozboot/mozboot/mach_commands.py", line 45, in bootstrap
    bootstrapper.bootstrap()
  File "c:\Users\dblohm7\src\mc2\python/mozboot\mozboot\bootstrap.py", line 578, in bootstrap
    checkout_root)
  File "c:\Users\dblohm7\src\mc2\python/mozboot\mozboot\bootstrap.py", line 421, in maybe_install_private_packages_or_exit
    self.instance.ensure_node_packages(state_dir, checkout_root)
  File "c:\Users\dblohm7\src\mc2\python/mozboot\mozboot\mozillabuild.py", line 243, in ensure_node_packages
    state_dir, checkout_root, node_artifact)
  File "c:\Users\dblohm7\src\mc2\python/mozboot\mozboot\base.py", line 356, in install_toolchain_artifact
    subprocess.check_call(cmd, cwd=state_dir)
  File "c:\Users\dblohm7\mozilla-build\python3\lib\subprocess.py", line 291, in check_call
    raise CalledProcessError(retcode, cmd)

Comment 1

[(No longer employed by Mozilla) Aaron Klotz]

Is this related to the virtualenv changes?

Comment 2

[Gerald Squelart (he/him) (not at Mozilla since 2022-09-15)]

I'm having the same issue, since I've rebased to central 508a0cc2f6d4.

And in case it's related, I can't even run ./mach busted:

$ ./mach busted
Error loading mozconfig: c:/mozilla-source/mozilla-central/mozconfig

Evaluation of your mozconfig exited with an error. This could be triggered
by a command inside your mozconfig failing. Please change your mozconfig
to not error and/or to catch errors in executed commands.

mozconfig output:

------BEGIN_ENV_BEFORE_SOURCE
c:/mozilla-source/mozilla-central/python/mozbuild/mozbuild/mozconfig_loader: line 29: c:\Users\Gerald: No such file or directory

My user directory has a space in it "Gerald Sq[...]". But this was not happening before.

Comment 3

[Jan Honza Odvarko [:Honza] (always need-info? me)]

I have the same problem.

Note that mach create-mach-environment seems to be working for me

Honza

Comment 4

[(No longer employed by Mozilla) Aaron Klotz]

(In reply to Jan Honza Odvarko [:Honza] (always need-info? me) from comment #3)

Note that mach create-mach-environment seems to be working for me

That did not work in my case.

Comment 5

[Bob Owen (:bobowen)]

mach create-mach-environment itself apparently worked for me, but build, bootstrap and even busted are still broken.
I also have a space in my user dir (oh, how I wish I'd recreated that when I first got this machine).

Comment 6

[Bob Owen (:bobowen)]

Not sure if bug 1659685 is a full duplicate.

Comment 7

[Ricky Stewart]

The regressing patch is the latest commit to bug 1636797. Reverting 98fddfef58ce locally is a workaround while we don't have a permanent solution.

Comment 8

[Ricky Stewart]

Bug 1659685 is unrelated. Take the conversation about "spaces in paths" there -- this bug will be about the subprocess issues in mach artifact/bootstrap.

Comment 9

[Ricky Stewart]

Attachment: Bug 1659602 - Fix type error in `mozversioncontrol` for Windows+`hg`+Python 3

These arguments will get converted to bytes later on if necessary; if we don't defer to hglib, then we need proper strings.

Comment 10

[Ricky Stewart]

I BELIEVE this patch should resolve this particular bug (again, not the "spaces in paths" thing, which the other bug will track). Please bug me if it ends up not doing so.

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/94ef23b99e50
Fix type error in `mozversioncontrol` for Windows+`hg`+Python 3 r=dmajor

Comment 13

[(No longer employed by Mozilla) Aaron Klotz]

(In reply to Ricky Stewart from comment #10)

I BELIEVE this patch should resolve this particular bug (again, not the "spaces in paths" thing, which the other bug will track). Please bug me if it ends up not doing so.

Do you have a Windows VM to test these things yourself? We're seeing far too many bugs lately affecting Windows users that could have been caught before they wasted everybody's time.

Comment 14

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/94ef23b99e50

Comment 15

[Ricky Stewart]

(In reply to Aaron Klotz [:aklotz][:dblohm7] from comment #13)

(In reply to Ricky Stewart from comment #10)

I BELIEVE this patch should resolve this particular bug (again, not the "spaces in paths" thing, which the other bug will track). Please bug me if it ends up not doing so.

Do you have a Windows VM to test these things yourself? We're seeing far too many bugs lately affecting Windows users that could have been caught before they wasted everybody's time.

I have a Windows machine (native, not a VM), and I did test the patch on Windows before landing, both myself and in CI. Unfortunately this bug does not affect 100% of Windows users.

If there were a process I could feasibly run through before landing EVERY patch to assure that each patch is completely bug-free, I assure you I would do so.

Comment 16

[(No longer employed by Mozilla) Aaron Klotz]

Understood. Sorry about that, I think that everything that has gone on over the past week has got me a bit testy.