1678064 - |./mach bootstrap| broken on Windows

Status: RESOLVED FIXED

(Firefox Build System :: Mach Core, defect)

Description

[Ryan VanderMeulen [:RyanVM]]

Bisected to bug 1676533. Reproducible with python 3.7 and 3.8.

$ ./mach bootstrap --no-system-changes --application-choice browser
Your version of Python 3 (3.7.4) is new enough.
Your version of Python 2 (2.7.18) is new enough.
virtualenv at c:/Users/Ryan/.mozbuild\_virtualenvs\mach is already up to date.
Requirement already satisfied: zstandard<=0.13.0,>=0.9.0 in c:\users\ryan\.mozbuild\_virtualenvs\mach\lib\site-packages (0.13.0)
Requirement already satisfied: cffi>=1.11 in c:\users\ryan\.mozbuild\_virtualenvs\mach\lib\site-packages (from zstandard<=0.13.0,>=0.9.0) (1.14.3)
Requirement already satisfied: pycparser in c:\users\ryan\.mozbuild\_virtualenvs\mach\lib\site-packages (from cffi>=1.11->zstandard<=0.13.0,>=0.9.0) (2.20)
WARNING: You are using pip version 20.2.2; however, version 20.2.4 is available.
You should consider upgrading via the 'C:\Users\Ryan\.mozbuild\_virtualenvs\mach\Scripts\python.exe -m pip install --upgrade pip' command.
Requirement already satisfied: glean_sdk~=32.3.1 in c:\users\ryan\.mozbuild\_virtualenvs\mach\lib\site-packages (32.3.2)
Requirement already satisfied: cffi>=1 in c:\users\ryan\.mozbuild\_virtualenvs\mach\lib\site-packages (from glean_sdk~=32.3.1) (1.14.3)
Requirement already satisfied: glean-parser==1.28.3 in c:\users\ryan\.mozbuild\_virtualenvs\mach\lib\site-packages (from glean_sdk~=32.3.1) (1.28.3)
Requirement already satisfied: pycparser in c:\users\ryan\.mozbuild\_virtualenvs\mach\lib\site-packages (from cffi>=1->glean_sdk~=32.3.1) (2.20)
Requirement already satisfied: yamllint>=1.18.0 in c:\users\ryan\.mozbuild\_virtualenvs\mach\lib\site-packages (from glean-parser==1.28.3->glean_sdk~=32.3.1) (1.25.0)
Requirement already satisfied: Click>=7 in c:\users\ryan\.mozbuild\_virtualenvs\mach\lib\site-packages (from glean-parser==1.28.3->glean_sdk~=32.3.1) (7.1.2)
Requirement already satisfied: appdirs>=1.4 in c:\users\ryan\.mozbuild\_virtualenvs\mach\lib\site-packages (from glean-parser==1.28.3->glean_sdk~=32.3.1) (1.4.4)
Requirement already satisfied: PyYAML>=3.13 in c:\users\ryan\.mozbuild\_virtualenvs\mach\lib\site-packages (from glean-parser==1.28.3->glean_sdk~=32.3.1) (5.3.1)
Requirement already satisfied: diskcache>=4 in c:\users\ryan\.mozbuild\_virtualenvs\mach\lib\site-packages (from glean-parser==1.28.3->glean_sdk~=32.3.1) (5.1.0)
Requirement already satisfied: Jinja2>=2.10.1 in c:\users\ryan\.mozbuild\_virtualenvs\mach\lib\site-packages (from glean-parser==1.28.3->glean_sdk~=32.3.1) (2.11.2)
Requirement already satisfied: jsonschema>=3.0.2 in c:\users\ryan\.mozbuild\_virtualenvs\mach\lib\site-packages (from glean-parser==1.28.3->glean_sdk~=32.3.1) (3.2.0)
Requirement already satisfied: setuptools in c:\users\ryan\.mozbuild\_virtualenvs\mach\lib\site-packages (from yamllint>=1.18.0->glean-parser==1.28.3->glean_sdk~=32.3.1) (49.6.0)
Requirement already satisfied: pathspec>=0.5.3 in c:\users\ryan\.mozbuild\_virtualenvs\mach\lib\site-packages (from yamllint>=1.18.0->glean-parser==1.28.3->glean_sdk~=32.3.1) (0.8.1)
Requirement already satisfied: MarkupSafe>=0.23 in c:\users\ryan\.mozbuild\_virtualenvs\mach\lib\site-packages (from Jinja2>=2.10.1->glean-parser==1.28.3->glean_sdk~=32.3.1) (1.1.1)
Requirement already satisfied: attrs>=17.4.0 in c:\users\ryan\.mozbuild\_virtualenvs\mach\lib\site-packages (from jsonschema>=3.0.2->glean-parser==1.28.3->glean_sdk~=32.3.1) (20.3.0)
Requirement already satisfied: importlib-metadata; python_version < "3.8" in c:\users\ryan\.mozbuild\_virtualenvs\mach\lib\site-packages (from jsonschema>=3.0.2->glean-parser==1.28.3->glean_sdk~=32.3.1) (2.0.0)
Requirement already satisfied: pyrsistent>=0.14.0 in c:\users\ryan\.mozbuild\_virtualenvs\mach\lib\site-packages (from jsonschema>=3.0.2->glean-parser==1.28.3->glean_sdk~=32.3.1) (0.17.3)
Requirement already satisfied: six>=1.11.0 in c:\users\ryan\.mozbuild\_virtualenvs\mach\lib\site-packages (from jsonschema>=3.0.2->glean-parser==1.28.3->glean_sdk~=32.3.1) (1.15.0)
Requirement already satisfied: zipp>=0.5 in c:\users\ryan\.mozbuild\_virtualenvs\mach\lib\site-packages (from importlib-metadata; python_version < "3.8"->jsonschema>=3.0.2->glean-parser==1.28.3->glean_sdk~=32.3.1) (3.4.0)
WARNING: You are using pip version 20.2.2; however, version 20.2.4 is available.
You should consider upgrading via the 'C:\Users\Ryan\.mozbuild\_virtualenvs\mach\Scripts\python.exe -m pip install --upgrade pip' command.
Python 3 mach environment created.
virtualenv at c:/Users/Ryan/.mozbuild\_virtualenvs\mach_py2 is already up to date.
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Requirement already satisfied: zstandard<=0.13.0,>=0.9.0 in c:\users\ryan\.mozbuild\_virtualenvs\mach_py2\lib\site-packages (0.13.0)
Requirement already satisfied: cffi>=1.11 in c:\users\ryan\.mozbuild\_virtualenvs\mach_py2\lib\site-packages (from zstandard<=0.13.0,>=0.9.0) (1.14.3)
Requirement already satisfied: pycparser in c:\users\ryan\.mozbuild\_virtualenvs\mach_py2\lib\site-packages (from cffi>=1.11->zstandard<=0.13.0,>=0.9.0) (2.20)
WARNING: You are using pip version 20.2.2; however, version 20.2.4 is available.
You should consider upgrading via the 'c:\Users\Ryan\.mozbuild\_virtualenvs\mach_py2\Scripts\python.exe -m pip install --upgrade pip' command.
Python 2 mach environment created.
abort: repository default does not exist!
abort: repository default does not exist!
Error running mach:

    ['artifact', 'toolchain', '--bootstrap', '--from-build', 'win64-node']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.
You can invoke |./mach busted| to check if this issue is already on file. If it
isn't, please use |./mach busted file artifact| to report it. If |./mach busted| is
misbehaving, you can also inspect the dependencies of bug 1543241.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

subprocess.CalledProcessError: Command '('c:\\mozilla-build\\python3\\Scripts\\hg.EXE', 'outgoing', '-r', '.', '--quiet', '--template', "{file_adds % '{file}\\n'}{file_dels % '{file}\\n'}{file_mods % '{file}\\n'}", 'default')' returned non-zero exit status 255.

  File "c:\Users\Ryan\repos\mozilla\python/mozbuild/mozbuild/artifact_commands.py", line 405, in artifact_toolchain
    toolchains = load_tasks_for_kind(params, "toolchain", root_dir=root_dir)
  File "c:\Users\Ryan\repos\mozilla\taskcluster\taskgraph\generator.py", line 472, in load_tasks_for_kind
    for task in tgg.full_task_set
  File "c:\Users\Ryan\repos\mozilla\taskcluster\taskgraph\generator.py", line 164, in full_task_set
    return self._run_until("full_task_set")
  File "c:\Users\Ryan\repos\mozilla\taskcluster\taskgraph\generator.py", line 417, in _run_until
    k, v = next(self._run)
  File "c:\Users\Ryan\repos\mozilla\taskcluster\taskgraph\generator.py", line 294, in _run
    self._write_artifacts,
  File "c:\Users\Ryan\repos\mozilla\taskcluster\taskgraph\generator.py", line 92, in load_tasks
    for task_dict in transforms(trans_config, inputs)
  File "c:\Users\Ryan\repos\mozilla\taskcluster\taskgraph\generator.py", line 80, in <listcomp>
    Task(
  File "c:\Users\Ryan\repos\mozilla\taskcluster\taskgraph\transforms\task.py", line 2110, in check_run_task_caches
    for task in tasks:
  File "c:\Users\Ryan\repos\mozilla\taskcluster\taskgraph\transforms\task.py", line 2018, in check_task_dependencies
    for task in tasks:
  File "c:\Users\Ryan\repos\mozilla\taskcluster\taskgraph\transforms\task.py", line 2004, in check_task_identifiers
    for task in tasks:
  File "c:\Users\Ryan\repos\mozilla\taskcluster\taskgraph\transforms\task.py", line 1985, in chain_of_trust
    for task in tasks:
  File "c:\Users\Ryan\repos\mozilla\taskcluster\taskgraph\transforms\task.py", line 1776, in build_task
    for task in tasks:
  File "c:\Users\Ryan\repos\mozilla\taskcluster\taskgraph\transforms\task.py", line 1767, in try_task_config_routes
    for task in tasks:
  File "c:\Users\Ryan\repos\mozilla\taskcluster\taskgraph\transforms\task.py", line 1757, in try_task_config_chemspill_prio
    for task in tasks:
  File "c:\Users\Ryan\repos\mozilla\taskcluster\taskgraph\transforms\task.py", line 1747, in try_task_config_env
    for task in tasks:
  File "c:\Users\Ryan\repos\mozilla\taskcluster\taskgraph\transforms\task.py", line 1709, in add_index_routes
    for task in tasks:
  File "c:\Users\Ryan\repos\mozilla\taskcluster\taskgraph\transforms\task.py", line 1504, in validate
    for task in tasks:
  File "c:\Users\Ryan\repos\mozilla\taskcluster\taskgraph\transforms\task.py", line 1481, in task_name_from_label
    for task in tasks:
  File "c:\Users\Ryan\repos\mozilla\taskcluster\taskgraph\transforms\task.py", line 1436, in set_defaults
    for task in tasks:
  File "c:\Users\Ryan\repos\mozilla\taskcluster\taskgraph\transforms\task.py", line 1415, in set_implementation
    for task in tasks:
  File "c:\Users\Ryan\repos\mozilla\taskcluster\taskgraph\transforms\cached_tasks.py", line 61, in cache_task
    for task in order_tasks(config, tasks):
  File "c:\Users\Ryan\repos\mozilla\taskcluster\taskgraph\transforms\cached_tasks.py", line 19, in order_tasks
    pending = deque(tasks)
  File "c:\Users\Ryan\repos\mozilla\taskcluster\taskgraph\transforms\job\__init__.py", line 391, in make_task_description
    config, job, taskdesc, job["worker"]["implementation"]
  File "c:\Users\Ryan\repos\mozilla\taskcluster\taskgraph\transforms\job\__init__.py", line 464, in configure_taskdesc_for_run
    func(config, job, taskdesc)
  File "c:\Users\Ryan\repos\mozilla\taskcluster\taskgraph\transforms\job\toolchain.py", line 156, in docker_worker_toolchain
    "digest-data": get_digest_data(config, run, taskdesc),
  File "c:\Users\Ryan\repos\mozilla\taskcluster\taskgraph\transforms\job\toolchain.py", line 80, in get_digest_data
    data = [hash_paths(GECKO, files)]
  File "c:\Users\Ryan\repos\mozilla\taskcluster\taskgraph\util\hash.py", line 63, in hash_paths
    dirty = get_dirty(base_path)
  File "c:\Users\Ryan\repos\mozilla\python/mozbuild\mozbuild\util.py", line 1040, in __call__
    self[args] = self.func(*args)
  File "c:\Users\Ryan\repos\mozilla\taskcluster\taskgraph\util\hash.py", line 46, in get_dirty
    return set(repo.get_outgoing_files()) | set(repo.get_changed_files())
  File "c:\Users\Ryan\repos\mozilla\python/mozversioncontrol\mozversioncontrol\__init__.py", line 445, in get_outgoing_files
    return_codes=(1,),
  File "c:\Users\Ryan\repos\mozilla\python/mozversioncontrol\mozversioncontrol\__init__.py", line 344, in _run
    return super(HgRepository, self)._run(*args, **runargs)
  File "c:\Users\Ryan\repos\mozilla\python/mozversioncontrol\mozversioncontrol\__init__.py", line 121, in _run
    universal_newlines=universal_newlines,
  File "c:\mozilla-build\python3\lib\subprocess.py", line 395, in check_output
    **kwargs).stdout
  File "c:\mozilla-build\python3\lib\subprocess.py", line 487, in run
    output=stdout, stderr=stderr)
abort: repository default does not exist!
Error running mach:

    ['bootstrap', '--no-system-changes', '--application-choice', 'browser']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.
You can invoke |./mach busted| to check if this issue is already on file. If it
isn't, please use |./mach busted file bootstrap| to report it. If |./mach busted| is
misbehaving, you can also inspect the dependencies of bug 1543241.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

subprocess.CalledProcessError: Command '['c:/Users/Ryan/.mozbuild\\_virtualenvs\\mach\\Scripts\\python.exe', 'c:\\Users\\Ryan\\repos\\mozilla\\mach', 'artifact', 'toolchain', '--bootstrap', '--from-build', 'win64-node']' returned non-zero exit status 1.

  File "c:\Users\Ryan\repos\mozilla\python/mozboot/mozboot/mach_commands.py", line 58, in bootstrap
    bootstrapper.bootstrap()
  File "c:\Users\Ryan\repos\mozilla\python/mozboot\mozboot\bootstrap.py", line 411, in bootstrap
    self.maybe_install_private_packages_or_exit(state_dir, checkout_root)
  File "c:\Users\Ryan\repos\mozilla\python/mozboot\mozboot\bootstrap.py", line 313, in maybe_install_private_packages_or_exit
    self.instance.ensure_node_packages(state_dir, checkout_root)
  File "c:\Users\Ryan\repos\mozilla\python/mozboot\mozboot\mozillabuild.py", line 218, in ensure_node_packages
    self.install_toolchain_artifact(state_dir, checkout_root, node_artifact)
  File "c:\Users\Ryan\repos\mozilla\python/mozboot\mozboot\base.py", line 399, in install_toolchain_artifact
    subprocess.check_call(cmd, cwd=state_dir)
  File "c:\mozilla-build\python3\lib\subprocess.py", line 347, in check_call
    raise CalledProcessError(retcode, cmd)

Comment 1

[Julien Cristau [:jcristau] (back April 22)]

Regressing bug was backed out.