1467605 - Bug 1460777 seems to break ./mach bootstrap on linux

Status: RESOLVED FIXED

(Firefox Build System :: Bootstrap Configuration, defect)

Description

[Michael Froman [:mjf]]

I'm running an Ubuntu 18.04 VM.  Also verified on 16.04 baremetal.
If I update to 306ce4906024 (the commit prior to 1460777 landing) ./mach bootstrap completes successfully.  If I update to 90dca0906337, ./mach bootstrap fails with the following output:

Error running mach:

    ['artifact', 'toolchain', '--from-build', 'linux64-clang']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.

You should consider filing a bug for this issue.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

Exception: Missing fetch job for toolchain-linux64-mingw32-gcc: binutils-2.27

  File "/home/mfroman/mozilla/moz-central/python/mozbuild/mozbuild/mach_commands.py", line 1389, in artifact_toolchain
    toolchains = tasks('toolchain')
  File "/home/mfroman/mozilla/moz-central/python/mozbuild/mozbuild/mach_commands.py", line 1383, in tasks
    tasks = Kind.load(root_path, graph_config, kind_name).load_tasks(params, {})
  File "/home/mfroman/mozilla/moz-central/taskcluster/taskgraph/generator.py", line 73, in load_tasks
    for task_dict in transforms(trans_config, inputs)]
  File "/home/mfroman/mozilla/moz-central/taskcluster/taskgraph/transforms/task.py", line 1710, in check_run_task_caches
    for task in tasks:
  File "/home/mfroman/mozilla/moz-central/taskcluster/taskgraph/transforms/task.py", line 1656, in check_task_dependencies
    for task in tasks:
  File "/home/mfroman/mozilla/moz-central/taskcluster/taskgraph/transforms/task.py", line 1644, in check_task_identifiers
    for task in tasks:
  File "/home/mfroman/mozilla/moz-central/taskcluster/taskgraph/transforms/task.py", line 1627, in chain_of_trust
    for task in tasks:
  File "/home/mfroman/mozilla/moz-central/taskcluster/taskgraph/transforms/task.py", line 1474, in build_task
    for task in tasks:
  File "/home/mfroman/mozilla/moz-central/taskcluster/taskgraph/transforms/task.py", line 1444, in add_index_routes
    for task in tasks:
  File "/home/mfroman/mozilla/moz-central/taskcluster/taskgraph/transforms/task.py", line 1266, in validate
    for task in tasks:
  File "/home/mfroman/mozilla/moz-central/taskcluster/taskgraph/transforms/task.py", line 1243, in task_name_from_label
    for task in tasks:
  File "/home/mfroman/mozilla/moz-central/taskcluster/taskgraph/transforms/task.py", line 1204, in set_defaults
    for task in tasks:
  File "/home/mfroman/mozilla/moz-central/taskcluster/taskgraph/transforms/job/__init__.py", line 136, in make_task_description
    for job in jobs:
  File "/home/mfroman/mozilla/moz-central/taskcluster/taskgraph/transforms/job/__init__.py", line 113, in rewrite_when_to_optimization
    for job in jobs:
  File "/home/mfroman/mozilla/moz-central/taskcluster/taskgraph/transforms/job/__init__.py", line 105, in validate
    for job in jobs:
  File "/home/mfroman/mozilla/moz-central/taskcluster/taskgraph/transforms/use_toolchains.py", line 32, in use_toolchains
    jobs = list(jobs)
  File "/home/mfroman/mozilla/moz-central/taskcluster/taskgraph/transforms/use_fetches.py", line 40, in use_fetches
    config.kind, job['name'], fetch))
Error running mach:

    ['bootstrap']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.

You should consider filing a bug for this issue.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

CalledProcessError: Command '['/usr/bin/python2.7', u'/home/mfroman/mozilla/moz-central/mach', u'artifact', u'toolchain', u'--from-build', u'linux64-clang']' returned non-zero exit status 1

  File "/home/mfroman/mozilla/moz-central/python/mozboot/mozboot/mach_commands.py", line 32, in bootstrap
    bootstrapper.bootstrap()
  File "/home/mfroman/mozilla/moz-central/python/mozboot/mozboot/bootstrap.py", line 360, in bootstrap
    self.instance.ensure_stylo_packages(state_dir, checkout_root)
  File "/home/mfroman/mozilla/moz-central/python/mozboot/mozboot/linux_common.py", line 18, in ensure_stylo_packages
    self.install_toolchain_artifact(state_dir, checkout_root, stylo.LINUX)
  File "/home/mfroman/mozilla/moz-central/python/mozboot/mozboot/base.py", line 277, in install_toolchain_artifact
    subprocess.check_call(cmd, cwd=state_dir)
  File "/usr/lib/python2.7/subprocess.py", line 190, in check_call
    raise CalledProcessError(retcode, cmd)

Comment 1

[Gregory Szorc [:gps]]

I seem to recall someone saying that it was dangerous for `mach artifact` to be making calls into low-level taskgraph code. This bug is a case in point.

This is definitely a regression from the changesets tracked by bug 1460777. I'll have a look.

Comment 2

[Mike Hommey [:glandium]]

the taskcluster code in the tree is in desperate need for a *fast* api that can pull out task information for a single kind.

Comment 3

[Phabricator Automation]

Attachment: Bug 1467605 - Disable processing of fetch dependencies; r?glandium

90dca0906337 accidentally broke `mach artifact toolchain --from-build`
because that code is attempting to load toolchain tasks in isolation.
The new "use_fetches" transform added to toolchain tasks requires
that "fetch" tasks are already processed and their references are
available to toolchain tasks.

This commit adds a mechanism to effectively disable the "use_fetches"
transform when called by `mach artifact toolchain`. It is a hack. I
suspect future planned work around artifacts/fetches will necessitate
additional changes to the `mach artifact toolchain` code. But this
can be deferred to a later day: this commit unbusts `mach artifact
toolchain` and isn't super hacky, so it seems more reasonable than
backing out fetch tasks completely.

Comment 4

[Phabricator Automation]

Comment on attachment 8984310 [details]
Bug 1467605 - Disable processing of fetch dependencies; r?glandium

Mike Hommey [:glandium] has approved the revision.

https://phabricator.services.mozilla.com/D1588

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/cdaaa5caac74
Disable processing of fetch dependencies; r=glandium

Comment 6

[Arthur Iakab [arthur_iakab]]

https://hg.mozilla.org/mozilla-central/rev/cdaaa5caac74

Comment 8

[YUKI "Piro" Hiroshi]

"status-firefox-esr60" is now marked as "unaffected", but I've seen quite same error: "Exception: Missing fetch job for toolchain-linux64-mingw32-gcc: binutils-2.27" when I've tried to run "./math bootstrap" for ESR60 on Windows 10. Manyally applied patch attached to this bug has solved the problem on my environment.

Comment 10

[Tom Prince [:tomprince]]

https://hg.mozilla.org/releases/mozilla-esr60/rev/a23dce703d2b