Closed Bug 1467605 Opened Last year Closed Last year

Bug 1460777 seems to break ./mach bootstrap on linux

Categories

(Firefox Build System :: Bootstrap Configuration, defect)

3 Branch
defect
Not set

Tracking

(firefox-esr52 unaffected, firefox-esr60 fixed, firefox60 unaffected, firefox61 unaffected, firefox62 fixed)

RESOLVED FIXED
mozilla62
Tracking Status
firefox-esr52 --- unaffected
firefox-esr60 --- fixed
firefox60 --- unaffected
firefox61 --- unaffected
firefox62 --- fixed

People

(Reporter: mjf, Assigned: gps)

References

Details

(Keywords: regression)

Attachments

(1 file)

I'm running an Ubuntu 18.04 VM.  Also verified on 16.04 baremetal.
If I update to 306ce4906024 (the commit prior to 1460777 landing) ./mach bootstrap completes successfully.  If I update to 90dca0906337, ./mach bootstrap fails with the following output:

Error running mach:

    ['artifact', 'toolchain', '--from-build', 'linux64-clang']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.

You should consider filing a bug for this issue.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

Exception: Missing fetch job for toolchain-linux64-mingw32-gcc: binutils-2.27

  File "/home/mfroman/mozilla/moz-central/python/mozbuild/mozbuild/mach_commands.py", line 1389, in artifact_toolchain
    toolchains = tasks('toolchain')
  File "/home/mfroman/mozilla/moz-central/python/mozbuild/mozbuild/mach_commands.py", line 1383, in tasks
    tasks = Kind.load(root_path, graph_config, kind_name).load_tasks(params, {})
  File "/home/mfroman/mozilla/moz-central/taskcluster/taskgraph/generator.py", line 73, in load_tasks
    for task_dict in transforms(trans_config, inputs)]
  File "/home/mfroman/mozilla/moz-central/taskcluster/taskgraph/transforms/task.py", line 1710, in check_run_task_caches
    for task in tasks:
  File "/home/mfroman/mozilla/moz-central/taskcluster/taskgraph/transforms/task.py", line 1656, in check_task_dependencies
    for task in tasks:
  File "/home/mfroman/mozilla/moz-central/taskcluster/taskgraph/transforms/task.py", line 1644, in check_task_identifiers
    for task in tasks:
  File "/home/mfroman/mozilla/moz-central/taskcluster/taskgraph/transforms/task.py", line 1627, in chain_of_trust
    for task in tasks:
  File "/home/mfroman/mozilla/moz-central/taskcluster/taskgraph/transforms/task.py", line 1474, in build_task
    for task in tasks:
  File "/home/mfroman/mozilla/moz-central/taskcluster/taskgraph/transforms/task.py", line 1444, in add_index_routes
    for task in tasks:
  File "/home/mfroman/mozilla/moz-central/taskcluster/taskgraph/transforms/task.py", line 1266, in validate
    for task in tasks:
  File "/home/mfroman/mozilla/moz-central/taskcluster/taskgraph/transforms/task.py", line 1243, in task_name_from_label
    for task in tasks:
  File "/home/mfroman/mozilla/moz-central/taskcluster/taskgraph/transforms/task.py", line 1204, in set_defaults
    for task in tasks:
  File "/home/mfroman/mozilla/moz-central/taskcluster/taskgraph/transforms/job/__init__.py", line 136, in make_task_description
    for job in jobs:
  File "/home/mfroman/mozilla/moz-central/taskcluster/taskgraph/transforms/job/__init__.py", line 113, in rewrite_when_to_optimization
    for job in jobs:
  File "/home/mfroman/mozilla/moz-central/taskcluster/taskgraph/transforms/job/__init__.py", line 105, in validate
    for job in jobs:
  File "/home/mfroman/mozilla/moz-central/taskcluster/taskgraph/transforms/use_toolchains.py", line 32, in use_toolchains
    jobs = list(jobs)
  File "/home/mfroman/mozilla/moz-central/taskcluster/taskgraph/transforms/use_fetches.py", line 40, in use_fetches
    config.kind, job['name'], fetch))
Error running mach:

    ['bootstrap']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.

You should consider filing a bug for this issue.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

CalledProcessError: Command '['/usr/bin/python2.7', u'/home/mfroman/mozilla/moz-central/mach', u'artifact', u'toolchain', u'--from-build', u'linux64-clang']' returned non-zero exit status 1

  File "/home/mfroman/mozilla/moz-central/python/mozboot/mozboot/mach_commands.py", line 32, in bootstrap
    bootstrapper.bootstrap()
  File "/home/mfroman/mozilla/moz-central/python/mozboot/mozboot/bootstrap.py", line 360, in bootstrap
    self.instance.ensure_stylo_packages(state_dir, checkout_root)
  File "/home/mfroman/mozilla/moz-central/python/mozboot/mozboot/linux_common.py", line 18, in ensure_stylo_packages
    self.install_toolchain_artifact(state_dir, checkout_root, stylo.LINUX)
  File "/home/mfroman/mozilla/moz-central/python/mozboot/mozboot/base.py", line 277, in install_toolchain_artifact
    subprocess.check_call(cmd, cwd=state_dir)
  File "/usr/lib/python2.7/subprocess.py", line 190, in check_call
    raise CalledProcessError(retcode, cmd)
Flags: needinfo?(gps)
I seem to recall someone saying that it was dangerous for `mach artifact` to be making calls into low-level taskgraph code. This bug is a case in point.

This is definitely a regression from the changesets tracked by bug 1460777. I'll have a look.
Assignee: nobody → gps
Status: NEW → ASSIGNED
Depends on: 1460777
Flags: needinfo?(gps)
the taskcluster code in the tree is in desperate need for a *fast* api that can pull out task information for a single kind.
90dca0906337 accidentally broke `mach artifact toolchain --from-build`
because that code is attempting to load toolchain tasks in isolation.
The new "use_fetches" transform added to toolchain tasks requires
that "fetch" tasks are already processed and their references are
available to toolchain tasks.

This commit adds a mechanism to effectively disable the "use_fetches"
transform when called by `mach artifact toolchain`. It is a hack. I
suspect future planned work around artifacts/fetches will necessitate
additional changes to the `mach artifact toolchain` code. But this
can be deferred to a later day: this commit unbusts `mach artifact
toolchain` and isn't super hacky, so it seems more reasonable than
backing out fetch tasks completely.
Comment on attachment 8984310 [details]
Bug 1467605 - Disable processing of fetch dependencies; r?glandium

Mike Hommey [:glandium] has approved the revision.

https://phabricator.services.mozilla.com/D1588
Attachment #8984310 - Flags: review+
Pushed by gszorc@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/cdaaa5caac74
Disable processing of fetch dependencies; r=glandium
https://hg.mozilla.org/mozilla-central/rev/cdaaa5caac74
Status: ASSIGNED → RESOLVED
Closed: Last year
Resolution: --- → FIXED
Target Milestone: --- → mozilla62
Duplicate of this bug: 1467773
"status-firefox-esr60" is now marked as "unaffected", but I've seen quite same error: "Exception: Missing fetch job for toolchain-linux64-mingw32-gcc: binutils-2.27" when I've tried to run "./math bootstrap" for ESR60 on Windows 10. Manyally applied patch attached to this bug has solved the problem on my environment.
Duplicate of this bug: 1505372
Keywords: regression
Version: Version 3 → 3 Branch
You need to log in before you can comment on or make changes to this bug.