1660381 - Could not find artifacts for a toolchain build named `linux64-node`

Status: RESOLVED WORKSFORME

(Firefox Build System :: Bootstrap Configuration, defect)

Description

[Jason Kratzer [:jkratzer]]

Running ./mach bootstrap on mozilla-central rev e375b85cfba3 fails with the following error message:

 0:05.67 Could not find artifacts for a toolchain build named `linux64-node`. Local commits, dirty/stale files, and other changes in your checkout may cause this error. Make sure you are on a fresh, current checkout of mozilla-central. If you are already, you may be able to avoid this error by running `mach clobber python`. Beware that commands like `mach bootstrap` and `mach artifact` are unlikely to work on any versions of the code besides recent revisions of mozilla-central.
Error running mach:

    ['bootstrap']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.
You can invoke |./mach busted| to check if this issue is already on file. If it
isn't, please use |./mach busted file bootstrap| to report it. If |./mach busted| is
misbehaving, you can also inspect the dependencies of bug 1543241.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

subprocess.CalledProcessError: Command '['/usr/bin/python3', '/home/user/source/mozilla-central/mach', 'artifact', 'toolchain', '--bootstrap', '--from-build', 'linux64-node']' returned non-zero exit status 1.

  File "/home/user/source/mozilla-central/python/mozboot/mozboot/mach_commands.py", line 45, in bootstrap
    bootstrapper.bootstrap()
  File "/home/user/source/mozilla-central/python/mozboot/mozboot/bootstrap.py", line 578, in bootstrap
    checkout_root)
  File "/home/user/source/mozilla-central/python/mozboot/mozboot/bootstrap.py", line 421, in maybe_install_private_packages_or_exit
    self.instance.ensure_node_packages(state_dir, checkout_root)
  File "/home/user/source/mozilla-central/python/mozboot/mozboot/linux_common.py", line 102, in ensure_node_packages
    self.install_toolchain_artifact(state_dir, checkout_root, node.LINUX)
  File "/home/user/source/mozilla-central/python/mozboot/mozboot/base.py", line 356, in install_toolchain_artifact
    subprocess.check_call(cmd, cwd=state_dir)
  File "/usr/lib/python3.6/subprocess.py", line 311, in check_call
    raise CalledProcessError(retcode, cmd)

Comment 1

[Mike Hommey [:glandium]]

Do you have modifications to taskcluster/scripts/misc/repack-node.sh or taskcluster/ci/toolchain/node.yml in your tree? That's about the only way this could happen.

Comment 2

[Jason Kratzer [:jkratzer]]

I do not. This is on a brand new clone of mozilla-central.

Comment 3

[Ricky Stewart]

Can you still reproduce the issue today?

If so, to be clear, hg parent shows that you're updated to the latest central, and hg status (and hg status -i) are clean?

If so, does mach clobber python fix it, or no? What about an hg purge --all (you can exclude your mozconfig and objdir to avoid losing a bunch of state if you need to)?

(Sorry for all the questions, I'm trying to isolate the source of the problem.)

Comment 4

[Jason Kratzer [:jkratzer]]

Attachment: bootstrap.log.txt

Comment 5

[Jason Kratzer [:jkratzer]]

Yes, I can still reproduce it today as of m-c 98f04ffcd64a. I've attached the full ./mach bootstrap log including the commands you suggested.

Comment 6

[Mitchell Hentges [:mhentges] 🦀]

Hey, can you paste the results of hg files -0? I wonder if unexpected files are being picked up 🤔
I've just done a fresh clone of mozilla-central, checked out 98f04ffcd64a, performed a ./mach clobber python and hg purge --all, and yet my bootstrap isn't reproducing the issue, which is a bit of a bummer.

Comment 7

[Mike Hommey [:glandium]]

In the case of node, there is no wildcard involved. The only file that is being hashed for it is taskcluster/scripts/misc/repack-node.sh. What is the sha256sum of that file in your tree?

Comment 8

[Jason Kratzer [:jkratzer]]

Attachment: files.log.zip

Comment 9

[Jason Kratzer [:jkratzer]]

(In reply to Mitchell Hentges [:mhentges] 🦀 from comment #6)

Hey, can you paste the results of hg files -0? I wonder if unexpected files are being picked up 🤔
I've just done a fresh clone of mozilla-central, checked out 98f04ffcd64a, performed a ./mach clobber python and hg purge --all, and yet my bootstrap isn't reproducing the issue, which is a bit of a bummer.

Mitchell, I've attached the output of hg files.

Comment 10

[Mitchell Hentges [:mhentges] 🦀]

Thanks! It looks like there's exactly one repack-node.sh detected by hg, so that's good!
Two other questions to help with investigation:

• Can you run sha256sum taskcluster/scripts/misc/repack-node.sh and paste the result?
• Which hg revision are you on (hd id)? I'm guessing 2944e59?

EDIT: My local hg files -0 result is identical to yours, which is a good sign :)
I'm very interested in your repack-node.sh hash. FWIW, mine is 68405772781a5f18a65ad1841053884ed23b9fd63313e1b3628bc4689798b298.

Comment 11

[Mitchell Hentges [:mhentges] 🦀]

Attachment: debug-artifact.patch

Also, if you don't mind, try applying this patch (hg import --no-commit ~/debug-artifact.patch) and paste the output.

Comment 12

[Jason Kratzer [:jkratzer]]

(In reply to Mitchell Hentges [:mhentges] 🦀 from comment #10)

Thanks! It looks like there's exactly one repack-node.sh detected by hg, so that's good!
Two other questions to help with investigation:

• Can you run sha256sum taskcluster/scripts/misc/repack-node.sh and paste the result?

$ sha256sum taskcluster/scripts/misc/repack-node.sh
68405772781a5f18a65ad1841053884ed23b9fd63313e1b3628bc4689798b298  taskcluster/scripts/misc/repack-node.sh

• Which hg revision are you on (hd id)? I'm guessing 2944e59?

$ hg id
63d860f118da tip

Comment 13

[Jason Kratzer [:jkratzer]]

(In reply to Mitchell Hentges [:mhentges] 🦀 from comment #11)

Created attachment 9171746 [details] [diff] [review]
debug-artifact.patch

Also, if you don't mind, try applying this patch (hg import --no-commit ~/debug-artifact.patch) and paste the output.

head_ref: 63d860f118da09320ed65431129775b4e3631ff3
repack-node.sh hash: 68405772781a5f18a65ad1841053884ed23b9fd63313e1b3628bc4689798b298
hash_paths(...): ['803f598fc69d38059626f273a4955b9f8d11b995946a2018c2a144e4fd578f92']
taskdesc["cache"]["digest-data"]: ['803f598fc69d38059626f273a4955b9f8d11b995946a2018c2a144e4fd578f92', 'public/build/node.tar.xz', 'deb8-toolchain-build', 'linux64', '10']
repack-node.sh hash: 68405772781a5f18a65ad1841053884ed23b9fd63313e1b3628bc4689798b298
repack-node.sh hash: 68405772781a5f18a65ad1841053884ed23b9fd63313e1b3628bc4689798b298
repack-node.sh hash: 68405772781a5f18a65ad1841053884ed23b9fd63313e1b3628bc4689798b298
task["dependencies"]: {}
linux64-node-10 digest: 644e95189325a06123aee8cbd0daef7a223350db9d81cf563b91e0bec30f9805
task.optimization["index-search"]: ['gecko.cache.level-3.toolchains.v3.linux64-node-10.hash.644e95189325a06123aee8cbd0daef7a223350db9d81cf563b91e0bec30f9805']
 0:05.39 Could not find artifacts for a toolchain build named `linux64-node`. Local commits, dirty/stale files, and other changes in your checkout may cause this error. Make sure you are on a fresh, current checkout of mozilla-central. If you are already, you may be able to avoid this error by running `mach clobber python`. Beware that commands like `mach bootstrap` and `mach artifact` are unlikely to work on any versions of the code besides recent revisions of mozilla-central.
Error running mach:

    ['bootstrap']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.
You can invoke |./mach busted| to check if this issue is already on file. If it
isn't, please use |./mach busted file bootstrap| to report it. If |./mach busted| is
misbehaving, you can also inspect the dependencies of bug 1543241.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

subprocess.CalledProcessError: Command '['/usr/bin/python3', '/home/user/source/mozilla-central/mach', 'artifact', 'toolchain', '--bootstrap', '--from-build', 'linux64-node']' returned non-zero exit status 1.

  File "/home/user/source/mozilla-central/python/mozboot/mozboot/mach_commands.py", line 45, in bootstrap
    bootstrapper.bootstrap()
  File "/home/user/source/mozilla-central/python/mozboot/mozboot/bootstrap.py", line 578, in bootstrap
    checkout_root)
  File "/home/user/source/mozilla-central/python/mozboot/mozboot/bootstrap.py", line 421, in maybe_install_private_packages_or_exit
    self.instance.ensure_node_packages(state_dir, checkout_root)
  File "/home/user/source/mozilla-central/python/mozboot/mozboot/linux_common.py", line 102, in ensure_node_packages
    self.install_toolchain_artifact(state_dir, checkout_root, node.LINUX)
  File "/home/user/source/mozilla-central/python/mozboot/mozboot/base.py", line 356, in install_toolchain_artifact
    subprocess.check_call(cmd, cwd=state_dir)
  File "/usr/lib/python3.6/subprocess.py", line 311, in check_call
    raise CalledProcessError(retcode, cmd)

Comment 14

[Mitchell Hentges [:mhentges] 🦀]

Attachment: debug-artifact-with-urls.patch

That's really helpful, thanks!
Looks like the task IDs are being generated correctly - can you load this link in your browser? I wonder if mach is failing to communicate with Taskcluster 🤔

I've added a new patch to test that^ theory. Can you apply it and paste your logs again?

Comment 15

[Jason Kratzer [:jkratzer]]

Welp, looks like I've found the issue. I had TASKCLUSTER_ROOT_URL=https://community-tc.services.mozilla.com set system wide which was causing the artifact URL to resolve to https://community-tc.services.mozilla.com/api/index/v1/task/gecko.cache.level-3.toolchains.v3.linux64-node-10.hash.644e95189325a06123aee8cbd0daef7a223350db9d81cf563b91e0bec30f9805.

Thanks so much for helping me debug this. I'll go ahead and close it as WFM.

Comment 16

[Mitchell Hentges [:mhentges] 🦀]

No worries, glad we got to the bottom of it. I'll submit a patch to improve the error message when artifacts can't be found 😁