1768066 - mach bootstrap fails in NON-artifact mode, complaining "Could not find artifacts" which makes no sense

Status: RESOLVED DUPLICATE of bug 1771269

(Firefox Build System :: Task Configuration, defect)

Description

[David A. Madore]

Running mach bootstrap for Firefox 100.0 release (Mercurial commit bb58f916e0964688d75619c1171bddf773a50d36) on Ubuntu 18.04.6 LTS and selecting NON-artifact mode fails with the error

Could not find artifacts for a toolchain build named `linux64-clang-tidy`

which obviously makes no sense at all since the point is to NOT use artifacts. Full output of mach bootstrap follows:

Note on Artifact Mode:

Artifact builds download prebuilt C++ components rather than building
them locally. Artifact builds are faster!

Artifact builds are recommended for people working on Firefox or
Firefox for Android frontends, or the GeckoView Java API. They are unsuitable
for those working on C++ code. For more information see:
https://firefox-source-docs.mozilla.org/contributing/build/artifact_builds.html.

Please choose the version of Firefox you want to build:
  1. Firefox for Desktop Artifact Mode [default]
  2. Firefox for Desktop
  3. GeckoView/Firefox for Android Artifact Mode
  4. GeckoView/Firefox for Android
  5. SpiderMonkey JavaScript engine
Your choice:2
Requirement already satisfied: glean-sdk==44.0.0 in /usr/src/local/mozilla-pseudohome/.mozbuild/srcdirs/mozilla-2a3bc27b5592/_virtualenvs/mach/lib/python3.6/site-packages (44.0.0)
Requirement already satisfied: cffi>=1.13.0 in /usr/src/local/mozilla-pseudohome/.mozbuild/srcdirs/mozilla-2a3bc27b5592/_virtualenvs/mach/lib/python3.6/site-packages (from glean-sdk==44.0.0) (1.15.0)
Requirement already satisfied: glean-parser==5.0.1 in /usr/src/local/mozilla-pseudohome/.mozbuild/srcdirs/mozilla-2a3bc27b5592/_virtualenvs/mach/lib/python3.6/site-packages (from glean-sdk==44.0.0) (5.0.1)
Requirement already satisfied: iso8601>=0.1.10 in ./third_party/python/iso8601 (from glean-sdk==44.0.0) (0.1.14)
Requirement already satisfied: diskcache>=4 in ./third_party/python/diskcache (from glean-parser==5.0.1->glean-sdk==44.0.0) (4.1.0)
Requirement already satisfied: Click>=7 in ./third_party/python/click (from glean-parser==5.0.1->glean-sdk==44.0.0) (7.1.2)
Requirement already satisfied: appdirs>=1.4 in ./third_party/python/appdirs (from glean-parser==5.0.1->glean-sdk==44.0.0) (1.4.4)
Requirement already satisfied: jsonschema>=3.0.2 in ./third_party/python/jsonschema (from glean-parser==5.0.1->glean-sdk==44.0.0) (3.2.0)
Requirement already satisfied: yamllint>=1.18.0 in ./third_party/python/yamllint (from glean-parser==5.0.1->glean-sdk==44.0.0) (1.23.0)
Requirement already satisfied: Jinja2>=2.10.1 in ./third_party/python/Jinja2 (from glean-parser==5.0.1->glean-sdk==44.0.0) (2.11.3)
Requirement already satisfied: PyYAML>=5.3.1 in ./third_party/python/PyYAML/lib3 (from glean-parser==5.0.1->glean-sdk==44.0.0) (5.4.1)
Requirement already satisfied: pycparser in /usr/src/local/mozilla-pseudohome/.mozbuild/srcdirs/mozilla-2a3bc27b5592/_virtualenvs/mach/lib/python3.6/site-packages (from cffi>=1.13.0->glean-sdk==44.0.0) (2.21)
Requirement already satisfied: MarkupSafe>=0.23 in ./third_party/python/MarkupSafe/src (from Jinja2>=2.10.1->glean-parser==5.0.1->glean-sdk==44.0.0) (1.1.1)
Requirement already satisfied: setuptools in ./third_party/python/setuptools (from jsonschema>=3.0.2->glean-parser==5.0.1->glean-sdk==44.0.0) (51.2.0)
Requirement already satisfied: importlib-metadata in ./third_party/python/importlib_metadata (from jsonschema>=3.0.2->glean-parser==5.0.1->glean-sdk==44.0.0) (1.7.0)
Requirement already satisfied: pyrsistent>=0.14.0 in ./third_party/python/pyrsistent (from jsonschema>=3.0.2->glean-parser==5.0.1->glean-sdk==44.0.0) (0.16.0)
Requirement already satisfied: six>=1.11.0 in ./third_party/python/six (from jsonschema>=3.0.2->glean-parser==5.0.1->glean-sdk==44.0.0) (1.13.0)
Requirement already satisfied: attrs>=17.4.0 in ./third_party/python/attrs (from jsonschema>=3.0.2->glean-parser==5.0.1->glean-sdk==44.0.0) (19.2.0)
Requirement already satisfied: pathspec>=0.5.3 in ./third_party/python/pathspec (from yamllint>=1.18.0->glean-parser==5.0.1->glean-sdk==44.0.0) (0.9.0)
Requirement already satisfied: zipp>=0.5 in ./third_party/python/zipp (from importlib-metadata->jsonschema>=3.0.2->glean-parser==5.0.1->glean-sdk==44.0.0) (3.4.1)
Requirement already satisfied: psutil<=5.8.0,>=5.4.2 in /usr/src/local/mozilla-pseudohome/.mozbuild/srcdirs/mozilla-2a3bc27b5592/_virtualenvs/mach/lib/python3.6/site-packages (5.8.0)
Requirement already satisfied: zstandard<=0.17.0,>=0.11.1 in /usr/src/local/mozilla-pseudohome/.mozbuild/srcdirs/mozilla-2a3bc27b5592/_virtualenvs/mach/lib/python3.6/site-packages (0.17.0)
Executing as root: sudo apt-get install build-essential libpython3-dev m4 unzip uuid zip
Executing as root: sudo apt-get install libasound2-dev libcurl4-openssl-dev libdbus-1-dev libdbus-glib-1-dev libdrm-dev libgtk-3-dev libpulse-dev libx11-xcb-dev libxt-dev xvfb
Executing as root: sudo apt-get update
Your version of Mercurial (4.5.3) is not modern enough.
(Older versions of Mercurial have known security vulnerabilities. Unless you are running a patched Mercurial version, you may be vulnerable.

Mercurial releases a new version every 3 months and your distro's package
may become out of date. This may cause incompatibility with some
Mercurial extensions that rely on new Mercurial features. As a result,
you may not have an optimal version control experience.

To have the best Mercurial experience possible, we recommend installing
Mercurial via the "pip" Python packaging utility. This will likely result
in files being placed in /usr/local/bin and /usr/local/lib.

How would you like to continue?
  1. Install a modern Mercurial via pip [default]
  2. Install a legacy Mercurial via apt
  3. Do not install Mercurial
Your choice: 3
Not installing Mercurial.
Your version of Rust (1.60.0) is new enough.
Rust supports x86_64-unknown-linux-gnu targets.

Mozilla recommends a number of changes to Mercurial to enhance your
experience with it.

Would you like to run a configuration wizard to ensure Mercurial is
optimally configured? (Yn): n
checking for vcs source checkout... hg
checking for a shell... /bin/sh
checking for host system type... x86_64-pc-linux-gnu
 0:04.16 ERROR!!!!!! Could not find artifacts for a toolchain build named `linux64-clang-tidy`. Local commits, dirty/stale files, and other changes in your checkout may cause this error. Make sure you are on a fresh, current checkout of mozilla-central. Beware that commands like `mach bootstrap` and `mach artifact` are unlikely to work on any versions of the code besides recent revisions of mozilla-central.
Error running mach:

    ['bootstrap']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.
You can invoke |./mach busted| to check if this issue is already on file. If it
isn't, please use |./mach busted file bootstrap| to report it. If |./mach busted| is
misbehaving, you can also inspect the dependencies of bug 1543241.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

subprocess.CalledProcessError: Command '['/usr/local/src/mozilla-pseudohome/.mozbuild/srcdirs/mozilla-2a3bc27b5592/_virtualenvs/mach/bin/python', '/usr/src/local/mozilla/mach', 'artifact', 'toolchain', '--bootstrap', '--from-build', 'linux64-clang-tidy']' returned non-zero exit status 1.

  File "/usr/src/local/mozilla/python/mozboot/mozboot/mach_commands.py", line 43, in bootstrap
    bootstrapper.bootstrap(command_context.settings)
  File "/usr/src/local/mozilla/python/mozboot/mozboot/bootstrap.py", line 367, in bootstrap
    self.maybe_install_private_packages_or_exit(application)
  File "/usr/src/local/mozilla/python/mozboot/mozboot/bootstrap.py", line 241, in maybe_install_private_packages_or_exit
    self.instance.ensure_clang_static_analysis_package()
  File "/usr/src/local/mozilla/python/mozboot/mozboot/linux_common.py", line 94, in ensure_clang_static_analysis_package
    self.install_toolchain_static_analysis(static_analysis.LINUX_CLANG_TIDY)
  File "/usr/src/local/mozilla/python/mozboot/mozboot/base.py", line 376, in install_toolchain_static_analysis
    self.install_toolchain_artifact_impl(clang_tools_path, toolchain_job)
  File "/usr/src/local/mozilla/python/mozboot/mozboot/base.py", line 434, in install_toolchain_artifact_impl
    subprocess.check_call(cmd, cwd=str(install_dir))
  File "/usr/lib/python3.6/subprocess.py", line 311, in check_call
    raise CalledProcessError(retcode, cmd)

Comment 1

[Alex Hochheiden [:ahochheiden]]

Hey David,

The naming convention is a bit misleading, but it does make sense. There's a difference between the Toolchain Artifacts and the Build Artifacts. When selecting 2, the Toolchain Artifacts need to be Bootstrapped because they're all necessary to complete the build. So even though you're not doing an Artifact Build (using Build Artifacts) the Toolchain Artifacts still need to be Bootstrapped.

Good news is that I was able to easily reproduce this. I think it's a problem with that particular revision (bb58f916e0964688d75619c1171bddf773a50d36) and it not having the Toolchain Artifacts built for it. I was able to Bootstrap successfully on the latest central (revision: 768b37c0ea1a).

Would you mind doing the following and trying again?

hg pull
hg up 768b37c0ea1a

(You should also be able to do hg up central instead, since I think the Toolchain Artifacts are always built for the head revision in the push group)

If need to be on the specific revision you mentioned, you should be able to swap back to it after the bootstrap and it should work fine. Let me know how it goes, if that doesn't solve your problem I'll investigate further.

Comment 2

[Mike Hommey [:glandium]]

What happened here is that this change (or another in the same push) triggered a change in the hashes for toolchains. But the toolchain tasks themselves weren't all triggered, so when you the command was run on Thursday, no corresponding clang-tidy task could be found. Since then, on Friday, the clang-tidy task was triggered via a searchfox indexing cron, so this fixed itself.

But what this tells me, really, is that tasks with a local-toolchain attribute should be triggered whether a task that depend on them is triggered or not.