1798993 - `./mach bootstap` failure, requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/W6yNR7UGRluCUYrsXe6x7Q/artifacts/public/chain-of-trust.json

Status: NEW

(Firefox Build System :: Bootstrap Configuration, defect, P3)

Description

[Alastor Wu [:alwu]]

On the latest Nightly on Windows, run $./mach bootstrap

The details of the failure are as follows:

requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/W6yNR7UGRluCUYrsXe6x7Q/artifacts/public/chain-of-trust.json

  File "c:\mozilla-source\mozilla-unified\python\mozbuild\mozbuild\artifact_commands.py", line 467, in artifact_toolchain
    record = ArtifactRecord(task_id, artifact_name)
  File "c:\mozilla-source\mozilla-unified\python\mozbuild\mozbuild\artifact_commands.py", line 326, in __init__
    cot.raise_for_status()
  File "c:\mozilla-source\mozilla-unified\third_party\python\requests\requests\models.py", line 943, in raise_for_status
    raise HTTPError(http_error_msg, response=self)

Sentry event ID: 121f0cc6e3304f5cbf9dba90ff57b38c
Error running mach:

    ['bootstrap']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.
You can invoke |./mach busted| to check if this issue is already on file. If it
isn't, please use |./mach busted file bootstrap| to report it. If |./mach busted| is
misbehaving, you can also inspect the dependencies of bug 1543241.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

subprocess.CalledProcessError: Command '['C:\\Users\\Alastor\\.mozbuild\\srcdirs\\mozilla-unified-f689344771d4\\_virtualenvs\\mach\\Scripts\\python.exe', 'C:\\mozilla-source\\mozilla-unified\\mach', 'artifact', 'toolchain', '--bootstrap', '--from-build', 'win64-clang-tidy']' returned non-zero exit status 1.

  File "C:\mozilla-source\mozilla-unified\python\mozboot\mozboot\mach_commands.py", line 43, in bootstrap
    bootstrapper.bootstrap(command_context.settings)
  File "c:\mozilla-source\mozilla-unified\python\mozboot\mozboot\bootstrap.py", line 384, in bootstrap
    self.maybe_install_private_packages_or_exit(application)
  File "c:\mozilla-source\mozilla-unified\python\mozboot\mozboot\bootstrap.py", line 258, in maybe_install_private_packages_or_exit
    self.instance.ensure_clang_static_analysis_package()
  File "c:\mozilla-source\mozilla-unified\python\mozboot\mozboot\mozillabuild.py", line 230, in ensure_clang_static_analysis_package
    self.install_toolchain_static_analysis(static_analysis.WINDOWS_CLANG_TIDY)
  File "c:\mozilla-source\mozilla-unified\python\mozboot\mozboot\base.py", line 392, in install_toolchain_static_analysis
    self.install_toolchain_artifact_impl(clang_tools_path, toolchain_job)
  File "c:\mozilla-source\mozilla-unified\python\mozboot\mozboot\base.py", line 429, in install_toolchain_artifact_impl
    subprocess.check_call(cmd, cwd=str(install_dir))
  File "C:\mozilla-build\python3\lib\subprocess.py", line 373, in check_call
    raise CalledProcessError(retcode, cmd)

Sentry event ID: 8a558ae0a44d4807af26058282f80bf6

Comment 1

[Mike Hommey [:glandium]]

Andrew, this sounds like this could be related to your CoT retriggers?

Comment 2

[Mike Hommey [:glandium]]

Yeah, what seems to be happening is that the CoT retrigger has triggered new runs, and artifact downloads will fail until those runs finish... that kind of sucks, I wonder what we can do to improve the situation.

Comment 3

[Andrew Halberstadt [:ahal]]

Hm yeah.. I guess we use "eager-indexes" so that the tasks populate the index as soon as they start (as opposed to when they finish). We do this to avoid the scenario where someone modifies a toolchain, then every subsequent autoland push starts rebuilding that same toolchain until the first one finishes.

Maybe we could have a separate index reserved for ./mach bootstrap that isn't eager. That way, it would always point to a finished task.

Comment 4

[Andrew Halberstadt [:ahal]]

Although this implies that only the TARGET_CACHE_INDEX is eager and the other two (latest and the build date one) are not. I would have assumed that ./mach bootstrap would be using that latest index.. so not sure what's going on.

Comment 5

[Mike Hommey [:glandium]]

no, bootstrap uses the exact one, not latest or date based.

Comment 6

[Andrew Halberstadt [:ahal]]

Ah ok. So if switching to latest isn't an option, we could make a copy of the digest based index that isn't eager and use that in bootstrap instead.

Comment 7

[Ana]

Hello was there a workaround that came up for this issue in any way?
I am trying to install a couple of files and gecko gives me this error post running ./mach bootstrap.