Open Bug 2060889 Opened 1 day ago Updated 1 day ago

chain of trust failures due to mismatched hashes

Categories

(Release Engineering :: Release Automation, defect, P1)

Product:
Release Engineering
Component:
Release Automation
Type:
defect

Tracking

(Not tracked)

People

(Reporter: bhearsum, Assigned: bhearsum)

References

Details

On today's Nightly graph we had a handful of chain of trust failures such as the following:

2026-08-05T00:54:45 CRITICAL - Chain of Trust verification error!
Traceback (most recent call last):
  File "/app/.venv/lib/python3.11/site-packages/scriptworker/cot/verify.py", line 2064, in verify_chain_of_trust
    await download_cot_artifacts(chain)
  File "/app/.venv/lib/python3.11/site-packages/scriptworker/cot/verify.py", line 804, in download_cot_artifacts
    mandatory_artifacts_paths = await raise_future_exceptions(mandatory_artifact_tasks)
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.11/site-packages/scriptworker/utils.py", line 419, in raise_future_exceptions
    succeeded_results, _ = await _process_future_exceptions(tasks, raise_at_first_error=True)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.11/site-packages/scriptworker/utils.py", line 453, in _process_future_exceptions
    raise exc
  File "/app/.venv/lib/python3.11/site-packages/scriptworker/cot/verify.py", line 753, in download_cot_artifact
    raise CoTError("BAD HASH on file {}: {}: Expected {} {}; got {}!".format(full_path, link.name, alg, expected_sha, real_sha))
scriptworker.exceptions.CoTError: 'BAD HASH on file /app/workdir/cot/XEWDZtNrTruNTWmqi8zilQ/public/build/be/target.tar.gz: scriptworker:scriptworker: Expected sha256 03071d53c110b3485c248e5f7db2667056156082ba1384eeb14ca2c5b04bebdc; got 337ed76b792b1e16082ab8363ad67405ff62d37b8bb3dd5b55774d2e1ea88e97!'
2026-08-05T00:54:45     INFO - Creating link artifact public/logs/live_backing.log -> public/logs/chain_of_trust.log
2026-08-05T00:54:45    ERROR - Hit ScriptWorkerException: 'BAD HASH on file /app/workdir/cot/XEWDZtNrTruNTWmqi8zilQ/public/build/be/target.tar.gz: scriptworker:scriptworker: Expected sha256

When inspecting the upstream artifacts, I found that even when downloading them directly from the CDN origin, the hashes were wrong compared with the chain-of-trust.json. The local files (one .tar.gz, 3 .pkg files) all show up as corrupt, and cannot be unpacked. Other files of the same type from the same task have no issues.

The tasks that have produced corrupt files are all iscript machines. These machines have not had any changes deployed to them in ~2 weeks.

Debugging with an LLM, I've found that all of the corrupt files are missing some leading parts of them, and that there's a novel bug in aiohttp causing it. At first glance the repro it came up with looks very plausible; I'm attempting to further verify this theory. If it's correct, we'll be waiting on an upstream fix, but we may be able to come up with a workaround in the meantime (eg: verify the uploads ourselves, and re-do them if they are incomplete).

Okay, I'm fairly confident in the explanation and problem now. I've filed https://github.com/aio-libs/aiohttp/issues/13329 for the upstream issue, and opened https://github.com/mozilla-releng/scriptworker/pull/815 for a workaround in scriptworker.

Assignee: nobody → bhearsum
Severity: -- → S2
Priority: -- → P1
You need to log in before you can comment on or make changes to this bug.