2005180 - Perma Linux Beetmover Repackage failing with exception with ERROR - Hit ScriptWorkerException: 'path public/build/langpack-en-US.noarch.rpm not in scriptworker:repackage JiibCMQrTx635AZryWhacw chain of trust artifacts!

Status: RESOLVED FIXED

(Release Engineering :: Release Automation, defect)

Description

[amarc]

Failure Log: https://firefox-ci-tc.services.mozilla.com/tasks/JKZ7_fCORSCgkxoqwqKmHw/runs/1/logs/public/logs/chain_of_trust.log

2025-12-10T12:07:59 INFO - Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/JiibCMQrTx635AZryWhacw/artifacts/public%2Fbuild%2Ftarget.rpm 2025-12-10T12:07:59 DEBUG - /app/workdir/cot/JiibCMQrTx635AZryWhacw/public/build/langpack-vi.noarch.rpm matches the expected sha256 fa16b56d21cf9a999321d73f3f95427ff782ccf125faf55de81438715eab1285 2025-12-10T12:07:59 DEBUG - /app/workdir/cot/JiibCMQrTx635AZryWhacw/public/build/langpack-wo.noarch.rpm matches the expected sha256 239e18c2427e911a28405a496d85b9a298b5f6560a2b579d499cf532941f98b7 2025-12-10T12:07:59 INFO - Done 2025-12-10T12:07:59 DEBUG - /app/workdir/cot/JiibCMQrTx635AZryWhacw/public/build/langpack-xh.noarch.rpm matches the expected sha256 ce154e373198de9a06ec1d449b022bf857dd933cf4e8b90c3300df4329e4199f 2025-12-10T12:07:59 INFO - Done 2025-12-10T12:07:59 INFO - Done 2025-12-10T12:07:59 DEBUG - /app/workdir/cot/JiibCMQrTx635AZryWhacw/public/build/langpack-zh-CN.noarch.rpm matches the expected sha256 a809db24817d3afe730f34fd47fac1dfb860d89d47c2b9d01b16d59db985ef85 2025-12-10T12:07:59 DEBUG - /app/workdir/cot/JiibCMQrTx635AZryWhacw/public/build/langpack-zh-TW.noarch.rpm matches the expected sha256 da688eac53719fb344fc3f9ddecc6275ccaf5e757cc19ba63376ec380420080d 2025-12-10T12:08:03 INFO - Done 2025-12-10T12:08:04 DEBUG - /app/workdir/cot/JiibCMQrTx635AZryWhacw/public/build/target.rpm matches the expected sha256 e10696b0334e4004f9498723b3ac4e229790c0a64207a5dd9d75d894609a9d26 2025-12-10T12:08:04 CRITICAL - Chain of Trust verification error! Traceback (most recent call last): File "/app/.venv/lib/python3.11/site-packages/scriptworker/cot/verify.py", line 2055, in verify_chain_of_trust await download_cot_artifacts(chain) File "/app/.venv/lib/python3.11/site-packages/scriptworker/cot/verify.py", line 799, in download_cot_artifacts mandatory_artifacts_paths = await raise_future_exceptions(mandatory_artifact_tasks) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/.venv/lib/python3.11/site-packages/scriptworker/utils.py", line 416, in raise_future_exceptions succeeded_results, _ = await _process_future_exceptions(tasks, raise_at_first_error=True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/.venv/lib/python3.11/site-packages/scriptworker/utils.py", line 450, in _process_future_exceptions raise exc File "/app/.venv/lib/python3.11/site-packages/scriptworker/cot/verify.py", line 731, in download_cot_artifact raise CoTError( scriptworker.exceptions.CoTError: 'path public/build/langpack-en-US.noarch.rpm not in scriptworker:repackage JiibCMQrTx635AZryWhacw chain of trust artifacts! This is likely a bug in JiibCMQrTx635AZryWhacw: docker-worker does not fail if a declared artifact was never generated by the task. Please download and make sure the artifact public/build/langpack-en-US.noarch.rpm is the one you expect. If it is, then please reach out to the Release Engineering team. For more information: https://github.com/taskcluster/taskgraph/issues/47' 2025-12-10T12:08:04 ERROR - Hit ScriptWorkerException: 'path public/build/langpack-en-US.noarch.rpm not in scriptworker:repackage JiibCMQrTx635AZryWhacw chain of trust artifacts! This is likely a bug in JiibCMQrTx635AZryWhacw: docker-worker does not fail if a declared artifact was never generated by the task. Please download and make sure the artifact public/build/langpack-en-US.noarch.rpm is the one you expect. If it is, then please reach out to the Release Engineering team. For more information: https://github.com/taskcluster/taskgraph/issues/47' 2025-12-10T12:08:04 DEBUG - "/app/artifacts/public/logs/chain_of_trust.log" is encoded with "None" and has mime/type "text/plain" 2025-12-10T12:08:04 INFO - "/app/artifacts/public/logs/chain_of_trust.log" can be gzip'd. Compressing...

Comment 1

[amarc]

hey Alex, could Bug 2003412 have caused this?

Comment 2

[Julien Cristau [:jcristau]]

Confirmed, 040998e094c02be42ee00f9ebd35854992568959 is the first bad commit.

Comment 3

[Julien Cristau [:jcristau]]

Attachment: Bug 2005180 - fix beetmover-repackage-rpm task definition. r?#releng

As far as I can tell the fix for bug 2003412 didn't account for the addition of
not_for_locales in bug 1999650, so we ended up checking the wrong locale
variable and adding a nonexistent en-US langpack rpm to the upstream artifacts.

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/bfa84722b17d
https://hg.mozilla.org/integration/autoland/rev/62e1bf3c7fe9
fix beetmover-repackage-rpm task definition. r=releng-reviewers,taskgraph-reviewers,Eijebong

Comment 5

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 2003412

Comment 6

[Iulian Moraru]

https://hg.mozilla.org/mozilla-central/rev/62e1bf3c7fe9

Comment 7

[Intermittent Failures Robot]

1 failures were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Table

	no_variant
linux64-x86/opt	1

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=2005180&startday=2025-12-08&endday=2025-12-14&tree=all