Closed
Bug 2003144
Opened 6 months ago
Closed 6 months ago
Perma Beetmover Repackages exceptions fail with malformed payload
Categories
(Release Engineering :: Release Automation, defect)
Release Engineering
Release Automation
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: SerbanS, Assigned: jcristau)
References
Details
Attachments
(2 files)
Push with failures: https://treeherder.mozilla.org/jobs?repo=mozilla-central&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel%2Crunning%2Cpending%2Crunnable&searchStr=beetmover&revision=5d9afcd27956a4b07b5bc5cba0b6407922727791&selectedTaskRun=E4_9L2h7SPu3XvqYZcCSBw.0
Failure log: https://firefox-ci-tc.services.mozilla.com/tasks/E4_9L2h7SPu3XvqYZcCSBw/runs/4/logs/public/logs/chain_of_trust.log
'taskId': None,
'tasks_for': 'hg-push'}
2025-11-29T17:55:59 CRITICAL - scriptworker:build:docker-image:parent bpxtF8TfRN-Hpncs9G2fdA: the runtime task doesn't match any rebuilt definition!
["[('change',\n"
" 'payload.image',\n"
' '
"('mozillareleases/gecko_decision:5.1.0@sha256:8879ecb7f859e1001b0aa67de440a18c3a896446bfd0e2e87688bb367e590667',\n"
' '
"'mozillareleases/gecko_decision:5.0.0@sha256:9e000c9fd31ee1723927aeea2b82b95ff2ecf32ebdb7a5d893300281a1452670')),\n"
" ('change',\n"
" ['payload', 'command', 0],\n"
" ('/builds/worker/bin/run-task-hg', '/builds/worker/bin/run-task'))]"]
2025-11-29T17:55:59 CRITICAL - Chain of Trust verification error!
Traceback (most recent call last):
File "/app/.venv/lib/python3.11/site-packages/scriptworker/cot/verify.py", line 1737, in verify_parent_task
await verify_parent_task_definition(chain, link)
File "/app/.venv/lib/python3.11/site-packages/scriptworker/cot/verify.py", line 1611, in verify_parent_task_definition
compare_jsone_task_definition(parent_link, rebuilt_definitions)
File "/app/.venv/lib/python3.11/site-packages/scriptworker/cot/verify.py", line 1694, in compare_jsone_task_definition
raise CoTError(error_msg)
scriptworker.exceptions.CoTError: 'scriptworker:build:docker-image:parent bpxtF8TfRN-Hpncs9G2fdA: the runtime task doesn\'t match any rebuilt definition!\n["[(\'change\',\\n"\n " \'payload.image\',\\n"\n \' \'\n "(\'mozillareleases/gecko_decision:5.1.0@sha256:8879ecb7f859e1001b0aa67de440a18c3a896446bfd0e2e87688bb367e590667\',\\n"\n \' \'\n "\'mozillareleases/gecko_decision:5.0.0@sha256:9e000c9fd31ee1723927aeea2b82b95ff2ecf32ebdb7a5d893300281a1452670\')),\\n"\n " (\'change\',\\n"\n " [\'payload\', \'command\', 0],\\n"\n " (\'/builds/worker/bin/run-task-hg\', \'/builds/worker/bin/run-task\'))]"]'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/app/.venv/lib/python3.11/site-packages/scriptworker/cot/verify.py", line 2055, in verify_chain_of_trust
await verify_task_types(chain)
File "/app/.venv/lib/python3.11/site-packages/scriptworker/cot/verify.py", line 1813, in verify_task_types
await valid_task_types[task_type](chain, obj)
File "/app/.venv/lib/python3.11/site-packages/scriptworker/cot/verify.py", line 1739, in verify_parent_task
raise CoTError(e)
scriptworker.exceptions.CoTError: CoTError('scriptworker:build:docker-image:parent bpxtF8TfRN-Hpncs9G2fdA: the runtime task doesn\'t match any rebuilt definition!\n["[(\'change\',\\n"\n " \'payload.image\',\\n"\n \' \'\n "(\'mozillareleases/gecko_decision:5.1.0@sha256:8879ecb7f859e1001b0aa67de440a18c3a896446bfd0e2e87688bb367e590667\',\\n"\n \' \'\n "\'mozillareleases/gecko_decision:5.0.0@sha256:9e000c9fd31ee1723927aeea2b82b95ff2ecf32ebdb7a5d893300281a1452670\')),\\n"\n " (\'change\',\\n"\n " [\'payload\', \'command\', 0],\\n"\n " (\'/builds/worker/bin/run-task-hg\', \'/builds/worker/bin/run-task\'))]"]')
2025-11-29T17:55:59 ERROR - Hit ScriptWorkerException: CoTError('scriptworker:build:docker-image:parent bpxtF8TfRN-Hpncs9G2fdA: the runtime task doesn\'t match any rebuilt definition!\n["[(\'change\',\\n"\n " \'payload.image\',\\n"\n \' \'\n "(\'mozillareleases/gecko_decision:5.1.0@sha256:8879ecb7f859e1001b0aa67de440a18c3a896446bfd0e2e87688bb367e590667\',\\n"\n \' \'\n "\'mozillareleases/gecko_decision:5.0.0@sha256:9e000c9fd31ee1723927aeea2b82b95ff2ecf32ebdb7a5d893300281a1452670\')),\\n"\n " (\'change\',\\n"\n " [\'payload\', \'command\', 0],\\n"\n " (\'/builds/worker/bin/run-task-hg\', \'/builds/worker/bin/run-task\'))]"]')
2025-11-29T17:55:59 DEBUG - "/app/artifacts/public/logs/chain_of_trust.log" is encoded with "None" and has mime/type "text/plain"
2025-11-29T17:55:59 INFO - "/app/artifacts/public/logs/chain_of_trust.log" can be gzip'd. Compressing...
|
Reporter | |
Comment 1•6 months ago
•
|
||
After the latest nightlies, there are still issues with Beetmover Repackages exceptions on Windows. On top of that, we have these failures now.
Hi Julien! Could you please check this bug? This was discussed in https://matrix.to/#/!SASNEOjTdjZfoqDIAv:mozilla.org/$JyMxK3ytQzQGeuQWGBbcLDj7LOkzP2IA0w9QgRA5FJQ?via=mozilla.org. We are interested in what has been done in the updated .taskcluster.yml.
Thank you!
Flags: needinfo?(jcristau)
| Comment hidden (Intermittent Failures Robot) |
|
||
Comment 3•6 months ago
|
||
|
|
||
Comment 4•6 months ago
|
||
|
Assignee | |
Updated•6 months ago
|
Assignee: nobody → jcristau
Status: NEW → ASSIGNED
|
|
Assignee | |
Comment 6•6 months ago
|
||
nightly buildid 20251201102759 should now be available for all platforms/locales.
Status: ASSIGNED → RESOLVED
Closed: 6 months ago
Resolution: --- → FIXED
It's still failling on macos shippable.
Flags: needinfo?(jcristau)
|
||
Comment 8•6 months ago
|
||
It's actually another bug, looks like workers are getting OoM'ed... I opened https://bugzilla.mozilla.org/show_bug.cgi?id=2003438
Flags: needinfo?(jcristau)
| Comment hidden (Intermittent Failures Robot) |
You need to log in
before you can comment on or make changes to this bug.
Description
•