Perma BMR exeptions ScriptWorkerException: 'path public/build/target.normandy.json not in scriptworker:build Y3WPVV1XQPuv4TE_RmcUJA chain of trust artifacts!'
Categories
(Release Engineering :: General, defect)
Tracking
(firefox75 fixed)
| Tracking | Status | |
|---|---|---|
| firefox75 | --- | fixed |
People
(Reporter: aiakab, Assigned: nthomas, NeedInfo)
References
(Regression)
Details
Attachments
(1 file)
2020-02-21T01:04:52 CRITICAL - Chain of Trust verification error!
Traceback (most recent call last):
File "/app/lib/python3.7/site-packages/scriptworker/cot/verify.py", line 1906, in verify_chain_of_trust
await download_cot_artifacts(chain)
File "/app/lib/python3.7/site-packages/scriptworker/cot/verify.py", line 719, in download_cot_artifacts
mandatory_artifacts_paths = await raise_future_exceptions(mandatory_artifact_tasks)
File "/app/lib/python3.7/site-packages/scriptworker/utils.py", line 398, in raise_future_exceptions
succeeded_results, _ = await _process_future_exceptions(tasks, raise_at_first_error=True)
File "/app/lib/python3.7/site-packages/scriptworker/utils.py", line 432, in _process_future_exceptions
raise exc
File "/app/lib/python3.7/site-packages/scriptworker/cot/verify.py", line 671, in download_cot_artifact
raise CoTError("path {} not in {} {} chain of trust artifacts!".format(path, link.name, link.task_id))
scriptworker.exceptions.CoTError: 'path public/build/target.normandy.json not in scriptworker:build Y3WPVV1XQPuv4TE_RmcUJA chain of trust artifacts!'
2020-02-21T01:04:52 ERROR - Hit ScriptWorkerException: 'path public/build/target.normandy.json not in scriptworker:build Y3WPVV1XQPuv4TE_RmcUJA chain of trust artifacts!'
2020-02-21T01:04:52 DEBUG - "/app/artifacts/public/logs/chain_of_trust.log" is encoded with "None" and has mime/type "text/plain"
2020-02-21T01:04:52 INFO - "/app/artifacts/public/logs/chain_of_trust.log" can be gzip'd. Compressing...
|
Reporter | |
Comment 1•5 years ago
|
||
This could be a fallout from:
https://hg.mozilla.org/mozilla-central/rev/7356ddff15bcfc9c186f17f3b1994898b2de2466
Bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1563564
Michael can you please take a look?
|
Assignee | |
Comment 2•5 years ago
|
||
|
||
Updated•5 years ago
|
|
|
Assignee | |
Comment 3•5 years ago
|
||
I've grabbed this because I'm 95% sure that the beetmover jobs build a list of expected artifacts from the manifests I've modified, and the target.normandy.json file is only generated on linux64 and win64 build jobs. The other desktop builds are cross compiles so don't go inside the ifdef blocks (eg https://hg.mozilla.org/mozilla-central/rev/7356ddff15bcfc9c186f17f3b1994898b2de2466#l5.12).
|
|
Assignee | |
Comment 5•5 years ago
|
||
mcooper, are you planning to track devedition releases as well as firefox ? The landed patch will miss those but we can land a followup if needed.
|
||
Comment 6•5 years ago
|
||
| bugherder | ||
|
||
Comment 7•5 years ago
|
||
Thanks for tackling this so quickly. It was really great to wake up to a fix instead of a backout.
This is actually something that came up in review. :froydnj pointed out that it was weird that the artifact was filtered in one place, but not in this place. Since it didn't fail any of the try builds I ran, I assumed it would be ok. Was there a try build that I could have run that would have caught this issue? Or was there something that should have been tested while this was on autoland? This seems like a short coming in our CI system.
mcooper, are you planning to track devedition releases as well as firefox ? The landed patch will miss those but we can land a followup if needed.
Ideally we'd want this for all channels, including devedition. This artifact is going to be used to build a linter for Normandy recipes. We want to make sure that if we target a given subset of Firefox clients, they are capable of doing what we ask them to do. Since we sometimes target Devedition, then we would need this information there as well.
That being said, this is also only running on Linux and Windows right now, since the current method is not compatible with cross compilation. Because of that I hope to eventually switch over to a newer, taskgraph based approach that could run on all platforms. This was beyond my ability right now though. It may be ok to ignore Devedition until that second version of the feature is done. (Also, to anyone watching, if adding an xpcshell task to CI is something you know how to do I'd love to talk through it with you, because when I tried, I got horribly lost).
|
|
Assignee | |
Comment 8•5 years ago
|
||
(In reply to Michael Cooper [:mythmon] from comment #7)
This is actually something that came up in review. :froydnj pointed out that it was weird that the artifact was filtered in one place, but not in this place. Since it didn't fail any of the try builds I ran, I assumed it would be ok. Was there a try build that I could have run that would have caught this issue? Or was there something that should have been tested while this was on autoland? This seems like a short coming in our CI system.
You could needinfo mtabara (with fallback to anyone in releng) if there are any queries about those manifests. Since the failures were in a downstream job of the actual build, and in a separate nightly-specific set of tasks triggered by a cron, I'm not sure how practical it is to test on try. tomprince, do you know of a method ?
That being said, this is also only running on Linux and Windows right now, since the current method is not compatible with cross compilation. Because of that I hope to eventually switch over to a newer, taskgraph based approach that could run on all platforms. This was beyond my ability right now though. It may be ok to ignore Devedition until that second version of the feature is done. (Also, to anyone watching, if adding an xpcshell task to CI is something you know how to do I'd love to talk through it with you, because when I tried, I got horribly lost).
Callek tells me that you'd just need to add linux64-devedition and win64-devedition to firefox_candidates.yml (for Firefox 75). I'll leave that up to you.
| Comment hidden (Intermittent Failures Robot) |
Description
•