1578289 - raise CoTError("path {} not in {} {} chain of trust artifacts!".format(path, link.name, link.task_id))

Status: RESOLVED FIXED

(Release Engineering :: Release Automation, defect, P1)

Description

[Alexandru Michis [:malexandru]]

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=mozilla-beta&searchStr=langpack&revision=cc694864a3fe8fa859142e0a81041a08e1f00d68&selectedJob=264609830

Failure log: https://taskcluster-artifacts.net/CVH2dULGQgqrYCxeBwduuw/0/public/logs/chain_of_trust.log

2019-09-02T17:34:46 DEBUG - Verifying public/task-graph.json is in PP7pkEQPSMKcSpyi1NQjew cot artifacts...
2019-09-02T17:34:46 INFO - Downloading Chain of Trust artifact:
https://queue.taskcluster.net/v1/task/PP7pkEQPSMKcSpyi1NQjew/artifacts/public%2Ftask-graph.json
2019-09-02T17:34:46 DEBUG - Verifying public/build/en-US/target.langpack.xpi is in FWChbg9YQMmSO8Ysi80Yiw cot artifacts...
2019-09-02T17:34:46 INFO - Downloading https://queue.taskcluster.net/v1/task/elOeM2LhRvyFuoX-l1niwA/artifacts/public%2Ftask-graph.json
2019-09-02T17:34:46 INFO - Downloading https://queue.taskcluster.net/v1/task/GFdUE_EPQ4ew5Dvdt358IQ/artifacts/public%2Factions.json
2019-09-02T17:34:46 INFO - Downloading https://queue.taskcluster.net/v1/task/GFdUE_EPQ4ew5Dvdt358IQ/artifacts/public%2Fparameters.yml
2019-09-02T17:34:46 INFO - Downloading https://queue.taskcluster.net/v1/task/GFdUE_EPQ4ew5Dvdt358IQ/artifacts/public%2Ftask-graph.json
2019-09-02T17:34:46 INFO - Downloading https://queue.taskcluster.net/v1/task/PP7pkEQPSMKcSpyi1NQjew/artifacts/public%2Factions.json
2019-09-02T17:34:46 INFO - Downloading https://queue.taskcluster.net/v1/task/PP7pkEQPSMKcSpyi1NQjew/artifacts/public%2Fparameters.yml
2019-09-02T17:34:46 INFO - Downloading https://queue.taskcluster.net/v1/task/PP7pkEQPSMKcSpyi1NQjew/artifacts/public%2Ftask-graph.json
2019-09-02T17:34:46 INFO - Done
2019-09-02T17:34:46 DEBUG - /builds/scriptworker/work/cot/PP7pkEQPSMKcSpyi1NQjew/public/parameters.yml matches the expected sha256 29840b8ecff3c37f62e1b31eacf932bf56b1e2a6af6bcfea1079fe5a18b2214b
2019-09-02T17:34:46 INFO - Done
2019-09-02T17:34:46 DEBUG - /builds/scriptworker/work/cot/PP7pkEQPSMKcSpyi1NQjew/public/actions.json matches the expected sha256 5425d484efa16c4624ec2e5049763a02b141d37defeba64c6510a89c242d470e
2019-09-02T17:34:46 INFO - Done
2019-09-02T17:34:46 INFO - Done
2019-09-02T17:34:46 DEBUG - /builds/scriptworker/work/cot/GFdUE_EPQ4ew5Dvdt358IQ/public/parameters.yml matches the expected sha256 bfc016f0102cda2e9d7ff95fa25c681f754b68a27bd69ef59d5ecff4ac101f33
2019-09-02T17:34:46 DEBUG - /builds/scriptworker/work/cot/GFdUE_EPQ4ew5Dvdt358IQ/public/actions.json matches the expected sha256 b1968d03f8a1d55b87f614b60ebc56bd68da293852191024a6948dc6c3c974ca
2019-09-02T17:34:46 INFO - Done
2019-09-02T17:34:46 DEBUG - /builds/scriptworker/work/cot/PP7pkEQPSMKcSpyi1NQjew/public/task-graph.json matches the expected sha256 ffadad8292fa8410d5adfd7e4f95c47f9277e8a0cd72417cb073be1f07870309
2019-09-02T17:34:47 INFO - Done
2019-09-02T17:34:47 DEBUG - /builds/scriptworker/work/cot/GFdUE_EPQ4ew5Dvdt358IQ/public/task-graph.json matches the expected sha256 a47fd8981131b34dafdedb37c5b997f44124826601fbf49535aeef2afdd6e2e8
2019-09-02T17:34:48 INFO - Done
2019-09-02T17:34:48 DEBUG - /builds/scriptworker/work/cot/elOeM2LhRvyFuoX-l1niwA/public/task-graph.json matches the expected sha256 6483c8ad95a9025accb39625e1c534410c0c747a05947e16e9a487868cd1a427
2019-09-02T17:34:48 CRITICAL - Chain of Trust verification error!
Traceback (most recent call last):
File "/builds/scriptworker/lib/python3.6/site-packages/scriptworker/cot/verify.py", line 2061, in verify_chain_of_trust
await download_cot_artifacts(chain)
File "/builds/scriptworker/lib/python3.6/site-packages/scriptworker/cot/verify.py", line 744, in download_cot_artifacts
mandatory_artifacts_paths = await raise_future_exceptions(mandatory_artifact_tasks)
File "/builds/scriptworker/lib/python3.6/site-packages/scriptworker/utils.py", line 327, in raise_future_exceptions
succeeded_results, _ = await _process_future_exceptions(tasks, raise_at_first_error=True)
File "/builds/scriptworker/lib/python3.6/site-packages/scriptworker/utils.py", line 361, in _process_future_exceptions
raise exc
File "/builds/scriptworker/lib/python3.6/site-packages/scriptworker/cot/verify.py", line 692, in download_cot_artifact
raise CoTError("path {} not in {} {} chain of trust artifacts!".format(path, link.name, link.task_id))
scriptworker.exceptions.CoTError: 'path public/build/en-US/target.langpack.xpi not in beetmover:signing FWChbg9YQMmSO8Ysi80Yiw chain of trust artifacts!'
2019-09-02T17:34:48 ERROR - Hit ScriptWorkerException: 'path public/build/en-US/target.langpack.xpi not in beetmover:signing FWChbg9YQMmSO8Ysi80Yiw chain of trust artifacts!'
2019-09-02T17:34:48 DEBUG - "/builds/scriptworker/artifacts/public/logs/chain_of_trust.log" is encoded with "None" and has mime/type "text/plain"
2019-09-02T17:34:48 INFO - "/builds/scriptworker/artifacts/public/logs/chain_of_trust.log" can be gzip'd. Compressing...

Comment 1

[Justin Wood (:Callek)]

Attachment: Bug 1578289 - fix addon beetmoving failure in declarative artifacts too. r=mtabara

Comment 2

[Julien Cristau [:jcristau]]

https://hg.mozilla.org/releases/mozilla-beta/rev/82e9cab54ba6

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/ccfe2aee8492
fix addon beetmoving failure in declarative artifacts too. r=mtabara

Comment 4

[Arthur Iakab [arthur_iakab]]

https://hg.mozilla.org/mozilla-central/rev/ccfe2aee8492

Comment 5

[Intermittent Failures Robot]

4 failures in 4408 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 4

Platform breakdown:

• win64-shippable: 1
• macosx64-shippable: 1
• win32-shippable: 1
• linux-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1578289&startday=2019-09-02&endday=2019-09-08&tree=all

Comment 6

[Emma Humphries ☕️🎸🧞‍♀️✨ (she/they) [:emceeaich] (Pacific Time) use needinfo]

This bug has been identified as part of a pilot on determining root causes of blocking and dot release drivers.

It needs a root-cause set for it. Please see the list at https://docs.google.com/document/d/1FFEGsmoU8T0N8R9kk-MXWptOPtXXXRRIe4vQo3_HgMw/.

Add the root cause as a whiteboard tag in the form [rca - <cause> ] and remove the rca-needed keyword.

If you have questions, please contact :tmaity.

Comment 7

[Aki Sasaki (not active)]

Due to the patch fixing this specific bug, this seems like a logical error. However, this error can crop up from bug 1512733, in which case it might be a missing requirement or conflicting design. We could also add something about wanting test coverage around declarative artifacts, as well as any other task-to-task artifact dependencies.