Closed Bug 1384134 Opened 7 years ago Closed 7 years ago

Chain of Trust verification error! Can't find task signing:decision

Categories

(Release Engineering :: General, defect)

Product:
Release Engineering
Component:
General
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: cbook, Unassigned)

References

(
URL
)

Details

https://public-artifacts.taskcluster.net/Xn2OVf4sTfeHZlSh-vcVjg/0/public/logs/chain_of_trust.log 2017-07-25T14:21:46 CRITICAL - Can't find task signing Xn2OVf4sTfeHZlSh-vcVjg in signing:decision fJM0K3RlQoSOM7Z5FymCeQ task-graph.json! 2017-07-25T14:21:46 CRITICAL - Chain of Trust verification error! Traceback (most recent call last): File "/builds/scriptworker/lib/python3.5/site-packages/scriptworker/cot/verify.py", line 1311, in verify_chain_of_trust task_count = await verify_task_types(chain) File "/builds/scriptworker/lib/python3.5/site-packages/scriptworker/cot/verify.py", line 1098, in verify_task_types await valid_task_types[task_type](chain, obj) File "/builds/scriptworker/lib/python3.5/site-packages/scriptworker/cot/verify.py", line 942, in verify_decision_task verify_link_in_task_graph(chain, link, target_link) File "/builds/scriptworker/lib/python3.5/site-packages/scriptworker/cot/verify.py", line 832, in verify_link_in_task_graph task_link.name, task_link.task_id, decision_link.name, decision_link.task_id File "/builds/scriptworker/lib/python3.5/site-packages/scriptworker/cot/verify.py", line 219, in raise_on_errors raise CoTError("\n".join(errors)) scriptworker.exceptions.CoTError: "Can't find task signing Xn2OVf4sTfeHZlSh-vcVjg in signing:decision fJM0K3RlQoSOM7Z5FymCeQ task-graph.json!" OS X Cross Compiled opt l10n signing it seems
We're working on making CoT able to deal with nightly retriggers. Until then, we think the best short term solution is to retrigger the nightly hook for failures.
(In reply to Aki Sasaki [:aki] from comment #1) > We're working on making CoT able to deal with nightly retriggers. Our current thinking is: 1. When we retrigger a task via treeherder, it creates an action task. Either the action task is CoT verifiable, or the action task creates a decision task that only submits a subgraph. Either way, this subgraph is then fully CoT verifiable. Ideally this would have a pointer back to the original graph. We'll probably end up with a 2nd taskGroupId; we'll need to find and address any assumptions about retriggers being in the same taskGroupId. 2. CoT verification should support verifying this subgraph. This likely involves supporting cot v2 (rebuilding the json-e template to verify the decision task comes from the tree). https://bugzilla.mozilla.org/show_bug.cgi?id=cot-v2
Depends on: 1267427
For the record, this breaks pretty much all Windows build retriggers at the moment. Which is particularly bad given the rather high rate of hitting bug 1381768 that we're seeing.
Do you think installing and running taskcluster-cli would be acceptable for sheriffs for windows for the short term? taskcluster signin # opens browser to have you log in to taskcluster via okta taskcluster task rerun -- TASKID # rerun the taskId specified. this doesn't break cot https://github.com/taskcluster/taskcluster-cli
Assuming we have the scopes to actually do that, sure. I've tried retriggering tasks from the "Task Actions" drop-down on the Task Inspector and was given insufficient scopes errors.
Updated status: RyanVM was able to log in, once he used my url with a modified port. This sounds like a taskcluster-cli bug, which we should file against taskcluster-cli. I was able to rerun several tasks in the meantime.
Latest update: I believe the sheriffs can now rerun tasks; scopes bug was bug 1390241.
See Also: → 1393277
I see many tasks (25) annotated with this yesterday: https://brasstacks.mozilla.com/orangefactor/index.html?display=Bug&bugid=1384134 is there a new problem, or is this expected?
Flags: needinfo?(aki)
This is due to people retriggering tasks that can not run successfully when retriggered, so this is expected. The problem still needs to be fixed; we still have a workaround of rerunning the task via taskcluster-cli.
Flags: needinfo?(aki)
Depends on: 1393277
Bug 1393277 fixes this for Fx>=58 once https://hg.mozilla.org/integration/autoland/rev/e87d535aebd7 merges around. To retrigger: go to `...` menu -> custom action -> retrigger. For Fx<58 or branches without e87d535aebd7, we still need to use the taskcluster-cli rerun.
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.