1544774 - COT ERROR - Hit ScriptWorkerException

Status: RESOLVED DUPLICATE of bug 1373013

(Firefox Build System :: Task Configuration, defect)

Description

[Andreea Pavel [:apavel]]

Treeherder link: https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&resultStatus=testfailed%2Cbusted%2Cexception&selectedJob=240620177

Failure log: https://tools.taskcluster.net/groups/RuA4eB6sQ76r6eObSSEnaA/tasks/aQE2ZZ2TSG6VSTh8vfJM1g/runs/0/logs/public%2Flogs%2Fchain_of_trust.log#L135

2019-04-16T10:51:29 CRITICAL - Can't find task signing aQE2ZZ2TSG6VSTh8vfJM1g in signing:parent RuA4eB6sQ76r6eObSSEnaA task-graph.json!
2019-04-16T10:51:29 CRITICAL - Chain of Trust verification error!
Traceback (most recent call last):
File "/builds/scriptworker/lib/python3.6/site-packages/scriptworker/cot/verify.py", line 2136, in verify_chain_of_trust
task_count = await verify_task_types(chain)
File "/builds/scriptworker/lib/python3.6/site-packages/scriptworker/cot/verify.py", line 1887, in verify_task_types
await valid_task_types[task_type](chain, obj)
File "/builds/scriptworker/lib/python3.6/site-packages/scriptworker/cot/verify.py", line 1668, in verify_parent_task
verify_link_in_task_graph(chain, link, target_link)
File "/builds/scriptworker/lib/python3.6/site-packages/scriptworker/cot/verify.py", line 936, in verify_link_in_task_graph
task_link.name, task_link.task_id, decision_link.name, decision_link.task_id
File "/builds/scriptworker/lib/python3.6/site-packages/scriptworker/cot/verify.py", line 301, in raise_on_errors
raise CoTError("\n".join(errors))
scriptworker.exceptions.CoTError: "Can't find task signing aQE2ZZ2TSG6VSTh8vfJM1g in signing:parent RuA4eB6sQ76r6eObSSEnaA task-graph.json!"
2019-04-16T10:51:29 ERROR - Hit ScriptWorkerException: "Can't find task signing aQE2ZZ2TSG6VSTh8vfJM1g in signing:parent RuA4eB6sQ76r6eObSSEnaA task-graph.json!"
2019-04-16T10:51:29 DEBUG - "/builds/scriptworker/artifacts/public/logs/chain_of_trust.log" is encoded with "None" and has mime/type "text/plain"
2019-04-16T10:51:29 INFO - "/builds/scriptworker/artifacts/public/logs/chain_of_trust.log" can be gzip'd. Compressing...

Comment 1

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Merge to central landed at 9:50:14 UTC.

First gecko decision https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&resultStatus=success%2Ctestfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel%2Crunnable&group_state=expanded&selectedJob=240592867 task got retried as https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&resultStatus=success%2Ctestfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel%2Crunnable&group_state=expanded&selectedJob=240593018 which was complete at 9:56:16 UTC.

The nightly desktop decision task started at 10:00:42 UTC and thus shouldn't have scheduled each job twice.

Justin, does the shippable script not skip the retried decision tasks?

Comment 2

[Tom Prince [:tomprince]]

This is unrelated to the shippable work. What happened is that the first run of the decision task started scheduling jobs and got interrupted. All the tasks depend on the decision task, so that if it fails they don't get started. However, the second run of the decision task succeeded, causing all the tasks scheduled by the first run to be started, along with all the tasks from the second. The task that failed was from the first run, and scriptworker (correctly) couldn't find it in the list of tasks from the succesful second run.

The solution would be to cancel or otherwise cause all the tasks from the first run of the decision tasks to not run. This seems like it happens rarely enough that it might not be worth spending much time on, though.

Comment 3

[Intermittent Failures Robot]

13 failures in 2673 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 13

Platform breakdown:

• linux64: 2
• linux32: 1
• windows2012-64: 3
• windows2012-aarch64: 1
• osx-cross: 1
• windows2012-32: 2
• windows2012-32-shippable: 1
• linux32-shippable: 1
• windows2012-64-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1544774&startday=2019-04-15&endday=2019-04-21&tree=all

Comment 4

[Intermittent Failures Robot]

7 failures in 4409 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 7

Platform breakdown:

• windows2012-aarch64: 2
• windows2012-32-shippable: 1
• linux64-shippable: 1
• linux32-shippable: 1
• windows2012-aarch64-shippable: 1
• windows2012-64-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1544774&startday=2019-06-24&endday=2019-06-30&tree=all

Comment 5

[Intermittent Failures Robot]

8 failures in 4182 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 8

Platform breakdown:

• windows2012-32: 1
• windows2012-64: 1
• osx-shippable: 1
• windows2012-aarch64: 1
• linux64-shippable: 1
• linux32-shippable: 1
• windows2012-32-shippable: 1
• windows2012-aarch64-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1544774&startday=2019-10-21&endday=2019-10-27&tree=all

Comment 6

[Intermittent Failures Robot]

1 failures in 4727 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform breakdown:

• android-4-2-x86-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1544774&startday=2020-05-18&endday=2020-05-24&tree=all

Comment 8

[Aki Sasaki (not active)]

Aryx
afaik decision tasks have to be rerun from the command line
aki
rerunning will cause bustage in tasks that have been scheduled but are blocked on run 0. rescheduling via the fxci replay-hg-push is probably the right solution
CosminS|sheriffduty
I think we're good without it, will be on the lookout for failures from that push
aki
ok. we'll see duplicate tasks, which waste $$, and bustage like https://bugzilla.mozilla.org/show_bug.cgi?id=1637750 , which is a dup of https://bugzilla.mozilla.org/show_bug.cgi?id=1373013 , and can be avoided by not rerunning the decision task