1690947 - Decision tasks can fail when an optimized task is about to expire

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, task)

Description

[Mike Hommey [:glandium]]

The decision task JTtpfthTQLKTtNROpJhCPQ failed with:

[task 2021-02-05T01:57:02.454Z] `task.dependencies` references tasks that expires
[task 2021-02-05T01:57:02.454Z] before `task.deadline` this is not allowed, see tasks: 
[task 2021-02-05T01:57:02.454Z]  * exXDr2J5TuSouPw0ax-I-A,
[task 2021-02-05T01:57:02.454Z] All taskIds in `task.dependencies` **must** have
[task 2021-02-05T01:57:02.454Z] `task.expires` greater than the `deadline` for this task.

The task in question, exXDr2J5TuSouPw0ax-I-A, has:

expires	"2021-02-05T02:52:40.752Z"

which, as of writing, is now the past, so this is going to work out ok for the next attempt at the same task graph, but was in the future back then but not enough in the future that all tasks' deadlines would be before that.

Comment 2

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1690947 - [taskgraph.optimize] Take dependent deadlines into account when replacing a task, r?#taskgraph-reviewers!

Comment 3

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1690947 - [taskgraph] Return full status in util.taskcluster:status_task, r?#taskgraph-reviewers!

This moves the return value of the old task to a new 'state_task' function and
updates consumers. We'll be using other parts of the status in later patches in
the stack.

Comment 4

[Intermittent Failures Robot]

1 failures in 3061 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• comm-central: 1

Platform and build breakdown:

• gecko-decision: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1690947&startday=2021-05-17&endday=2021-05-23&tree=all

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/9de86787dc53
[taskgraph] Return full status in util.taskcluster:status_task, r=taskgraph-reviewers,aki
https://hg.mozilla.org/integration/autoland/rev/89909afc3777
[taskgraph.optimize] Take dependent deadlines into account when replacing a task, r=taskgraph-reviewers,bhearsum

Comment 6

[Mike Hommey [:glandium]]

This busted bootstrap via e.g. mach artifact toolchain because it uses should_replace_task but its call site hasn't been updated.

Comment 7

[Mike Hommey [:glandium]]

mach taskcluster-load-image is probably similarly busted.

Comment 8

[Marian-Vasile Laza]

https://hg.mozilla.org/mozilla-central/rev/9de86787dc53
https://hg.mozilla.org/mozilla-central/rev/89909afc3777

Comment 9

[Andrew Halberstadt [:ahal]]

This got fixed in bug 1715493 and is currently on central. Thanks Emilio!

Comment 10

[Intermittent Failures Robot]

1 failures in 3115 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr78: 1

Platform and build breakdown:

• gecko-decision: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1690947&startday=2021-06-21&endday=2021-06-27&tree=all