1400223 - "Add new jobs" do not run any jobs if original build failed

Status: RESOLVED FIXED

(Taskcluster Graveyard :: Scheduler, enhancement)

Description

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Please have a look at the following try build:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=50e29da978d246f6f7e742f065803be319987bc6&selectedJob=131119752

The original build for Linux64 failed because of infra issues (S3 outage) yesterday. So I retriggered the build once the problems went away.

Later I wanted to add some Firefox UI functional jobs, and did so by selecting via "Add new jobs". Submitting the request worked fine and the "add-new" jobs appear. But none of the selected test jobs is getting scheduled, because all relies on the original build.

We talked with Armen on IRC and as he also mentioned it was working fine with mozci earlier because in such a case it would have created a new build job. But that is no longer the case with the re-implementation.

Brian can you please have a look? Thanks.

Comment 1

[Brian Stack [:bstack]]

Ah, that is indeed a change in behavior in actions.json. I'll work on a patch now! Thanks for finding this, there will probably be a long-tail of differences in implementations.

Comment 2

[Brian Stack [:bstack]]

Attachment: Bug 1400223 - Merge tasks added by action tasks into graphs used for subsequent tasks

Review commit: https://reviewboard.mozilla.org/r/181890/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/181890/

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8910450 [details]
Bug 1400223 - Merge tasks added by action tasks into graphs used for subsequent tasks

https://reviewboard.mozilla.org/r/181890/#review187916

::: taskcluster/taskgraph/actions/util.py:75
(Diff revision 1)
>                                                                  params,
>                                                                  to_run,
>                                                                  label_to_taskid)
> +    write_artifact('task-graph.json', optimized_task_graph.to_json())
> +    write_artifact('label-to-taskid.json', label_to_taskid)
> +    write_artifact('to-run.json', list(to_run))

Is there a doc to update about these?

::: taskcluster/taskgraph/util/taskcluster.py:113
(Diff revision 1)
> +            data = {'continuationToken': response.get('continuationToken')}
> +        else:
> +            break
> +
> +    # We can sort on expires because all of these tasks should be created with the
> +    # same expires time so they end up in order from earliest to latest action

clever!

Comment 4

[Brian Stack [:bstack]]

Comment on attachment 8910450 [details]
Bug 1400223 - Merge tasks added by action tasks into graphs used for subsequent tasks

https://reviewboard.mozilla.org/r/181890/#review187916

> clever!

Now that I think about this, I realize this is not generically true across all namespaces. Within the context of action tasks this should be true, but this is a function in taskgraph/util and not action specific. Should we leave it as-is and call it an implementation detail to sort out later or maybe I should do the sorting in action tasks?

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

I think you can document the function as sorting by expiration, and leave it up to other users to sort differently if desired.

Comment 6

[Brian Stack [:bstack]]

Comment on attachment 8910450 [details]
Bug 1400223 - Merge tasks added by action tasks into graphs used for subsequent tasks

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/181890/diff/1-2/

Comment 7

[Brian Stack [:bstack]]

Comment on attachment 8910450 [details]
Bug 1400223 - Merge tasks added by action tasks into graphs used for subsequent tasks

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/181890/diff/2-3/

Comment 8

[Brian Stack [:bstack]]

Comment on attachment 8910450 [details]
Bug 1400223 - Merge tasks added by action tasks into graphs used for subsequent tasks

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/181890/diff/3-4/

Comment 9

[Brian Stack [:bstack]]

Comment on attachment 8910450 [details]
Bug 1400223 - Merge tasks added by action tasks into graphs used for subsequent tasks

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/181890/diff/4-5/

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/112de4d9e9e6
Merge tasks added by action tasks into graphs used for subsequent tasks r=dustin

Comment 11

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/112de4d9e9e6