1654591 - Ensure 'manifest-scheduling' honours backstop pushes

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, task, P1)

Description

[Andrew Halberstadt [:ahal]]

When performing test chunking during the taskgraph phase, we need to make sure all tests are run if we're scheduling on a backstop push.

Ideally we'd have a robust mechanism for determining whether the generation is running on a backstop or not that gets shared with the backstop optimizer. Though this likely means inventing a new concept in the taskgraph, especially if we want different tasks running on different backstop intervals.

For now I plan to add a hardcoded 10th push backstop in the chunking algorithm to match up with the default backstop.

Comment 1

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1654591 - [taskgraph] Move logic that determines a backstop to utility file, r?jmaher

Comment 2

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1654591 - [taskgraph] Ensure manifest chunking doesn't prune manifests on backstop pushes, r?marco

Depends on D84598

Comment 3

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1654591 - Format 'taskcluster/test' with black, r?marco

Depends on D84598

Comment 4

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1654591 - [taskgraph] Implement a 'relevant_tests' based manifest loader and use it as the bugbug fallback, r?marco

Depends on D84599

Comment 5

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1654591 - [taskgraph] Fallback all manifests if there was a timeout in the bugbug loader, r?marco

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/0f69abd89331
[taskgraph] Move logic that determines a backstop to utility file, r=jmaher
https://hg.mozilla.org/integration/autoland/rev/00920823fc42
Format 'taskcluster/test' with black, r=marco
https://hg.mozilla.org/integration/autoland/rev/ee59f46aea2c
[taskgraph] Ensure manifest chunking doesn't prune manifests on backstop pushes, r=marco
https://hg.mozilla.org/integration/autoland/rev/2bd75664acd6
[taskgraph] Fallback all manifests if there was a timeout in the bugbug loader, r=marco

Comment 7

[Noemi Erli[:noemi_erli]]

https://hg.mozilla.org/mozilla-central/rev/0f69abd89331
https://hg.mozilla.org/mozilla-central/rev/00920823fc42
https://hg.mozilla.org/mozilla-central/rev/ee59f46aea2c
https://hg.mozilla.org/mozilla-central/rev/2bd75664acd6