1608834 - Enable chunking in the taskgraph for all mochitest suites

Status: NEW

(Firefox Build System :: Task Configuration, task, P2)

Description

[Andrew Halberstadt [:ahal]]

+++ This bug was initially created as a clone of Bug #1608833 +++

In bug 1583353 we started chunking test suites in the taskgraph (rather than at test runtime). Though I only enabled it for a subset of suites.

This bug tracks getting the rest of the mochitest suites enabled. To enable taskgraph chunking, remove the suite from this config:
https://searchfox.org/mozilla-central/rev/dd1dafd5c9c05640e76af30b58749076e0199704/taskcluster/taskgraph/transforms/tests.py#1280

I normally like to test that we still run the same set of tests before and after. To do this I:

1. Push to try scheduling all tests in the suite, both with and without this change.
2. Clone https://github.com/mozilla/ci-recipes and follow installation instructions.
3. Run poetry run adr compare_pushes --branch try -r1 <rev1> -r2 <rev2>
4. It will take awhile (~10-20min).
5. Make sure the output says all tasks are the same. If the output shows one task ran more manifests than another, then we need to investigate.

The remaining mochitest suites weren't enabled simply because I never got around to them. They might be trivial. Though browser-chrome fission mochitests seemed to fail intermittently a bit more frequently than before. A bit of care will be needed with those.

Comment 1

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1608834 - [ci] Chunk mochitest-a11y and plain in the taskgraph, r?gbrown

Comment 2

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/a80ee2878a79
[ci] Chunk mochitest-a11y and plain in the taskgraph, r=gbrown

Comment 3

[Andrew Halberstadt [:ahal]]

Just noticed that all four mochitest-webgl* suites are entirely defined inside a single manifest:
https://searchfox.org/mozilla-central/source/dom/canvas/test/webgl-conf/generated-mochitest.ini

Taskgraph chunking means we start chunkng by manifests, therefore the maximum number of chunks is <= the number of manifests. Therefore we'll either need to:

1. Reduce all these suites to a single chunk (looks like ~60min combined runtime).
2. Generate several manifests, one per chunk we desire.
3. Implement special logic for this suite, to group them by some metric other than manifests (might need to do this for WPT anyway).
4. Leave them in the BLACKLIST indefinitely.

Comment 4

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/a80ee2878a79

Comment 5

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1608834 - [taskgraph] Enable taskgraph chunking for mochitest-browser-chrome, r?jmaher

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/60e82c1bd6b1
[taskgraph] Enable taskgraph chunking for mochitest-browser-chrome, r=jmaher

Comment 7

[Stefan Hindli [:stefan_hindli]]

https://hg.mozilla.org/mozilla-central/rev/60e82c1bd6b1

Comment 8

[BugBot [:suhaib / :marco/ :calixte]]

The leave-open keyword is there and there is no activity for 6 months.
:ahal, maybe it's time to close this bug?

Comment 9

[Andrew Halberstadt [:ahal]]

We still need to support mochitest-webgl here.

Comment 10

[BugBot [:suhaib / :marco/ :calixte]]

The leave-open keyword is there and there is no activity for 6 months.
:ahal, maybe it's time to close this bug?

Comment 11

[Andrew Halberstadt [:ahal]]

Still a few flavors left, at least mochitest-a11y, possibly a couple others.