2015054 - Update chunking to use the new manifests.json file

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, task)

Description

[Florian Quèze [:florian]]

Chunking of test jobs uses an algorithm that seems reasonable, but attempts to use run time data that was last updated in 2020. Instead of files in the tree, let's fetch data from the taskcluster job added in bug 2015029.

Comment 1

[Florian Quèze [:florian]]

Attachment: Bug 2015054 - Update chunking to use the new manifests.json file, r=ahal.

Comment 2

[Florian Quèze [:florian]]

Attachment: Bug 2015054 - Spread manifests with unknown runtimes evenly across chunks. r=ahal

When falling back to a similar platform's runtime data, some manifests
may be missing and end up with 0ms. Instead of letting the greedy
chunking algorithm pile them into the lightest chunks, round-robin
distribute them across all chunks to limit imbalance when they actually
take significant time.

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/391d03ef5d5e
https://hg.mozilla.org/integration/autoland/rev/ca3b91b5715e
Update chunking to use the new manifests.json file, r=ahal,taskgraph-reviewers.
https://github.com/mozilla-firefox/firefox/commit/11f9a1662d8a
https://hg.mozilla.org/integration/autoland/rev/f47e9a0d2919
Spread manifests with unknown runtimes evenly across chunks. r=ahal,taskgraph-reviewers

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/b97cbbabe8c4
https://hg.mozilla.org/integration/autoland/rev/6fb98e41ba45
Fix.

Comment 5

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/mozilla-central/rev/ca3b91b5715e
https://hg.mozilla.org/mozilla-central/rev/f47e9a0d2919
https://hg.mozilla.org/mozilla-central/rev/6fb98e41ba45

Comment 6

[Florian Quèze [:florian]]

Attachment: Screenshot of the changes on the dashboard for Linux mochitest-plain

The impact of the change can be seen on this dashboard: https://tests.firefox.dev/mochitest-jobs.html