Update build/test job optimization to use a strategy vs a list of included files
Categories
(Thunderbird :: Build Config, task)
Tracking
(thunderbird_esr78 wontfix, thunderbird84 wontfix)
People
(Reporter: rjl, Assigned: rjl)
References
Details
Attachments
(1 file)
Building on bug 1661493 - implement a job optimization strategy that has the same effect as the "when.files" used now to avoid builds for Suite-only pushes. "when.files" works, but there's too many places where the list has to be kept in sync. This is leading to builds not happening when they're expected.
Additionally a strategy to build win64-plain and ASan builds only one daily. Can either use a combo of the above with a backstop, or maybe something else altogether.
|
Assignee | |
Comment 1•4 years ago
|
||
Taskcluster does not build on suite-only pushes. Using the 'when.files-changed'
method is prone to missing things due to new directories being added and is
not easy to combine with other strategies such as a backstop.
The skip-suite-only optimization is effectively the same as what is done with
when.files-changed. The changed files in a push are pulled from hg.mo and then
any files matching the suite/editor exclude patterns are removed from the list.
If the changed files list is empty after that process, then no build is done.
|
||
Updated•4 years ago
|
|
|
Assignee | |
Comment 2•4 years ago
|
||
Testing info:
The list of changed files is pulled from hg.mo by revision hash, so all that really matters when testing locally is that comm_head_revision in your parameters.yml file points to a Thunderbird code push or a suite-only push.
You'll need a parameters.yml file from a push to comm-central that was a "suite-only" push. This is the most recent one:
https://treeherder.mozilla.org/jobs?repo=comm-central&revision=13a747e586b815745aadb240da321d54522acebe
The Decision task artifacts will have the parameters.yml file, download and save someplace.
Full testing will require four runs of mach taskgraph optimized.
Run 1: Thunderbird code push with current code
time ./mach taskgraph optimized -p project=comm-central --root comm/taskcluster/ci -o /tmp/run1.txt
Run 2: Suite-only code push with current code
time ./mach taskgraph optimized -p /tmp/suiteonly.yml --root comm/taskcluster/ci -o /tmp/run2.txt
run1.txt should contain ~170 jobs that would run (give or take depending on if there are toolchain jobs and such to run).
run2.txt should have two source-test-* jobs.
Apply the patch from this bug locally and repeat the above as run3 and run4.
Run 3: Thunderbird code push with patch
time ./mach taskgraph optimized -p project=comm-central --root comm/taskcluster/ci -o /tmp/run3.txt
Run 4: Suite-only code push with patch
time ./mach taskgraph optimized -p /tmp/suiteonly.yml --root comm/taskcluster/ci -o /tmp/run4.txt
run3.txt should have the same jobs as run1.txt (sort the files before comparing), and run4.txt should match run2.txt.
The Nightly build taskgraphs should be checked as well. That's going to be more challenging. Grab the parameters.yml from the latest Nightly. It's decision task is cron(Nd). Run mach taskgraph optimized with that parameters file without the patch then with the patch. To test a suite-only "nightly" push you'll need to make a copy of that nightly parameters file and change comm_head_revision to the one from your suite-only paramters from above. Then run mach taskgraph optimized without and with the patch.
Nightly builds are different. If the most recent push was a Suite-only push, the builds should still run. So there should not be any differences in the optimized job lists.
Pushed by thunderbird@calypsoblue.org:
https://hg.mozilla.org/comm-central/rev/1b7dae9b88ed
Use 'skip-suite-only' build/test job optimization strategy. r=justdave
Pushed by thunderbird@calypsoblue.org: https://hg.mozilla.org/comm-central/rev/7473fd8d06d9 Followup: Fix Python formatting with black. r=linting DONTBUILD
|
|
||
Updated•3 years ago
|
Description
•