Stop running shippable and other unnecessary builds with |mach try auto|
Categories
(Firefox Build System :: Task Configuration, task, P2)
Tracking
(Not tracked)
People
(Reporter: ahal, Assigned: ahal)
References
(Blocks 1 open bug)
Details
Attachments
(7 files, 1 obsolete file)
Bug 1641131 - [taskgraph] Make 'upload-generated-sources' tasks copy their dependency's optimization
47 bytes,
text/x-phabricator-request
|
Details | Review | |
47 bytes,
text/x-phabricator-request
|
Details | Review | |
47 bytes,
text/x-phabricator-request
|
Details | Review | |
47 bytes,
text/x-phabricator-request
|
Details | Review | |
47 bytes,
text/x-phabricator-request
|
Details | Review | |
Bug 1641131 - [ci] Don't use shippable builds as dependencies for source-test tasks on try, r?jmaher
47 bytes,
text/x-phabricator-request
|
Details | Review | |
47 bytes,
text/x-phabricator-request
|
Details | Review |
Ben Hearsum recently landed a change to stop running shippable builds/tests on try. However they're still being selected with |mach try auto| (though no test / perf tasks are).
I believe this is due to the build-signing and upload-generated-symbols tasks.
Assignee | ||
Comment 1•8 months ago
|
||
Removing build-signing + upload-generated-symbols wasn't enough to get rid of shippable builds, but it did remove some DevEdition builds. Unrelatedly I discovered that the win64-aarch
builds are still running because they have Be
as a dependent which in turn has a diffoscope task (DWE
on treeherder) as a dependent. Not sure what that is, but doubt it's important enough to run on all |mach try auto| pushes.
Assignee | ||
Comment 2•8 months ago
|
||
Assignee | ||
Comment 3•8 months ago
|
||
Depends on D76988
Comment 4•8 months ago
|
||
To remove the DevEdition builds, we could use filter_out_devedition
in target_tasks_try_auto
.
Assignee | ||
Comment 5•8 months ago
|
||
I had an idea.. we could create a "default" optimization that aliases to the "never" strategy and is applied to all tasks that don't have any other optimizations. Then the tryselect override can change it to "always". So we'd only run tasks that have an explicit optimization set.
In some cases it be worth getting dependents to copy their parent's optimization in addition.
Assignee | ||
Comment 6•8 months ago
|
||
(In reply to Marco Castelluccio [:marco] from comment #4)
To remove the DevEdition builds, we could use
filter_out_devedition
intarget_tasks_try_auto
.
They're currently filtered out by bugbug_reduced, I think they were just running due to the build-signing tasks. Seems worth keeping it that way in case they're actually relevant to some patch.
Assignee | ||
Comment 7•8 months ago
|
||
Turns out the build-signing tasks already copy their dependencies' optimization:
https://searchfox.org/mozilla-central/source/taskcluster/taskgraph/transforms/signing.py#174
So that shouldn't be why the shippable tasks are still running.. Indeed they're still running even after my patch above.
Updated•8 months ago
|
Updated•8 months ago
|
Assignee | ||
Comment 8•8 months ago
|
||
Note: DevEdition builds are still running on some |mach try auto| pushes solely due to test-verify.
Assignee | ||
Updated•8 months ago
|
Comment 9•8 months ago
|
||
(In reply to Andrew Halberstadt [:ahal] from comment #6)
(In reply to Marco Castelluccio [:marco] from comment #4)
To remove the DevEdition builds, we could use
filter_out_devedition
intarget_tasks_try_auto
.They're currently filtered out by bugbug_reduced, I think they were just running due to the build-signing tasks. Seems worth keeping it that way in case they're actually relevant to some patch.
(In reply to Andrew Halberstadt [:ahal] from comment #8)
Note: DevEdition builds are still running on some |mach try auto| pushes solely due to test-verify.
They don't run on autoland (AFAICS), so they'd never be selected by bugbug anyway. Maybe it's best to filter them out so they don't run just because of test-verify.
Assignee | ||
Comment 10•8 months ago
|
||
Depends on D76988
Assignee | ||
Comment 11•8 months ago
|
||
The Backstop optimization doesn't take any arguments, yet the schema for 'push-optimization-*'
requires a 'schedules-component' that goes ignored. Fix this.
Updated•8 months ago
|
Assignee | ||
Comment 12•8 months ago
|
||
Depends on D77048
Assignee | ||
Updated•8 months ago
|
Comment 13•8 months ago
|
||
Pushed by ahalberstadt@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/48e73dbc588e [taskgraph] Fix the 'push-interval-*' schema to not require an argument, r=jmaher https://hg.mozilla.org/integration/autoland/rev/2e034897a64f [taskgraph] Make 'upload-generated-sources' tasks copy their dependency's optimization r=Callek https://hg.mozilla.org/integration/autoland/rev/6f3a65196d88 [taskgraph] Make 'upload-symbols' tasks copy their dependency's optimization r=Callek
Comment 14•8 months ago
|
||
Backed out 3 changesets (bug 1641131) for gecko decision task bustage.
Backout link: https://hg.mozilla.org/integration/autoland/rev/9dae371f2b336893faea10148a86fc1fca325584
Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=304228405&repo=autoland&lineNumber=612
[task 2020-05-29T08:19:45.977Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/util/schema.py", line 34, in validate_schema
[task 2020-05-29T08:19:45.977Z] raise Exception('\n'.join(msg) + '\n' + pprint.pformat(obj))
[task 2020-05-29T08:19:45.977Z] Exception: In build kind task u'android-x86_64-gcp/opt':
[task 2020-05-29T08:19:45.977Z] extra keys not allowed @ data[u'optimization'][u'push-interval-25']
[task 2020-05-29T08:19:45.977Z] required key not provided @ data[u'optimization'][u'always']
[task 2020-05-29T08:19:45.977Z] {u'attributes': {u'accepted-mar-channel-ids': u'firefox-mozilla-central',
[task 2020-05-29T08:19:45.977Z] u'artifact_map': u'taskcluster/taskgraph/manifests/fennec_geckoview.yml',
[task 2020-05-29T08:19:45.977Z] u'build_platform': u'android-x86_64-gcp',
[task 2020-05-29T08:19:45.977Z] u'build_type': u'opt',
[task 2020-05-29T08:19:45.977Z] u'mar-channel-id': u'firefox-mozilla-central'},
[task 2020-05-29T08:19:45.977Z] u'description': u'Android 5.0 x86-64 Opt - built on GCP',
[task 2020-05-29T08:19:45.977Z] u'fetches': {u'toolchain': [u'android-gradle-dependencies',
[task 2020-05-29T08:19:45.977Z] u'android-ndk-linux',
[task 2020-05-29T08:19:45.977Z] u'android-sdk-linux',
[task 2020-05-29T08:19:45.977Z] u'linux64-rust-android',
[task 2020-05-29T08:19:45.978Z] u'linux64-rust-size',
[task 2020-05-29T08:19:45.978Z] u'linux64-cbindgen',
[task 2020-05-29T08:19:45.978Z] u'linux64-nasm',
[task 2020-05-29T08:19:45.978Z] u'linux64-node',
[task 2020-05-29T08:19:45.978Z] u'linux64-clang',
[task 2020-05-29T08:19:45.978Z] u'linux64-sccache']},
[task 2020-05-29T08:19:45.978Z] u'index': {u'job-name': u'android-x86_64-gcp-opt', u'product': u'mobile'},
[task 2020-05-29T08:19:45.978Z] u'job-from': u'android.yml',
...
Assignee | ||
Comment 15•8 months ago
|
||
This was a merge conflict. Will rebase and re-land.
Comment 16•8 months ago
|
||
Pushed by ahalberstadt@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/2c18e64584af [taskgraph] Fix the 'push-interval-*' schema to not require an argument, r=jmaher https://hg.mozilla.org/integration/autoland/rev/f0d405cc42cb [taskgraph] Make 'upload-generated-sources' tasks copy their dependency's optimization r=Callek https://hg.mozilla.org/integration/autoland/rev/ee6cb8a2f458 [taskgraph] Make 'upload-symbols' tasks copy their dependency's optimization r=Callek https://hg.mozilla.org/integration/autoland/rev/683ec8a0929d [taskgraph] Make 'repackage' and 'repackage-signing' tasks copy their dependency's optimization, r=Callek
Assignee | ||
Comment 17•8 months ago
|
||
Ugh, that push-interval
patch bitrotted again despite an update to latest central. I'm trying to land that schema change at a time when people are really focusing on adding it to more places :p
I'll rebase on top of latest autoland next time.
Comment 18•8 months ago
|
||
Backed out changeset 2c18e64584af (bug 1641131) for gecko decision task bustage
Backout link: https://hg.mozilla.org/integration/autoland/rev/189788e0dec2b204d8aae83b35ce0078c2b172b8
Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=304277763&repo=autoland&lineNumber=683
[task 2020-05-29T15:18:46.055Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/transforms/base.py", line 87, in __call__
[task 2020-05-29T15:18:46.055Z] validate_schema(self.schema, task, error)
[task 2020-05-29T15:18:46.055Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/util/schema.py", line 34, in validate_schema
[task 2020-05-29T15:18:46.055Z] raise Exception('\n'.join(msg) + '\n' + pprint.pformat(obj))
[task 2020-05-29T15:18:46.055Z] Exception: In build kind task u'linux64-asan/debug':
[task 2020-05-29T15:18:46.055Z] extra keys not allowed @ data[u'optimization'][u'push-interval-10']
[task 2020-05-29T15:18:46.055Z] required key not provided @ data[u'optimization'][u'always']
[task 2020-05-29T15:18:46.055Z] {u'attributes': {u'accepted-mar-channel-ids': u'firefox-mozilla-central',
[task 2020-05-29T15:18:46.055Z] u'build_platform': u'linux64-asan',
[task 2020-05-29T15:18:46.055Z] u'build_type': u'debug',
[task 2020-05-29T15:18:46.055Z] u'mar-channel-id': u'firefox-mozilla-central'},
[task 2020-05-29T15:18:46.055Z] u'description': u'Linux64 Debug ASAN',
[task 2020-05-29T15:18:46.055Z] u'fetches': {u'toolchain': [u'linux64-binutils',
[task 2020-05-29T15:18:46.055Z] u'linux64-clang',
[task 2020-05-29T15:18:46.055Z] u'linux64-rust',
[task 2020-05-29T15:18:46.055Z] u'linux64-rust-size',
[task 2020-05-29T15:18:46.055Z] u'linux64-cbindgen',
[task 2020-05-29T15:18:46.055Z] u'linux64-sccache',
[task 2020-05-29T15:18:46.055Z] u'linux64-nasm',
[task 2020-05-29T15:18:46.055Z] u'linux64-node',
[task 2020-05-29T15:18:46.055Z] u'linux64-lucetc',
[task 2020-05-29T15:18:46.055Z] u'wasi-sysroot',
[task 2020-05-29T15:18:46.055Z] u'linux64-fix-stacks']},
[task 2020-05-29T15:18:46.055Z] u'index': {u'job-name': u'linux64-asan-debug', u'product': u'firefox'},
[task 2020-05-29T15:18:46.055Z] u'job-from': u'linux.yml',
[task 2020-05-29T15:18:46.055Z] u'name': u'linux64-asan/debug',
Assignee | ||
Updated•8 months ago
|
Assignee | ||
Comment 19•8 months ago
|
||
With the optimization
patches sticking, things are looking a bit more manageable. Here's an example push:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=d1003b11355a85ea0366eb541c596c7aac3c2fe6
You'll see some shippable builds still ran, but no more signing/repackage tasks at least.
Shippable builds on Linux / Macosx
Ran because of source-test tasks depending on them (i.e the mochitest unittests and others). I assume we switched them to use shippable
since opt
was no longer being built on autoland
. We'll likely want to have a transform that uses shippable
if it would have been scheduled anyway, but otherwise depends on opt
. Or maybe just hardcode them to opt
on try.
Generate profile on all platforms
Note the actual shippable builds didn't run on Windows, so that's good! But the profile generation tasks ran because they're needed by the DevEdition
builds which apparently uses PGO (should these builds have shippable in their labels?). So once we optimize `DevEdition properly, theses should also go away.
Comment 20•8 months ago
|
||
bugherder |
Comment 21•8 months ago
|
||
Pushed by ahalberstadt@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/01507268c72f [taskgraph] Fix the 'push-interval-*' schema to not require an argument, r=jmaher
Assignee | ||
Comment 22•8 months ago
|
||
Comment 23•8 months ago
|
||
bugherder |
Comment 24•8 months ago
|
||
Pushed by ahalberstadt@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/db92ded549e9 [taskgraph] Filter out devedition builds and tests with |mach try auto|, r=marco
Comment 25•8 months ago
|
||
bugherder |
Assignee | ||
Comment 26•7 months ago
|
||
Assignee | ||
Comment 27•7 months ago
|
||
Depends on D79854
Comment 28•7 months ago
|
||
Pushed by ahalberstadt@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/6122792219ec [ci] Don't use shippable builds as dependencies for source-test tasks on try, r=jmaher https://hg.mozilla.org/integration/autoland/rev/8d6b87a4b6d3 [ci] Don't run test-verify tasks on projects where the associated build is skipped, r=gbrown
Comment 29•7 months ago
|
||
bugherder |
Assignee | ||
Comment 30•6 months ago
|
||
I'm going to call this fixed for now. I'll open a new bug if I see further work is needed.
Description
•