Closed Bug 1641131 Opened 8 months ago Closed 6 months ago

Stop running shippable and other unnecessary builds with |mach try auto|

Categories

(Firefox Build System :: Task Configuration, task, P2)

Product:
Firefox Build System
Component:
Task Configuration
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: ahal, Assigned: ahal)

References

(Blocks 1 open bug)

Details

Attachments

(7 files, 1 obsolete file)

Bug 1641131 - [taskgraph] Make 'upload-generated-sources' tasks copy their dependency's optimization
47 bytes, text/x-phabricator-request
Details | Review
Bug 1641131 - [taskgraph] Make 'upload-symbols' tasks copy their dependency's optimization
47 bytes, text/x-phabricator-request
Details | Review
Bug 1641131 - [taskgraph] Fix the 'push-interval-*' schema to not require an argument, r?jmaher
47 bytes, text/x-phabricator-request
Details | Review
Bug 1641131 - [taskgraph] Make 'repackage' and 'repackage-signing' tasks copy their dependency's optimization, r?Callek
47 bytes, text/x-phabricator-request
Details | Review
Bug 1641131 - [taskgraph] Filter out devedition builds and tests with |mach try auto|, r?marco
47 bytes, text/x-phabricator-request
Details | Review
Bug 1641131 - [ci] Don't use shippable builds as dependencies for source-test tasks on try, r?jmaher
47 bytes, text/x-phabricator-request
Details | Review
Bug 1641131 - [ci] Don't run test-verify tasks on projects where the associated build is skipped, r?gbrown
47 bytes, text/x-phabricator-request
Details | Review

Ben Hearsum recently landed a change to stop running shippable builds/tests on try. However they're still being selected with |mach try auto| (though no test / perf tasks are).

I believe this is due to the build-signing and upload-generated-symbols tasks.

Removing build-signing + upload-generated-symbols wasn't enough to get rid of shippable builds, but it did remove some DevEdition builds. Unrelatedly I discovered that the win64-aarch builds are still running because they have Be as a dependent which in turn has a diffoscope task (DWE on treeherder) as a dependent. Not sure what that is, but doubt it's important enough to run on all |mach try auto| pushes.

Status: ASSIGNED → NEW
Summary: Stop running shippable builds with |mach try auto| → Stop running shippable and other unnecessary builds with |mach try auto|

To remove the DevEdition builds, we could use filter_out_devedition in target_tasks_try_auto.

I had an idea.. we could create a "default" optimization that aliases to the "never" strategy and is applied to all tasks that don't have any other optimizations. Then the tryselect override can change it to "always". So we'd only run tasks that have an explicit optimization set.

In some cases it be worth getting dependents to copy their parent's optimization in addition.

(In reply to Marco Castelluccio [:marco] from comment #4)

To remove the DevEdition builds, we could use filter_out_devedition in target_tasks_try_auto.

They're currently filtered out by bugbug_reduced, I think they were just running due to the build-signing tasks. Seems worth keeping it that way in case they're actually relevant to some patch.

Turns out the build-signing tasks already copy their dependencies' optimization:
https://searchfox.org/mozilla-central/source/taskcluster/taskgraph/transforms/signing.py#174

So that shouldn't be why the shippable tasks are still running.. Indeed they're still running even after my patch above.

Attachment #9151973 - Attachment description: Bug 1641131 - [taskgraph] Add 'push-interval-10' optimization to win64-aarch-devedition builds → Bug 1641131 - [taskgraph] Add 'push-interval-10' optimization to win64-aarch-devedition builds, r?jmaher
Attachment #9151972 - Attachment description: Bug 1641131 - [taskgraph] Always optimize 'build-signing' and 'upload-generated-sources' tasks on try → Bug 1641131 - [taskgraph] Make 'upload-generated-sources' tasks copy their dependency's optimization

Note: DevEdition builds are still running on some |mach try auto| pushes solely due to test-verify.

Status: NEW → ASSIGNED

(In reply to Andrew Halberstadt [:ahal] from comment #6)

(In reply to Marco Castelluccio [:marco] from comment #4)

To remove the DevEdition builds, we could use filter_out_devedition in target_tasks_try_auto.

They're currently filtered out by bugbug_reduced, I think they were just running due to the build-signing tasks. Seems worth keeping it that way in case they're actually relevant to some patch.

(In reply to Andrew Halberstadt [:ahal] from comment #8)

Note: DevEdition builds are still running on some |mach try auto| pushes solely due to test-verify.

They don't run on autoland (AFAICS), so they'd never be selected by bugbug anyway. Maybe it's best to filter them out so they don't run just because of test-verify.

The Backstop optimization doesn't take any arguments, yet the schema for 'push-optimization-*'
requires a 'schedules-component' that goes ignored. Fix this.

Attachment #9151973 - Attachment is obsolete: true
Pushed by ahalberstadt@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/48e73dbc588e
[taskgraph] Fix the 'push-interval-*' schema to not require an argument, r=jmaher
https://hg.mozilla.org/integration/autoland/rev/2e034897a64f
[taskgraph] Make 'upload-generated-sources' tasks copy their dependency's optimization r=Callek
https://hg.mozilla.org/integration/autoland/rev/6f3a65196d88
[taskgraph] Make 'upload-symbols' tasks copy their dependency's optimization r=Callek

Backed out 3 changesets (bug 1641131) for gecko decision task bustage.

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&selectedTaskRun=Npd5DGgFT1SQE6NlqtN0fQ-0&fromchange=bfb3aee94a88794cfd51b5a80bf162357b2606cf&tochange=9dae371f2b336893faea10148a86fc1fca325584&searchStr=gecko%2Cdecision%2Ctask%2Copt%2Cgecko%2Cdecision%2Ctask%2C%28d%29

Backout link: https://hg.mozilla.org/integration/autoland/rev/9dae371f2b336893faea10148a86fc1fca325584

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=304228405&repo=autoland&lineNumber=612

[task 2020-05-29T08:19:45.977Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/util/schema.py", line 34, in validate_schema
[task 2020-05-29T08:19:45.977Z]     raise Exception('\n'.join(msg) + '\n' + pprint.pformat(obj))
[task 2020-05-29T08:19:45.977Z] Exception: In build kind task u'android-x86_64-gcp/opt':
[task 2020-05-29T08:19:45.977Z] extra keys not allowed @ data[u'optimization'][u'push-interval-25']
[task 2020-05-29T08:19:45.977Z] required key not provided @ data[u'optimization'][u'always']
[task 2020-05-29T08:19:45.977Z] {u'attributes': {u'accepted-mar-channel-ids': u'firefox-mozilla-central',
[task 2020-05-29T08:19:45.977Z]                  u'artifact_map': u'taskcluster/taskgraph/manifests/fennec_geckoview.yml',
[task 2020-05-29T08:19:45.977Z]                  u'build_platform': u'android-x86_64-gcp',
[task 2020-05-29T08:19:45.977Z]                  u'build_type': u'opt',
[task 2020-05-29T08:19:45.977Z]                  u'mar-channel-id': u'firefox-mozilla-central'},
[task 2020-05-29T08:19:45.977Z]  u'description': u'Android 5.0 x86-64 Opt - built on GCP',
[task 2020-05-29T08:19:45.977Z]  u'fetches': {u'toolchain': [u'android-gradle-dependencies',
[task 2020-05-29T08:19:45.977Z]                              u'android-ndk-linux',
[task 2020-05-29T08:19:45.977Z]                              u'android-sdk-linux',
[task 2020-05-29T08:19:45.977Z]                              u'linux64-rust-android',
[task 2020-05-29T08:19:45.978Z]                              u'linux64-rust-size',
[task 2020-05-29T08:19:45.978Z]                              u'linux64-cbindgen',
[task 2020-05-29T08:19:45.978Z]                              u'linux64-nasm',
[task 2020-05-29T08:19:45.978Z]                              u'linux64-node',
[task 2020-05-29T08:19:45.978Z]                              u'linux64-clang',
[task 2020-05-29T08:19:45.978Z]                              u'linux64-sccache']},
[task 2020-05-29T08:19:45.978Z]  u'index': {u'job-name': u'android-x86_64-gcp-opt', u'product': u'mobile'},
[task 2020-05-29T08:19:45.978Z]  u'job-from': u'android.yml',
...
Flags: needinfo?(ahal)

This was a merge conflict. Will rebase and re-land.

Flags: needinfo?(ahal)
Pushed by ahalberstadt@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/2c18e64584af
[taskgraph] Fix the 'push-interval-*' schema to not require an argument, r=jmaher
https://hg.mozilla.org/integration/autoland/rev/f0d405cc42cb
[taskgraph] Make 'upload-generated-sources' tasks copy their dependency's optimization r=Callek
https://hg.mozilla.org/integration/autoland/rev/ee6cb8a2f458
[taskgraph] Make 'upload-symbols' tasks copy their dependency's optimization r=Callek
https://hg.mozilla.org/integration/autoland/rev/683ec8a0929d
[taskgraph] Make 'repackage' and 'repackage-signing' tasks copy their dependency's optimization, r=Callek

Ugh, that push-interval patch bitrotted again despite an update to latest central. I'm trying to land that schema change at a time when people are really focusing on adding it to more places :p

I'll rebase on top of latest autoland next time.

Backed out changeset 2c18e64584af (bug 1641131) for gecko decision task bustage

Push with failure: https://treeherder.mozilla.org/#/jobs?repo=autoland&fromchange=683ec8a0929d4b56d24eabd51c6c39207af4e41f&tochange=189788e0dec2b204d8aae83b35ce0078c2b172b8&searchStr=gecko%2Cdecision%2Ctask%2Copt%2Cgecko%2Cdecision%2Ctask%2C%28d%29&selectedTaskRun=Am_R0tVcREyXichgN8q1sA-0

Backout link: https://hg.mozilla.org/integration/autoland/rev/189788e0dec2b204d8aae83b35ce0078c2b172b8

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=304277763&repo=autoland&lineNumber=683

[task 2020-05-29T15:18:46.055Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/transforms/base.py", line 87, in __call__
[task 2020-05-29T15:18:46.055Z]     validate_schema(self.schema, task, error)
[task 2020-05-29T15:18:46.055Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/util/schema.py", line 34, in validate_schema
[task 2020-05-29T15:18:46.055Z]     raise Exception('\n'.join(msg) + '\n' + pprint.pformat(obj))
[task 2020-05-29T15:18:46.055Z] Exception: In build kind task u'linux64-asan/debug':
[task 2020-05-29T15:18:46.055Z] extra keys not allowed @ data[u'optimization'][u'push-interval-10']
[task 2020-05-29T15:18:46.055Z] required key not provided @ data[u'optimization'][u'always']
[task 2020-05-29T15:18:46.055Z] {u'attributes': {u'accepted-mar-channel-ids': u'firefox-mozilla-central',
[task 2020-05-29T15:18:46.055Z]                  u'build_platform': u'linux64-asan',
[task 2020-05-29T15:18:46.055Z]                  u'build_type': u'debug',
[task 2020-05-29T15:18:46.055Z]                  u'mar-channel-id': u'firefox-mozilla-central'},
[task 2020-05-29T15:18:46.055Z]  u'description': u'Linux64 Debug ASAN',
[task 2020-05-29T15:18:46.055Z]  u'fetches': {u'toolchain': [u'linux64-binutils',
[task 2020-05-29T15:18:46.055Z]                              u'linux64-clang',
[task 2020-05-29T15:18:46.055Z]                              u'linux64-rust',
[task 2020-05-29T15:18:46.055Z]                              u'linux64-rust-size',
[task 2020-05-29T15:18:46.055Z]                              u'linux64-cbindgen',
[task 2020-05-29T15:18:46.055Z]                              u'linux64-sccache',
[task 2020-05-29T15:18:46.055Z]                              u'linux64-nasm',
[task 2020-05-29T15:18:46.055Z]                              u'linux64-node',
[task 2020-05-29T15:18:46.055Z]                              u'linux64-lucetc',
[task 2020-05-29T15:18:46.055Z]                              u'wasi-sysroot',
[task 2020-05-29T15:18:46.055Z]                              u'linux64-fix-stacks']},
[task 2020-05-29T15:18:46.055Z]  u'index': {u'job-name': u'linux64-asan-debug', u'product': u'firefox'},
[task 2020-05-29T15:18:46.055Z]  u'job-from': u'linux.yml',
[task 2020-05-29T15:18:46.055Z]  u'name': u'linux64-asan/debug',
Flags: needinfo?(ahal)
Flags: needinfo?(ahal)

With the optimization patches sticking, things are looking a bit more manageable. Here's an example push:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=d1003b11355a85ea0366eb541c596c7aac3c2fe6

You'll see some shippable builds still ran, but no more signing/repackage tasks at least.

Shippable builds on Linux / Macosx

Ran because of source-test tasks depending on them (i.e the mochitest unittests and others). I assume we switched them to use shippable since opt was no longer being built on autoland. We'll likely want to have a transform that uses shippable if it would have been scheduled anyway, but otherwise depends on opt. Or maybe just hardcode them to opt on try.

Generate profile on all platforms

Note the actual shippable builds didn't run on Windows, so that's good! But the profile generation tasks ran because they're needed by the DevEdition builds which apparently uses PGO (should these builds have shippable in their labels?). So once we optimize `DevEdition properly, theses should also go away.

Pushed by ahalberstadt@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/01507268c72f
[taskgraph] Fix the 'push-interval-*' schema to not require an argument, r=jmaher
Pushed by ahalberstadt@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/db92ded549e9
[taskgraph] Filter out devedition builds and tests with |mach try auto|, r=marco
Pushed by ahalberstadt@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/6122792219ec
[ci] Don't use shippable builds as dependencies for source-test tasks on try, r=jmaher
https://hg.mozilla.org/integration/autoland/rev/8d6b87a4b6d3
[ci] Don't run test-verify tasks on projects where the associated build is skipped, r=gbrown

I'm going to call this fixed for now. I'll open a new bug if I see further work is needed.

Status: ASSIGNED → RESOLVED
Closed: 6 months ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.