1641131 - Stop running shippable and other unnecessary builds with |mach try auto|

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, task, P2)

Description

[Andrew Halberstadt [:ahal]]

Ben Hearsum recently landed a change to stop running shippable builds/tests on try. However they're still being selected with |mach try auto| (though no test / perf tasks are).

I believe this is due to the build-signing and upload-generated-symbols tasks.

Comment 1

[Andrew Halberstadt [:ahal]]

Removing build-signing + upload-generated-symbols wasn't enough to get rid of shippable builds, but it did remove some DevEdition builds. Unrelatedly I discovered that the win64-aarch builds are still running because they have Be as a dependent which in turn has a diffoscope task (DWE on treeherder) as a dependent. Not sure what that is, but doubt it's important enough to run on all |mach try auto| pushes.

Comment 2

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1641131 - [taskgraph] Make 'upload-generated-sources' tasks copy their dependency's optimization

Comment 3

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1641131 - [taskgraph] Add 'push-interval-10' optimization to win64-aarch-devedition builds, r?jmaher

Depends on D76988

Comment 4

[Marco Castelluccio [:marco]]

To remove the DevEdition builds, we could use filter_out_devedition in target_tasks_try_auto.

Comment 5

[Andrew Halberstadt [:ahal]]

I had an idea.. we could create a "default" optimization that aliases to the "never" strategy and is applied to all tasks that don't have any other optimizations. Then the tryselect override can change it to "always". So we'd only run tasks that have an explicit optimization set.

In some cases it be worth getting dependents to copy their parent's optimization in addition.

Comment 6

[Andrew Halberstadt [:ahal]]

(In reply to Marco Castelluccio [:marco] from comment #4)

To remove the DevEdition builds, we could use filter_out_devedition in target_tasks_try_auto.

They're currently filtered out by bugbug_reduced, I think they were just running due to the build-signing tasks. Seems worth keeping it that way in case they're actually relevant to some patch.

Comment 7

[Andrew Halberstadt [:ahal]]

Turns out the build-signing tasks already copy their dependencies' optimization:
https://searchfox.org/mozilla-central/source/taskcluster/taskgraph/transforms/signing.py#174

So that shouldn't be why the shippable tasks are still running.. Indeed they're still running even after my patch above.

Comment 8

[Andrew Halberstadt [:ahal]]

Note: DevEdition builds are still running on some |mach try auto| pushes solely due to test-verify.

Comment 9

[Marco Castelluccio [:marco]]

(In reply to Andrew Halberstadt [:ahal] from comment #6)

(In reply to Marco Castelluccio [:marco] from comment #4)

To remove the DevEdition builds, we could use filter_out_devedition in target_tasks_try_auto.

They're currently filtered out by bugbug_reduced, I think they were just running due to the build-signing tasks. Seems worth keeping it that way in case they're actually relevant to some patch.

(In reply to Andrew Halberstadt [:ahal] from comment #8)

Note: DevEdition builds are still running on some |mach try auto| pushes solely due to test-verify.

They don't run on autoland (AFAICS), so they'd never be selected by bugbug anyway. Maybe it's best to filter them out so they don't run just because of test-verify.

Comment 10

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1641131 - [taskgraph] Make 'upload-symbols' tasks copy their dependency's optimization

Depends on D76988

Comment 11

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1641131 - [taskgraph] Fix the 'push-interval-*' schema to not require an argument, r?jmaher

The Backstop optimization doesn't take any arguments, yet the schema for 'push-optimization-*'
requires a 'schedules-component' that goes ignored. Fix this.

Comment 12

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1641131 - [taskgraph] Make 'repackage' and 'repackage-signing' tasks copy their dependency's optimization, r?Callek

Depends on D77048

Comment 13

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/48e73dbc588e
[taskgraph] Fix the 'push-interval-*' schema to not require an argument, r=jmaher
https://hg.mozilla.org/integration/autoland/rev/2e034897a64f
[taskgraph] Make 'upload-generated-sources' tasks copy their dependency's optimization r=Callek
https://hg.mozilla.org/integration/autoland/rev/6f3a65196d88
[taskgraph] Make 'upload-symbols' tasks copy their dependency's optimization r=Callek

Comment 14

[Bogdan Tara[:bogdan_tara | bogdant]]

Backed out 3 changesets (bug 1641131) for gecko decision task bustage.

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&selectedTaskRun=Npd5DGgFT1SQE6NlqtN0fQ-0&fromchange=bfb3aee94a88794cfd51b5a80bf162357b2606cf&tochange=9dae371f2b336893faea10148a86fc1fca325584&searchStr=gecko%2Cdecision%2Ctask%2Copt%2Cgecko%2Cdecision%2Ctask%2C%28d%29

Backout link: https://hg.mozilla.org/integration/autoland/rev/9dae371f2b336893faea10148a86fc1fca325584

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=304228405&repo=autoland&lineNumber=612

[task 2020-05-29T08:19:45.977Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/util/schema.py", line 34, in validate_schema
[task 2020-05-29T08:19:45.977Z]     raise Exception('\n'.join(msg) + '\n' + pprint.pformat(obj))
[task 2020-05-29T08:19:45.977Z] Exception: In build kind task u'android-x86_64-gcp/opt':
[task 2020-05-29T08:19:45.977Z] extra keys not allowed @ data[u'optimization'][u'push-interval-25']
[task 2020-05-29T08:19:45.977Z] required key not provided @ data[u'optimization'][u'always']
[task 2020-05-29T08:19:45.977Z] {u'attributes': {u'accepted-mar-channel-ids': u'firefox-mozilla-central',
[task 2020-05-29T08:19:45.977Z]                  u'artifact_map': u'taskcluster/taskgraph/manifests/fennec_geckoview.yml',
[task 2020-05-29T08:19:45.977Z]                  u'build_platform': u'android-x86_64-gcp',
[task 2020-05-29T08:19:45.977Z]                  u'build_type': u'opt',
[task 2020-05-29T08:19:45.977Z]                  u'mar-channel-id': u'firefox-mozilla-central'},
[task 2020-05-29T08:19:45.977Z]  u'description': u'Android 5.0 x86-64 Opt - built on GCP',
[task 2020-05-29T08:19:45.977Z]  u'fetches': {u'toolchain': [u'android-gradle-dependencies',
[task 2020-05-29T08:19:45.977Z]                              u'android-ndk-linux',
[task 2020-05-29T08:19:45.977Z]                              u'android-sdk-linux',
[task 2020-05-29T08:19:45.977Z]                              u'linux64-rust-android',
[task 2020-05-29T08:19:45.978Z]                              u'linux64-rust-size',
[task 2020-05-29T08:19:45.978Z]                              u'linux64-cbindgen',
[task 2020-05-29T08:19:45.978Z]                              u'linux64-nasm',
[task 2020-05-29T08:19:45.978Z]                              u'linux64-node',
[task 2020-05-29T08:19:45.978Z]                              u'linux64-clang',
[task 2020-05-29T08:19:45.978Z]                              u'linux64-sccache']},
[task 2020-05-29T08:19:45.978Z]  u'index': {u'job-name': u'android-x86_64-gcp-opt', u'product': u'mobile'},
[task 2020-05-29T08:19:45.978Z]  u'job-from': u'android.yml',
...

Comment 15

[Andrew Halberstadt [:ahal]]

This was a merge conflict. Will rebase and re-land.

Comment 16

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/2c18e64584af
[taskgraph] Fix the 'push-interval-*' schema to not require an argument, r=jmaher
https://hg.mozilla.org/integration/autoland/rev/f0d405cc42cb
[taskgraph] Make 'upload-generated-sources' tasks copy their dependency's optimization r=Callek
https://hg.mozilla.org/integration/autoland/rev/ee6cb8a2f458
[taskgraph] Make 'upload-symbols' tasks copy their dependency's optimization r=Callek
https://hg.mozilla.org/integration/autoland/rev/683ec8a0929d
[taskgraph] Make 'repackage' and 'repackage-signing' tasks copy their dependency's optimization, r=Callek

Comment 17

[Andrew Halberstadt [:ahal]]

Ugh, that push-interval patch bitrotted again despite an update to latest central. I'm trying to land that schema change at a time when people are really focusing on adding it to more places :p

I'll rebase on top of latest autoland next time.

Comment 18

[Bogdan Tara[:bogdan_tara | bogdant]]

Backed out changeset 2c18e64584af (bug 1641131) for gecko decision task bustage

Push with failure: https://treeherder.mozilla.org/#/jobs?repo=autoland&fromchange=683ec8a0929d4b56d24eabd51c6c39207af4e41f&tochange=189788e0dec2b204d8aae83b35ce0078c2b172b8&searchStr=gecko%2Cdecision%2Ctask%2Copt%2Cgecko%2Cdecision%2Ctask%2C%28d%29&selectedTaskRun=Am_R0tVcREyXichgN8q1sA-0

Backout link: https://hg.mozilla.org/integration/autoland/rev/189788e0dec2b204d8aae83b35ce0078c2b172b8

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=304277763&repo=autoland&lineNumber=683

[task 2020-05-29T15:18:46.055Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/transforms/base.py", line 87, in __call__
[task 2020-05-29T15:18:46.055Z]     validate_schema(self.schema, task, error)
[task 2020-05-29T15:18:46.055Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/util/schema.py", line 34, in validate_schema
[task 2020-05-29T15:18:46.055Z]     raise Exception('\n'.join(msg) + '\n' + pprint.pformat(obj))
[task 2020-05-29T15:18:46.055Z] Exception: In build kind task u'linux64-asan/debug':
[task 2020-05-29T15:18:46.055Z] extra keys not allowed @ data[u'optimization'][u'push-interval-10']
[task 2020-05-29T15:18:46.055Z] required key not provided @ data[u'optimization'][u'always']
[task 2020-05-29T15:18:46.055Z] {u'attributes': {u'accepted-mar-channel-ids': u'firefox-mozilla-central',
[task 2020-05-29T15:18:46.055Z]                  u'build_platform': u'linux64-asan',
[task 2020-05-29T15:18:46.055Z]                  u'build_type': u'debug',
[task 2020-05-29T15:18:46.055Z]                  u'mar-channel-id': u'firefox-mozilla-central'},
[task 2020-05-29T15:18:46.055Z]  u'description': u'Linux64 Debug ASAN',
[task 2020-05-29T15:18:46.055Z]  u'fetches': {u'toolchain': [u'linux64-binutils',
[task 2020-05-29T15:18:46.055Z]                              u'linux64-clang',
[task 2020-05-29T15:18:46.055Z]                              u'linux64-rust',
[task 2020-05-29T15:18:46.055Z]                              u'linux64-rust-size',
[task 2020-05-29T15:18:46.055Z]                              u'linux64-cbindgen',
[task 2020-05-29T15:18:46.055Z]                              u'linux64-sccache',
[task 2020-05-29T15:18:46.055Z]                              u'linux64-nasm',
[task 2020-05-29T15:18:46.055Z]                              u'linux64-node',
[task 2020-05-29T15:18:46.055Z]                              u'linux64-lucetc',
[task 2020-05-29T15:18:46.055Z]                              u'wasi-sysroot',
[task 2020-05-29T15:18:46.055Z]                              u'linux64-fix-stacks']},
[task 2020-05-29T15:18:46.055Z]  u'index': {u'job-name': u'linux64-asan-debug', u'product': u'firefox'},
[task 2020-05-29T15:18:46.055Z]  u'job-from': u'linux.yml',
[task 2020-05-29T15:18:46.055Z]  u'name': u'linux64-asan/debug',

Comment 19

[Andrew Halberstadt [:ahal]]

With the optimization patches sticking, things are looking a bit more manageable. Here's an example push:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=d1003b11355a85ea0366eb541c596c7aac3c2fe6

You'll see some shippable builds still ran, but no more signing/repackage tasks at least.

Shippable builds on Linux / Macosx

Ran because of source-test tasks depending on them (i.e the mochitest unittests and others). I assume we switched them to use shippable since opt was no longer being built on autoland. We'll likely want to have a transform that uses shippable if it would have been scheduled anyway, but otherwise depends on opt. Or maybe just hardcode them to opt on try.

Generate profile on all platforms

Note the actual shippable builds didn't run on Windows, so that's good! But the profile generation tasks ran because they're needed by the DevEdition builds which apparently uses PGO (should these builds have shippable in their labels?). So once we optimize `DevEdition properly, theses should also go away.

Comment 20

[Cristina Coroiu [:ccoroiu]]

https://hg.mozilla.org/mozilla-central/rev/f0d405cc42cb
https://hg.mozilla.org/mozilla-central/rev/ee6cb8a2f458
https://hg.mozilla.org/mozilla-central/rev/683ec8a0929d

Comment 21

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/01507268c72f
[taskgraph] Fix the 'push-interval-*' schema to not require an argument, r=jmaher

Comment 22

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1641131 - [taskgraph] Filter out devedition builds and tests with |mach try auto|, r?marco

Comment 23

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/01507268c72f

Comment 24

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/db92ded549e9
[taskgraph] Filter out devedition builds and tests with |mach try auto|, r=marco

Comment 25

[Dorel Luca [:dluca]]

https://hg.mozilla.org/mozilla-central/rev/db92ded549e9

Comment 26

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1641131 - [ci] Don't use shippable builds as dependencies for source-test tasks on try, r?jmaher

Comment 27

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1641131 - [ci] Don't run test-verify tasks on projects where the associated build is skipped, r?gbrown

Depends on D79854

Comment 28

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/6122792219ec
[ci] Don't use shippable builds as dependencies for source-test tasks on try, r=jmaher
https://hg.mozilla.org/integration/autoland/rev/8d6b87a4b6d3
[ci] Don't run test-verify tasks on projects where the associated build is skipped, r=gbrown

Comment 29

[Razvan Maries]

https://hg.mozilla.org/mozilla-central/rev/6122792219ec
https://hg.mozilla.org/mozilla-central/rev/8d6b87a4b6d3

Comment 30

[Andrew Halberstadt [:ahal]]

I'm going to call this fixed for now. I'll open a new bug if I see further work is needed.