1638014 - don't run shippable builds or tests on try in most cases

Status: RESOLVED FIXED

(Testing :: General, enhancement)

Description

[[e-mail] (:bhearsum)]

opt & shippable builds are very much alike in build configuration, and there aren't many cases where we should be running both of them on try. Opt builds are much, much faster, so they should be the preferred build for running mochitest/reftest/etc. on. Marco did some investigation and found very few cases where testing shippable builds found unique failures compared to opt.

We do still need the ability to run shippable builds & tests on try for certain types of work. I propose that shippable builds & tests should not be scheduled unless:
a) performance tests are requested (we don't do these for opt)
or
b) shippable builds are explicitly requested

This means that if, eg "mochitest" is selected, shippable builds & tests will not be run unless an explicit action is taken.

Comment 1

[[e-mail] (:bhearsum)]

glandium, I started digging into this and it looks like we can probably just add an entry to https://searchfox.org/mozilla-central/source/taskcluster/taskgraph/target_tasks.py#21 that matches all of the shippable tasks that we also have opt tasks for. Does that sound right to you?

Comment 2

[Mike Hommey [:glandium]]

I'm not entirely sure. That might prevent the talos and raptor tasks from being in the target set.

Comment 3

[[e-mail] (:bhearsum)]

Attachment: Bug 1638014: blacklist shippable tests from try when there is an opt version of the test available

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/45f3db139940
blacklist shippable tests from try when there is an opt version of the test available r=ahal

Comment 5

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/45f3db139940

Comment 6

[[e-mail] (:bhearsum)]

I've been using https://datastudio.google.com/reporting/1QG2CTQV1KCu0Vavdr0HZZNzKF5ykxocn/page/kxCQB/edit and https://console.cloud.google.com/bigquery?project=moz-fx-data-taskclu-prod-8fbf&supportedpurview=project&j=bq:US:bquxjob_675116de_17257dc13d5&page=queryresults to help calculate savings here.