Closed Bug 1638014 Opened 2 years ago Closed 1 year ago

don't run shippable builds or tests on try in most cases

Categories

(Testing :: General, enhancement)

Version 3
enhancement

Tracking

(firefox78 fixed)

RESOLVED FIXED
mozilla78
Tracking Status
firefox78 --- fixed

People

(Reporter: bhearsum, Assigned: bhearsum, NeedInfo)

References

(Blocks 1 open bug)

Details

(Whiteboard: [ci-costs-2020:done])

Attachments

(1 file)

opt & shippable builds are very much alike in build configuration, and there aren't many cases where we should be running both of them on try. Opt builds are much, much faster, so they should be the preferred build for running mochitest/reftest/etc. on. Marco did some investigation and found very few cases where testing shippable builds found unique failures compared to opt.

We do still need the ability to run shippable builds & tests on try for certain types of work. I propose that shippable builds & tests should not be scheduled unless:
a) performance tests are requested (we don't do these for opt)
or
b) shippable builds are explicitly requested

This means that if, eg "mochitest" is selected, shippable builds & tests will not be run unless an explicit action is taken.

Whiteboard: [ci-costs-2020:todo]
Assignee: nobody → bhearsum

glandium, I started digging into this and it looks like we can probably just add an entry to https://searchfox.org/mozilla-central/source/taskcluster/taskgraph/target_tasks.py#21 that matches all of the shippable tasks that we also have opt tasks for. Does that sound right to you?

Flags: needinfo?(mh+mozilla)

I'm not entirely sure. That might prevent the talos and raptor tasks from being in the target set.

Flags: needinfo?(mh+mozilla) → needinfo?(mozilla)
Depends on: 1638945
Pushed by bhearsum@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/45f3db139940
blacklist shippable tests from try when there is an opt version of the test available r=ahal
Status: NEW → RESOLVED
Closed: 1 year ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla78
Whiteboard: [ci-costs-2020:todo] → [ci-costs-2020:done]
See Also: → 1645288
You need to log in before you can comment on or make changes to this bug.