Closed Bug 1638014 Opened 2 years ago Closed 1 year ago

don't run shippable builds or tests on try in most cases


(Testing :: General, enhancement)

Version 3


(firefox78 fixed)

Tracking Status
firefox78 --- fixed


(Reporter: bhearsum, Assigned: bhearsum, NeedInfo)


(Blocks 1 open bug)


(Whiteboard: [ci-costs-2020:done])


(1 file)

opt & shippable builds are very much alike in build configuration, and there aren't many cases where we should be running both of them on try. Opt builds are much, much faster, so they should be the preferred build for running mochitest/reftest/etc. on. Marco did some investigation and found very few cases where testing shippable builds found unique failures compared to opt.

We do still need the ability to run shippable builds & tests on try for certain types of work. I propose that shippable builds & tests should not be scheduled unless:
a) performance tests are requested (we don't do these for opt)
b) shippable builds are explicitly requested

This means that if, eg "mochitest" is selected, shippable builds & tests will not be run unless an explicit action is taken.

Whiteboard: [ci-costs-2020:todo]
Assignee: nobody → bhearsum

glandium, I started digging into this and it looks like we can probably just add an entry to that matches all of the shippable tasks that we also have opt tasks for. Does that sound right to you?

Flags: needinfo?(mh+mozilla)

I'm not entirely sure. That might prevent the talos and raptor tasks from being in the target set.

Flags: needinfo?(mh+mozilla) → needinfo?(mozilla)
Depends on: 1638945
Pushed by
blacklist shippable tests from try when there is an opt version of the test available r=ahal
Closed: 1 year ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla78
Whiteboard: [ci-costs-2020:todo] → [ci-costs-2020:done]
See Also: → 1645288
You need to log in before you can comment on or make changes to this bug.