Closed Bug 1531032 Opened 5 years ago Closed 5 years ago

try syntax shouldn't trigger tier 2 jobs by default

Categories

(Firefox Build System :: Task Configuration, task)

task
Not set
normal

Tracking

(firefox68 fixed)

RESOLVED FIXED
mozilla68
Tracking Status
firefox68 --- fixed

People

(Reporter: catlee, Assigned: catlee)

References

Details

Attachments

(1 file)

On try when someone pushes with '-p linux64', we end up scheduling all the linux64-qr tests, even though those aren't run on autoland/inbound by default.

I'd like to change the try syntax scheduling so that tier 2 jobs aren't scheduled unless specifically requested. For example, for linux64-qr tests, this would look something like:

try: -b o -p linux64 -u web-platform-tests[linux64-qr]

Assignee: nobody → catlee

kats, does this seem reasonable to only run linux64-qr tests when requested on try? I'm not sure how often we're catching unique regressions there.

Flags: needinfo?(kats)

If we do this I'd like to promote the web-platform reftest suite back to tier-1 and have it run on inbound/autoland. Those tests do catch regressions, while the other WPT tests generally do not. Does that sound reasonable?

Flags: needinfo?(kats)

this sounds good to me; lets turn off wpt and make wr tier-1

My expectation is that the status of wpt (testharness) tests would match mochitest-plain and Wr (wpt reftests) would match the reftest suite. Deviations from that setup resulting in (any flavour of) wpt test running less often than the corresponding Mozilla-only tests would need some jusification since the corresponding suites have similar capabilities and we are trying to move more coverage into the shared wpt tests.

If the proposal here doesn't imply such a difference I don't have any concerns, if it does then I think more consideration is required.

the proposal here is to run for -qr tests that we run web-platform-tests-reftests all the time as tier-1, just like non -qr configs. In addition, the proposal is to not run web-platform-tests (plain) on -qr configs, as there is little to no value in doing so.

I think for what :jgraham mentions we are making this uniform for wpt-reftests, but not uniform for wpt plain in the -qr case.

Pushed by catlee@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/4de00f3a05ad
disable tier2 jobs from running with try syntax unless specifically enabled. r=tomprince
Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla68
Regressions: 1550866
Regressions: 1573239
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: