try syntax shouldn't trigger tier 2 jobs by default
Categories
(Firefox Build System :: Task Configuration, task)
Tracking
(firefox68 fixed)
Tracking | Status | |
---|---|---|
firefox68 | --- | fixed |
People
(Reporter: catlee, Assigned: catlee)
References
Details
Attachments
(1 file)
On try when someone pushes with '-p linux64', we end up scheduling all the linux64-qr tests, even though those aren't run on autoland/inbound by default.
I'd like to change the try syntax scheduling so that tier 2 jobs aren't scheduled unless specifically requested. For example, for linux64-qr tests, this would look something like:
try: -b o -p linux64 -u web-platform-tests[linux64-qr]
Assignee | ||
Updated•5 years ago
|
Assignee | ||
Comment 1•5 years ago
|
||
kats, does this seem reasonable to only run linux64-qr tests when requested on try? I'm not sure how often we're catching unique regressions there.
Comment 2•5 years ago
|
||
If we do this I'd like to promote the web-platform reftest suite back to tier-1 and have it run on inbound/autoland. Those tests do catch regressions, while the other WPT tests generally do not. Does that sound reasonable?
Comment 3•5 years ago
|
||
this sounds good to me; lets turn off wpt and make wr tier-1
Comment 4•5 years ago
|
||
My expectation is that the status of wpt (testharness) tests would match mochitest-plain and Wr (wpt reftests) would match the reftest suite. Deviations from that setup resulting in (any flavour of) wpt test running less often than the corresponding Mozilla-only tests would need some jusification since the corresponding suites have similar capabilities and we are trying to move more coverage into the shared wpt tests.
If the proposal here doesn't imply such a difference I don't have any concerns, if it does then I think more consideration is required.
Comment 5•5 years ago
|
||
the proposal here is to run for -qr tests that we run web-platform-tests-reftests all the time as tier-1, just like non -qr configs. In addition, the proposal is to not run web-platform-tests (plain) on -qr configs, as there is little to no value in doing so.
I think for what :jgraham mentions we are making this uniform for wpt-reftests, but not uniform for wpt plain in the -qr case.
Assignee | ||
Comment 6•5 years ago
|
||
Enable WPT-reftests as tier1
Pushed by catlee@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/4de00f3a05ad disable tier2 jobs from running with try syntax unless specifically enabled. r=tomprince
Comment 8•5 years ago
|
||
bugherder |
Description
•