Closed Bug 1363344 Opened 4 years ago Closed 4 years ago

Make linux64-qr reftests tier-1 and run them on autoland/inbound

Categories

(Core :: Graphics: WebRender, enhancement, P3)

Other Branch
enhancement

Tracking

()

RESOLVED FIXED
mozilla55
Tracking Status
firefox55 --- fixed

People

(Reporter: kats, Assigned: kats)

References

Details

(Whiteboard: [gfx-noted])

Attachments

(1 file)

Lately we've had a number of cases where after a merge from m-c to graphics we end up with permafail or high-frequency intermittent failures on QR tests. These take time to bisect and is not a reliable process because of the potential intermittency. Avoid this is one reason I want to start running the QR reftests on autoland/inbound and make them tier-1, so that anything that regresses them gets detected earlier and backed out.

Also eventually we're going to want this anyway because we'd like to stop working on the graphics branch and move to working directly on central. Turning on the tests on central/integration branches is a necessary prerequisite for this.

The downside to doing this, of course, is we'll need more machine time. Right now all we want to enable are linux64 tests which I hear is the cheapest to scale up. If we can use SETA to further reduce the number of times it runs that's fine by me, as long as the sheriffs have the ability to backfill and bisect as needed.
I also plan to fix bug 1342488 soonish, which will remove the QR builds as a separate build job and just use the regular non-QR build with the pref flipped on. That should save some machine time.
r? to dustin for the actual config file changes, r? to catlee for the fact that this will increase load on AWS and that we're ok with doing this. I talked with Milan about this yesterday and he said to flag you for review on this.
Assignee: nobody → bugmail
Comment on attachment 8865877 [details]
Bug 1363344 - Update linux64-qr test jobs on central and integration branches.

https://reviewboard.mozilla.org/r/137478/#review140658

I like it -- it's nice to see some of the special-cases going away
Attachment #8865877 - Flags: review?(dustin) → review+
Comment on attachment 8865877 [details]
Bug 1363344 - Update linux64-qr test jobs on central and integration branches.

Based on an estimated 23h of additional test load per push, about 3200 pushes on inbound/autoland per month = 73,600h. We pay about $0.02/hr, so that's about an extra $1500 per month, which is fine.
Attachment #8865877 - Flags: review?(catlee) → review+
Pushed by kgupta@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/f4aa943a5bc9
Update linux64-qr test jobs on central and integration branches. r=dustin,catlee
https://hg.mozilla.org/mozilla-central/rev/f4aa943a5bc9
Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla55
You need to log in before you can comment on or make changes to this bug.