Open Bug 1365250 (stage-wr-tests) Opened 3 years ago Updated 6 months ago

[meta][stage-wr-tests] Green up the windows QR try run

Categories

(Core :: Graphics: WebRender, enhancement, P3)

Other Branch
enhancement

Tracking

()

Tracking Status
firefox56 --- unaffected
firefox57 --- unaffected

People

(Reporter: kats, Unassigned)

References

(Depends on 4 open bugs, Blocks 1 open bug)

Details

(Keywords: meta, Whiteboard: [gfx-noted])

Bug 1362397 added some jobs on the graphics branch for windows reftests with webrender enabled. These are running now but are failing. We need to get them passing.

Here are a few examples:

https://treeherder.mozilla.org/logviewer.html#?job_id=99352304&repo=graphics&lineNumber=2991
https://treeherder.mozilla.org/logviewer.html#?job_id=99352434&repo=graphics&lineNumber=9462
https://treeherder.mozilla.org/logviewer.html#?job_id=99352435&repo=graphics&lineNumber=2453

At first glance it looks like all the failures are due to fuzz with a max difference of 1, so maybe we should just bake that into the reftest harness.
What's worrisome is that the tests that are failing are different from one run to the next. Not sure why that's happening, but it seems like there's some element of randomness here.
I'm sort of working on this. The reftests seem to pass locally on my Windows machine. I'll have to debug on tryserver.
Assignee: nobody → bugmail
I took a closer look at these failures. It looks like all the failures are in the image/test/reftest suite, so maybe for now I'll just skip that folder. The exact set of failures seems to change from run to run, as does the number of different pixels on a given failure. The number of failures is around ~240 on the few opt runs that I sampled.
(In reply to Kartikaya Gupta (email:kats@mozilla.com) from comment #5)
> It looks like all the failures are
> in the image/test/reftest suite

That's wrong. The test is actually getting aborted because the log file is too big (because of all data URLs from the failing reftests). If I skip the image/test/refest folder there are more failures in other places.
For future reference, I picked a random failure that looked interesting and investigated it. Since I could reproduce it locally consistently, I chased it down and it turned out to be related to bug 748228. There's some discussion on that bug regarding what to do about that particular failure. However the fix for that test doesn't fix a large swath of the text-fuzz failures I was hoping it would. The best course of action here is still to figure out what's causing the nondeterminism in these reftests on Windows and fix that first.
Also, disabling subpixel AA in webrender doesn't fix the text fuzz:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=6816087f4906e683320c3b2d6c29be407cea19a2

Neither disabling AA entirely:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=6da9f03a3f1d76612744d003b43ad58564e21175

(Look at the reftest analyzer view for the debug R-e10s2 job, you don't have to go very far to find differences around text).
Alias: stage-wr-tests
Summary: Green up the windows QR reftests → [meta][stage-wr-tests] Green up the windows QR try run
Summary: [meta][stage-wr-tests] Green up the windows QR try run → [meta] Green up the windows QR try run
Summary: [meta] Green up the windows QR try run → [meta][stage-wr-tests] Green up the windows QR try run
Keywords: meta
I'm not really working on this at the moment. I continue to do windows reftest try pushes as part of the WR update testing so if anybody wants to look at a recent windows reftest run go to whatever the current wr-future-update bug is.
Assignee: bugmail → nobody
Should we be starting discussing getting permanent windows-qr CI set up?

We still have a lot of failures (that seem to mostly fall into a few categories?), but unless we're ready to flip the switch (which if I recall, requires some administrative legwork?) it seems like we're always going to be regressing them.
(In reply to Alexis Beingessner [:Gankro] from comment #12)
> Should we be starting discussing getting permanent windows-qr CI set up?
> 

We probably should, yeah. Except that this bug isn't in the MVP backlog! But yeah if we can figure out the classes of failures here that would be worthwhile I think.
You need to log in before you can comment on or make changes to this bug.