1365250 - (stage-wr-tests) [meta][stage-wr-tests] Green up the windows QR try run

Status: RESOLVED FIXED

(Core :: Graphics: WebRender, enhancement, P3)

Description

[Kartikaya Gupta (email:[e-mail])]

Bug 1362397 added some jobs on the graphics branch for windows reftests with webrender enabled. These are running now but are failing. We need to get them passing.

Here are a few examples:

https://treeherder.mozilla.org/logviewer.html#?job_id=99352304&repo=graphics&lineNumber=2991
https://treeherder.mozilla.org/logviewer.html#?job_id=99352434&repo=graphics&lineNumber=9462
https://treeherder.mozilla.org/logviewer.html#?job_id=99352435&repo=graphics&lineNumber=2453

At first glance it looks like all the failures are due to fuzz with a max difference of 1, so maybe we should just bake that into the reftest harness.

Comment 1

[Kartikaya Gupta (email:[e-mail])]

What's worrisome is that the tests that are failing are different from one run to the next. Not sure why that's happening, but it seems like there's some element of randomness here.

Comment 2

[Intermittent Failures Robot]

18 failures in 166 pushes (0.108 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* graphics: 18

Platform breakdown:
* windows8-64: 18

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1365250&startday=2017-05-18&endday=2017-05-18&tree=all

Comment 3

[Kartikaya Gupta (email:[e-mail])]

I'm sort of working on this. The reftests seem to pass locally on my Windows machine. I'll have to debug on tryserver.

Comment 4

[Intermittent Failures Robot]

32 failures in 777 pushes (0.041 failures/push) were associated with this bug in the last 7 days. 

This is the #42 most frequent failure this week.  

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. ** 

Repository breakdown:
* graphics: 32

Platform breakdown:
* windows8-64: 32

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1365250&startday=2017-05-15&endday=2017-05-21&tree=all

Comment 5

[Kartikaya Gupta (email:[e-mail])]

I took a closer look at these failures. It looks like all the failures are in the image/test/reftest suite, so maybe for now I'll just skip that folder. The exact set of failures seems to change from run to run, as does the number of different pixels on a given failure. The number of failures is around ~240 on the few opt runs that I sampled.

Comment 6

[Kartikaya Gupta (email:[e-mail])]

(In reply to Kartikaya Gupta (email:[e-mail]) from comment #5)
> It looks like all the failures are
> in the image/test/reftest suite

That's wrong. The test is actually getting aborted because the log file is too big (because of all data URLs from the failing reftests). If I skip the image/test/refest folder there are more failures in other places.

Comment 7

[Intermittent Failures Robot]

11 failures in 891 pushes (0.012 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* graphics: 11

Platform breakdown:
* windows8-64: 11

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1365250&startday=2017-05-22&endday=2017-05-28&tree=all

Comment 8

[Intermittent Failures Robot]

11 failures in 820 pushes (0.013 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* graphics: 11

Platform breakdown:
* windows8-64: 11

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1365250&startday=2017-05-29&endday=2017-06-04&tree=all

Comment 9

[Kartikaya Gupta (email:[e-mail])]

For future reference, I picked a random failure that looked interesting and investigated it. Since I could reproduce it locally consistently, I chased it down and it turned out to be related to bug 748228. There's some discussion on that bug regarding what to do about that particular failure. However the fix for that test doesn't fix a large swath of the text-fuzz failures I was hoping it would. The best course of action here is still to figure out what's causing the nondeterminism in these reftests on Windows and fix that first.

Comment 10

[Kartikaya Gupta (email:[e-mail])]

Also, disabling subpixel AA in webrender doesn't fix the text fuzz:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=6816087f4906e683320c3b2d6c29be407cea19a2

Neither disabling AA entirely:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=6da9f03a3f1d76612744d003b43ad58564e21175

(Look at the reftest analyzer view for the debug R-e10s2 job, you don't have to go very far to find differences around text).

Comment 11

[Kartikaya Gupta (email:[e-mail])]

I'm not really working on this at the moment. I continue to do windows reftest try pushes as part of the WR update testing so if anybody wants to look at a recent windows reftest run go to whatever the current wr-future-update bug is.

Comment 12

[Aria Beingessner [:Gankra]]

Should we be starting discussing getting permanent windows-qr CI set up?

We still have a lot of failures (that seem to mostly fall into a few categories?), but unless we're ready to flip the switch (which if I recall, requires some administrative legwork?) it seems like we're always going to be regressing them.

Comment 13

[Kartikaya Gupta (email:[e-mail])]

(In reply to Alexis Beingessner [:Gankro] from comment #12)
> Should we be starting discussing getting permanent windows-qr CI set up?
> 

We probably should, yeah. Except that this bug isn't in the MVP backlog! But yeah if we can figure out the classes of failures here that would be worthwhile I think.