Make webrender reftests deterministic enough to get consistent pass/fail results

RESOLVED FIXED in mozilla54



Graphics: WebRender
10 months ago
8 months ago


(Reporter: kats, Assigned: kats)


(Blocks: 1 bug)

53 Branch
Dependency tree / graph

Firefox Tracking Flags

(firefox53 affected)


(Whiteboard: [gfx-noted])

In bug 1319188 I'm making the reftest harness assume that all tests that are EXPECTED_PASS or EXPECTED_FUZZY can randomly fail when webrender is enabled. That's because we don't seem to be producing deterministic results from webrender. We need to fix that, so that we can start figuring out which tests are actually passing and which are actually failing.
Can you point me to some randomless logs? I wonder if it's just bug 1322796. Thanks!
Flags: needinfo?(bugmail)
It might be. I'll re-do some try pushes and see if I get random results or not. Leaving ni on me for now.
Might also be related to 1323143.
Depends on: 1323143
It looks all the nondeterminism is in the layout/reftests/position-dynamic-changes folder. For example, in the try push at [1] there were a bunch of failures. I took all the opt e10s failures, marked them fails-if(webrender) and ran it again [2]. If the failures were the same every time, this try push should have all green opt-e10s reftests. However, there are some additional UNEXPECTED-FAIL and some UNEXPECTED-PASS results in the position-dynamic-changes reftests.

Also note that in [1] the opt non-e10s, debug e10s, and debug non-e10s have different sets of failing position-dynamic-changes tests.

I'll do another try push with all the position-dynamic-changes ones marked as random-if(webrender) and see if that fares better.

[2] seems to be better. Almost too good to be true, actually...
Flags: needinfo?(bugmail)


10 months ago
Assignee: nobody → bugmail

Comment 6

10 months ago
Pushed by
Mark many position-dynamic-changes reftests as random-if(webrender). r=gfx?
Mark reftests failing on WR builds as fails-if(webrender). r=gfx?
Mark zoomed video tests as failing with non-e10s WR. r=gfx?
Stop assuming all reftests behave randomly on WR builds. r=gfx?
Last Resolved: 10 months ago
Resolution: --- → FIXED

Comment 7

10 months ago
Pushed by
Mark a couple more tests as random-if(webrender) since they fail intermittently. r=gfx?


8 months ago
Target Milestone: --- → mozilla54
You need to log in before you can comment on or make changes to this bug.