Closed Bug 1322817 Opened 7 years ago Closed 7 years ago

Make webrender reftests deterministic enough to get consistent pass/fail results

Categories

(Core :: Graphics: WebRender, defect, P3)

53 Branch
All
Linux
defect

Tracking

()

RESOLVED FIXED
mozilla54
Tracking Status
firefox53 --- affected

People

(Reporter: kats, Assigned: kats)

References

Details

(Whiteboard: [gfx-noted])

In bug 1319188 I'm making the reftest harness assume that all tests that are EXPECTED_PASS or EXPECTED_FUZZY can randomly fail when webrender is enabled. That's because we don't seem to be producing deterministic results from webrender. We need to fix that, so that we can start figuring out which tests are actually passing and which are actually failing.
Can you point me to some randomless logs? I wonder if it's just bug 1322796. Thanks!
Flags: needinfo?(bugmail)
It might be. I'll re-do some try pushes and see if I get random results or not. Leaving ni on me for now.
Might also be related to 1323143.
Depends on: 1323143
It looks all the nondeterminism is in the layout/reftests/position-dynamic-changes folder. For example, in the try push at [1] there were a bunch of failures. I took all the opt e10s failures, marked them fails-if(webrender) and ran it again [2]. If the failures were the same every time, this try push should have all green opt-e10s reftests. However, there are some additional UNEXPECTED-FAIL and some UNEXPECTED-PASS results in the position-dynamic-changes reftests.

Also note that in [1] the opt non-e10s, debug e10s, and debug non-e10s have different sets of failing position-dynamic-changes tests.

I'll do another try push with all the position-dynamic-changes ones marked as random-if(webrender) and see if that fares better.

[1] https://treeherder.mozilla.org/#/jobs?repo=try&revision=f1797c4f3dd4bc4005cb4d037d19e30ace93e5c2
[2] https://treeherder.mozilla.org/#/jobs?repo=try&revision=5796575fee2b7e87b2bec2f841f4899dbc4d3844
https://treeherder.mozilla.org/#/jobs?repo=try&revision=fb46a049e932fc266a9c98ca1a4bee5f6e592023 seems to be better. Almost too good to be true, actually...
Flags: needinfo?(bugmail)
Assignee: nobody → bugmail
Pushed by kgupta@mozilla.com:
https://hg.mozilla.org/projects/graphics/rev/6c6f13287120
Mark many position-dynamic-changes reftests as random-if(webrender). r=gfx?
https://hg.mozilla.org/projects/graphics/rev/a2040adb5434
Mark reftests failing on WR builds as fails-if(webrender). r=gfx?
https://hg.mozilla.org/projects/graphics/rev/b6a73b6f177e
Mark zoomed video tests as failing with non-e10s WR. r=gfx?
https://hg.mozilla.org/projects/graphics/rev/92f621d51f1f
Stop assuming all reftests behave randomly on WR builds. r=gfx?
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
Pushed by kgupta@mozilla.com:
https://hg.mozilla.org/projects/graphics/rev/186e023dadf6
Mark a couple more tests as random-if(webrender) since they fail intermittently. r=gfx?
Target Milestone: --- → mozilla54
You need to log in before you can comment on or make changes to this bug.