Make webrender reftests deterministic enough to get consistent pass/fail results

RESOLVED FIXED in mozilla54

Status

()

Core
Graphics: WebRender
P3
normal
RESOLVED FIXED
10 months ago
8 months ago

People

(Reporter: kats, Assigned: kats)

Tracking

(Blocks: 1 bug)

53 Branch
mozilla54
All
Linux
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(firefox53 affected)

Details

(Whiteboard: [gfx-noted])

In bug 1319188 I'm making the reftest harness assume that all tests that are EXPECTED_PASS or EXPECTED_FUZZY can randomly fail when webrender is enabled. That's because we don't seem to be producing deterministic results from webrender. We need to fix that, so that we can start figuring out which tests are actually passing and which are actually failing.
Can you point me to some randomless logs? I wonder if it's just bug 1322796. Thanks!
Flags: needinfo?(bugmail)
It might be. I'll re-do some try pushes and see if I get random results or not. Leaving ni on me for now.
Might also be related to 1323143.
Depends on: 1323143
It looks all the nondeterminism is in the layout/reftests/position-dynamic-changes folder. For example, in the try push at [1] there were a bunch of failures. I took all the opt e10s failures, marked them fails-if(webrender) and ran it again [2]. If the failures were the same every time, this try push should have all green opt-e10s reftests. However, there are some additional UNEXPECTED-FAIL and some UNEXPECTED-PASS results in the position-dynamic-changes reftests.

Also note that in [1] the opt non-e10s, debug e10s, and debug non-e10s have different sets of failing position-dynamic-changes tests.

I'll do another try push with all the position-dynamic-changes ones marked as random-if(webrender) and see if that fares better.

[1] https://treeherder.mozilla.org/#/jobs?repo=try&revision=f1797c4f3dd4bc4005cb4d037d19e30ace93e5c2
[2] https://treeherder.mozilla.org/#/jobs?repo=try&revision=5796575fee2b7e87b2bec2f841f4899dbc4d3844
https://treeherder.mozilla.org/#/jobs?repo=try&revision=fb46a049e932fc266a9c98ca1a4bee5f6e592023 seems to be better. Almost too good to be true, actually...
Flags: needinfo?(bugmail)
(Assignee)

Updated

10 months ago
Assignee: nobody → bugmail

Comment 6

10 months ago
Pushed by kgupta@mozilla.com:
https://hg.mozilla.org/projects/graphics/rev/6c6f13287120
Mark many position-dynamic-changes reftests as random-if(webrender). r=gfx?
https://hg.mozilla.org/projects/graphics/rev/a2040adb5434
Mark reftests failing on WR builds as fails-if(webrender). r=gfx?
https://hg.mozilla.org/projects/graphics/rev/b6a73b6f177e
Mark zoomed video tests as failing with non-e10s WR. r=gfx?
https://hg.mozilla.org/projects/graphics/rev/92f621d51f1f
Stop assuming all reftests behave randomly on WR builds. r=gfx?
Status: NEW → RESOLVED
Last Resolved: 10 months ago
Resolution: --- → FIXED

Comment 7

10 months ago
Pushed by kgupta@mozilla.com:
https://hg.mozilla.org/projects/graphics/rev/186e023dadf6
Mark a couple more tests as random-if(webrender) since they fail intermittently. r=gfx?
(Assignee)

Updated

8 months ago
Target Milestone: --- → mozilla54
You need to log in before you can comment on or make changes to this bug.