1319188 - Add jobs in automation for Linux reftests with QuantumRender enabled

Status: RESOLVED FIXED

(Core :: Graphics: WebRender, defect, P3)

Description

[Kartikaya Gupta (email:[e-mail])]

Now that we have automated builds going, we need automated tests with webrender enabled. Most important are reftests. Mason landed some stuff to make reftests work. I have a try push going with TC jobs to run reftests. Let's see how those go.

https://treeherder.mozilla.org/#/jobs?repo=try&revision=1934b6c19c05b3f0764c3677eb7f3af69f1c46f4

Comment 1

[Kartikaya Gupta (email:[e-mail])]

Bug 1319303 should fix the failures in the try push above. New try push with that cset:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=f8779af0b02a4f21642e408510bc403f0be5c151

Comment 2

[Kartikaya Gupta (email:[e-mail])]

There's still some crashes because it seems like in the QR builds we're sometimes destroying the UI process compositor-side objects before the content process compositor-side objects. So e.g. we destroy the CompositorBridgeParent, leaving dangling pointers for the cross-process entries in sIndirectLayerTrees. When those are accessed to shut them down, they crash. I'm not sure if there's anything in the codebase that actually guarantees that the CPCBPs will get torn down before the corresponding CBP, so I think it might be better to just make LayerTreeState hold RefPtrs to as many things as possible instead of raw pointers.

I also hit a crash in WebRenderBridgeParent where ScheduleComposition is called after mCompositorScheduler is nulled.

Try push with those hopefully fixed:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=5f6384283fc5e6df64eb0b2f9cf2bcb851fc7ea1

Comment 3

[Kartikaya Gupta (email:[e-mail])]

Still one GLX crash remaining, here's an attempt to fix it:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=7614b2dd5db876799ec0c0bc46d0302ae82e3d97

Comment 4

[Kartikaya Gupta (email:[e-mail])]

Ok, crashes are fixed. Now we just have a slew of failures to deal with:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=4a9511fc031f61fccb54ccc695e050c2c0bfbfdb

Comment 5

[Kartikaya Gupta (email:[e-mail])]

There are still some crashes, unfortunately. I may not be able to get this done before I go on PTO, but here's my latest WIP.

https://treeherder.mozilla.org/#/jobs?repo=try&revision=6fdccc04c6bd3b14ac19dddde065d566e26bf797

Comment 6

[Kartikaya Gupta (email:[e-mail])]

Hopefully better: https://treeherder.mozilla.org/#/jobs?repo=try&revision=aa7615d0895cba523b1fae291354fdff08636dfc

Comment 7

[Kartikaya Gupta (email:[e-mail])]

Also for the record the commands I used to update the reftest.list files (i.e. to generate the patch at [1]) can be found at [2]. If you want to use it, read the comments and exercise caution.

[1] https://hg.mozilla.org/try/rev/c4180011c5eccd4466795cdfdf8cf653938ad79c
[2] https://github.com/staktrace/moz-scripts/blob/fad6382447890d30851a313de52d87c1811da104/mark-failing-reftests.sh

Comment 8

[Kartikaya Gupta (email:[e-mail])]

So given the results from before my PTO I suspect we weren't getting deterministic results from the reftests - after I fails-if'd the ones that were failing, a different set started failing and some of the failing ones were passing. So that'll need investigation. However it looks like we're back to having some crashes now:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=692105fb7299b46274b59827d9b87baeb5bf4f96

Comment 9

[Kartikaya Gupta (email:[e-mail])]

The R1 failure at least is at https://hg.mozilla.org/try/file/692105fb7299b46274b59827d9b87baeb5bf4f96/third_party/rust/gleam/src/gl.rs#l96

Comment 10

[Kartikaya Gupta (email:[e-mail])]

Getting closer: https://treeherder.mozilla.org/#/jobs?repo=try&revision=479ddb1d38bd3885fbc3f0899dc8bf902560a8e9

There's some set of tests that seem to spin forever in invalidation loops.

Comment 11

[Kartikaya Gupta (email:[e-mail])]

After some chasing, it looks like the "spin forever" happens because of OMTA not being properly handled in the webrender builds. In particular I chased it down to the condition at [1]. When running the transform-3d/animate-preserve3d-child.html reftest locally in the WebRender build, I see that layer->GetAnimationGeneration is 0 while frameGeneration is 1. In a Gecko build layer->GetAnimationGeneration is 1. This difference causes different branches to be taken in Gecko vs WebRender, and so WebRender ends up continually scheduling flushes and painting.

[1] http://searchfox.org/mozilla-central/rev/3e157ac240611f80df409ae76221d46a31c20dfc/layout/base/RestyleManager.cpp#1544

Comment 12

[Kartikaya Gupta (email:[e-mail])]

Expanding the condition at [1] to include LAYERS_WR fixed it for me locally. Try push at [2].

[1] http://searchfox.org/mozilla-central/rev/3e157ac240611f80df409ae76221d46a31c20dfc/layout/painting/nsDisplayList.cpp#712
[2] https://treeherder.mozilla.org/#/jobs?repo=try&revision=4bb735dfbf97bac7fe8c9267510a4e034f1f17e4

Comment 13

[Kartikaya Gupta (email:[e-mail])]

That seems to have done the job. Note that these patches don't actually test that the reftests all pass - there's a change to the reftest harness so that any == or != reftest that's expected to produce a particular result is allowed to fail. The reason I want to land these anyway is because if we introduce crashes it will cause these tests to start failing, so it's a good regression detector in that sense. I kicked off another try push with more stuff to make sure these patches don't accidentally regress anything else.

https://treeherder.mozilla.org/#/jobs?repo=try&revision=3d53be6f4a640f8a615d58c73a5a675ffaf87fad

Comment 14

[Kartikaya Gupta (email:[e-mail])]

It looks like after all that layout/reftests/bugs/1114526-1.html is still failing sometimes. I can't reproduce it locally, so for now I'm just going to skip-if(webrender) the test, and file a follow-up to figure out what's going on there.

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/projects/graphics/rev/ab736a6ae87e
Assume LAYERS_WR can handle OMTA. r=gfx?
https://hg.mozilla.org/projects/graphics/rev/89bd46bf1d0e
Move the QR build jobs into their own row on TH. r=gfx?
https://hg.mozilla.org/projects/graphics/rev/5dad3acb7790
Add Linux64-QR reftests. r=gfx?
https://hg.mozilla.org/projects/graphics/rev/eca282e174e5
Update reftest sandbox conditions for webrender. r=gfx?
https://hg.mozilla.org/projects/graphics/rev/a28daf8892fa
Add skip-if(webrender) annotations to crashing reftests. r=gfx?
https://hg.mozilla.org/projects/graphics/rev/8750ab4d2152
Expect random non-crash results from QR reftests. r=gfx?
https://hg.mozilla.org/projects/graphics/rev/9563f383566a
Skip a reftest that intermittently fails in QR builds. r=gfx?

Comment 16

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/projects/graphics/rev/85efd4833716
Add Linux64-QR debug reftest jobs. r=gfx?