1592026 - Make reftests work in the WR OS compositor configuration on macOS

Status: RESOLVED FIXED

(Core :: Graphics: WebRender, enhancement, P3)

Description

[Markus Stange [:mstange]]

Reftests need to read back the entire window contents into a main memory buffer. When we draw into OS compositor layers, there is no such single buffer to read back from. We'll need to find a solution so that we can enable gfx.webrender.compositor by default without breaking reftests.

I think there are a number of potential solutions:

1. Read back the update rect contents from each surface individually, and do a simple "software compositing" step to assemble the window buffer on the CPU.
2. Make the reftest drawing step trigger an extra WR render which goes down the traditional path and doesn't use OS compositor surfaces.
3. Make it Gecko's responsibility to ask for the OS's compositing of our OS surfaces.
4. Others?

Solution 1 only works as long as the OS compositor is simple enough, i.e. as long as it doesn't support opacity / transforms or other effects. Solutions 1 and 2 would miss bugs that are introduced at the OS compositor level.
Solution 3 might not be possible on all platforms.

It would be good to find out what EdgeHTML and Safari do for their reftests.

Comment 1

[Sotaro Ikeda [:sotaro]]

On windows, reftest seemed to work with os compositor by Bug 1585619.
https://treeherder.mozilla.org/#/jobs?repo=try&selectedJob=279479858&revision=3a5cea30145a9b720fa2b20352d999e97e610651

Comment 2

[Markus Stange [:mstange]]

I'm still investigating options for Mac. It looks like Safari does not test their CoreAnimation path in their reftests. At least web platform tests seem to be going through Safari's main thread rendering path, which does not support 3d transforms. As a result, tests which contain 3d transforms that can't be flattened fail in Safari, such as this one: https://wpt.fyi/results/css/css-transforms/css-transform-3d-transform-style.html?label=experimental&label=master&aligned

https://github.com/WebKit/webkit/blob/a71e69ec66df96246fad2b984ad5fcdef62677b0/Source/WebCore/rendering/RenderLayer.cpp#L1029-L1033

Comment 3

[Markus Stange [:mstange]]

I'm morphing this bug into the macOS version, since bug 1585619 took care of Windows.

Comment 4

[Markus Stange [:mstange]]

I explored two options for macOS. I wanted to have option 3, i.e. rather than doing a "fake" rendering, read back exactly what the native layers put into the window, including any compositing effects from those native layers.

1. First I tried using CGWindowListCreateImage. When called as CGWindowListCreateImage(CGRectNull, kCGWindowListOptionIncludingWindow, windowID, kCGWindowImageBoundsIgnoreFraming | kCGWindowImageBestResolution), it successfully rendered the window contents, even if the window was covered by other windows on the screen. However, there was a big problem: I frequently got old frames. And I couldn't find a way to wait for the window server to render our CATransaction. I also couldn't find any other way of establishing a fence between committing a CATransaction and the CGWindowListCreateImage render. There may still be a way to do this, but I stopped looking at some point. (For example, I didn't investigate what the CAContext fence port is.)
2. Then I explored using CARenderer. CARenderer can synchronously render our CALayer tree into an OpenGL context. This worked! I'm going to attach patches that use it. I ran into a few problems along the way:
   • CARenderer only picks up CALayer changes that have truly been "committed" by the root CATransaction. On the main thread, the root CATransaction is always an implicit transaction that gets committed by the event loop. So we can't render use CARenderer on the main thread, at least not in a synchronous fashion. But that's fine, we want to do this on the Renderer thread anyway.
   • Whether IOSurfaces render upside down or not is affected by the geometryFlipped property on ancestor CALayers. The WebRender CALayers in the Firefox window are rooted at the backing layer of an NSView which has isFlipped == YES. In order to get the same results when rendering layers with CARenderer, the layers need to be rooted at a CALayer which has geometryFlipped set to YES.
   • CALayers can be either on screen (i.e. somehow attached to an NSView's layer) or they can be attached to a CARenderer. Not both at the same time! That's because the CARenderer has its own CAContext, and a CALayer can only be associated with one CAContext at any given time. This requirement means that our NativeLayerRoot now needs to maintain two CALayer trees: an on-screen tree and an off-screen tree. The on-screen tree is attached to the NSView, and the off-screen tree is attached to the CARenderer.

Comment 5

[Markus Stange [:mstange]]

Attachment: Bug 1592026 - Move AsyncCATransaction suspension into NativeLayerRootCA.

Comment 6

[Markus Stange [:mstange]]

Attachment: Bug 1592026 - Move NativeLayerRoot::Commit call from PostRender into the compositors.

This makes it more similar to how SwapBuffers was used.
This patch also makes us call glFlush directly when using native layers, rather than going through the misleadingly-named GLContext::SwapBuffers method.

Depends on D56668

Comment 7

[Markus Stange [:mstange]]

Attachment: Bug 1592026 - Make NativeLayerCA::Impl

Depends on D56669

Comment 8

[Markus Stange [:mstange]]

Attachment: Bug 1592026 - Add mOffScreenLayer

Depends on D56670

Comment 9

[Markus Stange [:mstange]]

Attachment: Bug 1592026 - Add NativeLayerRoot::ReadbackPixels and implement it with CARenderer.

Depends on D56671

Comment 10

[Markus Stange [:mstange]]

Attachment: patch that uses CGWindowListCreateImage, for future reference

(This is the approach that returned old frames. It's no good. I'm just attaching it for future reference.)

Comment 11

[Markus Stange [:mstange]]

Attachment: Bug 1592026 - Reorder some methods and add a getter for the backing scale. r=jrmuizel

Depends on D54863

Comment 12

[Markus Stange [:mstange]]

Attachment: Bug 1592026 - Move AsyncCATransaction suspension into NativeLayerRootCA. r=jrmuizel

Depends on D57060

Comment 13

[Markus Stange [:mstange]]

Attachment: Bug 1592026 - Move NativeLayerRoot::CommitToScreen call from PostRender into the compositors. r=jrmuizel

This makes it more similar to how SwapBuffers was used.
This patch also makes us call glFlush directly when using native layers, rather than going through the misleadingly-named GLContext::SwapBuffers method.

Depends on D57061

Comment 14

[Markus Stange [:mstange]]

Attachment: Bug 1592026 - Create AutoCATransaction. r=jrmuizel

Depends on D57062

Comment 15

[Markus Stange [:mstange]]

Attachment: Bug 1592026 - Clear sublayers on layer root destruction. This addresses a fixme. r=jrmuizel

Depends on D57063

Comment 16

[Markus Stange [:mstange]]

Attachment: Bug 1592026 - Replace separate mReadySurface field with a bool mMutatedFrontSurface. r=jrmuizel

Depends on D57064

Comment 17

[Markus Stange [:mstange]]

Attachment: Bug 1592026 - Move the responsibility of applying NativeLayerCA changes to native CALayers into a Representation struct. r=jrmuizel

This will allow us to have two representations per NativeLayerCA in the next patch.

Depends on D57065

Comment 18

[Markus Stange [:mstange]]

Attachment: Bug 1592026 - Make NativeLayer(Root)CA build two representations: an onscreen representation and an offscreen representation. r=jrmuizel

The onscreen representation is attached to the NSView.
The offscreen representation is free-floating but will be used in a CARenderer in an upcoming patch.

Each representation is only updated on demand.

Depends on D57066

Comment 19

[Markus Stange [:mstange]]

Attachment: Bug 1592026 - Add a NativeLayerRootSnapshotter API and implement it with CARenderer. r=jrmuizel

Suggestions for a better name than "snapshotter" are welcome.
This is a separate object so that the lifetime of its GLContext isn't governed by the lifetime of the NativeLayerRootCA.
The NativeLayerRootCA gets destroyed on the main thread, but GLContext uses non-threadsafe weak pointer support, so it wants to be destroyed on the same thread that it was created on.
So now the GLContext lives on the snapshotter, which is created and destroyed on the renderer thread.

Depends on D57067

Comment 20

[Markus Stange [:mstange]]

Attachment: Bug 1592026 - When using the OS compositor with WebRender on macOS, use NativeLayerRootSnapshotter for reftest readback. r=jrmuizel

Depends on D57068

Comment 21

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/5433f2ea1f4b
Reorder some methods and add a getter for the backing scale. r=jrmuizel
https://hg.mozilla.org/integration/autoland/rev/f478a2e999e4
Move AsyncCATransaction suspension into NativeLayerRootCA. r=jrmuizel
https://hg.mozilla.org/integration/autoland/rev/b5cf7c3c4771
Move NativeLayerRoot::CommitToScreen call from PostRender into the compositors. r=jrmuizel
https://hg.mozilla.org/integration/autoland/rev/90d4d9ab6493
Create AutoCATransaction. r=jrmuizel
https://hg.mozilla.org/integration/autoland/rev/2effeae528d3
Clear sublayers on layer root destruction. This addresses a fixme. r=jrmuizel
https://hg.mozilla.org/integration/autoland/rev/b041f2395c05
Replace separate mReadySurface field with a bool mMutatedFrontSurface. r=jrmuizel
https://hg.mozilla.org/integration/autoland/rev/8d7fd1220a4d
Move the responsibility of applying NativeLayerCA changes to native CALayers into a Representation struct. r=jrmuizel
https://hg.mozilla.org/integration/autoland/rev/c15a3eb03899
Make NativeLayer(Root)CA build two representations: an onscreen representation and an offscreen representation. r=jrmuizel
https://hg.mozilla.org/integration/autoland/rev/2bb6e9c84513
Add a NativeLayerRootSnapshotter API and implement it with CARenderer. r=jrmuizel
https://hg.mozilla.org/integration/autoland/rev/781f53bf9c78
When using the OS compositor with WebRender on macOS, use NativeLayerRootSnapshotter for reftest readback. r=jrmuizel

Comment 22

[Razvan Maries]

https://hg.mozilla.org/mozilla-central/rev/5433f2ea1f4b
https://hg.mozilla.org/mozilla-central/rev/f478a2e999e4
https://hg.mozilla.org/mozilla-central/rev/b5cf7c3c4771
https://hg.mozilla.org/mozilla-central/rev/90d4d9ab6493
https://hg.mozilla.org/mozilla-central/rev/2effeae528d3
https://hg.mozilla.org/mozilla-central/rev/b041f2395c05
https://hg.mozilla.org/mozilla-central/rev/8d7fd1220a4d
https://hg.mozilla.org/mozilla-central/rev/c15a3eb03899
https://hg.mozilla.org/mozilla-central/rev/2bb6e9c84513
https://hg.mozilla.org/mozilla-central/rev/781f53bf9c78