Closed Bug 1484812 Opened 6 years ago Closed 4 years ago

Do partial compositing on Android using EGL_KHR_swap_buffers_with_damage

Categories

(Core :: Graphics: Layers, enhancement, P3)

Product:
Core
Component:
Graphics: Layers
Type:
enhancement

Tracking

()

Status:
RESOLVED FIXED
Milestone:
mozilla74
Tracking Flags:
Tracking Status
firefox63 --- wontfix
firefox74 --- fixed

People

(Reporter: mstange, Assigned: jnicol)

References

Details

(Whiteboard: gfx-noted)

Attachments

(1 file, 1 obsolete file)

Bug 1484812 - Use SwapBuffersWithDamage on EGL platforms (Wayland/Android)
47 bytes, text/x-phabricator-request
Details | Review 
EGL has an extension called EGL_KHR_swap_buffers_with_damage which exposes a function called eglSwapBuffersWithDamageKHR, which can be used to specify an invalid area since the last swap. This extension is supported on the Moto G5 reference phone, and it's exactly what we need for partial compositing.

https://www.khronos.org/registry/EGL/extensions/KHR/EGL_KHR_swap_buffers_with_damage.txt
Jamie, is that your area?
Assignee: nobody → jnicol
Priority: -- → P3
Whiteboard: gfx-noted

EGL_KHR_swap_buffers_with_damage (or EGL_EXT_swap_buffers_with_damage)
is an EGL extension that allows the application to inform the display
server (system compositor) which areas of the window have changed.

EGL_EXT_buffer_age is also used to avoid sending damage when
the compositor cannot accept it.

Tested on Wayland, should also help Android.

See Also: → 1575765

Is this supposed to already have a visible effect :myfreeweb? I just tested (rebased to current tip) on Webrender and OpenGL, but unfortunately FF still damages the whole surface on Wayland. Or is that expected and the backends simply do not yet supply proper damage regions?

Tested on Mutter (git master) with PAINT_DAMAGE_REGION set, which properly shows surface-/buffer-damage for other Wayland apps.

(In reply to robert.mader from comment #4)

Is this supposed to already have a visible effect :myfreeweb? I just tested (rebased to current tip) on Webrender and OpenGL, but unfortunately FF still damages the whole surface on Wayland. Or is that expected and the backends simply do not yet supply proper damage regions?

Tested on Mutter (git master) with PAINT_DAMAGE_REGION set, which properly shows surface-/buffer-damage for other Wayland apps.

Without Webrender (currently not using it due to bug 1578598), on wlroots (Wayfire) the damage debugging shows only a relatively small rectangle around an animation that's running. A good test is https://tobiasahlin.com/spinkit/ Don't expect it to have an effect when, like, scrolling :)

Well this is odd, I see a full repaint whenever anything on screen changes...will need to verify that Mutter doesn't do a full repaint on EGL_EXT_swap_buffers_with_damage then. All other tests I ran still use SHM buffers (which benefit even more from partial damage as they make copies from CPU to GPU).

Ouch, partial redraw with EGL is broken on Mutter, see https://gitlab.gnome.org/GNOME/mutter/issues/947#note_656221

Alright, fixed in Mutter 3.34.2, due in a couple of days [1]. Was a regression from a big rewrite in 3.34.

So I just tested this and can confirm it works well for OpenGL. My favourite online music player now only updates a few pixels instead of the whole window and therefore screen. Also, during scrolling only the page content is invalidated, not browser parts like the url bar. Overall it behaves like with 'basic' composition.
Unfortunately this doesn't apply to Webrender. Is partial damage still hidden behind some preference? Or will it maybe only work with the compositor integration stuff?

1: https://gitlab.gnome.org/GNOME/mutter/merge_requests/948

Seems like WebRender needs an extra pref indeed (gfx.webrender.max-partial-present-rects), and I don't know if does partial on non-Windows. bug 1480172, bug 1595014.

Thanks for testing! :)

Jamie: Can you take a look at this? Looks like this needs a review.

Flags: needinfo?(jnicol)

Greg, did you have a look into bug 1582624 ? Do you think that could be wired up here?

(In reply to robert.mader from comment #11)

Greg, did you have a look into bug 1582624 ? Do you think that could be wired up here?

This is just the API inside WR. The more relevant code would be the Windows impl: https://hg.mozilla.org/mozilla-central/rev/1e151275792d (bug 1575159).

Of course that can be wired up too, but I'm not using WR right now due to bug 1578598, so I'm not touching anything WR related yet. Let's land this for GL layers compositing first.

Now that the xwayland problem is out of the way (https://phabricator.services.mozilla.com/D57474), I'm trying to connect WR to this. Very glitchy so far :) checking buffer age == 0 (after setting it on frame begin) in RequestFullRender helps a bit (removes fully transparent glitches) but it's still not right..

Sooo, WebRender actually only renders the damage since the last frame (since the current front buffer). Into the back buffer. Which contains the frame before the previous one. SwapBuffersWithDamage does not guarantee that the damage will be the only thing taken from the buffer. Oops.

How do I tell WR to actually render everything (or, ideally, damage since N frames ago) but still tell me the damage since last frame? Returning true from RequestFullRender causes it to consider the whole buffer as damage :(

Huh? Reading through https://www.khronos.org/registry/EGL/extensions/KHR/EGL_KHR_swap_buffers_with_damage.txt, I thought the point of this extension was that webrender would not have to worry about buffers, and that just worrying about "damage since the last frame" would be ok. Specifically, this section:

Terminology

    This extension and the EGL_KHR_partial_update extension both use the word
    "damage" for subtly but significantly different purposes:

    "Surface damage" is what the EGL_KHR_swap_buffers_with_damage extension
    is concerned with. This is the area of the *surface* that changes between
    frames for that surface. It concerns the differences between two buffers -
    the current back buffer and the current front buffer. It is useful only to
    the consumer.

    "Buffer damage" is what the EGL_KHR_partial_update extension is concerned
    with. This is the area of a particular buffer that has changed since that
    same buffer was last used. As it only concerns changes to a single buffer,
    there is no dependency on the next or previous frames or any other buffer.
    It therefore cannot be used to infer anything about changes to the surface,
    which requires linking one frame or buffer to another. Buffer damage is
    therefore only useful to the producer.

    Following are examples of the two different damage types. Note that the
    final surface content is the same in both cases, but the damaged areas
    differ according to the type of damage being discussed.

Surface damage example (EGL_KHR_swap_buffers_with_damage)

    The surface damage for frame n is the difference between frame n and frame
    (n-1), and represents the area that a compositor must recompose.

      Frame 0     Frame 1     Frame 2     Frame 3     Frame 4
    +---------+ +---------+ +---------+ +---------+ +---------+
    |         | |#########| |#########| |#########| |#########|
    |         | |         | |#########| |#########| |#########| Final surface
    |         | |         | |         | |#########| |#########|   content
    |         | |         | |         | |         | |#########|
    +---------+ +---------+ +---------+ +---------+ +---------+

    +---------+ +---------+ +---------+ +---------+ +---------+
    |@@@@@@@@@| |@@@@@@@@@| |         | |         | |         |
    |@@@@@@@@@| |         | |@@@@@@@@@| |         | |         | Surface damage
    |@@@@@@@@@| |         | |         | |@@@@@@@@@| |         |
    |@@@@@@@@@| |         | |         | |         | |@@@@@@@@@|
    +---------+ +---------+ +---------+ +---------+ +---------+

Buffer damage example (EGL_KHR_partial_update)

    The buffer damage for a frame is the area changed since that same buffer was
    last used. If the buffer has not been used before, the buffer damage is the
    entire area of the buffer.

    The buffer marked with an 'X' in the top left corner is the buffer that is
    being used for that frame. This is the buffer to which the buffer age and
    the buffer damage relate.

    Note that this example shows a double buffered surface - the actual number
    of buffers could be different and variable throughout the lifetime of the
    surface. The age *must* therefore be queried for every frame.

      Frame 0     Frame 1     Frame 2     Frame 3     Frame 4
    +---------+ +---------+ +---------+ +---------+ +---------+
    |         | |#########| |#########| |#########| |#########|
    |         | |         | |#########| |#########| |#########| Final surface
    |         | |         | |         | |#########| |#########|   content
    |         | |         | |         | |         | |#########|
    +---------+ +---------+ +---------+ +---------+ +---------+

    X---------+ +---------+ X---------+ +---------+ X---------+
    |         | |         | |#########| |#########| |#########|
    |         | |         | |#########| |#########| |#########| Buffer 1 content
    |         | |         | |         | |         | |#########|
    |         | |         | |         | |         | |#########|
    +---------+ +---------+ +---------+ +---------+ +---------+

                X---------+ +---------+ X---------+ +---------+
                |#########| |#########| |#########| |#########|
                |         | |         | |#########| |#########| Buffer 2 content
                |         | |         | |#########| |#########|
                |         | |         | |         | |         |
                +---------+ +---------+ +---------+ +---------+

         0           0           2           2           2      Buffer age

    +---------+ +---------+ +---------+ +---------+ +---------+
    |@@@@@@@@@| |@@@@@@@@@| |@@@@@@@@@| |         | |         |
    |@@@@@@@@@| |@@@@@@@@@| |@@@@@@@@@| |@@@@@@@@@| |         | Buffer damage
    |@@@@@@@@@| |@@@@@@@@@| |         | |@@@@@@@@@| |@@@@@@@@@|
    |@@@@@@@@@| |@@@@@@@@@| |         | |         | |@@@@@@@@@|
    +---------+ +---------+ +---------+ +---------+ +---------+

Or are you saying that the computation of the damage region itself is all right, but that we need to draw more than the damage region?

(In reply to Markus Stange [:mstange] from comment #16)

Or are you saying that the computation of the damage region itself is all right, but that we need to draw more than the damage region?

Yes, exactly! The whole buffer needs to be up to date.

The damage is only a hint, "the area that a compositor must recompose" — but the compositor often needs to get pixels from undamaged areas as well, there's nothing saying "the compositor MUST NOT recompose other areas". And it would use pixels from the front buffer, because it can't exactly access the back buffer the application is currently drawing into :)

I'm surprised Windows DXGI works differently — I guess that means the Windows desktop compositor keeps a hidden copy of the previous frame for itself…

From the spec:

        To be clear; the entire contents
        of the back buffer will still be swapped to the front so
        applications using this API must still ensure that the entire
        back buffer is consistent. The rectangles are only a hint for
        the system compositor so it can avoid recomposing parts of the
        surface that haven't really changed.

Indeed!

This makes it quite a bit less convenient as an API from the producer's perspective, because we now do have to worry about tracking per-buffer damage and buffer age. Oh well.

(In reply to greg v [:myfreeweb] from comment #17)

I'm surprised Windows DXGI works differently — I guess that means the Windows desktop compositor keeps a hidden copy of the previous frame for itself…

I think DXGI issues a copy of the missing pieces between the buffers when the swap function is called. On macOS we do the same in our custom swap chain implementation: https://searchfox.org/mozilla-central/rev/331f0c3b25089c9a16be65f4dc8c601aeaac8cc4/gfx/layers/NativeLayerCA.mm#487-490

(In reply to Markus Stange [:mstange] from comment #19)

I think DXGI issues a copy of the missing pieces between the buffers when the swap function is called. On macOS we do the same in our custom swap chain implementation: https://searchfox.org/mozilla-central/rev/331f0c3b25089c9a16be65f4dc8c601aeaac8cc4/gfx/layers/NativeLayerCA.mm#487-490

https://www.khronos.org/registry/EGL/extensions/KHR/EGL_KHR_partial_update.txt
a competing extension to buffer_age supposedly allows the same kind of thing.
Unfortunately, Mesa only supports buffer_age for now, while Android drivers only support partial_update.
I think I saw a merge request implementing partial_update in Mesa, if that's working I could try implementing support for both.
Thanks for the link, that was probably the missing piece for buffer_age.

Oh wait, that Mac code has explicit front and back surfaces, while in EGL that's all managed implicitly.

How do I tell WebRender to just render the non-damaged region? (That should be fast anyway because of caching, right?)
I want to basically just have the option to RequestFullRender but get the damage info. (That's what I asked originally actually…)

(In reply to greg v [:myfreeweb] from comment #21)

Oh wait, that Mac code has explicit front and back surfaces, while in EGL that's all managed implicitly.

Exactly, the Mac code cannot really be used as a reference here. It's also not used for "the window", it's used for individual native layers.

How do I tell WebRender to just render the non-damaged region?

I don't think WebRender currently has a way to do this. You'd need to add it.
But rendering the non-damaged region isn't what you want, either, is it? You want WebRender to render "the union of damaged regions since the last time this buffer was used", so you need to keep a list of regions around and pick the right one based on the current frame's buffer age.
Another complication is the fact that WebRender prepares its GL render commands on the RenderBackend thread and then executes them on the Renderer thread. But the RenderBackend thread needs to know the buffer age in order to compute the correct render region. But the buffer age can only be known on the Renderer thread.
I think all of this will be easier to implement for non-WebRender first.

(That should be fast anyway because of caching, right?)

That's right.

I think we should attempt to implement this for CompositorOGL first, and then file a new bug for the WebRender part and ask for the necessary WebRender functionality.

(In reply to Markus Stange [:mstange] from comment #22)

How do I tell WebRender to just render the non-damaged region?

I don't think WebRender currently has a way to do this. You'd need to add it.
But rendering the non-damaged region isn't what you want, either, is it? You want WebRender to render "the union of damaged regions since the last time this buffer was used", so you need to keep a list of regions around and pick the right one based on the current frame's buffer age.
Another complication is the fact that WebRender prepares its GL render commands on the RenderBackend thread and then executes them on the Renderer thread. But the RenderBackend thread needs to know the buffer age in order to compute the correct render region. But the buffer age can only be known on the Renderer thread.
I think all of this will be easier to implement for non-WebRender first.

Ok, thinking about this again, all the complications I listed are only relevant if you want to avoid re-rendering the entire buffer. But I realize now that that's not what you were intending to do. You want to render the entire buffer, but allow the window manager to only "present" the damaged region. This gives you around half of the potential win. That's not optimal, but it's definitely worth doing.

(In reply to greg v [:myfreeweb] from comment #21)

How do I tell WebRender to just render the non-damaged region?

Alternatively, maybe WebRender can keep rendering only the damaged region, and then the GLContext could copy valid content from the previous buffer using glBlitFramebuffer. Is this possible by using glReadBuffer(GL_FRONT)?

Attachment #9106013 - Attachment description: Bug 1484812 - Partial compositing (damage) on EGL platforms → Bug 1484812 - Partial compositing (damage) on EGL platforms (Wayland/Android)

could copy valid content from the previous buffer using glBlitFramebuffer. Is this possible by using glReadBuffer(GL_FRONT)?

That would've made the "full win" easier, but unfortunately glReadBuffer doesn't support GL_FRONT in GLES3, only in desktop GL.

There's also the EGL_BUFFER_PRESERVED EGL_SWAP_BEHAVIOR which would've made the "half win" trivial with how WR works now, if I understand it correctly… guess what, it's so disliked that it's not supported by Mesa at all.

implement this for CompositorOGL

Cleaned up the attached patch for that, now it doesn't include buffer_age.

[WIP: WR patch included here for simplicity, will create github PR soon]

EGL requires the application to keep the front buffer fully consistent.
This means we have to draw the previous frame's damage as well.
(But we don't need to include it in the hint we're sending to the
system compositor via SwapBuffersWithDamage.)

I have WebRender working now, it wasn't that hard to make it render the previous frame's damage. I'm still seeing artifacts when switching tabs / when the current tab is not loaded..

(In reply to greg v [:myfreeweb] from comment #26)

Created attachment 9123043 [details]
[WIP: WR patch included here for simplicity, will create github PR soon]

It's not necessary to create a github PR; mozilla-central is the canonical source for WebRender now. (If you land changes to the WR directories, a bot will sync your changes into the github repository automatically.)

(In reply to greg v [:myfreeweb] from comment #27)

I have WebRender working now, it wasn't that hard to make it render the previous frame's damage. I'm still seeing artifacts when switching tabs / when the current tab is not loaded..

It's definitely possible that there are still bugs in WebRender's partial damage implementation. This code path is not used on any platform at the moment.

(In reply to Markus Stange [:mstange] from comment #28)

It's not necessary to create a github PR; mozilla-central is the canonical source for WebRender now. (If you land changes to the WR directories, a bot will sync your changes into the github repository automatically.)

Yeah, I saw that the commits on github were pulled from a bot, I removed that part from the Phabricator description right after posting it :D

It's definitely possible that there are still bugs in WebRender's partial damage implementation. This code path is not used on any platform at the moment.

It's probably somewhere closer to gecko, in bindings… like, I'm seeing partial renders when WebRender "hasn't responded yet". For example I'm seeing a part of the address bar suggestions left over after pressing enter and until WR starts rendering the new page.

Attachment #9106013 - Attachment description: Bug 1484812 - Partial compositing (damage) on EGL platforms (Wayland/Android) → Bug 1484812 - Use SwapBuffersWithDamage on EGL platforms (Wayland/Android)
Pushed by btara@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/c7072bfa62d6
Use SwapBuffersWithDamage on EGL platforms (Wayland/Android) r=jnicol,mstange,jgilbert

https://hg.mozilla.org/mozilla-central/rev/c7072bfa62d6

Status: NEW → RESOLVED
Closed: 4 years ago
status-firefox74: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla74

Comment on attachment 9123043 [details]
Bug 1484812 - Partial compositing (damage) with EGL_EXT_buffer_age in WebRender

Revision D61062 was moved to bug 1620076. Setting attachment 9123043 [details] to obsolete.

Attachment #9123043 - Attachment is obsolete: true
Blocks: 1620076
Flags: needinfo?(jnicol)
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: