1563075 - Implement FrameSource compositing abstraction

Status: REOPENED

(Core :: Widget: Gtk, enhancement, P3)

Description

[Kenny Levinsen :kennylevinsen]

Not all platforms present us with vsync as its means of communicating an appropriate time for composition.

Wayland, for example, provides what is known as a "frame callback". A frame-callback is a one-shot notification on a surface that you register when you commit/swap a buffer, which notifies you when you should get to work for this surface next. This frame-callback is not called consistently on vsync, but simply when and if the compositor wishes for you to render. It is, however, guaranteed to be called a time granting us the best chance of reaching the next surface sampling deadline (i.e. vblank).

This flow works like this:

1. Render some content
2. Configure callback on a surface
3. Swap buffers
4. On callback, go to #1

A huge difference here is that there are no "idle" events: There will be no callbacks unless you swap a buffer. This means that if no callback is requested, we must composite immediately, and we cannot postpone for next vsync. If we stop compositing, the only thing that will wake us up is if someone from the outside re-triggers us.

A solution to this is to create a higher-level abstraction, rather than relying on exposing hardware vsync as we try to do now. This also gives us a chance to remove oddities from the interface, such as the exposure of a "Display", and instead let that be implementation defined (for Wayland, for example, there is no "display", just you and an arbitrary rate of callbacks).

The abstraction proposed is a FrameSource, with a simple interface: RequestFrame(), called just before a swap to signal that we would like to be called in the future, PendingFrame() to see if we should composite immediately or wait for a future frame callback, and SetCallback(FrameObserver*), which sets the callback to be called.

For classical platforms, this is easily implemented as a wrapper around a VsyncSource, where Wayland will have a direct implementation. Other platforms where applicable can then slowly be ported.

Non-compositing will currently rely on existing Vsync implementations (and a software timer on Wayland), until a later date.

Comment 1

[Kenny Levinsen :kennylevinsen]

Attachment: Implement FrameSource composition timer

Replaces VsyncSource with the higher level FrameSource abstraction for composition timing.

Comment 2

[Kenny Levinsen :kennylevinsen]

WIP review posted. Feel free to provide preliminary feedback.

Comment 3

[Kenny Levinsen :kennylevinsen]

Hmm, stuck on a dependency issue. Snippet from build log:

2019-07-02T18:38:05.393Z] 18:38:05    ERROR -  /builds/worker/workspace/build/src/widget/gtk/nsWaylandDisplay.cpp:338:5: error: use of undeclared identifier 'wl_proxy_wrapper_destroy'; did you mean 'wl_proxy_destroy'?
[task 2019-07-02T18:38:05.394Z] 18:38:05     INFO -      wl_proxy_wrapper_destroy(mDisplayWrapper);
[task 2019-07-02T18:38:05.394Z] 18:38:05     INFO -      ^~~~~~~~~~~~~~~~~~~~~~~~
[task 2019-07-02T18:38:05.394Z] 18:38:05     INFO -      wl_proxy_destroy
[task 2019-07-02T18:38:05.395Z] 18:38:05     INFO -  /usr/include/wayland-client.h:128:6: note: 'wl_proxy_destroy' declared here

It would appear that the build machines contain a version of the wayland headers too old to have support for proxy wrappers. Proxy wrappers were introduced in wayland core version 1.11 (june 2016), see https://lists.freedesktop.org/archives/wayland-devel/2016-June/029163.html.

Proxy wrappers are needed when sharing wayland objects across threads and queues.

Comment 4

[Kenny Levinsen :kennylevinsen]

Everything is coming along nicely, but a bunch of reftests fail on Linux and MacOS under my random testing. This is likely due to a missing composite.

Comment 5

[Yariv]

(In reply to Kenny Levinsen :kennylevinsen from comment #4)

Everything is coming along nicely, but a bunch of reftests fail on Linux and MacOS under my random testing. This is likely due to a missing composite.

Is it possible to generate a test build? It would help me check something w.r.t smooth scrolling.

Comment 6

[Kenny Levinsen :kennylevinsen]

This one should work: https://queue.taskcluster.net/v1/task/DJmQKI18RvOo434RvFYwYA/runs/0/artifacts/public/build/target.tar.bz2

Still a few reftests that fail due to a missing frame.

Comment 7

[Robert Mader [:rmader]]

Can this be closed as a duplicate of bug 1542808 or do you still plan to work on this?

Comment 8

[Kenny Levinsen :kennylevinsen]

Closing as duplicate. This would still be a more correct approach, but bug 1542808 does the job well enough for now.

Comment 9

[Robert Mader [:rmader]]

Reopening as it could be helpful for future vsync improvements.

Comment 10

[Robert Mader [:rmader]]

Kenny, do you think the presentation time extension (1) would be helpful for the Wayland implementation? If so I'd look into getting it wired up in Mutter.

1: https://gitlab.freedesktop.org/wayland/wayland-protocols/-/blob/master/stable/presentation-time/presentation-time.xml

Comment 11

[Kenny Levinsen :kennylevinsen]

Well, we'd have to know what to use the timing information they provide for. Generally, it's there to allow you to do: next_present = cur_present + (cur_present - past_present), and then e.g. prepare the video frame that should be shown at the next present time to match an audio track.

Examples for ways we could use it:

1. Provide some mechanism to query next estimated presentation time if the frame deadline is met, based on recorded presentation times.
2. Extend the VsyncSource to keep track of presentation events, and set the vsync event time to the last recorded presentation time, but still fire on frame callbacks.
3. Run the VsyncSource off presentation events. The timing related to frame callbacks is compositor and configuration specific. On Weston, it would result in about half the render time available in the default configuration on a 60Hz monitor.

The last two would require the vsync consumers to keep track of the presentation times themselves to perform the calculation (and would replace the current bogus timestamp on Wayland), and would do no good if the vsync event time isn't used for anything.

It might be interesting to see what Chromium does here in their Ozone Wayland implementation. It is closer to the "FrameSource" model, but uses both frame callbacks and presentation feedback.

Comment 12

[Kenny Levinsen :kennylevinsen]

Note: Presentation feedback might not arrive at all if there are no new buffers attached, which can make things a tricky with the VsyncSource model. It would make #3 impossible to implement, and #2 tricky as presentation information would not be available whenever frames are missing.

At least, talk in #wayland on IRC suggests that this would be compositor-defined.

Comment 13

[Markus Stange [:mstange]]

(In reply to Kenny Levinsen :kennylevinsen from comment #0)

This frame-callback is not called consistently on vsync, but simply when and if the compositor wishes for you to render. It is, however, guaranteed to be called a time granting us the best chance of reaching the next surface sampling deadline (i.e. vblank).

I'm curious, does this callback come with some kind of "budget" information, i.e. a CPU time and GPU time budget, that we need to stay under in order to hit the deadline?

Comment 14

[Kenny Levinsen :kennylevinsen]

(In reply to Markus Stange [:mstange] from comment #13)

(In reply to Kenny Levinsen :kennylevinsen from comment #0)

This frame-callback is not called consistently on vsync, but simply when and if the compositor wishes for you to render. It is, however, guaranteed to be called a time granting us the best chance of reaching the next surface sampling deadline (i.e. vblank).

I'm curious, does this callback come with some kind of "budget" information, i.e. a CPU time and GPU time budget, that we need to stay under in order to hit the deadline?

No, when the next deadline will occur is not known ahead of time. For example, on a VRR-enabled compositor like sway, the refresh rate might be 45Hz one moment, but 144Hz the next due to moving the mouse. It's entirely controlled by whether and when applications on that screen post buffers, as well as the refresh rate limits of the monitor.

The frame callback itself only carries a timestamp (without defined base, useful for simple animations), while precise presentation timing (with well-defined base) can also be requested for delivery after a frame gets presented. This can be used to estimate the frame time and guess when the next presentation (and thus deadline) is likely to occur (which, for fixed frame-rate setups, is of course very precise).