1594860 - Move Compositor into Android UI thread

Status: RESOLVED WONTFIX

(GeckoView :: General, enhancement, P2)

Description

[James Willcox (:snorp) ([e-mail]) (he/him)]

Currently, GeckoView uses a SurfaceView for drawing to the screen. This essentially creates a new window in the Android OS compositor (surfaceflinger) and stacks it on top of the rest of the app UI which is drawn in a separate window. We can position and resize the SurfaceView, but it doesn't happen in lockstep with the rest of the UI, which can cause undesirable gaps or overlaps. Additionally, transformations like scale or rotation don't work on the SurfaceView. It's also difficult to stack SurfaceViews, as there is no defined zorder. You can tell one SurfaceView that it should be on top, but that's it.

The Android WebView manages to avoid all of this by embedding the Chromium compositor directly inside the View.onDraw() method of the WebView. This allows them to draw in lockstep with the rest of the app, allowing it to behave like any other "normal" View. AFAICT, the way this is accomplished is by using the EGLContext that is current at the time View.onDraw() is invoked. I think reading the WebView code that does all of this will be very educational[1].

[1] https://cs.chromium.org/chromium/src/android_webview/browser/aw_contents.cc?type=cs&q=onDraw+file:%5Esrc/android_webview/+package:%5Echromium$&g=0&l=1002

Comment 1

[James Willcox (:snorp) ([e-mail]) (he/him)]

Randall, is this something that could help FxR too? You wouldn't use View.onDraw(), but maybe you can integrate with the OS compositor in a similar way?

Comment 2

[James Willcox (:snorp) ([e-mail]) (he/him)]

Jessie I have this in the GV product, but we're probably not going to do most of the work here. If you'd rather track it in GFX, that's fine.

Comment 3

[Randall Barker [:rbarker]]

(In reply to James Willcox (:snorp) ([e-mail]) (he/him) from comment #1)

Randall, is this something that could help FxR too? You wouldn't use View.onDraw(), but maybe you can integrate with the OS compositor in a similar way?

It's possible, but the compositor/renderer would have to run at device frame rate (72-75Hz currently). I know the hope is to one day use webrender and apply the distortion at that level. This would improve text rendering and allow us to bypass the distortion render which would remove the last copy in the pipeline. This is might be a good first step towards that goal?

Comment 4

[Jeff Muizelaar [:jrmuizel]]

Would this work be useful if we only didn't when WebRender was enabled?

Comment 5

[James Willcox (:snorp) ([e-mail]) (he/him)]

(In reply to Jeff Muizelaar [:jrmuizel] from comment #4)

Would this work be useful if we only didn't when WebRender was enabled?

I'd rather it work everywhere, because we'll need to change the public API for GeckoView to accomodate this. We'd probably settle for WR-only, though.

Comment 6

[Chris Peterson [:cpeterson]]

James says Graphics team will work on this in Q1.

This would allow the GeckoView class to work like other views and help with screen tearing. The GeckoDisplay API will need to change.

Comment 7

[Jessie [:jbonisteel] pls NI]

We are taking a look to come up with an estimate to figure out where we might be able to slot this in during Q1

Comment 8

[Sotaro Ikeda [:sotaro]]

I looked into chromium source code around AwContents class and created some diagrams.

• https://github.com/sotaroikeda/chromium-diagrams/wiki/Chromium-diagrams

AwContents::OnDraw() posts rendering to android hwui RenderThread when canvas is hardware accelerated. For it, Functor class is used. The Functor's function is called on android hwui RenderThread. It actually does rendering. Functor's architecture was changed since android Q. It supports OpenGLES and Vulkan.

But there is a problem for using Functor in gecko. Functor related APIs are not public API.

Comment 9

[Sotaro Ikeda [:sotaro]]

Attachment: overview diagram if gecko renders in a similar way to chromium

To render gecko in a similar way to chromium, we need to use several hidden API like android.webkit.
WebViewDeletage and functors.

It seems that there is a way to bypass hidden api restrictions. But I am not sure if it is possible to do it in post android Q.
https://www.xda-developers.com/android-development-bypass-hidden-api-restrictions/ureData/GPUVideoTextureHost is allocated in GPU process.
https://stackoverflow.com/questions/55970137/bypass-androids-hidden-api-restrictions

Comment 10

[Jeff Muizelaar [:jrmuizel]]

Sotaro, do we know how Chromium gets access to the hidden APIs?

Comment 11

[Sotaro Ikeda [:sotaro]]

(In reply to Jeff Muizelaar [:jrmuizel] from comment #10)

Sotaro, do we know how Chromium gets access to the hidden APIs?

I am going to look into it.

Comment 12

[Sotaro Ikeda [:sotaro]]

(In reply to Jeff Muizelaar [:jrmuizel] from comment #10)

Sotaro, do we know how Chromium gets access to the hidden APIs?

For java side, chromium has some implementations of com.android.webview.chromium as glue. For native side, chromium has definitions like the following.
+ https://cs.chromium.org/chromium/src/android_webview/public/browser/draw_gl.h
+ https://cs.chromium.org/chromium/src/android_webview/public/browser/draw_sw.h
+ https://cs.chromium.org/chromium/src/android_webview/public/browser/draw_fn.h

---

Some details

• com.android.webview.chromium
  • chromium source code implements part of com.android.webview.chromium and provide access to com.android.webview.chromium from org.chromium.
  • https://cs.chromium.org/chromium/src/android_webview/glue/java/src/com/android/webview/chromium/
  • Java functor handling also exists here.
• com.android.webview.chromium.GraphicsUtils
  • It is used to get function tables of SW and HW rendering.
  • It is not a public method, then we could not access to the class from another package.
  • Java class is defined in chromium, but native implementation is in android framework.
  • https://android.googlesource.com/platform/frameworks/base/+/master/native/webview/plat_support/graphics_utils.cpp#94
  • In chromium, com.android.webview.chromium.SharedWebViewChromium is defined as a public class, and it provides access to the GraphicsUtils.
  • Native side definitions exist in chromium in the following.
    https://cs.chromium.org/chromium/src/android_webview/public/browser/draw_gl.h
    https://cs.chromium.org/chromium/src/android_webview/public/browser/draw_sw.h
• Get DrawFunctor table is defined in chromium.
  • https://cs.chromium.org/chromium/src/android_webview/glue/java/src/com/android/webview/chromium/DrawFunctor.java
  • Native side of DrawFunctor is defined in android.
  • https://android.googlesource.com/platform/frameworks/base/+/master/native/webview/plat_support/draw_functor.cpp#219
  • Native side of draw functor definitions exist in chromium in the following.
  • https://cs.chromium.org/chromium/src/android_webview/public/browser/draw_fn.h

Comment 13

[Jessie [:jbonisteel] pls NI]

The conclusion from the above comments:

It seems that we could use native hidden api as relatively stable API. On Java side, it seems that we could use it similar way as chromium by implementing glue in com.android.webview.chromium.

However, this needs to be confirmed by creating a sample app that uses the Functor API. This would help further determine the scope and risks of this approach.

Snorp, following up on our convo on Slack - GFX is currently in the midst of a significant piece of work for unblocking WebRender on Intel and won’t have capacity to do much more here until later in Q1, at best. One suggestion would be for someone on the GV team to try out the sample app and see what we can learn from that.

Comment 14

[James Willcox (:snorp) ([e-mail]) (he/him)]

I filed 1607847 for the sample app. I'll try to get to it soonish.

Comment 15

[James Willcox (:snorp) ([e-mail]) (he/him)]

I think we can close this for now. I tried to get a prototype going, but it looks like we basically need to paste the entire part of WebView that does this, which is not reasonable.