Closed Bug 1594860 Opened 1 year ago Closed 9 months ago

Move Compositor into Android UI thread

Categories

(GeckoView :: General, enhancement, P2)

Unspecified
All
enhancement

Tracking

(Not tracked)

RESOLVED WONTFIX

People

(Reporter: snorp, Unassigned)

References

Details

Attachments

(1 file)

Currently, GeckoView uses a SurfaceView for drawing to the screen. This essentially creates a new window in the Android OS compositor (surfaceflinger) and stacks it on top of the rest of the app UI which is drawn in a separate window. We can position and resize the SurfaceView, but it doesn't happen in lockstep with the rest of the UI, which can cause undesirable gaps or overlaps. Additionally, transformations like scale or rotation don't work on the SurfaceView. It's also difficult to stack SurfaceViews, as there is no defined zorder. You can tell one SurfaceView that it should be on top, but that's it.

The Android WebView manages to avoid all of this by embedding the Chromium compositor directly inside the View.onDraw() method of the WebView. This allows them to draw in lockstep with the rest of the app, allowing it to behave like any other "normal" View. AFAICT, the way this is accomplished is by using the EGLContext that is current at the time View.onDraw() is invoked. I think reading the WebView code that does all of this will be very educational[1].

[1] https://cs.chromium.org/chromium/src/android_webview/browser/aw_contents.cc?type=cs&q=onDraw+file:%5Esrc/android_webview/+package:%5Echromium$&g=0&l=1002

Randall, is this something that could help FxR too? You wouldn't use View.onDraw(), but maybe you can integrate with the OS compositor in a similar way?

Flags: needinfo?(rbarker)

Jessie I have this in the GV product, but we're probably not going to do most of the work here. If you'd rather track it in GFX, that's fine.

Flags: needinfo?(jbonisteel)

(In reply to James Willcox (:snorp) (jwillcox@mozilla.com) (he/him) from comment #1)

Randall, is this something that could help FxR too? You wouldn't use View.onDraw(), but maybe you can integrate with the OS compositor in a similar way?

It's possible, but the compositor/renderer would have to run at device frame rate (72-75Hz currently). I know the hope is to one day use webrender and apply the distortion at that level. This would improve text rendering and allow us to bypass the distortion render which would remove the last copy in the pipeline. This is might be a good first step towards that goal?

Flags: needinfo?(rbarker)

Would this work be useful if we only didn't when WebRender was enabled?

Flags: needinfo?(snorp)

(In reply to Jeff Muizelaar [:jrmuizel] from comment #4)

Would this work be useful if we only didn't when WebRender was enabled?

I'd rather it work everywhere, because we'll need to change the public API for GeckoView to accomodate this. We'd probably settle for WR-only, though.

Flags: needinfo?(snorp)

James says Graphics team will work on this in Q1.

This would allow the GeckoView class to work like other views and help with screen tearing. The GeckoDisplay API will need to change.

Rank: 11
Priority: -- → P2
Whiteboard: [geckoview:m2001]?

We are taking a look to come up with an estimate to figure out where we might be able to slot this in during Q1

Flags: needinfo?(sotaro.ikeda.g)

I looked into chromium source code around AwContents class and created some diagrams.

AwContents::OnDraw() posts rendering to android hwui RenderThread when canvas is hardware accelerated. For it, Functor class is used. The Functor's function is called on android hwui RenderThread. It actually does rendering. Functor's architecture was changed since android Q. It supports OpenGLES and Vulkan.

But there is a problem for using Functor in gecko. Functor related APIs are not public API.

To render gecko in a similar way to chromium, we need to use several hidden API like android.webkit.
WebViewDeletage and functors.

It seems that there is a way to bypass hidden api restrictions. But I am not sure if it is possible to do it in post android Q.
https://www.xda-developers.com/android-development-bypass-hidden-api-restrictions/ureData/GPUVideoTextureHost is allocated in GPU process.
https://stackoverflow.com/questions/55970137/bypass-androids-hidden-api-restrictions

Flags: needinfo?(sotaro.ikeda.g)

Sotaro, do we know how Chromium gets access to the hidden APIs?

Flags: needinfo?(sotaro.ikeda.g)

(In reply to Jeff Muizelaar [:jrmuizel] from comment #10)

Sotaro, do we know how Chromium gets access to the hidden APIs?

I am going to look into it.

(In reply to Jeff Muizelaar [:jrmuizel] from comment #10)

Sotaro, do we know how Chromium gets access to the hidden APIs?

For java side, chromium has some implementations of com.android.webview.chromium as glue. For native side, chromium has definitions like the following.
+ https://cs.chromium.org/chromium/src/android_webview/public/browser/draw_gl.h
+ https://cs.chromium.org/chromium/src/android_webview/public/browser/draw_sw.h
+ https://cs.chromium.org/chromium/src/android_webview/public/browser/draw_fn.h


Some details

Flags: needinfo?(sotaro.ikeda.g)

The conclusion from the above comments:

It seems that we could use native hidden api as relatively stable API. On Java side, it seems that we could use it similar way as chromium by implementing glue in com.android.webview.chromium.

However, this needs to be confirmed by creating a sample app that uses the Functor API. This would help further determine the scope and risks of this approach.

Snorp, following up on our convo on Slack - GFX is currently in the midst of a significant piece of work for unblocking WebRender on Intel and won’t have capacity to do much more here until later in Q1, at best. One suggestion would be for someone on the GV team to try out the sample app and see what we can learn from that.

Flags: needinfo?(jbonisteel) → needinfo?(snorp)
Whiteboard: [geckoview:m2001]? → [geckoview:m74]

I filed 1607847 for the sample app. I'll try to get to it soonish.

Flags: needinfo?(snorp)
Rank: 15
Priority: P1 → P2
Whiteboard: [geckoview:m74]

I think we can close this for now. I tried to get a prototype going, but it looks like we basically need to paste the entire part of WebView that does this, which is not reasonable.

Status: NEW → RESOLVED
Closed: 9 months ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.