686611 - Use texture sharing

Status: RESOLVED WONTFIX

(Firefox for Android Graveyard :: General, defect)

Description

[James Willcox (:snorp) ([e-mail]) (he/him)]

Android has some special sauce for sharing GL textures (or really, EGLImage) that it uses internally.  If we're clever, we can use this in order to share the viewport texture between content and chrome processes.

Android has a system called 'gralloc' (graphics alloc) which exposes some of the graphics driver memory management to user space.  The main header for this can be found here: https://github.com/android/platform_hardware_libhardware/blob/master/include/hardware/gralloc.h

Usage is relatively straightforward.  You allocate a buffer of some size, pixelformat, and intended usage.  You can call lock() on the returned handle which gives you exclusive access and a pointer for manipulating the pixel data.  Calling unlock() essentially commits the changes and invalidates the pointer.

There is an Android-specific EGL extension which can be used to create a EGLImage from one of these buffer handles (sort of).  The EGL extension actually requires usage of a higher-level Android structure, found here: https://github.com/android/platform_frameworks_base/blob/master/include/ui/GraphicBuffer.h

The GraphicBuffer class is essentially a wrapper for gralloc.  It also implements a Flattenable interface, useful for serializing to a string for marshaling over IPC.

The EGL extension mentioned above is called 'EGL_ANDROID_image_native_buffer'.  There is virtually no information available about it on the interwebs, I guess because it isn't an official Khronos extension?  However, you can see how Android uses here: https://github.com/android/platform_frameworks_base/blob/master/services/surfaceflinger/TextureManager.cpp#L134

If you look at the 'getNativeBuffer' method in GraphicBuffer, all it does is cast itself to 'android_native_buffer_t*'.  Therefore, you must have a GraphicBuffer instance in order to use the EGL extension.

I propose something like the following:

1) In the chrome process:

    - GraphicBuffer is private API, so we need dlopen /system/lib/libui.so
      (where GraphicBuffer lives), and dlsym any methods necessary to
      create and use GraphicBuffer instances.  Package it nicely in some
      GraphicBufferWrapper class or something.

    - Use the above to allocate the texture for the viewport and pass it to
      the content process via the flatten/unflatten methods.

    - Create GL texture from the EGLImage and splat it on the screen using
      using whatever methods necessary.  I guess this would just use GL layers?

2) In the content process:

    - Receive GraphicBuffer as above
    - Create EGLImage using the nifty extension, and convert to GL texture
    - Use texture as target for FBO
    - Render all content using GL layers into the FBO
    - Win?

There are surely some details that I'm missing here, but I believe that the gist of this should be possible.  The result should, in theory, be very fast.

Comment 1

[James Willcox (:snorp) ([e-mail]) (he/him)]

BTW, github is not the canonical location for the Android source, but kernel.org is still hosed so it's all we have :)

Comment 2

[Benoit Jacob [:bjacob] (mostly away)]

I know we had agreed to go for GL command remoting, but this is pretty big news to me and puts this back into question. If this API is not too unstable then I believe we want to do that over GL command remoting for some reasons:

 1) It could be a lot simpler than doing GL command remoting

 2) It could also run faster. GL command remoting means we have to cache GL commands in a command buffer to mitigate the cost of IPC and means we have to store any buffers, e.g. for texImage2D, somewhere in that command buffer. This is hard to get right performance wise. Chromium is still having trouble with it, see
http://code.google.com/p/chromium/issues/detail?id=92388#c8

 3) It could use less memory. The GL command buffer, with its storage for buffers of e.g. texture image data, will use memory. Chromium's using more than 1 MB already (above comment) and that is still not enough (above comment again).

Comment 3

[Benoit Girard (:BenWa)]

Sounds similar to IOSurface on mac. I will work on a patch next week.

Comment 4

[Dan Mosedale (:dmosedale, :dmose)]

CCing vlad in case the problems he mentioned in bug 613566 should happen to be germaine here.

Comment 5

[James Willcox (:snorp) ([e-mail]) (he/him)]

Ugh, the comments in bug 613566 certainly sound bad.  I have been able to successfully create and lock/unlock a GraphicBuffer (gralloc buffer) already, though, so maybe there is hope?  I can't see why eglCreateImageKHR wouldn't be allowed....

Comment 6

[James Willcox (:snorp) ([e-mail]) (he/him)]

I should add that grallocing with appropriate flags for usage as a texture appears to work fine on my Honeycomb tablet, but it does *not* work on my Gingerbread phone.  Maybe something changed there?

Comment 7

[James Willcox (:snorp) ([e-mail]) (he/him)]

Attachment: Add AndroidGraphicBuffer, WIP

Comment 8

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

(In reply to James Willcox (:snorp) ([e-mail]) from comment #6)
> I should add that grallocing with appropriate flags for usage as a texture
> appears to work fine on my Honeycomb tablet, but it does *not* work on my
> Gingerbread phone.  Maybe something changed there?

Entirely possible.  I was last looking at this in the Eclair/Froyo era, and I spoke with a few people In The Know (tm).  From what I remember, the problems I ran into were -- 1) getting permissions to talk to SurfaceFlinger and associated machinery to be able to do a gralloc correctly; this was apparently impossible on an actual device at the time; 2) having to poke at some entirely internal methods (like, static methods) inside various libraries to be able to turn the gralloc'd thing into something that I could use as a texture -- reading the android source it seemed like this is straightforward, but most of it applies only to the pure software renderer.  Once HW accel is involved, it takes a very different path, most of which was hidden in the vendor-specific EGL drivers.  So even if you can do the gralloc, being able to generate the token through Binder that you can pass to another process is likely still going to be a challenge, and turning the gralloc'd memory into something that GL knows about is going to be another..

I'd have to dig through some emails for more details though, which I might not even have any more...

Comment 9

[Benoit Girard (:BenWa)]

Currently the consensus is that this isn't needed as we're switching towards a single mutli threaded process for mobile. No longer blocking opengl-mobile.

Comment 10

[BMO Automation]

Closing all opened bug in a graveyard component