477456 - Provide API for direct rendering of windowless plugins to mozilla layout context

Status: RESOLVED INCOMPLETE

(Core Graveyard :: Plug-ins, defect)

Description

[Oleg Romashin (:romaxa)]

Steps:
Provide to RenderDocument gfxImageSurface (24bit) context.

Gecko plugins rendering pipeline in this case will be next:
1) Create temporary xlib surface.
2) Push temp_xlib_surface drawable to XEvent and call HandleEvent
3) Plugins rendering data to drawable (X[Shm]PutImage).
4) Gecko getting image from temp_xlib_surface and painting it to main layout context (gfxImageSurface 24bit).

In this case plugins rendering speed will be very slow (on n810 7x slower) comparing with normal windowed mode rendering.

Comment 1

[Oleg Romashin (:romaxa)]

Here is the profile data for use case described above:
1637      0.0831  libxul.so                _moz_pixman_transform_point_3d
13299     0.6754  libc-2.5.so              memcpy
26147     1.3279  libxul.so                _moz_pixman_fill
56787     2.8840  Xomap                    /usr/bin/Xomap
80810     4.1040  libdiamondx.so          
/usr/lib/browser/plugins/libdiamondx.so
84554     4.2941  oprofiled                /usr/bin/oprofiled
120718    6.1307  libX11.so.6.2.0          _XGetPixel16
144354    7.3311  libxul.so                fbCombineOverU
180265    9.1549  libxul.so                fbFetchPixel_x8r8g8b8
252211   12.8087  libxul.so                _get_image_surface
270616   13.7434  no-vmlinux               /no-vmlinux
609195   30.9384  libxul.so                fbFetchTransformed

Comment 2

[Oleg Romashin (:romaxa)]

I think we should provide some API for rendering plguins data directly to gecko layout in next way:
1) Check that main context is gfxImageSurface
2) Ask plugins for direct rendering support (GetValue(nsPluginInstanceVariable_WindowlessLocalBool,...))
3) Get pointer to image surface data with plugin position offset, and put it in exposeEvent.drawable.
4) Provide stride of main context image to plugin.

Comment 3

[Oleg Romashin (:romaxa)]

Attachment: WIP, by using XEvent structure

Probably in this case XEvent structure is not very good implementation for this API.
I think we have to introduce some structure like
NP_ImageDrawable {
char *data;
NPRect exposeRect;
int stride;
};

Comment 4

[Robert O'Callahan (:roc) (email my personal email if necessary)]

If we're going to extend the plugin API, IMHO it would make more sense to add an API to let the plugin give us an RGBA memory surface directly, so we do the compositing. IIRC Adobe preferred that to having them to do the compositing in the plugin.

Comment 5

[Oleg Romashin (:romaxa)]

> an API to let the plugin give us an RGBA memory surface directly, so we do the
> compositing.
But in this case we always have to do at least one composite, also it is not very clear who will take care about destroying of that RGBA memory surface....

If we will provide our drawable surface to plugins, then plugins could draw their content (primitives, images) even without any temporary buffer... then we avoiding at least one memory copy (or composite).

Comment 6

[Robert O'Callahan (:roc) (email my personal email if necessary)]

I think typically plugins already have their own buffer internally.

Comment 7

[Robert O'Callahan (:roc) (email my personal email if necessary)]

Well, I'm not 100% sure. I think this is something you need to work with jst and the plugin vendors on. We can't decide it here.

Comment 8

[Oleg Romashin (:romaxa)]

(In reply to comment #6)
> I think typically plugins already have their own buffer internally.
I think this is not true, because if it is true then plugins should pre-render own buffer before sending Invalidate... but in real life invalidate can be ignored (skip frames, or window is not active, or by some other reason...), and plugins will just waste CPU time in this case. (ex: playing video)

Comment 9

[Oleg Romashin (:romaxa)]

Attachment: WIP 2

Use new API only when we are painting with imageSurface.
Created new structure NPImageExpose, with provide pointer to original surface, expose rect values, scale and translate matrix values.
Now plugins can paint their content directly to main context.

Here is the one problem about how to send properly NPImageExpose structure with HandleEvent API.

Comment 10

[Oleg Romashin (:romaxa)]

Comment on attachment 362710 [details] [diff] [review]
WIP 2

Also plugins now able to provide picture with high quality (Fonts in flash plugin)...
In current implementation whole plugin picture scaled with bilinear quality on gecko side.

Comment 11

[Stuart Parmenter]

another thing that would be useful here would be providing the final output size to the plugin, so even if we're scaling flash could just scale ahead of time to that size.

Comment 12

[Benjamin Smedberg]

Resolving old bugs which are likely not relevant any more, since NPAPI plugins are deprecated.