679366 - Video to WebGL texture conversion is inefficient

Status: RESOLVED DUPLICATE of bug 814524

(Core :: Graphics: CanvasWebGL, defect)

Description

[Benoit Jacob [:bjacob] (mostly away)]

Let's summarize what currently happens when you do webgl.texImage2d(videoElement, format):

1. the video frame has been decoded to YCbCr data
2. assuming GL layers, it's been uploaded to a GL texture, and used in a fragment shader that converts to RGB format, to get a RGB texture
3. the RGB texture is read back to a gfxImageSurface in main memory (nsLayoutUtils::SurfaceFromElement, called from WebGLContext::DOMElementToImageSurface)
4. the gfxImageSurface is run through our WebGL texel format conversion code, to match the format passed to texImage2D. This consists in two first steps:
  4a. Conversion from video RGB format to RGBA8
  4b. Conversion from RGBA8 to the format passed to texImage2D

In order to be fast, we need most importantly to avoid the readback (step 3).

Secondarily, it would be nice to reduce the number of format conversions, which is currently 3 (Steps 2, 4a, 4b).

Finally, since we are able to convert from YCbCr to RGB in a shader, I wonder why we can't do all texture conversions this way. Note that the integer arithmetic is taken care of by the GPU. When you paint a RGB8 texture in a shader on a framebuffer that's backed by a RGB565 texture, the integer arithmetic is implicitly performed.

Comment 1

[Jeff Muizelaar [:jrmuizel]]

(In reply to Benoit Jacob [:bjacob] from comment #0)
> Let's summarize what currently happens when you do
> webgl.texImage2d(videoElement, format):
> 
> 1. the video frame has been decoded to YCbCr data
> 2. assuming GL layers, it's been uploaded to a GL texture, and used in a
> fragment shader that converts to RGB format, to get a RGB texture
> 3. the RGB texture is read back to a gfxImageSurface in main memory
> (nsLayoutUtils::SurfaceFromElement, called from
> WebGLContext::DOMElementToImageSurface)
> 4. the gfxImageSurface is run through our WebGL texel format conversion
> code, to match the format passed to texImage2D. This consists in two first
> steps:
>   4a. Conversion from video RGB format to RGBA8
>   4b. Conversion from RGBA8 to the format passed to texImage2D

I'm not sure these steps are true.

Comment 2

[Benoit Jacob [:bjacob] (mostly away)]

First step here: figure if the above ^ is true.

Comment 3

[Kelsey Gilbert [:jgilbert]]

Step 3 is incorrect. We simply get a copy of the current image surface, and convert it to RGB(A) on the CPU.

I do like the idea of doing texture conversions on the GPU, though.
We should almost certainly do the trivial case of conversions between gl-supported formats. The other cases we should investigate, too, though.