495727 - Firefox doesn't provide any GPU Overlay/XVideo accelerated fallback for <video/>

Status: RESOLVED DUPLICATE of bug 1210726

(Core :: Audio/Video: Playback, enhancement)

Description

[Ben Schwartz]

User-Agent:       Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.10) Gecko/2009050400 Gentoo Firefox/3.0.10
Build Identifier: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.10) Gecko/2009050400 Gentoo Firefox/3.0.10

Firefox 3.5 series implements the HTML5 <video/> tag entirely in software, running on the CPU.  Almost no other video player behaves this way.  All common video players at least use overlay acceleration (called XVideo in X11).  Virtually every computer sold since 1996 includes a graphics card that provides acceleration of YUV->RGB conversion and output scaling.  These steps are very expensive to perform on the CPU, often using more CPU time than the entire remainder of the decode pipeline.  Overlay acceleration allows them to be performed on the GPU instead.

The reason Firefox does not use overlay acceleration, as noted in bug 462667 comment 2, is that HTML5 permits arbitrary manipulations to be applied to the video after it is decoded.  Overlay acceleration typically supports scaling and clipping but not transparency or arbitrary rotations.

The HTML5 spec says that user agents are permitted to show video "in manners more suitable to the user (e.g. full-screen or in an independent resizable window)".  Such a display, overlay accelerated, would be a valuable addition for users with limited CPU speed or battery power.

Note that I am not referring to the sort of shader-style 3D acceleration available on very modern graphics hardware.  Those systems are flexible enough to implement the entire HTML5 spec, but they are not available on the low-power hardware where acceleration is actually needed.

I also discussed this problem in the thread at http://lists.whatwg.org/pipermail/whatwg-whatwg.org/2009-March/019064.html

Reproducible: Always

Comment 1

[Robert O'Callahan (:roc) (email my personal email if necessary)]

I agree this would be nice, but it's hard to do. We've done a fair bit of work to make sure that YUV conversion is as efficient as possible on the CPU using SIMD instructions (SSE2).

To use overlays effectively we really have to create a separate player that targets XVideo and have a way to switch into it automatically or manually. That's quite a bit of work. I proposed it as a Summer of Code project, but no pickup. Got any volunteers handy?

Comment 2

[Ben Schwartz]

"Hard" is maybe an overstatement. SDL, for example, makes the overlay process close to trivial.  Nonetheless, it's definitely separate from the main rendering pipeline.*  It might even be useful to prototype as an extension.

I'll keep an eye open for volunteers.

*: One can imagine implementing overlay in-place, passing the pixels through the renderer in the overlay color and then initiating overlay at the appropriate position on the page... but I'm not holding my breath for that.

Comment 3

[Robert O'Callahan (:roc) (email my personal email if necessary)]

It should be easy to write an extension that adds a context menu item to open the video URL in an external player.

I don't think XVideo exposes the right APIs to it the way you're suggesting. Even if it did, it would get hairy with overlapping videos and would look bad with our controls bar (and the controls bars created by other sites), which use translucency; we really do want a completely separate player to work around those issues.