Feature Request: HTML5 video GPU acceleration

UNCONFIRMED
Unassigned

Status

()

Core
Audio/Video: Playback
--
enhancement
UNCONFIRMED
8 years ago
13 days ago

People

(Reporter: ryan14, Unassigned)

Tracking

Trunk
x86
Linux
Points:
---

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

8 years ago
User-Agent:       Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.3) Gecko/20100401 SUSE/3.6.3-1.2 Firefox/3.6.3
Build Identifier: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.3) Gecko/20100401 SUSE/3.6.3-1.2 Firefox/3.6.3

There should be a feature that enables HTML5 video to use GPU acceleration so this will take stress off the CPU. It should work on Ubuntu, OpenSuse, Fedora, etc, and it should work with all graphics cards including onboard video.

Reproducible: Always

Comment 1

8 years ago
This may depend on or be a dupe of bug 556027

Comment 2

8 years ago
This is filed before, and even partially implemented (see bug 555839 for instance), but there's still lots of work to do. Especially on the cross-platform behaviour (most work is done in the windows side at the moment).

Comment 3

8 years ago
Also see bug 495727.
Component: General → Video/Audio
Product: Firefox → Core
QA Contact: general → video.audio
Version: unspecified → Trunk
Component: Audio/Video → Audio/Video: Playback

Comment 4

3 years ago
Hello,

I was surprised to see that GStreamer is no longer needed for video playback and that only FFmpeg/Libav is needed now:

https://bugzilla.mozilla.org/show_bug.cgi?id=1207429

Which is very nice.

But I have a question:

There's a bug regarding hardware accelerated video playback with GStreamer, see:

https://bugzilla.mozilla.org/show_bug.cgi?id=894372

So, I was hoping that Firefox would support fully hardware accelerated video playback now that GStreamer is no longer required and went ahead and installed the latest Firefox Nightly 44.0a1 on Ubuntu 15.04 to test it.

Video playback indeed was working without GStreamer, but the video playback still didn't seem to be fully hardware accelerated ;(.

Why is that?

Why is the video playback on Linux still not fully hardware accelerated :(?

Regards

Comment 5

3 years ago
I'm on Fedora 23, ffmpeg and vaapi libs installed. Intel Sandy-Bridge hd3000. HW decoding with libva works in native apps.

Testing FF 45 nightly , in about:support I see : "Supports Hardware H264 Decoding	No;"

Is there a way to test that ffmpeg is in fact used ? And for for HW accel?
Jean-Yves, can you help answer? 
We could also maybe document this on SUMO.
Flags: needinfo?(mschmidt)
Flags: needinfo?(jyavenard)
There is no unique nor "official" software path to have hardware acceleration on linux.
Every chip makers have designed their own framework to do so; none of them compatible with one another.

You may want to track bug 1210729 or bug 1210727.

The only GStreamer plugin allowing hardware acceleration decoding was a VAAPI plugin: it was buggy and extremely unstable nor did it provide any significant speed improvement.

You're much better off using ffmpeg
Flags: needinfo?(jyavenard)

Comment 8

3 years ago
As far as I know, there is 2 main distinct framework for hardware acceleration on linux - VDPAU and VAAPI. Eg. VLC or Mplayer supports this, but not Firefox.

I read some comments, that GStreamer + VAAPI plugin works for someone in Firefox, but personally I could never manage it to work (video plays, but not HW accelerated), move to ffmpeg seems to be a good decision.

As far as I know, VLC also uses ffmpeg internally and is able to use HW acceleration frameworks, so I suppose there should be a way for Firefox to use it also.
Now video playback in Firefox on Linux means quite high CPU usage (for me - fullHD H.264 HTML5 video on recent Broadwell i7 CPU takes ~130% of CPU - more than entire one CPU core).

Comment 9

3 years ago
(In reply to Jean-Yves Avenard [:jya] from comment #7)
> There is no unique nor "official" software path to have hardware
> acceleration on linux.
> Every chip makers have designed their own framework to do so; none of them
> compatible with one another.

That isn't really true, or at best several years out of date. There are effectively only two frameworks, VDPAU (supported by NVidia's proprietary drivers and adopted by the open source NVidia and AMD drivers) and VAAPI (for Intel). Each can use the other as a back-end if necessary, but as far as I know you could be right that it doesn't work well. AMD's proprietary driver supposedly has bindings for something else, but they never provided any support or documentation, so that can be ignored.

> You may want to track bug 1210729 or bug 1210727.
> 
> The only GStreamer plugin allowing hardware acceleration decoding was a
> VAAPI plugin: it was buggy and extremely unstable nor did it provide any
> significant speed improvement.

VAAPI is a little behind VDPAU in terms of maturity and take-up, but both now work well in every other significant Linux video player. I heard the reason it was slow in Firefox was because they made it render into main RAM and copied the data to the framebuffer entirely in software. Use VAAPI properly and the CPU usage barely registers.

> You're much better off using ffmpeg

Only in versions of ffmpeg that support VDPAU and VAAPI. I'd be very happy for Firefox to use that route to gain GPU acceleration.

Comment 10

3 years ago
For what it's worth, embedded systems / SoC vendors often provide support for hardware-accelerated video decoding via OpenMax (omx), in addition to any vendor-specific APIs that may also exist.
(In reply to Tony Houghton from comment #9)
> > You're much better off using ffmpeg
> 
> Only in versions of ffmpeg that support VDPAU and VAAPI. I'd be very happy
> for Firefox to use that route to gain GPU acceleration.

This is irrelevant to what ffmpeg does, nor if it was compiled to support either vaapi or vdpau. Compiling FFmpeg with vdpau or vaapi support won't automagically make applications using ffmpeg start using hardware acceleration unfortunately.

The P in VDPAU stands for Presentation. The only way you can paint an image decoded with VDPAU is to *P*resent it directly using VDPAU. It can't be rendered the way all the other images are typically painted.
VAAPI is similar (There's also the issue that an entire generation of intel GPUs are buggy)

We do not have a VDPAU compositor (nor a OpenGL-VAAPI), so we currently have no other options but copying the decoded image out of the GPU memory into a software surface (which is exactly what the vaapi gstreamer plugin does).

Doing so almost always annihilate any benefits, as copying the image back from the GPU is typically slow (and even more so with nvidia devices)

To make things worse, at this stage, firefox on linux doesn't have hardware accelerated compositor either (OpenGL isn't enabled by default).

We won't be using FFmpeg's hwaccel layer to access vaapi or vdpau (it's only a helper)

(BTW, AMD's own XvBA is perfectly documented)
(In reply to t.i.m from comment #10)
> For what it's worth, embedded systems / SoC vendors often provide support
> for hardware-accelerated video decoding via OpenMax (omx), in addition to
> any vendor-specific APIs that may also exist.

Yes. We are currently developer an OpenMax decoder (bug 1224887). This is for our own FirefoxOS devices, and I'm not sure how easy it will be to enable on other platforms, but the majority of the work will be there.

Comment 13

3 years ago
(In reply to Jean-Yves Avenard [:jya] from comment #11)
> (In reply to Tony Houghton from comment #9)
> VAAPI is similar (There's also the issue that an entire generation of intel GPUs are buggy)

Maybe I haven't noticed this issue, but I can say, that I didn't noticed problems on Ivy Bridge, Haswell and Broadwell GPUs (VLC with VAAPI works just fine).

> We do not have a VDPAU compositor (nor a OpenGL-VAAPI), so we currently have
> no other options but copying the decoded image out of the GPU memory into a
> software surface (which is exactly what the vaapi gstreamer plugin does).
> 
> Doing so almost always annihilate any benefits, as copying the image back
> from the GPU is typically slow (and even more so with nvidia devices)

How is the situation different on Windows? Just trying to get, why is it such a problem.

> To make things worse, at this stage, firefox on linux doesn't have hardware
> accelerated compositor either (OpenGL isn't enabled by default).

OpenGL on linux can be enabled manually, I'm using it for months without apparent problems.
GPU Accelerated Windows: 1/1 OpenGL (OMTC) (in about:support)

If OpenGL compositor on Linux will be working without problems, would be still needed to copy image from GPU to software, or can it be avoided? If I remember right, I read some article about gstreamer+vaapi in Firefox and it stated, that this copying is killing VAAPI performance.

I'm just trying to figure out, what is needed to get VAAPI/VDPAU working...
(In reply to martin from comment #13)

> > Doing so almost always annihilate any benefits, as copying the image back
> > from the GPU is typically slow (and even more so with nvidia devices)
> 
> How is the situation different on Windows? Just trying to get, why is it
> such a problem.

On windows we go through the Windows Media Foundation framework for which the intel driver provides a hardware decoder and plug in automatically to WMF.

So the decoder returns a DXVA surface, which we can render immediately on the screen as we have a DXVA and D3D compositor.

On linux, there's no such common framework.

> OpenGL on linux can be enabled manually, I'm using it for months without
> apparent problems.
> GPU Accelerated Windows: 1/1 OpenGL (OMTC) (in about:support)
> 
> If OpenGL compositor on Linux will be working without problems, would be
> still needed to copy image from GPU to software, or can it be avoided?

VAAPI has an API to map a VAAPI surface to an OpenGL surface, it could then be drawn without having to first copy it to software.
Though, IIRC, Intel was talking about deprecating that API so that you must go through the VAAPI framework to draw the surface. If they did that, we would have to do the same work as we need for VDPAU

>  If I remember right, I read some article about gstreamer+vaapi in Firefox and it
> stated, that this copying is killing VAAPI performance.
this is what I stated earlier on !

> 
> I'm just trying to figure out, what is needed to get VAAPI/VDPAU working...

So for VDPAU, we need a VDPAU compositor. Something that would allow us to draw a VDPAU surface into the screen.
The way VDPAU works typically is that you create an OpenGL surface, assign it a colorkey and then tell VDPAU to draw on that color key. That allows to draw the VDPAU surface into the OpenGL surface and then render that surface on the screen using OpenGL.
It's similar to your blue screen stuff used on TV and movies.

Those are all things that need to be done if we want fully accelerated hardware decoding. Where we use the GPU to decode a video frame, and stay within the GPU to render it on the screen.
Right now, we can't.

Comment 15

3 years ago
(In reply to Jean-Yves Avenard [:jya] from comment #11)
> (In reply to Tony Houghton from comment #9)
> > > You're much better off using ffmpeg
> > 
> > Only in versions of ffmpeg that support VDPAU and VAAPI. I'd be very happy
> > for Firefox to use that route to gain GPU acceleration.
> 
> This is irrelevant to what ffmpeg does, nor if it was compiled to support
> either vaapi or vdpau. Compiling FFmpeg with vdpau or vaapi support won't
> automagically make applications using ffmpeg start using hardware
> acceleration unfortunately.

I know that, but doesn't ffmpeg provide some sort of API wrapper so that if you're already using it for software decoding it's easier to adapt your code for acceleration instead of starting again from scratch? And even if not, its demuxers and audio decoders should still be useful alongside VDPAU/VAAPI.

> The P in VDPAU stands for Presentation. The only way you can paint an image
> decoded with VDPAU is to *P*resent it directly using VDPAU. It can't be
> rendered the way all the other images are typically painted.
> VAAPI is similar (There's also the issue that an entire generation of intel
> GPUs are buggy)

That's just one type of GPU, there are many others that could be supported without that sort of hardware/driver problem.

> We do not have a VDPAU compositor (nor a OpenGL-VAAPI), so we currently have
> no other options but copying the decoded image out of the GPU memory into a
> software surface (which is exactly what the vaapi gstreamer plugin does).
> 
> Doing so almost always annihilate any benefits, as copying the image back
> from the GPU is typically slow (and even more so with nvidia devices)
> 
> To make things worse, at this stage, firefox on linux doesn't have hardware
> accelerated compositor either (OpenGL isn't enabled by default).

But OpenGL can be enabled, and Firefox's support for WebGL on Linux seems quite good. Seeing as you can use an accelerated surface for WebGL canvases, is it really that much harder to set one up for video elements so you can use VDPAU or VAAPI effectively?

I really can't agree that we're better off with software-only ffmpeg and that you shouldn't make every effort with GPU decoding. I've experienced that even some very recent generation desktop CPUs (eg Core i3 @ 3GHz) can't maintain the proper frame rate with some HD video. And 4K videos are already appearing on youtube. (Although we'd presumably have to wait for Firefox's DASH support for these).

> We won't be using FFmpeg's hwaccel layer to access vaapi or vdpau (it's only
> a helper)
> 
> (BTW, AMD's own XvBA is perfectly documented)

Nevertheless, there seems to be something blocking its adoption. But I think there was talk of exposing it via VAAPI if and when it does get supported properly, anyway.

Comment 16

3 years ago
Moreover hardware accelerated decoding seems to fit with project candle (rather than software decoding)
(In reply to Tony Houghton from comment #15)

> I really can't agree that we're better off with software-only ffmpeg and
> that you shouldn't make every effort with GPU decoding. I've experienced

What I said was that you're better of using ffmpeg over gstreamer with the vaapi plugin
Memory bandwidth with 4K quickly becomes the bottleneck. On windows in a current work in progress simply removing a single frame copy sped up the decoding frame rate by 6%; this is huge.

At not time did I state that we didn't want to do full GPU acceleration.

Just that things aren't trivial as what you appear to think. Why do you think chrome dropped it entirely very recently?
If it was that easy and worked well, it would be done already.

Comment 18

3 years ago
(In reply to Jean-Yves Avenard [:jya] from comment #17)
> (In reply to Tony Houghton from comment #15)
> 
> > I really can't agree that we're better off with software-only ffmpeg and
> > that you shouldn't make every effort with GPU decoding. I've experienced
> 
> What I said was that you're better of using ffmpeg over gstreamer with the
> vaapi plugin
> Memory bandwidth with 4K quickly becomes the bottleneck. On windows in a
> current work in progress simply removing a single frame copy sped up the
> decoding frame rate by 6%; this is huge.
> 
> At not time did I state that we didn't want to do full GPU acceleration.

OK. I just got the impression you weren't interested in supporting it. I hope you do manage to get it working.

> Just that things aren't trivial as what you appear to think. Why do you
> think chrome dropped it entirely very recently?

Have they dropped it altogether, even for ChromeOS, now? I know they disabled it for Linux, but didn't remove it before, and Chromium could be recompiled with it enabled with a few changes to some #if statements etc. True, it has been very unreliable, but in the sense that it never works in some versions/systems and falls back to software rather than in the sense that it keeps crashing; when it does work it's stable enough. But the reason they gave for disabling it in Linux was that they had nobody to work on it, not that they couldn't get it working.

> If it was that easy and worked well, it would be done already.

It works well in mpv, mplayer, vlc, kodi etc etc. I realise having to support it together with all the other things a browser has to do makes it harder, but surely not bordering on impossible, and I'm sure at Mozilla you have the programming talent to make it work.
(In reply to Tony Houghton from comment #18)

> It works well in mpv, mplayer, vlc, kodi etc etc. I realise having to

well, they have a single window to worry about. I got it working just fine in mythtv too (as well as vdpau).
They don't have to worry about compositing with any other objects, images or handle someone who has decided to make the video element move around in an html page.

Comment 20

3 years ago
(In reply to Jean-Yves Avenard [:jya] from comment #14)
> (In reply to martin from comment #13)

> 
> VAAPI has an API to map a VAAPI surface to an OpenGL surface, it could then
> be drawn without having to first copy it to software.
> Though, IIRC, Intel was talking about deprecating that API so that you must
> go through the VAAPI framework to draw the surface. If they did that, we
> would have to do the same work as we need for VDPAU

If you are talking about this api:http://cgit.freedesktop.org/vaapi/libva/tree/va/egl/va_egl.h,
Then yes, it is not implemented in driver.

But there are other ways to map deocoded va sufaces to eglimage through dmabuf and vaAcquireBufferHandle.

A detailed implementation with comments here: https://github.com/01org/gstreamer-vaapi/blob/master/gst-libs/gst/vaapi/gstvaapisurface_egl.c

Anther implementation for the same here: https://github.com/01org/libyami/blob/master/egl/egl_vaapi_image.cpp

Comment 21

3 years ago
(In reply to Jean-Yves Avenard [:jya] from comment #14)
> (In reply to martin from comment #13)
> VAAPI has an API to map a VAAPI surface to an OpenGL surface, it could then
> be drawn without having to first copy it to software.
> Though, IIRC, Intel was talking about deprecating that API so that you must
> go through the VAAPI framework to draw the surface. If they did that, we
> would have to do the same work as we need for VDPAU

> So for VDPAU, we need a VDPAU compositor. Something that would allow us to
> draw a VDPAU surface into the screen.
> The way VDPAU works typically is that you create an OpenGL surface, assign
> it a colorkey and then tell VDPAU to draw on that color key. That allows to
> draw the VDPAU surface into the OpenGL surface and then render that surface
> on the screen using OpenGL.
> It's similar to your blue screen stuff used on TV and movies.

Ok, thank you for explaining.
So, if I understood correctly, the blocker now is the OpenGL compositor, which is now disabled by default on linux. I guess there are some reasons for that, some issues? 
When OpenGL compositor will be working fine, then it could be possible to use VAAPI (using that - maybe in future deprecated - API to map VAAPI surface to OpenGL surface) - that would be the (relatively) easy way, right?

In fact I'm not sure, what VDPAU/VAAPI compositor means - does it mean entire new compositor (like the OpenGL one, which probably still has some issues (guessing from it's not enabled by default)), or is it some "overlay" on OpenGL compositor?
The "entirely new" option would be bad, I'm afraid it would take very long time... It's really a problem on linux, I also experienced lags and dropped frames in some HD videos, so I hope it wouldn't take years to support video HW acceleration.

Comment 22

3 years ago
(In reply to Jean-Yves Avenard [:jya] from comment #19)
> (In reply to Tony Houghton from comment #18)
> 
> > It works well in mpv, mplayer, vlc, kodi etc etc. I realise having to
> 
> well, they have a single window to worry about. I got it working just fine
> in mythtv too (as well as vdpau).
> They don't have to worry about compositing with any other objects, images or

Actually they do, most of those other players can overlay GUIs etc on the video.

> handle someone who has decided to make the video element move around in an
> html page.

Ick. But can't they also do that with canvases, so you would already have come up with some solution for WebGL?
(In reply to Tony Houghton from comment #22)
> (In reply to Jean-Yves Avenard [:jya] from comment #19)
> > handle someone who has decided to make the video element move around in an
> > html page.
> 
> Ick. But can't they also do that with canvases, so you would already have
> come up with some solution for WebGL?

Think about scrolling a video on a touch screen. The video needs to scroll coherently with the rest of the image as well as be able to support pinch-zoom. This becomes more difficult if the video is presented through a parallel presentation path as it would be with VDPAU. Ultimately the accelerated decoding on Linux doesn't match with Firefox's compositor model. In order to support accelerated decoding on Linux, someone needs to write some code to bridge that gap. It is not simply a matter of enabling acceleration.

> > They don't have to worry about compositing with any other objects, images or
> 
> Actually they do, most of those other players can overlay GUIs etc on the
> video.

You are ignoring the simple fact that video players are architecturally simpler than browsers. Video players have no other purpose than playing video.

We are definitely interested in getting hardware acceleration to work on Linux. If it is as easy as you seem to think it is and you've got some spare time on your hands then it might be a fun project for you.

Comment 24

3 years ago
(In reply to Anthony Jones (:kentuckyfriedtakahe, :k17e) from comment #23)
> (In reply to Tony Houghton from comment #22)
> > 
> > Ick. But can't they also do that with canvases, so you would already have
> > come up with some solution for WebGL?
> 
> Think about scrolling a video on a touch screen. The video needs to scroll
> coherently with the rest of the image as well as be able to support
> pinch-zoom. This becomes more difficult if the video is presented through a
> parallel presentation path as it would be with VDPAU. Ultimately the
> accelerated decoding on Linux doesn't match with Firefox's compositor model.
> In order to support accelerated decoding on Linux, someone needs to write
> some code to bridge that gap. It is not simply a matter of enabling
> acceleration.

I know, but what I was saying is that you already have a way of dealing with specialised rendering surfaces for WebGL canvases. What's so different about VDPAU and VAAPI surfaces? Or does Firefox's WebGL implementation have the same issue that the gstreamer-vaapi implementation did with inefficient copying from an off-screen GPU buffer to the window surface?

> > > They don't have to worry about compositing with any other objects, images or
> > 
> > Actually they do, most of those other players can overlay GUIs etc on the
> > video.
> 
> You are ignoring the simple fact that video players are architecturally
> simpler than browsers. Video players have no other purpose than playing
> video.

They may not have to deal with scrolling and clipping, but some of them can do a lot more. Kodi has a very sophisticated OSD, supporting non-rectangular transparency and arbitrary downloaded images. It can run full-screen or in a window and either play the video taking up the entire window/screen, or just part of it.

But I take your point, it would have been designed as a video player that has to fit static elements over and around the video, which is back-to-front compare to a browser.

> We are definitely interested in getting hardware acceleration to work on
> Linux.

Thank you for making an effort. I got the wrong impression to start with, because I'm more used to the GNOME bugzilla, where developers pointing out difficulties is usually their making excuses to ignore important issues for years. Ironically though, their browser does seem to support VAAPI at least, but of course it's a long way behind Firefox in other areas.

> If it is as easy as you seem to think it is and you've got some spare
> time on your hands then it might be a fun project for you.

It wouldn't be an efficient use of developer time. I don't mean that I've got better things to do, but that you probably have better people for the job. I don't already know how to use VDPAU and/or VAAPI anyway, but even if I did, I think it would be easier for Firefox developers to learn VDPAU/VAAPI than a VDPAU/VAAPI expert to learn their way around Firefox's code base.

Comment 25

2 years ago
i am new in the bugzilla, hello to all the world. i want to say that with archlinux all plugins installed, firefox nightly for testing too, nothing have accelerated video rendering(and my intel integradted is capable for that) I know that you, all the develops have much work.. but.. it seems that windows have better hardware capabilities, and not only in this things is better, in the html5 support too I think, boys.. please.. in Linux the predetermined browser is firefox while in windows google chrome is the most used, chrome/chromium develops aren't interested in hardware acceleration either for linux..I think is the time for the change..not only for this request, but for all request of linux that windows have implemented now

Comment 26

2 years ago
and don't tell me the drivers isn't good enoug because intel drivers are very well, amd open source too, and nvidia propietary drivers too, create a black list with amd cloused drivers and nvidia open drivers andfixed for the stablity

Comment 27

2 years ago
I encourage everyone here. It is the only remaining flaw in Linux

Comment 28

2 years ago
Aparently there is low to none interest from devs about this issue, it's been almost 3 months without any response.

Meanwhile I tried Chromium, that can support VAAPI (well not oficially, but VAAPI support is there, it has to be patched to work not only in Chrome OS). Since Chromium devs aren't interested in supporting linux, it's not really straightforward to get it to work, but I've made it and HW acceleration is working (CPU usage drops in video playback). I tried it only to be sure it's working elsewhere (and that it could be done in Firefox too).

VAAPI also works well in normal video players like VLC (on Intel Broadwell IGPU), so I don't think buggy drivers is valid reason for not implementing that. There could be configuration option to turn HW accel off, if on some machine it's causing a problem.

It's a shame that Firefox devs don't care much about Linux too. Firefox is imo still most used browser in Linux and video playback with Firefox is really a pain.
It's really frustrating that even with new Intel Broadwell i7, playback of 1080p video causes high CPU usage (thus low life time on battery) and that the video is still not fluent, it still shutters/lags.

I don't know what else to say. I think it's one of most serious issues of Firefox in Linux now.
Are there any developers interested in this, are there any plans about fixing this issue sometime in near future?

Comment 29

2 years ago
Chromium's VAAPI support seems quite flaky. I don't mean it crashes on vidoes, it either works (almost) perfectly or falls back to CPU with no diagnostic messages. It's a pity Mozilla abandoned gstreamer instead of upgrading to 1.0 because this seems to work quite well now, although I haven't tried it with the VDPAU backend on a non-Intel GPU. Totem and even "Web" (GNOME's browser) seamlessly use VAAPI. Sadly Web is lacking in other areas. I don't know whether Konqueror supports acceleration because it doesn't support fullscreen video, so it's useless anyway.

Comment 30

2 years ago
Chromium with VAAPI is not currently working for me, even with the patched version that enables VAAPI (I am trying with chromium from https://launchpad.net/~saiarcot895/+archive/ubuntu/chromium-dev/ on Ubuntu 14.04). I have a NVidia GTX560Ti with NVidia drivers, with VDPAU and VA-API working perfectly.

About gstreamer: now that gstreamer1.0-vaapi 0.7.0 seems to resolve many issues, should it work with Firefox if disabling ffmpeg in about::config?
(can not try this myself, on 14.04 I still have gstreamer1.0-vaapi 0.5.7, will try when I upgrade to 16.04).

Comment 31

2 years ago
It's curios (or sad) to see that playing a youtube video with flash player 11.2 (with EnableLinuxHWVideoDecode=1 and OverrideGPUValidation=1 in /etc/adobe/mms.cfg) works perfectly with hw acceleration.
Need to go back to go further...
Gstreamer's vaapi plugin was one of the biggest crasher!

Additionally, performance improvement were almost nil as for videos where it matters, the time required to copy from the GPU surface took all the benefits away.

Comment 33

2 years ago
I've also been using the saiarcot895 PPA for Chromium, but on newer versions of Ubuntu. My experience of it is that some versions of it seem to work reliably for a while (but maybe need restarting to restore GPU support), some don't.(In reply to Davide Capodaglio from comment #30)
> Chromium with VAAPI is not currently working for me, even with the patched
> version that enables VAAPI (I am trying with chromium from
> https://launchpad.net/~saiarcot895/+archive/ubuntu/chromium-dev/ on Ubuntu
> 14.04). I have a NVidia GTX560Ti with NVidia drivers, with VDPAU and VA-API
> working perfectly.

I've also been using that PPA for Chromium, but on newer versions of Ubuntu. My experience of it is that some versions of it seem to work reliably for a while (but maybe need restarting to restore GPU support), some don't.

> About gstreamer: now that gstreamer1.0-vaapi 0.7.0 seems to resolve many
> issues, should it work with Firefox if disabling ffmpeg in about::config?
> (can not try this myself, on 14.04 I still have gstreamer1.0-vaapi 0.5.7,
> will try when I upgrade to 16.04).

No, I think Firefox was using the old beta API 0.x for gstreamer, so it would need quite a bit of patching to work with 1.x. Plus the software copying issue.

Comment 34

2 years ago
(In reply to Davide Capodaglio from comment #31)
> It's curios (or sad) to see that playing a youtube video with flash player
> 11.2 (with EnableLinuxHWVideoDecode=1 and OverrideGPUValidation=1 in
> /etc/adobe/mms.cfg) works perfectly with hw acceleration.
> Need to go back to go further...

Most sites reject Flash 11 as too old. And most of the time the hw acceleration was pointless because something bottlenecked it so badly that in fullscreen it couldn't even play SD with a double figure FPS on a  decent Core i5.

Comment 35

2 years ago
(In reply to Jean-Yves Avenard [:jya] from comment #32)
> Gstreamer's vaapi plugin was one of the biggest crasher!
> 
> Additionally, performance improvement were almost nil as for videos where it
> matters, the time required to copy from the GPU surface took all the
> benefits away.

I don't think the copying bottleneck was specific to gstreamer, it's a problem that needs solving whether they use VAAPI or VDPAU directly or via any other library, which is probably why they're not using the VAAPI and VDPAU support in ffmpeg either.

Comment 36

2 years ago
Firefox 44 on ubuntu 14.04 is using gstreamer-1.0.
I tried on youtube by adding &nohtml5=1, and with flash I get about 15% cpu usage, while with ffmpeg I get about 100%.

Comment 37

2 years ago
(In reply to Tony Houghton from comment #34)
> Most sites reject Flash 11 as too old. And most of the time the hw
> acceleration was pointless because something bottlenecked it so badly that
> in fullscreen it couldn't even play SD with a double figure FPS on a  decent
> Core i5.

Well, I'm not sure about "most sites", I tried youtube with abobe-flash-11.2.202.569, that works correctly.
I made a comparison of the same full HD youtube video played using adobe-flash and html5 FF player.
adobe-flash uses VDPAU to accelerate video decoding (actually I use libvdpau-va-gl to get VDPAU on Intel iGPU, it works fine). VDPAU video decoding acceleration of adobe-flash can be toogled in /etc/adobe/mms.cfg by EnableLinuxHWVideoDecode option.
I've made sure that the same H264 full HD video is played (I disabled webm/VP9 support, which is normally html5 default).

First I tried firefox HTML5 video player:
- If I don't move mouse over video (buttons are not shown), CPU usage is 81% on average.
- If I move mouse over it and the buttons must be rendered, CPU usage goes to 143% on average.

Then I tried to play it using adobe-flash with VDPAU decoding disabled - it says "accelerated video rendering, software video decoding"
- If I don't move mouse over video (buttons are not shown), CPU usage is 40% on average.
- If I move mouse over it and the buttons must be rendered, CPU usage goes to 65% on average.

Finally I tried to play it using adobe-flash with VDPAU decoding disabled - it says "accelerated video rendering, accelerated video decoding"
- If I don't move mouse over video (buttons are not shown), CPU usage is 12% on average.
- If I move mouse over it and the buttons must be rendered, CPU usage goes to 40% on average.

(I summed up CPU usages of firefox+plugin_container for CPU usage of adobe-flash.)

That's quite a difference. The HTML5 video is not always fluent, especially when the buttons are shown, the video noticeably lags. I didn't noticed any lags while using adobe-flash.
It's quite sad that deprecated adobe-flash is significantly better that "recomended and cool" html5 player.

I don't think that gstreamer could help here, regardless of its version. gstreamer-vaapi suffered from copying issue and performance gain was neglible.
Gstreamer is not a solution, solution is to decode image using VAAPI/VDPAU and render it directly, without copying it back from GPU.
(In reply to Davide Capodaglio from comment #36)
> Firefox 44 on ubuntu 14.04 is using gstreamer-1.0.
> I tried on youtube by adding &nohtml5=1, and with flash I get about 15% cpu
> usage, while with ffmpeg I get about 100%.

flash has proper VDPAU support, that is it will not only decode but also paint using HW acceleration.

We currently don't have OpenGL layers enabled on Linux.

So the only way we can use VAAPI or VDPAU at this stage, is to decode using the GPU and then copy the GPU surface back into RAM. This has a high cost taking away most of the benefits of HW decoding.

VAAPI will be the simplest to support at this stage, and likely the first one we will work on.
(In reply to Davide Capodaglio from comment #36)
> Firefox 44 on ubuntu 14.04 is using gstreamer-1.0.
> I tried on youtube by adding &nohtml5=1, and with flash I get about 15% cpu
> usage, while with ffmpeg I get about 100%.

I should add:
when you use HTML5 on Linux with YouTube, you will get webm with VP9. There are no full VP9 hardware decoding available yet (it's just partially accelerated in Broadwell and Skylake anyway).
So CPU usage when watching YouTube will always be higher.

Flash is being served h264 which can be fully HW accelerated. 

The difference you're seeing in CPU usage on Linux is mostly a matter of h264 vs VP9.

Comment 40

2 years ago
(In reply to Jean-Yves Avenard [:jya] from comment #38)
> We currently don't have OpenGL layers enabled on Linux.
> 
> So the only way we can use VAAPI or VDPAU at this stage, is to decode using
> the GPU and then copy the GPU surface back into RAM. This has a high cost
> taking away most of the benefits of HW decoding.
> 
> VAAPI will be the simplest to support at this stage, and likely the first
> one we will work on.

Ok, I see it's bad to copy the surface back to RAM from GPU and the performance benefits will be small.

But what about the OpenGL compositor, are there any ongoing works on that or what are the reasons that it's not enabled by deafult?
Maybe on older machines it could cause problems, but I believe it should work on newer systems.
I'm using OpenGL compositor in Firefox for half a year (MOZ_USE_OMTC=1, GPU Accelerated Windows: 1/1 OpenGL in about:support).
I didn't noticed any problems with that settings for half a year.

(In reply to Jean-Yves Avenard [:jya] from comment #39)
> when you use HTML5 on Linux with YouTube, you will get webm with VP9.
Yes, in default you will get VP9. But you can force YouTube to use H264 (I managed this for the tests above by setting media.mediasource.webm.enabled to false).
There are also other ways - eg. there is extension (only for Chromium currently, but I'm sure it can be done for Firefox too) h264ify, which does precisely this.

It will be interesting to see how the new hybrid VP9 decoding works.

(In reply to Jean-Yves Avenard [:jya] from comment #39)
> The difference you're seeing in CPU usage on Linux is mostly a matter of h264 vs VP9.
Sorry, but I think that I cannot agree with that, you can see my test with CPU usages 3 posts above. 

I made sure I use H264 version for all tests (you can see stats for HTML5 player and there you can find used codec).
Both tests with HTML5 and adobe-flash used H264 video and the difference in CPU usage was quite big.
When you're playing a video with youtube; right click on the player and select "stats for nerds"

If the video has VP9 encoding (and most are) this is what you will get, as YouTube favour vp9 over h264.

As for OpenGL compositor, it's in the planning, and once this is enabled we should be able to do VAAPI and VDPAU hardware acceleration shortly after.

Comment 42

2 years ago
I think that this bug should be tagged regarding Project Candle ?
https://wiki.mozilla.org/Performance/Project_Candle

Comment 43

2 years ago
I think the same as antistress, this bug should be tagged with project candle because the increase of battery life if we use the hardware decode should be good, and not a little...

Updated

2 years ago
Flags: needinfo?(me+mozilla)

Comment 44

2 years ago
Just remove (uninstall) if your hardware is i915 (or not suported):

gstreamer1.0-vaapi

and optional:

i965-va-driver

Then delete the folder:

rm -r ~/.cache/gstreamer-1.0

Also if codecs are still missing, after reboot, install:

ubuntu-restricted-extras

or if in other distro change from:

oxideqt-codecs

to:

oxideqt-codecs-extra

commands to get more info:

/usr/bin/gst-inspect-1.0
/usr/bin/glxinfo
/usr/bin/vainfo

Comment 45

2 years ago
(In reply to Anthony Jones (:kentuckyfriedtakahe, :k17e) from comment #23)

> We are definitely interested in getting hardware acceleration to work on
> Linux. If it is as easy as you seem to think it is and you've got some spare
> time on your hands then it might be a fun project for you.

If everyone better qualified is too busy on other features (I see it's still assigned to nobody), maybe this isn't such a bad idea after all. I have experimented with writing a video player before and had partial success with ffmpeg several years ago, but that was before VDPA and VA-API existed, and I don't know my way around Firefox's code either. So I might not be able to contribute anything useful.

Comment 46

2 years ago
One thing I have thought of, if one of the big problems is getting a hardware-accelerated surface amid the jumble of a web page, would it be easier to only support hwaccel in full-screen mode? That could be at least a step in the right direction and a reasonable compromise. Or would it throw up more problems switching between sw and hw mid-stream?

Comment 47

2 years ago
see this other bugs related to this bug
https://bugzilla.mozilla.org/show_bug.cgi?id=1210727

Comment 48

2 years ago
I think this feature is really important for laptops.. the battery life can be improved

Comment 49

2 years ago
Why would you rewrite video player from scratch, struggling with acceleration for years, when there's a bunch of existing open source ones, *supporting video acceleration*? MPlayer, VLC to name a few. FYI, VLC can even directly play youtube videos.

Perhaps would it make sense to embed into Firefox existing player, to α) share efforts with community, and β) easier maintenance?

Comment 50

2 years ago
Firefox used to use plugins to play videos, and there were VLC- and MPlayer-based ones. But that's not how HTML5 video is supposed to work. They're doing the next best thing by using the library component of ffmpeg. Switching away from gstreamer may be better in the long run, but I think they could have got acceptable results sooner by fixing the buffer copying issue and upgrading to gstreamer 1.0, which is now a big improvement on 0.3x AFAICT.
(In reply to Hi-Angel from comment #49)
> Why would you rewrite video player from scratch, struggling with
> acceleration for years, when there's a bunch of existing open source ones,
> *supporting video acceleration*? MPlayer, VLC to name a few. FYI, VLC can
> even directly play youtube videos.
> 

Fine; I'll take the bet. 

Can any of your standalone players manage decoding multiple videos at once on a canvas; allow you to apply CSS transformation and transparency layers on the fly. 
Take a video element and move it around. Or record it while playing via simple JS code? Or stream it via WebRTC 

The level of complexity is several order of magnitude greater. 
The issue isn't much about the hardware decoding part. It's the ability to render the decoded video in hardware. 

Currently hardware accelerated layers aren't yet enabled on Linux. It will be soon. Once this is done, we will start working on hardware decoding. I have a personal timeline of a couple of months to get this done

Comment 52

2 years ago
(In reply to Jean-Yves Avenard [:jya] from comment #51)

I shall say beforehand: I didn't have background in that area, so feel free to point out if I am wrong at anything.

> Can any of your standalone players manage decoding multiple videos at once
> on a canvas

 Both mplayer and VLC support many different surfaces; the most extreme one is "ascii" output, allowing one to watch video with no graphic at all. Canvas would just be such a surface.

> ;allow you to apply CSS transformation and transparency layers on the fly. 
> Take a video element and move it around.

I think it would be done just as nowadays Compositors do. Transparency example: I can run a video in vlc, then to switch to another window; compositor makes VLC transparent. Transformation example: kwin and Compiz makes vlc window to wobble upon dragging it around.

The single purpose of a player is to play a video. Whatever happends with the surface later would determine browser, as compositors do in examples.

> Or record it while playing via simple JS code?

If I understood you right, you want to either α) redirect video to multiple surfaces, or β) stream as usual to canvas, and to additionally record whatever there to a file.

> Or stream it via WebRTC 

The same as previous: e.g. additionally output a surface to an IP.

> The level of complexity is several order of magnitude greater. 
> The issue isn't much about the hardware decoding part. It's the ability to
> render the decoded video in hardware. 

Sorry, I'm not sure I understood you: isn't "hardware decoding part" is the "ability to render the decoded video in hardware"?
 
> Currently hardware accelerated layers aren't yet enabled on Linux. It will
> be soon. Once this is done, we will start working on hardware decoding. I
> have a personal timeline of a couple of months to get this done

Honestly, I think video acceleration should have higher priority than acceleration of layers. Because whilst browsing without layers acceleration is mostly fine, watching video isn't. If I'm trying to watch on youtube anything 1920×1080 and higher, I'm getting full CPU load, and lags on Firefox, though my CPU is Core i5.

Comment 53

2 years ago
> Sorry, I'm not sure I understood you: isn't "hardware decoding part" is the "ability to render the
> decoded video in hardware"?

My bad, I see, "decoding" vs "decoded". Anyway, usage of an existing player would solve the both problems at once.
(In reply to Hi-Angel from comment #53)
> > Sorry, I'm not sure I understood you: isn't "hardware decoding part" is the "ability to render the
> > decoded video in hardware"?
> 
> My bad, I see, "decoding" vs "decoded". Anyway, usage of an existing player
> would solve the both problems at once.

Well, seeing your expertise on the matter, we're of course open to contributions.
I'm sure google would like this as well seeing they even removed it from their code due to the complexity of the task.

> I think video acceleration should have higher priority than acceleration of layers. Because whilst browsing without layers acceleration

you do realise that hardware decoding without hardware accelerated layer (that is what actually display the frame) is rather pointless?

> I shall say beforehand: I didn't have background in that area, so feel free to point out if I am wrong at anything.

well, that apply to pretty much everything here :)

Comment 55

2 years ago
(In reply to Jean-Yves Avenard [:jya] from comment #54)
> Well, seeing your expertise on the matter, we're of course open to
> contributions.

…and then later the text you mention that I wrong at pretty much everything :) I'd actually like to, but my plan for contributions is busy for, probably, long time. I wanted to get rid of Emacs+Evil in favor of Yi (would take much work, but I suddenly realized: if I'd started it ½ year ago instead of keep complaining of how I tired of Emacs, many things would be different). Then I want to get Compose key working in FCITX, and then tiliing in Enlightenment (the tiling is pretty much broken atm).
 
> you do realise that hardware decoding without hardware accelerated layer
> (that is what actually display the frame) is rather pointless?
 
Hm… Well. The problem is that decoded frame is rendered with CPU, right? I'm just speculating here, but when (e.g.) VLC is not fullscreen on X11, and there's compositioning enabled (which gives those effects, like transparency, burning, wobbling, etc), I think every VLC frame have to be copied by CPU to apply effects. Guess what: even 1080 video on vlc (given it is decoded via vaapi) does not give me full CPU load. So if we'd compare, then decoding is, probably, more important than rendering.
 
> well, that apply to pretty much everything here :)

It would be interesting to hear, why. FWIW, I came from a phoronix comments thread¹  and there was some speculation about why doesn't firefox use existing videoplayer for that. I asked that myself some time ago (and even searched for addons which does replace the player); so, as, turns out, I'm not the only one wondering, I decided to ask.

[1] https://www.phoronix.com/forums/forum/phoronix/latest-phoronix-articles/890996-firefox-49-to-offer-linux-widevine-support-firefox-also-working-on-webp-support

Comment 56

2 years ago
(In reply to Hi-Angel from comment #55)
> (In reply to Jean-Yves Avenard [:jya] from comment #54)
> > you do realise that hardware decoding without hardware accelerated layer
> > (that is what actually display the frame) is rather pointless?
>  
> Hm… Well. The problem is that decoded frame is rendered with CPU, right? I'm
> just speculating here, but when (e.g.) VLC is not fullscreen on X11, and
> there's compositioning enabled (which gives those effects, like
> transparency, burning, wobbling, etc), I think every VLC frame have to be
> copied by CPU to apply effects. Guess what: even 1080 video on vlc (given it
> is decoded via vaapi) does not give me full CPU load. So if we'd compare,
> then decoding is, probably, more important than rendering.

I'm not an expert for VLC and it's filters, but I agree that HW decoding of video and then rendering it on CPU is quite pointless.  Copying the decoded frame to GPU to decode and then back to CPU is a big overhead and the actual benefit would be very small. I suppose that VLC is trying to do as much work as it can on the GPU and avoid copying from GPU back to CPU.
Back then, when Firefox used gstreamer to decode video, there was some unofficial experiments with using gstreamer-vaapi to get HW decoding with firefox. The actual benefit of doing it was not big, it was exactly this copying issue mentioned above. On one side, you saved some CPU with doing decoding on GPU, but on the other side you had to do another copying between CPU and GPU and that used additional CPU time.
So I agree that the only way how to do this properly is use HW decoding and then rendering on HW accelerated layers.

Comment 57

2 years ago
(In reply to martin from comment #56)

Hmm, though a little research shows that some effects indeed don't need an additional copy to be applied, so it is possible that compton doesn't copy every VLC frame. I have to check it out, though don't know how off hands.

Comment 58

2 years ago
Well, I asked on Freenode #wayland channel, so, here's some corrections about the rendering from the discussion.

1) Compositioning always does, at least, a single copy. But it doesn't have to use CPU, because the copy might as well be done GPU → GPU. I don't know though whether Compton uses CPU or GPU.
2) Quoting "pq> the rule of thumb is: every time you switch between GPU and CPU processing in a pipeline, it causes a great overhead". There also were guesses about a big number of internal layers in Firefox; so, it seems enabling accelerated layers first makes sense.

Also, I've no experience in graphics, so this request I'd probably pass directly by quoting:

	pq> Hi-Angel, also remind them about the experimental Wayland linux-dmabuf protocol which should allow one to show a decoded YUV buffer without a trip through EGL and GL rendering in the app.
The GPU->GPU copy is only usable if you also have hardware accelerated composition.

Comment 60

2 years ago
(In reply to Jean-Yves Avenard [:jya] from comment #51)
> Can any of your standalone players manage decoding multiple videos at once
> on a canvas; allow you to apply CSS transformation and transparency layers
> on the fly. 
> Take a video element and move it around. Or record it while playing via
> simple JS code? Or stream it via WebRTC 
> 
> The level of complexity is several order of magnitude greater. 
> The issue isn't much about the hardware decoding part. It's the ability to
> render the decoded video in hardware. 

What I don't understand is that if you've already solved nearly all these problems for WebGL canvases, why can't you apply the same solutions to video elements with VA-API etc?

> Currently hardware accelerated layers aren't yet enabled on Linux. It will
> be soon. Once this is done, we will start working on hardware decoding. I
> have a personal timeline of a couple of months to get this done

Thanks for the feedback. It's good to hear it is going to be worked on soon.

Comment 61

2 years ago
I still have hopes to see full VDPAU/VAAVI support in order to have full video acceleration.

[urlhttps://wiki.mozilla.org/Performance/Project_Candle]Project Candle[/url] + [url=http://www.servo.org[/url] + 100% WebExtensions compatibility (featuring a consortium in W3C for this standard, full desktop+mobile support and make UI differences the less problem possible and propose lots more improvements over Google Chrome implementation, maybe even a embedded language like LUA?) + don't mess with bad UX = A really competent Firefox AGAIN.

Comment 62

2 years ago
Would it be possible do enable the GPU acceleration when playing full screen video? In that case the video is not being composed into a page and there is no scrolling to worry about.

Comment 63

2 years ago
I still have hope too to see full VDPAU/VAAVI support in order to have hardware video acceleration.

Comment 64

a year ago
Hi

Someone could find this useful, so I thought I'd mention it: we've managed to enable GPU video acceleration in our Digital Signage platform by combining Firefox with an ancient, almost forgotten NPAPI plugin that makes it rely on MPlayer for video playback:

http://mplayerplug-in.sourceforge.net/

(with just a small patch to accelerate video scaling https://sourceforge.net/p/mplayerplug-in/patches/25/)

You can see it working on Beabloo booth, this year's ISE in Amsterdam (https://www.beabloo.com/omnichannel/ise-2017/).

Please don't kill NPAPI just yet. Until GPU acceleration is integrated with HTML video, it still has some very interesting uses.
NPAPI is already dead; it's also completely incompatible with HTML5
Comment hidden (obsolete)
Comment hidden (obsolete)
Comment hidden (advocacy, me-too)
Comment hidden (me-too)
Comment hidden (me-too)

Comment 71

8 months ago
Will WebRender integration in Firefox help addressing this? https://github.com/servo/webrender/wiki
Comment hidden (obsolete)
You need to log in before you can comment on or make changes to this bug.