894372 - Gstreamer backend don't use hardware accelerate (VAAPI)

Status: RESOLVED WONTFIX

(Core :: Audio/Video: Playback, defect, P5)

Description

[vova.kravets]

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:25.0) Gecko/20130715 Firefox/25.0 (Nightly/Aurora)
Build ID: 20130715100109

Steps to reproduce:

1. Install Gstreamer 0.10
2. Install Gstreamer VAAPI plugin
3. Install Firefox 25.0a1
4. Play any HTML5 video 1080p/720p
5. Look at CPU usage



Actual results:

CPU usage is big. It seems VAAPI is not used.

Note: If I use below command I see that VAAPI is used.

gst-launch playbin2 <url>


Expected results:

VAAPI should use and Hardware Accelerate should use during rendering also.

Comment 1

[dE]

Dont you think FF uses an internal copy of GST?

Comment 2

[Alessandro Decina]

I haven't looked into this, but it's expected that it doesn't work yet. It requires some work in the backend and in the firefox gfx bits to pass around VAAPI backed textures, instead of decoding frames into main memory.

Comment 3

[Marco Trevisan (Treviño)]

With the majority of linux drivers now supporting VA-API (both directly and using mesa vdpau drivers throught vdpau-va-driver) and with HTML5 standard being used more widely (thank God), this should really be addressed ASAP, since the CPU usage of firefox when decoding and rendering videos is quite a lot (even compared to the "evil" flash player, with hw capabilities).

Comment 4

[vulcain]

(In reply to vova.kravets from comment #0)
> User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:25.0) Gecko/20130715
> Firefox/25.0 (Nightly/Aurora)
> Build ID: 20130715100109
> 
> Steps to reproduce:
> 
> 1. Install Gstreamer 0.10
> 2. Install Gstreamer VAAPI plugin 
> [...]
> 
> VAAPI should use and Hardware Accelerate should use during rendering also.

If i am not wrong, hardware acceleration need 0.5.7 et 0.5.8 of gstreamer-vaapi and Gtsreamer 1.x
http://lists.freedesktop.org/archives/gstreamer-devel/2014-May/047657.html
> Support for hardware codecs and special memory types has been
>      improved with bugfixes and feature additions in various plugins
>      and base classes.

Comment 5

[Alejandro Cobo López]

(In reply to vulcain from comment #4)
> (In reply to vova.kravets from comment #0)
> > User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:25.0) Gecko/20130715
> > Firefox/25.0 (Nightly/Aurora)
> > Build ID: 20130715100109
> > 
> > Steps to reproduce:
> > 
> > 1. Install Gstreamer 0.10
> > 2. Install Gstreamer VAAPI plugin 
> > [...]
Since now Firefox uses Gstreamer 1.0, adding video hardware decoding for HTML5 should be a priority.


> > 
> > VAAPI should use and Hardware Accelerate should use during rendering also.
> 
> If i am not wrong, hardware acceleration need 0.5.7 et 0.5.8 of
> gstreamer-vaapi and Gtsreamer 1.x
> http://lists.freedesktop.org/archives/gstreamer-devel/2014-May/047657.html
> > Support for hardware codecs and special memory types has been
> >      improved with bugfixes and feature additions in various plugins
> >      and base classes.

Comment 6

[Marco Trevisan (Treviño)]

That's true, however I don't think it's much something that is related to firefox anymore.

In fact using the latest firefox release, together with the current gstreamer1.0-vaapi git¹, as soon as I open a youtube html5 video or other html5 videos around², I get in my console:

libva info: VA-API version 0.35.0
libva info: va_getDriverName() returns 0
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/i965_drv_video.so
libva info: Found init function __vaDriverInit_0_35
libva info: va_openDriver() returns 0

Which is actually a syntom that things are correctly played with my intel vaapi driver.

I don't know exactly what gstreamer-vaapi revision introduced the changes needed to make firefox to be aware of the vaapi acceleration: for sure it didn't work with 0.5.7, but I didn't try 0.5.8.

Anyway, it seems that most of the work to get the hw acceleartion has to be done outside firefox, as this new gstreamer1.0 support already gives us the framework needed.

[1] For ubuntu amd64 you might try these debs if you want http://people.ubuntu.com/~3v1n0/gstreamer-vaapi/
[2] That said, youtube works, vimeo also, but other videos were not playing video or with some glitches, while in other cases, I got a crash³, but as bonus the CPU usage is really low.
[3] What said is valid only for intel in my tests, because using radeon throught the vdpau-va-driver this always crashes (and the same does totem by using that version of gstreamer-vaapi with any mp4 video). I still had no time to debug it and to check weather the problem is in the radeon vdpau driver or somewhere else; input from nvidia and nouveau users would be welcome.

Comment 7

[Jan Alexander Steffens [:heftig]]

The GStreamer 1.0 backend does autoplug vaapidecode if it's available, however CPU usage of Firefox is much higher here when it is: ~120% with vaapi vs. ~30% without, for Youtube 720p.

Comment 8

[Mariusz Libera]

gstreamer-vaapi 5.9 has been released and in NEWS file we can read among others:
* Fix support for Firefox >= 30 when built with GStreamer 1.0 APIs
And it indeed now works without crashing Firefox, but apparently it's not optimal and requires improvements on Firefox's side. Here is a comment by Gwenole Beauchesne who works on gstreamer-vaapi:

Firefox will not use hardware accelerated rendering and requires the decoded frames to be transferred to their own buffers. i.e. reading every single frame from GPU memory is necessary. If you want to compare Firefox behaviour's to the "optimal" one, you could probably use gst-launch-1.0 uri=file:///path/to/Video.mp4 video-sink="ximagesink" (similar to firefox behaviour), vs. gst-launch-1.0 uri=file:///path/to/Video/mp4 video-sink="vaapisink" (the default).

Yes, we have NV12 -> I420 (HW accelerated through GPU memory) -> read back to I420 buffer -> I420 to RGB32 conversion through SSE2 code it seems. Not the fastest pipeline, but patches are welcome for Firefox. 

source:http://www.phoronix.com/forums/showthread.php?104498-GStreamer-VA-API-Plug-In-Update-Adds-New-Features&p=431726#post431726

So, could Firefox devs work on this? Because right now there is no noticeable drop of CPU usage.

Comment 9

[Geert Bosman]

(In reply to Marco Trevisan (Treviño) from comment #6)
> .... input from nvidia and nouveau users would be
> welcome.
Here are my findings with a Nvidia GT 430 graphics card and a lowly Pentium 4 3Ghz and the following software versions:
Ubuntu 14.04.1 (AMD64)
Firefox 31.0
gStreamer-vaapi 5.9 (from http://people.ubuntu.com/~3v1n0/gstreamer-vaapi/ )
Nvidia driver 331.38

youtube and vimeo work. I tested some other HTML5 videos on other sites which also worked, but most of the times crashed when changeing resolution or pausing/playing the video (this also happens with youtube). For this hardware setup the difference between version 5.9 and 5.7 is that HD720 and HD1080p video's are now playable! While on the old (5.7) version of the gstreamer-vaapi, playing these video's in Firefox was impossible (i.e. a dia show and 100% cpu).
I also tested Totem with mp4 content. It crashed within 2 seconds with version 5.9 as you already found out.

Comment 10

[Mathew Hodson]

According to comment 4, this needs GStreamer 1.0, so it should depend on bug 947287.

Comment 11

[equeim]

Doesn't work with gstreamer-vaapi 0.5.10 and 0.6.0 (green background instead of video).

(firefox:31337): GStreamer-CRITICAL **: gst_mini_object_lock: assertion 'object != NULL' failed
(firefox:31337): GStreamer-CRITICAL **: gst_memory_unmap: assertion 'mem != NULL' failed
(firefox:31337): GStreamer-CRITICAL **: gst_memory_get_sizes: assertion 'mem != NULL' failed
(firefox:31337): GStreamer-CRITICAL **: gst_mini_object_unlock: assertion 'object != NULL' failed
(firefox:31337): GStreamer-CRITICAL **: gst_mini_object_unref: assertion 'mini_object != NULL' failed

gst-play-1.0 command works with hardware acceleration without any issues.

Mesa 11.0.0_rc1 with Gallium3D VAAPI state tracker, radeonsi.

Comment 12

[dE]

(In reply to Mathew Hodson from comment #10)
> According to comment 4, this needs GStreamer 1.0, so it should depend on bug
> 947287.

Correct.

Comment 13

[glunardi]

Installed Firefox 40.0 from OpenSUSE 13.2 repositories (32bit)

When I visit facebook.com, after a few scrolls on the page, Firefox crashes with only output VAAPI and GStreamer related messages.

libva info: VA-API version 0.34.0
libva info: va_getDriverName() returns 0
libva info: Trying to open /usr/lib/dri/i965_drv_video.so
libva info: Found init function __vaDriverInit_0_34
libva info: va_openDriver() returns 0
(firefox:3891): GStreamer-CRITICAL **: gst_mini_object_lock: assertion 'object != NULL' failed
(firefox:3891): GStreamer-CRITICAL **: gst_memory_unmap: assertion 'mem != NULL' failed
(firefox:3891): GStreamer-CRITICAL **: gst_memory_get_sizes: assertion 'mem != NULL' failed

Comment 14

[glunardi]

Attachment: glunardi-bug-894372-crashreport.txt

This is the crashreport that gets generated after I visit facebook.com using:
 MozillaFirefox-40.0-38.1.i586
 Mesa-10.3.7-91.6.1.i586
 GStreamer-1.4.3-1.4.i586
 libgstvaapi-1_4-2-0.6.0-9.2.i586
 libgstvaapi-drm-1_4-2-0.6.0-9.2.i586
 libgstvaapi-glx-1_4-0-0.5.9-4.1.i586
 libgstvaapi-x11-1_4-0-0.5.9-4.1.i586
 libgstvaapi-x11-1_4-2-0.6.0-9.2.i586

Comment 15

[Marco Castelluccio [:marco]]

(In reply to glunardi from comment #13)
> When I visit facebook.com, after a few scrolls on the page, Firefox crashes
> with only output VAAPI and GStreamer related messages.

Can you file a new bug?

Comment 16

[glunardi]

(In reply to Marco Castelluccio [:marco] from comment #15)
> Can you file a new bug?
Created https://bugzilla.mozilla.org/show_bug.cgi?id=1202140

Thanks!

Comment 17

[Stebs]

Shouldn't this bug have "[Power]" added to the bug whiteboard?
https://wiki.mozilla.org/Performance/Project_Candle

Comment 18

[Tony Houghton]

(In reply to Mariusz Libera from comment #8)
> gstreamer-vaapi 5.9 has been released and in NEWS file we can read among
> others:
> * Fix support for Firefox >= 30 when built with GStreamer 1.0 APIs
> And it indeed now works without crashing Firefox, but apparently it's not
> optimal and requires improvements on Firefox's side. Here is a comment by
> Gwenole Beauchesne who works on gstreamer-vaapi:

[Snip]

> So, could Firefox devs work on this? Because right now there is no
> noticeable drop of CPU usage.

Agreed. I don't even know whether it's using acceleration or not because the CPU usage is so high. Linux desperately needs a mainstream browser to support efficient HTML5 video playback. Chrome/Chromium devs aren't interested in supporting Linux either.

Comment 19

[N. W.]

Hello,

I guess now that GStreamer will be deprecated and FFmpeg/Libav will be used directly:

https://bugzilla.mozilla.org/show_bug.cgi?id=1207429

the proper bug to discuss hardware accelerated video playback now would be:

https://bugzilla.mozilla.org/show_bug.cgi?id=563206

?

Regards

Comment 20

[Jean-Yves Avenard [:jya]]

gstreamer is going in bug 1234092