Closed Bug 1292374 Opened 8 years ago Closed 8 years ago

Add support for full hardware acceleration in VP9

Categories

(Core :: Audio/Video: Playback, defect, P3)

Product:
Core
Component:
Audio/Video: Playback
Platform:
x86
Windows 10
Type:
defect

Tracking

()

Status:
RESOLVED FIXED
Milestone:
mozilla52
Tracking Flags:
Tracking Status
platform-rel --- +
firefox52 --- fixed

People

(Reporter: joseph.k.olivas, Assigned: joseph.k.olivas)

References

(Depends on 1 open bug)

Details

(Whiteboard: [platform-rel-nVidia][platform-rel-Intel][platform-rel-AMD])

Attachments

(3 files, 7 obsolete files)

Bug 1292374: P1. Add support for full hardware acceleration in VP9.
58 bytes, text/x-review-board-request
jya
: review+
Details
Bug 1292374: P2. Change preference name as it's no longer restricted to intel.
58 bytes, text/x-review-board-request
mozbugz
: review+
Details
Bug 1292374: P3. Enable VP9 hardware acceleration on windows.
58 bytes, text/x-review-board-request
mozbugz
: review+
Details 
Intel Kaby Lake and newer systems have full hardware decode of VP9, exposed on developer preview versions of Windows 10 through a new GUID.

This will replace the previous partially accelerated decode on Broadwell and Skylake platforms.
Attached patch Replace VP9 GUID, allow DX11 (obsolete) — — Splinter Review 
This uses the VP9 GUID exposed in the Windows 10 developer preview (build 14393), and allows DX11 for VP9, since this MFT only works with DX11 or newer.
Component: General → Audio/Video
Product: Firefox → Core
Attached patch Removing magic numbers (obsolete) — — Splinter Review 
This version is basically the same, but uses the VideoInfo to determine width and height, rather than the hardcoded values.
Attachment #8778005 - Attachment is obsolete: true
Attached patch bug1292374v3.patch (obsolete) — — Splinter Review 

        Attachment #8778418 -
        Attachment is obsolete: true

        Attachment #8778494 -
        Flags: review?(cpearce) → review+

    
    

    
  
Could this please get higher priority?
Both Edge and Chrome nowadays support hardware acceleration for VP9, it already works without issues on modern Nvidia GPUs (GTX 960, 950, 1080, 1070, 1060, Titan X Pascal).

    
    

    
  
(In reply to walmartguy from comment #4)
> Could this please get higher priority?
> Both Edge and Chrome nowadays support hardware acceleration for VP9, it
> already works without issues on modern Nvidia GPUs (GTX 960, 950, 1080,
> 1070, 1060, Titan X Pascal).

This ticket doesn't relate to nVidia GPUs because they use a different API.

    
    

    
  
Comment on attachment 8778494 [details] [diff] [review]
bug1292374v3.patch

Review of attachment 8778494 [details] [diff] [review]:
-----------------------------------------------------------------

this needs fixing as per comment

::: dom/media/platforms/wmf/WMFVideoMFTManager.cpp
@@ +389,5 @@
>  bool
>  WMFVideoMFTManager::InitInternal(bool aForceD3D9)
>  {
>    mUseHwAccel = false; // default value; changed if D3D setup succeeds.
> +  bool useDxva = InitializeDXVA(aForceD3D9);

why this change?

AFAIK, we must use dxva with vp8 or vp9, otherwise it could create an invalid decoder which would break software decoding using ffvp9

@@ +463,5 @@
>  
>    hr = inputType->SetUINT32(MF_MT_INTERLACE_MODE, MFVideoInterlace_MixedInterlaceOrProgressive);
>    NS_ENSURE_TRUE(SUCCEEDED(hr), hr);
>  
> +  // MSFT MFT needs this frame size set for VP9?

it's not just VP9, VP8 too. Actually this is something that got broken with bug 1297311 and need fixing.

        Attachment #8778494 -
        Flags: review-
Flags: needinfo?(joseph.k.olivas)

    
    

    
  
(In reply to Jean-Yves Avenard [:jya] from comment #6)

> > +  bool useDxva = InitializeDXVA(aForceD3D9);
> 
> why this change?

The previous version was forcing D3D9 for VP8 and VP9, but the MFT only works with D3D11 (at least in my testing this was the case). Forcing it to D3D9 made it fail to load, but removing the special case for VP8/9 worked as expected.

> >  
> > +  // MSFT MFT needs this frame size set for VP9?
> 
> it's not just VP9, VP8 too. Actually this is something that got broken with
> bug 1297311 and need fixing.

Even in a standalone test program, I had to set the frame size, otherwise, the MFT would fail on SetInputType with MF_E_INVALIDMEDIATYPE. But it can be set for both.

In addition, this MFT works with both VP8 and VP9, but I didn't enable it for VP8 in this case.
Flags: needinfo?(joseph.k.olivas)

    
    

    
  
(In reply to Anthony Jones (:kentuckyfriedtakahe, :k17e) from comment #5)
> This ticket doesn't relate to nVidia GPUs because they use a different API.
So, what now? Create a new ticket for it?

    
    

    
  
(In reply to walmartguy from comment #8)
> (In reply to Anthony Jones (:kentuckyfriedtakahe, :k17e) from comment #5)
> > This ticket doesn't relate to nVidia GPUs because they use a different API.
> So, what now? Create a new ticket for it?

That's bug 1273902.

Adding nvidia support is much more complicated as they don't provide an API compatible with our existing decoder (we only support MFT right now)

    
    

    
  

      
      
      
      

        Attached patch
          
          
        bug1292374v4.patch (obsolete)
        — 
          — Splinter Review
      

    


  
Adding/combining VP8 support and removing old GUIDs.

        Attachment #8778494 -
        Attachment is obsolete: true

    
    

    
  
(In reply to Jean-Yves Avenard [:jya] from comment #9) 
> That's bug 1273902.
> 
> Adding nvidia support is much more complicated as they don't provide an API
> compatible with our existing decoder (we only support MFT right now)
Thanks. Why does Firefox not use DX9 DXVA2 VP9 decoder?
LAV Filters supports it for quite some time now.

    
    

    
  
Comment on attachment 8803021 [details] [diff] [review]
bug1292374v4.patch

Review of attachment 8803021 [details] [diff] [review]:
-----------------------------------------------------------------

so if I understand properly, this disable VPX hardware decoding on system earlier than windows 10 anniversary edition?

What about earlier OS, aren't there be improvement in the earlier MFT. We had to disable those because it was crashy, but could we still use those with newer drivers?

        Attachment #8803021 -
        Flags: review+
Assignee: nobody → joseph.k.olivas

    
    

    
  
(In reply to Jean-Yves Avenard [:jya] from comment #12)
> so if I understand properly, this disable VPX hardware decoding on system
> earlier than windows 10 anniversary edition?
> 
> What about earlier OS, aren't there be improvement in the earlier MFT. We
> had to disable those because it was crashy, but could we still use those
> with newer drivers?

VP9 is only available with Win10 anniversary and newer on Kaby Lake, so perhaps best to separate it from an attempt to re-enable VP8 on the older platforms/OS (Broadwell and Sky Lake), since it doesn't use the Microsoft MFT.

Dumb question: how do I proceed to land and enable this on Nightly? Usually another Mozilla engineer takes things over at this point since I don't have permissions :)

    
    

    
  
(In reply to Joe Olivas from comment #13)
> VP9 is only available with Win10 anniversary and newer on Kaby Lake, so
> perhaps best to separate it from an attempt to re-enable VP8 on the older
> platforms/OS (Broadwell and Sky Lake), since it doesn't use the Microsoft
> MFT.

Ambiguous word choice clarification: VP8 doesn't use Microsoft MFT, but VP9 does.

    
    

    
  

      
      
      
      

        Attached patch
          
          
        bug1292374v5.patch (obsolete)
        — 
          — Splinter Review
      

    


  
Falling back to ffvpx if hw MFT is not available.

        Attachment #8803021 -
        Attachment is obsolete: true

    
    

    
  

      
      
      
      

        Attached patch
          
          
        bug1292374v5.patch (obsolete)
        — 
          — Splinter Review
      

    


  
Fixing whitespace issues.

        Attachment #8805180 -
        Attachment is obsolete: true

    
    

    
  

      
      
      
      

        Attached patch
          
          
        bug1292374v6.patch (obsolete)
        — 
          — Splinter Review
      

    


  
Ok, last one :)

Moving the check for failed hw acceleration to a better place since the previous may not have been hit, but still failed to be hw accelerated.

        Attachment #8805183 -
        Attachment is obsolete: true

    
    

    
      
  


  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
  
Comment on attachment 8805426 [details]
Bug 1292374: P1. Add support for full hardware acceleration in VP9.

https://reviewboard.mozilla.org/r/89152/#review88288

        Attachment #8805426 -
        Flags: review?(jyavenard) → review+

    
    

    
  
Comment on attachment 8805190 [details] [diff] [review]
bug1292374v6.patch

rebased and put in reviewboard

        Attachment #8805190 -
        Attachment is obsolete: true

    
    

    
  
Comment on attachment 8805427 [details]
Bug 1292374: P2. Change preference name as it's no longer restricted to intel.

https://reviewboard.mozilla.org/r/89154/#review88294

r+ with nit:

::: dom/media/MediaPrefs.h:123
(Diff revision 2)
>    DECL_MEDIA_PREF("media.ffvpx.enabled",                      PDMFFVPXEnabled, bool, true);
>  #endif
>  #ifdef XP_WIN
>    DECL_MEDIA_PREF("media.wmf.enabled",                        PDMWMFEnabled, bool, true);
>    DECL_MEDIA_PREF("media.decoder-doctor.wmf-disabled-is-failure", DecoderDoctorWMFDisabledIsFailure, bool, false);
> -  DECL_MEDIA_PREF("media.webm.intel_decoder.enabled",         PDMWMFIntelDecoderEnabled, bool, false);
> +  DECL_MEDIA_PREF("media.wmf.vp9.enabled",                    PDMWMFIntelDecoderEnabled, bool, false);

You should change the name of the function accordingly, e.g.: 'PDMWMFIntelDecoderEnabled' -> 'PDMWMFVP9Enabled'.
(Obviously you'll need to update it in other files.)

        Attachment #8805427 -
        Flags: review?(gsquelart) → review+

    
    

    
  
Comment on attachment 8805428 [details]
Bug 1292374: P3. Enable VP9 hardware acceleration on windows.

https://reviewboard.mozilla.org/r/89156/#review88296

        Attachment #8805428 -
        Flags: review?(gsquelart) → review+

    
    

    
      
  

    
    

    
      
  

    
    

    
  
Comment on attachment 8805427 [details]
Bug 1292374: P2. Change preference name as it's no longer restricted to intel.

https://reviewboard.mozilla.org/r/89154/#review88294

> You should change the name of the function accordingly, e.g.: 'PDMWMFIntelDecoderEnabled' -> 'PDMWMFVP9Enabled'.
> (Obviously you'll need to update it in other files.)

done

    
    

    
  
Pushed by jyavenard@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/6495adc039e1
P1. Add support for full hardware acceleration in VP9. r=jya
https://hg.mozilla.org/integration/autoland/rev/cb838996a5dd
P2. Change preference name as it's no longer restricted to intel. r=gerald
https://hg.mozilla.org/integration/autoland/rev/288ef9097e01
P3. Enable VP9 hardware acceleration on windows. r=gerald

    
    

    
  
(In reply to Joe Olivas from comment #17)
> Created attachment 8805190 [details] [diff] [review]
> bug1292374v6.patch
> 
> Ok, last one :)
> 
> Moving the check for failed hw acceleration to a better place since the
> previous may not have been hit, but still failed to be hw accelerated.

Actually, that way is rather inefficient, as we now always attempt to create a WMF decoder when it could have been refused in the SupportsMimeType called.

But I guess that's an improvement that can be done later.
platform-rel: --- → +
Whiteboard: [platform-rel-nVidia][platform-rel-Intel][platform-rel-AMD]

    
    

    
  
https://hg.mozilla.org/mozilla-central/rev/6495adc039e1
https://hg.mozilla.org/mozilla-central/rev/cb838996a5dd
https://hg.mozilla.org/mozilla-central/rev/288ef9097e01
Status: UNCONFIRMED → RESOLVED
Closed: 8 years ago

          status-firefox52:
          --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla52

    
    

    
  
Should it work now with Nvidia in FF 52.0a1?
I noticed that the config entry for Intel has been replaced with VP9 WMF, but it still uses CPU decoding here on 1070.

    
    

    
  
(In reply to walmartguy from comment #35)
> Should it work now with Nvidia in FF 52.0a1?
> I noticed that the config entry for Intel has been replaced with VP9 WMF,
> but it still uses CPU decoding here on 1070.

An updated nightly will only be available tomorrow.
You also need to use Windows 10 anniversary edition. 

You can test with this build: https://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-central-win64-pgo/1477751472/firefox-52.0a1.en-US.win64.installer.exe
Note that this is not nightly, it will not automatically update. I suggest that you install it in a different path than the default.

Our plan is to uplift this so it's available in 51

BTW, there are still issues with the d3d11 compositor, and it's not yet 100%; so 4K @ 60fps may not always perform better than software decoding.
You can use the media.wmf.vp9.enabled preference to disable the HW accelerated code path.

    
    

    
  
I can confirm now GPU's VPU is used instead of CPU, very good. And indeed there seems to be some problem with the renderer remaining, 60fps video looks more like 30fps video or even less.
This happens also with less damanding videos like 1080p 60fps.

    
    

    
  
Yes, the issue isn't with the dimension of the frame nor the time it takes to decide, but the speed of rendering.

A guess is how slow nvidia is to allocate textures and we should recycle them instead.

    
  
Depends on: 1314458

    
    

    
  
I'm running Firefox 52.0b6/win64-EME-free and I don't notice any acceleration under Windows 7 64 with GTX 1060 6GB and the latest NVIDIA drivers (378.66).

Is hardware acceleration of VP9 only supported under Windows 10?

    
    

    
  
While watching 4K youtube videos hundreds of frames get dropped and CPU usage is close to 100% (i.e. all four cores of my Intel Core i5 2500 are busy).

Also is it too much to ask to show VP9 hardware acceleration status in about:support? Currently it only shows "Supports Hardware H264 Decoding: YES/NO".

And I'm interested in all other supported codecs as well, not just AVC/H.264 and VP9.

    
    

    
      
  

    
    

    
  
As of today, running Firefox 54.0.1, there is still no VP9 hardware acceleration support.  This is not a feature request, 4K videos in YouTube actually do not run properly in full-screen mode, they are very choppy and freeze occasionally because as mentioned in this ticket, Firefox CPU and memory usage are very high because it has to do software decoding.  Anyone with a 4K monitor run this video in full-screen mode in Firefox:

https://www.youtube.com/watch?v=tO01J-M3g0U&t

It sucks.  Playing YouTube 4K videos in Edge and Chrome have almost no CPU usage whatsoever and videos play very smoothly. Something like this should be highest on the priority list of things to fix, this is one of the quickest ways to lose users if YouTube videos cannot play properly.  And yes 4K displays are common nowadays, its becoming the new standard over HD.

    
    

    
  
Apologies, forgot to add in previous comment.  It shows in this ticket that this was supposedly fixed/added in Firefox 52.  As of Firefox 54 I don't see any VP9 hardware acceleration.

    
    

    
  
(In reply to Leandro from comment #42)
> As of today, running Firefox 54.0.1, there is still no VP9 hardware
> acceleration support.  This is not a feature request, 4K videos in YouTube
> actually do not run properly in full-screen mode, they are very choppy and
> freeze occasionally because as mentioned in this ticket, Firefox CPU and
> memory usage are very high because it has to do software decoding.  Anyone
> with a 4K monitor run this video in full-screen mode in Firefox:
> 
> https://www.youtube.com/watch?v=tO01J-M3g0U&t
> 
> It sucks.  Playing YouTube 4K videos in Edge and Chrome have almost no CPU

Edge by default uses H264. AFAIK Chrome also only supports VP9 HW acceleration on Windows 10 (that currently only works for Inten and Nvidia), for AMD it's behind an experimental feature.

You have two choices in Firefox.
1- Let it use its default (since it appears you have a slow machine, it will default to H264 that will use HW acceleration), or
2- Wait until 55 that comes out tomorrow and if you have a supported hardware configuration with Windows 10 (anniversary edition), or set the pref media.wmf.vp9.enabled to true

    
    

    
  
(In reply to Jean-Yves Avenard [:jya] from comment #44)
> (In reply to Leandro from comment #42)
> > As of today, running Firefox 54.0.1, there is still no VP9 hardware
> > acceleration support.  This is not a feature request, 4K videos in YouTube
> > actually do not run properly in full-screen mode, they are very choppy and
> > freeze occasionally because as mentioned in this ticket, Firefox CPU and
> > memory usage are very high because it has to do software decoding.  Anyone
> > with a 4K monitor run this video in full-screen mode in Firefox:
> > 
> > https://www.youtube.com/watch?v=tO01J-M3g0U&t
> > 
> > It sucks.  Playing YouTube 4K videos in Edge and Chrome have almost no CPU
> 
> Edge by default uses H264. AFAIK Chrome also only supports VP9 HW
> acceleration on Windows 10 (that currently only works for Inten and Nvidia),
> for AMD it's behind an experimental feature.
> 
> You have two choices in Firefox.
> 1- Let it use its default (since it appears you have a slow machine, it will
> default to H264 that will use HW acceleration), or
> 2- Wait until 55 that comes out tomorrow and if you have a supported
> hardware configuration with Windows 10 (anniversary edition), or set the
> pref media.wmf.vp9.enabled to true

Hi and thanks for the quick follow-up.  The thing is I don't have a slow machine, it's brand new, with Intel i7, plenty of RAM and an SSD.  If Firefox 54 is using H.264 acceleration by default wow it isn't hardware accelerating :) much. If you play the above YouTube video link in full-screen mode in 4K (2160p60 quality) on a 4K monitor it's very choppy and not responsive and Firefox has 60% CPU utilization of all cores.

If you play the same video in Edge it has completely smooth playback and is responsive and Edge only has 3% CPU utilization.  That is what you would expect when hardware acceleration is enabled.  If, as you say, both Firefox and Edge are defaulting to the same H264 codec then Firefox is doing something wrong.

    
    

    
  
Edge will be getting H264 by default.
You can check by right clicking on the player video and select "Stats for nerds", you seel video/webm if vp9 or video/mp4 if h264.

Unless you have an intel i7-7xxx or an nvidia Pascal (nvidia 10xx) and some 9xx), there won't be hardware acceleration : it isn't possible.

In Firefox, if we deem your machine fast enough to decode VP9, we enable VP9 availability. So youtube gives you VP9.
Chrome always gets VP9.

Chrome would have the same downside as Firefox as it has the same restriction on which hardware is supported.

So either you wait for Firefox 55 which has VP9 HW acceleration enabled by default or you set the preference media.wmf.vp9.enabled to true.
Alternatively, you can disable vp9 in YouTube by setting media.benchmark.vp9.fps to a value lower than 150 (150 being our threshold to enable VP9)

    
    

    
  
Youtube does not support >1080p with h264, vp9 is required for 2160p+ so the comparison between edge and firefox is nullified.

    
    

    
  
Thanks Jean-Yves and Danial for all the interesting detailed info on this topic.

I do have a "Kaby Lake" i7-7xxx series CPU with hardware acceleration support for VP9 and H.265, so will be looking forward to testing this functionality in Firefox 55 and comparing to other leading browsers and reporting back to you.

BTW on the NVidia GPU side, I believe in addition to the Pascal-based GPUs as you mentioned there are some of the latest generation Maxwell (GM206) GPUs that do have VP9 hardware acceleration support.  This page I found on Nvidia has a summary although I don't see all recent Quadro GPUs there but it has most:

https://developer.nvidia.com/video-encode-decode-gpu-support-matrix

    
    

    
  
P.S. I also found the Nvidia parent page to that support matrix with additional information

https://developer.nvidia.com/nvidia-video-codec-sdk

    
    

    
  
I downloaded and tested Firefox 55.0beta from https://www.mozilla.org/en-US/firefox/channel/desktop/ and tested with the YouTube 4K video I referred to

https://www.youtube.com/watch?v=tO01J-M3g0U&t

as well as other videos.  I double checked the about:config vp9 settings and Firefox 55.0beta enabled everything for hardware acceleration on my machine.  I can confirm that now CPU utilization and RAM usage are indicative of VP9 hardware acceleration and similar to what I see in Edge and Chrome.

But... there still is a major problem.  Every YouTube 4K video I tested has very choppy playback and less than acceptable responsiveness in 4K full-screen mode.  Its not as bad as in Firefox 54 and previous versions when it doesn't have VP9 acceleration but it's still so choppy that it's unwatchable.  The exact same videos in Edge and Chrome have buttery smooth playback.  Not sure what this issue is now since it's choppy even when you can tell from CPU utilization and memory usage that hardware acceleration is taking place.
Depends on: 1387358

    
    

    
  
I've opened bug 1387358, we already have a few bugs on playback being choppy.

Here on my Dell XPS 15 9560, playback is perfect, it has some occasional jitter but it's very minimal.

    
    

    
  
I found the source of the issue Jean-Yves, my machine is a laptop (HP ZBook 15 G4) and when I'm using battery that's when it is choppy/has jitter.  I'm using production Firefox 55.0.1 now.  Once I plug in the laptop, even in the middle of YouTube video playback, the jitter goes completely away.  I have the following settings:

Intel HD Graphics Control Panel

On Battery:
- Power Plan: Maximum Battery Life
- Display Power Saving Mode: Enable
- Bar set to one notch to the right of Maximum Quality
- Enhanced Power Saving Mode: Enable
- Extended Battery Life for Gaming: Enable

My Windows Power Plan

- Multimedia Settings
-- Video playback quality bias
--- On battery: Video playback power-saving bias
-- When playing video
--- On battery: Optimize power savings

I still think this issue is worth looking into since with Edge and Chrome, using battery and the settings I have above, there is no playback jitter whatsoever.

          You need to log in
          before you can comment on or make changes to this bug.
        






  

    
  







  

    

      
Attachment

      

      
      
      
      
    

    

      

        

          

            
General

            

              Creator: 



            

            
Created: 

            
Updated: 

            
Size: