1141139 - Enable low latency decoding for MFTDecoder

Status: RESOLVED FIXED

(Core :: Audio/Video, defect)

Description

[schoenefeld]

User Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.76 Safari/537.36

Steps to reproduce:

From testing the Media Source Extension in firefox trunk with:
media.mediasource.enabled;true
media.mediasource.mp4.enabled;true
media.mediasource.youtubeonly;false

Playing a live h.264/mp.4 video stream using Media Source Extension (MSE), latency on Win7 and Win8 is very high (>1s). This is a known behavior of the MFT/DXVA decoder used on Windows, where it always buffers frames for 1 second before returning the first decoded frame.


Actual results:

The high latency makes use of MSE with mp4 unusable for low latency use-cases like webchat, desktop remoting...


Expected results:

Using the CODECAPI_AVLowLatencyMode flag when initializing the decoder will buffer less (0 frames on Win8, 4-10 frames on Win7), making it usable for low latency live stream decoding. 

Note that while the AVLowLatencyMode documentation says that it will only work on >=Win8 ,the flag will also have a positive effect on Windows 7, as long as SP2 is installed.

For reference: The AVLowLatencyMode flag has also been recently added to Chrome Canary build (dxva_video_decoder_accelerator.cc:887)

Comment 1

[Jean-Yves Avenard [:jya]]

Yes, I saw the video IMF filter sometimes buffer in excess of 20 frames before returning anything

Comment 2

[Mason Chang [Inactive] [:mchang]]

Just a note in reference to comment 0. This seems to be only valid on Windows 8 and Windows 7 SP 2 doesn't exist (http://windows.microsoft.com/en-US/windows/service-packs-download#sptabs=win7).

Maybe a typo and he meant SP 1?

Comment 3

[Mason Chang [Inactive] [:mchang]]

I am able to set CODECAPI_AVDecNumWorkerThreads, which is also reported as Windows 8 only, on Windows 7 SP 1. When setting CODECAPI_AVLowLatencyMode, I actually get an S_OK return code if I add a SetUINT32 at [1]. Also, I see from Visual Studio Concurrency Analyzer that the decoder threads are getting scheduled in short frequent bursts versus before I would see long decodes with 50-100 ms between each decode. So it seems to work on Windows 7 SP 1.

[1] http://mxr.mozilla.org/mozilla-central/source/dom/media/platforms/wmf/WMFVideoMFTManager.cpp#206

Comment 4

[Mason Chang [Inactive] [:mchang]]

Also, Chromium just recently enabled this - https://groups.google.com/a/chromium.org/forum/#!topic/feature-media-reviews/3fFal9GVhRg

Comment 5

[Jean-Yves Avenard [:jya]]

Attachment: Enable low latency decoding on Windows. r=cpearce

Comment 6

[Jean-Yves Avenard [:jya]]

Comment on attachment 8644756 [details] [diff] [review]
Enable low latency decoding on Windows. r=cpearce

Will do thorough testing before pushing...

Comment 7

[Jean-Yves Avenard [:jya]]

appears to work very well, we pass all the tests now (with a couple of adjustments on the test: e.g. properly calling endOfStream())

Comment 8

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/ac350571e7b8

Comment 9

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/ac350571e7b8

Comment 10

[Jean-Yves Avenard [:jya]]

Unfortunately, it appears that win7 try machines only have SP1 installed ; as such we still fail the test for bug 1180935.

Back to the drawing board.

Comment 11

[Virtual_ManPL [:Virtual] 🇵🇱 - (please needinfo? me - so I will see your comment/reply/question/etc.)]

SP1 is the latest SP for Win7.

Comment 12

[Jean-Yves Avenard [:jya]]

I went by the original description mentioning SP2.

So try machines don't have any SP installed then.

I verified that with SP1 it works as expected. The decoder output a frame after 10 Frames as opposed to 25+ without

Comment 13

[Chris Pearce [:cpearce (Not reading bugmail)]]

We've had a surge of crashes in msmpeg2vdec.dll since this landed.

https://crash-stats.mozilla.com/search/?product=Firefox&signature=~msmpeg2vdec&_facets=signature&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#facet-signature

So in bug 1205083 I'm proposing we remove enabling low latency video decoding.

Comment 14

[Danial Horton]

I believe the crashes mentioned previously have actually been fixed elsewhere in Firefox somewhere along the line, Low Latency Decoding should be given another shot.

Comment 15

[Jean-Yves Avenard [:jya]]

And it would be relying on an undefined behaviour of the decoder.
Per Microsoft documentation, it can't do content with B-frame.. and yet it does (and Chrome uses it)...

Comment 16

[Danial Horton]

Chrome apparently doesn't play nice with B Frames as per https://bugs.chromium.org/p/chromium/issues/detail?id=703019

Surely the decoder could fall back in the presence of b frames to the standard decoder, or only use low latency for live streaming.

I've heard of a few sites refraining from switching to full html5 players because of the latency differences between various web browsers, which isn't helping to kill Flash faster.

Comment 17

[Jean-Yves Avenard [:jya]]

Hmmmm, that bug is more about the stream having wrong pts set. You would see the same kind of jittery playback in Firefox on macos...