1133633 - Poor video decoding performance on mac when using VideoToolbox

Status: RESOLVED FIXED

(Core :: Audio/Video, defect, P2)

Description

[Jean-Yves Avenard [:jya]]

There appears to be severe performance issue with the Apple VideoToolbox decoder even with hardware acceleration is enabled.

A 4K YouTube videos show lots of dropped frame when using VideoToolBox with Intel GPU.

On a late-2013 macbook air (Intel HD5000), the average time it takes to decode a 4K frame is:
- Software decode: 95ms average, max 469ms
- VideoToolbox with HW acceleration: 31ms average, max 326ms
- VDA (pure HW acceleration): .63ms average, max 97ms

To not have dropped frames, we must decode in less than 30ms a 4K @ 30Hz. As it is the VideoToolbox decoder is borderline in the average case, and way over at its worse.

This patch track the attempt to prefer the VDA decoder for higher resolution video.

Comment 1

[Jean-Yves Avenard [:jya]]

Attachment: Part1. Add logging reporting if we are using HW accelerated decode

Add detection if decoding is hardware accelerated and related log

Comment 2

[Ralph Giles (:rillian)]

Comment on attachment 8565776 [details] [diff] [review]
Part1. Add logging reporting if we are using HW accelerated decode

Review of attachment 8565776 [details] [diff] [review]:
-----------------------------------------------------------------

r=me with question addressed.

::: dom/media/fmp4/apple/AppleVTDecoder.cpp
@@ +317,5 @@
> +    LOG("AppleVTDecoder: %s hardware accelerated decoding",
> +        (rv == noErr && isUsingHW == kCFBooleanTrue) ? "using" : "not using");
> +  } else {
> +    LOG("AppleVTDecoder: using software decoding "
> +        "(platform doesn't support HW acceleration)");

Is this true? Do we know MacOS X 10.8 and earlier won't use hardware acceleration, despite the property key not being defined in the API?

Comment 3

[Jean-Yves Avenard [:jya]]

As far as I can say, is that there's no way to tell.
Even the doc about kVTVideoDecoderSpecification_EnableHardwareAcceleratedVideoDecoder are confusing at best as the only guaranteed behaviour is when you set it to false, in which case HW acceleration will be disabled.

It is suggested that the VT decoder will, when it's true or not set decide what to use and when.
However, reports like bug 1132994, indicates to me that unless it's set to true, we won't get HW acceleration (as that's the only difference between 35 and 36)

On my mac pro, while the key exist, I can't query it, and when requesting all VTSession keys anything related to HW is missing (and VDA always return an error)

Comment 4

[Ralph Giles (:rillian)]

Who knows then. I just wanted to make sure we didn't leave a confusing message in the log. How about "Couldn't determine hardware acceleration status."?

Comment 5

[Jean-Yves Avenard [:jya]]

remote:   https://hg.mozilla.org/integration/mozilla-inbound/rev/5d68cdc80ddf

Comment 6

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/5d68cdc80ddf

Comment 7

[Ralph Giles (:rillian)]

Comment on attachment 8565776 [details] [diff] [review]
Part1. Add logging reporting if we are using HW accelerated decode

Requestint 36 uplift of teh initial logging patch.

Approval Request Comment
[Feature/regressing bug #]: MSE and mp4 playback on Mac.
[User impact if declined]: Slow video decoding on some systems.
[Describe test coverage new/current, TreeHerder]: Landed on m-c.
[Risks and why]: This just adds logging plumbing. Risk should be low.
[String/UUID change made/needed]: None.

Comment 8

[Lawrence Mandel [:lmandel] (use needinfo)]

Comment on attachment 8565776 [details] [diff] [review]
Part1. Add logging reporting if we are using HW accelerated decode

Looks like this may do a bit more than just logging but we definitely want the change in 37 Beta 1 in which we're testing MSE on OSX. 37 is now Beta so clearing Aurora request and approving for Beta.

Beta+

Comment 9

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-beta/rev/85e3bc280be6

Comment 10

[Jean-Yves Avenard [:jya]]

Attachment: Part2. Enable async decoding on mac

enable async decoding on apple VT.

Comment 11

[Jean-Yves Avenard [:jya]]

remote:   https://hg.mozilla.org/integration/mozilla-inbound/rev/a05f09431d9c

Comment 12

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/a05f09431d9c

Comment 13

[Jean-Yves Avenard [:jya]]

Comment on attachment 8585883 [details] [diff] [review]
Part2. Enable async decoding on mac

Approval Request Comment
[Feature/regressing bug #]: 1133633
[User impact if declined]: stuttering playback, dropped frames
[Describe test coverage new/current, TreeHerder]: This patch significantly improve the performance of the decoder on mac, a 4K video will go from 60% dropped frames to 0%. Patch will have to be rebase for beta.
[Risks and why]: Low.
[String/UUID change made/needed]: None

Comment 14

[Sylvestre Ledru [:Sylvestre]]

Jean-Yves, are you planning to provide the beta patch?

Comment 15

[Jean-Yves Avenard [:jya]]

Comment on attachment 8585883 [details] [diff] [review]
Part2. Enable async decoding on mac

I remove the request for beta uplift.

MSE isn't enabled on mac in beta, so let's urgent there.

Comment 16

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-aurora/rev/8ddb0e3d862d

Comment 17

[Jean-Yves Avenard [:jya]]

Comment on attachment 8585883 [details] [diff] [review]
Part2. Enable async decoding on mac

We may enable MSE in 38.. So we would want that in as well.

Comment 18

[Sylvestre Ledru [:Sylvestre]]

Comment on attachment 8585883 [details] [diff] [review]
Part2. Enable async decoding on mac

Taking it in case we decide to do something with MSE under Mac.
Should be in 38 beta 4.

Comment 19

[Jean-Yves Avenard [:jya]]

remote:   https://hg.mozilla.org/releases/mozilla-beta/rev/10f75583d21a