1905878 - VP9 playback stalls

Status: RESOLVED FIXED

(Core :: Audio/Video: Playback, defect, P2)

Description

[Jake Archibald [:jakea]]

Attachment: screen.webm

Steps to reproduce:

https://static-misc-3.glitch.me/alpha-video-test/vp9-video.html

Actual results:

The video plays and loops as expected on Android Chrome, but on Android Firefox the video freezes (tested on a Pixel 8 pro).

(note: the video displays incorrectly in Chrome due to a different issue https://issues.chromium.org/issues/349610465)

Expected results:

The video should not freeze.

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

The severity field is not set for this bug.
:jimm, could you have a look please?

For more information, please visit BugBot documentation.

Comment 2

[Paul Adenot (:padenot)]

https://share.firefox.dev/3AHrHJ5 is a profile of this by alwu, we can repro on all devices because this falls back to software decoding, and the decoding is annoyingly too close to the frame budget. Surely we can make the queue a bit deeper a smooth out the slowness.

Comment 3

[Paul Adenot (:padenot)]

This now works (e.g. on my Pixel 4 that is starting to show its age), likely thanks to Karl's recent work on video frame scheduling. Alastor, can you quickly check on the same device you used in comment 2? It was a Pixel 6.

Comment 4

[Alastor Wu [:alwu]]

It's still same for me, many video frames were dropped during playback. Another profile on Nightly 135.

In addition, shouldn't we use hw vp9 decoder on Android? it was still sw decoding in my both profiles, and I wonder if it's related with bug 1931941. John, could you provide some thought here? Thanks!

Comment 5

[Paul Adenot (:padenot)]

Hrm I updated my personal device to a Samsung S23, and now the video doesn't play for me...

Comment 6

[Jim Mathies [:jimm]]

We are having some trouble reproducing this.

Comment 7

[John Lin [:jhlin][:jolin]]

(In reply to Alastor Wu [:alwu] from comment #4)

It's still same for me, many video frames were dropped during playback. Another profile on Nightly 135.

In addition, shouldn't we use hw vp9 decoder on Android? it was still sw decoding in my both profiles, and I wonder if it's related with bug 1931941. John, could you provide some thought here? Thanks!

I think it falls back to sw decoding because AndroidDecoderModule explicitly rejects video with alpha.

The WebM has no audio track so I don't think our policy of dropping video frames to maintain a/v sync really matter much here. Maybe we can add an option to VideoSink to ignore the mock audio/system clock so it won't drop frames in scenario like this?

Comment 8

[John Lin [:jhlin][:jolin]]

Attachment: Bug 1905878 - increase default video queue max length on Android.

The default size was picked a long time ago and the Android system
has relaxed the limitation. This also mitigates frame-dropping issue
caused by SW/slow decoder with deeper queue.

Comment 9

[John Lin [:jhlin][:jolin]]

Attachment: Bug 1905878 - don't use SW VPX MediaCodec decoders.

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/1519a0346a0a
https://hg.mozilla.org/integration/autoland/rev/0735b01aaa22
don't use SW VPX MediaCodec decoders. r=media-playback-reviewers,alwu
https://github.com/mozilla-firefox/firefox/commit/6ccc0315b9f7
https://hg.mozilla.org/integration/autoland/rev/30d1c2d35eb7
increase default video queue max length on Android. r=geckoview-reviewers,nalexander,sotaro

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/a6fda1786121
https://hg.mozilla.org/integration/autoland/rev/cde4e2c0d669
Revert "Bug 1905878 - increase default video queue max length on Android. r=geckoview-reviewers,nalexander,sotaro" for causing mda failures at test_seamless_looping_cancel_looping_future_frames.html

Comment 12

[Cristina Horotan [:chorotan]]

Backed out for causing mda failures at test_seamless_looping_cancel_looping_future_frames.htm

Backout link

Push with failures

Failure log
Failure log wpt

Comment 13

[John Lin [:jhlin][:jolin]]

Attachment: Bug 1905878 - p2: test decoding properties.

Comment 14

[John Lin [:jhlin][:jolin]]

Attachment: Bug 1905878 - p1: let MediaDataDecoder define video decoding related properties.

MediaDecoderStateMachine uses global/static values for video queue sizes
(https://searchfox.org/firefox-main/rev/4e8871487a9f97992e00d5c8105ef1d8cf3c88f3/dom/media/MediaDecoderStateMachine.cpp#152-165),
which are actually constraints of platform decoders. In some cases the same content
process could have more than one MDSMs that uses different platform decoders (e.g.
on Android one video can be decoded by MediaCodec but another by libvpx) and causes
suboptimal results. Moving the definition to PlatformDecoderModule for each
MDSM to get its own value ensures playback smoothness.

Comment 15

[John Lin [:jhlin][:jolin]]

Attachment: Bug 1905878 - p2: test decoding properties.

Comment 16

[John Lin [:jhlin][:jolin]]

Attachment: Bug 1905878 - p3: Use decode property when available.

Comment 17

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/4ee3b55e4f10
https://hg.mozilla.org/integration/autoland/rev/4b009af5c10f
don't use SW VPX MediaCodec decoders. r=media-playback-reviewers,alwu
https://github.com/mozilla-firefox/firefox/commit/dbecb0c4758d
https://hg.mozilla.org/integration/autoland/rev/6444bcb100b5
p1: let MediaDataDecoder define video decoding related properties. r=media-playback-reviewers,chunmin
https://github.com/mozilla-firefox/firefox/commit/7248bea6c4fc
https://hg.mozilla.org/integration/autoland/rev/375675280358
p2: test decoding properties. r=media-playback-reviewers,chunmin
https://github.com/mozilla-firefox/firefox/commit/f0c2b93a2345
https://hg.mozilla.org/integration/autoland/rev/3eacaf37016d
p3: Use decode property when available. r=media-playback-reviewers,alwu

Comment 18

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/6576267517eb
https://hg.mozilla.org/integration/autoland/rev/7fd855c62761
Revert "Bug 1905878 - p3: Use decode property when available. r=media-playback-reviewers,alwu" for causing gtest failures on MFTDecoder

Comment 19

[Cosmin Sabou [:CosminS]]

Backed out for causing gtest failures on MFTDecoder.

Push with failures

Failure log

Backout link

Comment 20

[John Lin [:jhlin][:jolin]]

Attachment: Bug 1905878 - initialize WMF video manager on MTA thread.

Current media playbck pipeline setup satisfies the condition
but it's not so when running gtest.

Comment 21

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/b0f0f1c9d2f3
https://hg.mozilla.org/integration/autoland/rev/98ecd0ce7900
don't use SW VPX MediaCodec decoders. r=media-playback-reviewers,alwu
https://github.com/mozilla-firefox/firefox/commit/f92988332aa6
https://hg.mozilla.org/integration/autoland/rev/6f045f9620f0
initialize WMF video manager on MTA thread. r=media-playback-reviewers,alwu
https://github.com/mozilla-firefox/firefox/commit/9562a2882db1
https://hg.mozilla.org/integration/autoland/rev/12c0ae608bd9
p1: let MediaDataDecoder define video decoding related properties. r=media-playback-reviewers,chunmin
https://github.com/mozilla-firefox/firefox/commit/735d657d3b96
https://hg.mozilla.org/integration/autoland/rev/79977273c588
p2: test decoding properties. r=media-playback-reviewers,chunmin
https://github.com/mozilla-firefox/firefox/commit/392a6f0444ce
https://hg.mozilla.org/integration/autoland/rev/79b5af5a718a
p3: Use decode property when available. r=media-playback-reviewers,alwu

Comment 22

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/98ecd0ce7900
https://hg.mozilla.org/mozilla-central/rev/6f045f9620f0
https://hg.mozilla.org/mozilla-central/rev/12c0ae608bd9
https://hg.mozilla.org/mozilla-central/rev/79977273c588
https://hg.mozilla.org/mozilla-central/rev/79b5af5a718a