1846546 - Only use low-latency mode in decoders when it's useful to do so

Status: REOPENED

(Core :: Audio/Video: Playback, defect)

Description

[Paul Adenot (:padenot)]

From bug 1520894:

this is something we should do. This will fix the issue for non-live Youtube and others, but there is still a bug here, that we should fix.

It's only useful to use low latency decoder when playing a live source. This can save a lot of resources, and it can be more resilient against load (depending on the situation).

Comment 1

[Paul Adenot (:padenot)]

Attachment: Bug 1846546 - Determine if an mp4 or a WebM is live, and pass that information to the decoder. r?alwu

Comment 2

[Paul Adenot (:padenot)]

Attachment: Bug 1846546 - Only enable low-latency video decoding mode on Windows when the media is live. r?alwu

Depends on D185053

Comment 3

[Alastor Wu [:alwu]]

I'm still kinda worry about enabling low latency only when media is live, especially consider now we don't have any flag in the API to force media element using low latency. In bug 1520894, it's about the low latency didn't work well on web gaming, which is not live video so might have non-infinite duration. If we only turn the low latency on when the duration is not set, then all similar websites using low latency for other purposes might have worse performance. But having non-live video with low latency would cause higher CPU/GPU usage is also not something we want. So maybe we can use a pref to control this feature? In case that in some situation we still want to use low latency on non-live video?

By checking Chromium's code, even if they set live flag, but I can't see that flag would directly affect [1] whether using CODECAPI_AVLowLatencyMode or not. In addition, based on [2], CODECAPI_AVLowLatencyMode seems only affecting hardware decoding even if mdsn doesn't explicitly mention that.

[1]
https://source.chromium.org/search?q=StreamLiveness::kLive&sq=&ss=chromium%2Fchromium%2Fsrc

[2]
https://github.com/w3c/media-source/issues/184#issuecomment-309054314
https://source.chromium.org/chromium/chromium/src/+/main:gpu/config/gpu_preferences.h;l=115-116?q=CODECAPI_AVLowLatencyMode&ss=chromium

Comment 4

[Paul Adenot (:padenot)]

(In reply to Alastor Wu [:alwu] from comment #3)

I'm still kinda worry about enabling low latency only when media is live, especially consider now we don't have any flag in the API to force media element using low latency. In bug 1520894, it's about the low latency didn't work well on web gaming, which is not live video so might have non-infinite duration.

Web Gaming (here, streamed) very much is live video, how could it have a duration?

If we only turn the low latency on when the duration is not set, then all similar websites using low latency for other purposes might have worse performance. But having non-live video with low latency would cause higher CPU/GPU usage is also not something we want. So maybe we can use a pref to control this feature? In case that in some situation we still want to use low latency on non-live video?

In the absence of a dedicated API to select low-latency decoder, we don't have a choice, so we do a guess based on the container.

By checking Chromium's code, even if they set live flag, but I can't see that flag would directly affect [1] whether using CODECAPI_AVLowLatencyMode or not. In addition, based on [2], CODECAPI_AVLowLatencyMode seems only affecting hardware decoding even if mdsn doesn't explicitly mention that.

It changes their internal buffering, they don't set it on the decoder. We can do both.

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/559a73043ec5
Determine if an mp4 or a WebM is live, and pass that information to the decoder. r=alwu
https://hg.mozilla.org/integration/autoland/rev/c4ff418eb757
Only enable low-latency video decoding mode on Windows when the media is live. r=alwu
https://hg.mozilla.org/integration/autoland/rev/4dc0830ae686
apply code formatting via Lando

Comment 6

[Cristian Tuns]

Backed out for causing build bustages in WebMDemuxer.cpp

• Backout link
• Push with failures
• Failure Log
• Failure line: /builds/worker/checkouts/gecko/dom/media/webm/WebMDemuxer.cpp:392:24: error: no member named 'media_low_latency_decoding_force_enabled' in namespace 'mozilla::StaticPrefs'

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/dc904b668c03
Determine if an mp4 or a WebM is live, and pass that information to the decoder. r=alwu
https://hg.mozilla.org/integration/autoland/rev/a546ce2c8996
Only enable low-latency video decoding mode on Windows when the media is live. r=alwu

Comment 8

[Cosmin Sabou [:CosminS]]

Backed out for causing assertion failures on mozilla/Casting.h.

Push with failures: https://treeherder.mozilla.org/jobs?repo=autoland&selectedTaskRun=DuNupdkESDOOY8cjB53QnA.0&resultStatus=testfailed%2Cbusted%2Cexception%2Cusercancel&revision=a546ce2c8996c24628b783748a63ec0d39b9ec2f

Failure log: https://treeherder.mozilla.org/logviewer?job_id=426322336&repo=autoland

Backout link: https://hg.mozilla.org/integration/autoland/rev/9e74806972fa877484f3ddba318642071635db72

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/0ca91e7b51ff
Determine if an mp4 or a WebM is live, and pass that information to the decoder. r=alwu
https://hg.mozilla.org/integration/autoland/rev/aac1f1d71d52
Only enable low-latency video decoding mode on Windows when the media is live. r=alwu

Comment 10

[Serban Stanca [:SerbanS]]

https://hg.mozilla.org/mozilla-central/rev/0ca91e7b51ff
https://hg.mozilla.org/mozilla-central/rev/aac1f1d71d52

Comment 11

[Cristian Tuns]

Backed out for causing Bug 1849396

• Backout link

Comment 12

[Iulian Moraru]

Backout merged to central: https://hg.mozilla.org/mozilla-central/rev/219515fc9b06

Comment 13

[Alexandru Ionescu (needinfo me) [:alexandrui]]

(In reply to Pulsebot from comment #7)

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/dc904b668c03
Determine if an mp4 or a WebM is live, and pass that information to the
decoder. r=alwu
https://hg.mozilla.org/integration/autoland/rev/a546ce2c8996
Only enable low-latency video decoding mode on Windows when the media is
live. r=alwu

== Change summary for alert #39355 (as of Sat, 19 Aug 2023 14:32:24 GMT) ==

Improvements:

Ratio	Test	Platform	Options	Absolute values (old vs new)
17%	basic_compositor_video	windows10-64-qr	e10s fission stylo webrender-sw	3.35 -> 2.79
17%	basic_compositor_video	windows10-64-shippable-qr	e10s fission stylo webrender	3.35 -> 2.79
17%	basic_compositor_video	windows10-64-shippable-qr	e10s fission stylo webrender-sw	3.35 -> 2.79

For up to date results, see: https://treeherder.mozilla.org/perfherder/alerts?id=39355

Comment 14

[Alexandru Ionescu (needinfo me) [:alexandrui]]

(In reply to Pulsebot from comment #9)

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/0ca91e7b51ff
Determine if an mp4 or a WebM is live, and pass that information to the
decoder. r=alwu
https://hg.mozilla.org/integration/autoland/rev/aac1f1d71d52
Only enable low-latency video decoding mode on Windows when the media is
live. r=alwu

== Change summary for alert #39357 (as of Mon, 21 Aug 2023 04:41:28 GMT) ==

Improvements:

Ratio	Test	Platform	Options	Absolute values (old vs new)
18%	basic_compositor_video	windows10-64-qr	e10s fission stylo webrender-sw	3.35 -> 2.74
18%	basic_compositor_video	windows10-64-shippable-qr	e10s fission stylo webrender	3.34 -> 2.74

For up to date results, see: https://treeherder.mozilla.org/perfherder/alerts?id=39357

Comment 15

[Alexandru Ionescu (needinfo me) [:alexandrui]]

(In reply to Cristian Tuns from comment #11)

Backed out for causing Bug 1849396

• Backout link

== Change summary for alert #39371 (as of Wed, 23 Aug 2023 09:37:13 GMT) ==

Regressions:

Ratio	Test	Platform	Options	Absolute values (old vs new)
22%	basic_compositor_video	windows10-64-qr	e10s fission stylo webrender-sw	2.74 -> 3.36
22%	basic_compositor_video	windows10-64-shippable-qr	e10s fission stylo webrender	2.74 -> 3.34
20%	basic_compositor_video	windows10-64-shippable-qr	e10s fission stylo webrender-sw	2.79 -> 3.34

For up to date results, see: https://treeherder.mozilla.org/perfherder/alerts?id=39371

Comment 16

[BugBot [:suhaib / :marco/ :calixte]]

There are some r+ patches which didn't land and no activity in this bug for 2 weeks.
:padenot, could you have a look please?
If you still have some work to do, you can add an action "Plan Changes" in Phabricator.
For more information, please visit BugBot documentation.

Comment 17

[Alastor Wu [:alwu]]

Yes, we still need more changes for patches in order to figure out the solution for breaking H264 playback.