1630569 - Make muting of videos of playback speed higher than 4x optional

Status: VERIFIED FIXED

(Core :: Audio/Video: Playback, enhancement, P3)

Description

[sirati]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:75.0) Gecko/20100101 Firefox/75.0

Steps to reproduce:

Install the plugin https://addons.mozilla.org/en-US/firefox/addon/videospeed/ or manually set the playback speed of a video e.g. youtube higher than 4x

Actual results:

The sound get muted automatically

Expected results:

Muting the audio should be left to the website programmer and or the user. I suggest adding a config option to disable this restriction or disabling it outright.

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

Bugbug thinks this bug should belong to this component, but please revert this change in case of error.

Comment 2

[Firefox Bug Husbandry Bot]

Resetting severity to default of --.

Comment 3

[Alastor Wu [:alwu]]

Per [1], we will mute audio when the playback rate is beyond/under our threshold because the audio library we used doesn't do well on those range of playback rate.

[1] https://searchfox.org/mozilla-central/rev/3446310d6cc5c85cde16a82eccf560e9b71a3d44/dom/html/HTMLMediaElement.cpp#172-177

Comment 4

[braham.snyder]

Chromium solved a similar issue with their audio by switching algorithms [1], now allowing much faster audio playback, which is often very useful.

The mentioned videospeed extension has 80k users already, and the chrome version has 2 million. There is already interest in yet faster audio [2]. I think that interest will continue to increase as more people use such extensions, and use them more often -- especially kids, whom I bet will learn to prefer faster and faster speeds for a while.

I'm guessing the code is nontrivial, but maybe this is something to keep in mind at least?

[1] https://developers.google.com/web/updates/2017/12/chrome-63-64-media-updates#remove-muting-extreme-playbackrates
[2] https://github.com/codebicycle/videospeed/issues?q=4x+audio

Comment 5

[jtagcat]

prefer faster and faster speeds

Going to plug accessibility, screen reader users tend to go way beyond 4x.

Comment 6

[wofwca]

This is also an obstacle for these extensions: https://github.com/WofWca/jumpcutter. It speed up only silent parts and high-quality pitch-shifting is not really a requirement when it's silent anyway.

Also when video.preservesPitch = true

Comment 7

[wofwca]

I mean
Also when video.preservesPitch = true (although AFAIK this feature is somewhat abandoned) it doesn't really make sense to mute audio. And my extension is happy with analyzing non-pitch-shifted sped-up audio.

The other extension I meant was https://github.com/vantezzen/skip-silence.

Comment 8

[Alastor Wu [:alwu]]

It sounds like we should revisit this enhancement, otherwise it would cause compatibility issue.

Comment 9

[Paul Adenot (:padenot)]

Yeah we really should have a better algorithm here, and/or tune it better. WSOLA is likely a good candidate.

Ideally we would get an implementation in safe Rust, that would be really nice.

Comment 10

[sirati]

(In reply to Paul Adenot (:padenot) from comment #9)

Yeah we really should have a better algorithm here, and/or tune it better. WSOLA is likely a good candidate.

Please make sure to test algorithms with users of fast audio before deciding. I do not know that algorithms the respective products use, but they do have major impact on understanding. I have used sped up audio on Firefox Win10, Firefox Android, Chrome Win10 and VLC

Of these I can say that from the point of understandability the current algorithm of Firefox is the best. Personally I can speed up Video on these products to the following speed and still understand if I understand at 1x speed:
Firefox Win10: 4x possibly even higher
Chrome Win10: 3.2x
VLC: 2.9x
Firefox Android: 2.3x

I personally value preserving understandability more than conserving correct pitch at high speeds.

PS: based on that i like the current firefox algorithm by far the best. I would like to hear how it performs at speed >4x where according to @Alestor Wu the current library does not do well, as I do not know by which metric this conclusion was formed.

Comment 11

[wofwca]

I believe Chromium uses WSOLA: https://github.com/chromium/chromium/blob/e5a38eddbdf45d7563a00d019debd11b803af1bb/media/filters/audio_renderer_algorithm.cc

Comment 12

[Alastor Wu [:alwu]]

Reopen it in order to discuss this on our triage meeting.

Comment 13

[Jim Mathies [:jimm]]

Added to our roadmapping and will take this to product for consideration.

Comment 14

[Paul Adenot (:padenot)]

Attachment: Bug 1630569 - Change the limit at which audio is muted with high or low playback rates, and make it and other parameters configurable via prefs. r?alwu

Comment 15

[Paul Adenot (:padenot)]

After fiddling with various number, it's clear to me that I'm the wrong person to determine what's acceptable and what's not, so I decided to:

• bump the limit to 8x, i.e., audio is not muted until playbackRate > 8
• make it configurable via a preference change in about:config: media.audio.playbackrate.muting_threshold, with this new default value of 8. The lower limit is the inverse of this number, so with e.g. 8 it's 0.125.
• make various other preferences for the time-stretching algorithm configurable in the same way, I'm quoting SoundTouch's header file for the description:
  • media.audio.playbackrate.soundtouch_sequence_ms, default value 10ms in Firefox: Time-stretch algorithm single processing sequence length in milliseconds. This determines to how long sequences the original sound is chopped in the time-stretch algorithm.
  • media.audio.playbackrate.soundtouch_seekwindow_ms, default value 15ms in Firefox: Time-stretch algorithm seeking window length in milliseconds for algorithm that finds the best possible overlapping location. This determines from how wide window the algorithm may look for an optimal joining location when mixing the sound sequences back together.
  • media.audio.playbackrate.soundtouch_overlap_ms: default value 8ms in Firefox: Time-stretch algorithm overlap length in milliseconds. When the chopped sound sequences are mixed back together, to form a continuous sound stream, this parameter defines over how long period the two consecutive sequences are let to overlap each other.

The algorithm configuration is the same for now, it's just configurable and doesn't mute until 8x. Let us know if better values suite some use-cases better !

Comment 16

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/2129a7c7739b
Change the limit at which audio is muted with high or low playback rates, and make it and other parameters configurable via prefs. r=alwu

Comment 17

[Alexandru Michis [:malexandru]]

https://hg.mozilla.org/mozilla-central/rev/2129a7c7739b

Comment 19

[Justin Peter]

Thank you! I've verified that this works on speeds (slightly) above 4x.

Comment 20

[Giorgia Nichita, Release Desktop QA]

Reproduced this issue on Windows 10 x64 using an affected Nightly build, 77.0a1 (20200416094846) and following the STR from Comment 0.

The issue is verified as fixed on latest Beta 97.0b9 (20220127193706) across the following platforms: Windows 10 x64, Ubuntu 21.04 and macOS 10.14. The sound does not get muted anymore if you set the playback speed of a video higher than 4x.