2006276 - Video playback stalls or desynchronizes when resuming if an AudioContext-based volume booster extension is active

Status: VERIFIED FIXED

(Core :: Audio/Video: Playback, defect, P1)

Description

[knoa]

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:147.0) Gecko/20100101 Firefox/147.0

Steps to reproduce:

1. Launch Firefox (I have reproduced this on 147 Beta and 148 Nightly).
2. Install a volume booster add-on or userscript that likely utilizes the AudioContext (Web Audio API).
   Examples include:
   • https://addons.mozilla.org/ja/firefox/addon/600-sound-volume/
   • https://addons.mozilla.org/ja/firefox/addon/volume-control-boost-volume/
   • https://greasyfork.org/en/scripts/549531-youtube-volume-curve-designer/code
3. Visit a video streaming site (e.g., YouTube).
   (Note: If using the "600% Sound Volume" add-on, adjust the slider to activate the effect.)
4. Play a video.
5. Pause the video.
6. Resume playback.

Actual results:

The video playback glitches for several seconds. Specifically, one of the following occurs:

1. Audio plays, but the video frame remains frozen. After several seconds, the video frame suddenly jumps to the current audio position (skipping the frames in between).
2. Both audio and video remain frozen, but the seekbar/timestamp continues to progress. After several seconds, playback starts normally from the current timestamp.

Expected results:

Playback should resume immediately and smoothly without any freezing or resynchronization.

Additional Information

• Regression: The issue appears to be a regression starting with Firefox 147 Beta.
  • Works correctly: Firefox 146 (Release and Beta).
  • Broken: Firefox 147 Beta and 148 Nightly.

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

The Bugbug bot thinks this bug should belong to the 'Core::Audio/Video: Playback' component, and is moving the bug to that component. Please correct in case you think the bot is wrong.

Comment 2

[Karl Tomlinson (:karlt)]

Thank you for the report. Would you be able to use mozregression to bisect to find the change the triggered this, please?
mozregression --good 146 --bad 147 --profile-persistence reuse

Comment 3

[knoa]

Apologies for the delay. This was my first time using mozregression (it's a great tool!), so it took me a while to get set up.

I ran mozregression, but it failed to bisect the merge commit (Unable to exploit the merge commit).

However, the regression range is identified as:
https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=8b434e72bd1e5995992c79398c5d4024c21383d4&tochange=88a09a47e50b681f0b673d5314113e2992f468e3

Looking at the pushlog, Bug 1771789 seems highly suspicious, as it involves changes to CubebInputStream and MediaStreamTrack, which could affect AudioContext behavior.

Suspected patches:

• Andreas Pehrson — Bug 1771789 - Pass CubebInputStream the same CubebHandle used to enumerate its device id. r=padenot
• Andreas Pehrson — Bug 1771789 - Make AudioInputProcessing::NumberOfChannels always valid. r=padenot,webrtc-reviewers,mjf
• Andreas Pehrson — Bug 1771789 - Add and use an API for MediaStreamTrackSource cloning. r=jib

Comment 4

[Karl Tomlinson (:karlt)]

That's very helpful, thanks. Possibly bug 1771789, but bug 1931328 is about A/V synchronization.

Comment 5

[Donal Meehan [:dmeehan]]

Next week is RC week for Fx147.
:jimm, any concerns with prioritizing an investigation in time for Fx147?
Bug 1998839 is riding the train with Fx147

Comment 6

[Karl Tomlinson (:karlt)]

Attachment: testcase

It is as if the video position is advancing while paused, so the effect is more obvious for longer pauses. On resume the audio resumes, but the video playback jumps and then is paused or stuttery until the audio catches up.

Comment 7

[Karl Tomlinson (:karlt)]

https://hg.mozilla.org/mozilla-central/rev/6151857e7ed8, which re-landed as https://hg.mozilla.org/mozilla-central/rev/f8c71c831710, changed the behavior.

Comment 8

[Karl Tomlinson (:karlt)]

A logging profile is at https://share.firefox.dev/4jkeCYl

Looking at the MediaDecoderStateMachine thread, and searching the Marker Table for "odecoded,sink", both audio and video decoded frame times advance by the duration of the pause, but aClockTime passed to VideoSink::RenderVideoFrames() does not.

This bug would affect more situations that bug 1931328.
There would be risk in landing a fix for this, if we determine what the appropriate fix should be.
Given the time constraints, backing out https://hg.mozilla.org/mozilla-central/rev/f8c71c831710 and dependent changesets from bug 1931328 seems the best option at this stage.

https://treeherder.mozilla.org/jobs?repo=try&revision=2cdc18b26bcb764466a8b752656eb0886a2e3954

Comment 9

[Karl Tomlinson (:karlt)]

Attachment: Bug 2006276 Revert DecodedStream position interpolation and dependent changesets r?#media-playback-reviewers

due to skips and freezes when resuming after playback pause.

Revert "Bug 1931328 - Account for playback rate when setting frame timestamps in DecodedStream. r=padenot"

This reverts commit 5df6a99098033eac5a05134437d4393200c8652b.

Revert "Bug 1931328 - Test the interpolated GetPosition in DecodedStream. r=padenot"

This reverts commit a8c2ffa3f7c13ce6533bc06c5bbe7ecb0bc7f90f.

Revert "Bug 1931328 - Allow for deterministic output time testing of DecodedStream. r=padenot"

This reverts commit 1a970ccb483397c434bd9f2a3d566d72b2444c82.

Revert "Bug 1931328 - Interpolate DecodedStream position based on current system time. r=padenot,webidl,smaug"

This reverts commit 6bc5f33ade36f71c360721f04c77744d662ace1b.

Comment 10

[Donal Meehan [:dmeehan]]

:karlt could you also put up a beta uplift request when ready?
I know the patch is still pending review, but at least if we have a beta uplift we can take it once the patch hits firefox-main.
It's too late for 147 beta builds, but we can take it before 147 goes to release.

Comment 11

[Karl Tomlinson (:karlt)]

An awkward workaround is to drag the playback position back over the portion that was skipped.

Comment 12

[Karl Tomlinson (:karlt)]

Attachment: Bug 2006276 Revert DecodedStream position interpolation and dependent changesets

due to skips and freezes when resuming after playback pause.

Revert "Bug 1931328 - Account for playback rate when setting frame timestamps in DecodedStream. r=padenot"

This reverts commit 5df6a99098033eac5a05134437d4393200c8652b.

Revert "Bug 1931328 - Test the interpolated GetPosition in DecodedStream. r=padenot"

This reverts commit a8c2ffa3f7c13ce6533bc06c5bbe7ecb0bc7f90f.

Revert "Bug 1931328 - Allow for deterministic output time testing of DecodedStream. r=padenot"

This reverts commit 1a970ccb483397c434bd9f2a3d566d72b2444c82.

Revert "Bug 1931328 - Interpolate DecodedStream position based on current system time. r=padenot,webidl,smaug"

This reverts commit 6bc5f33ade36f71c360721f04c77744d662ace1b.

Original Revision: https://phabricator.services.mozilla.com/D277683

Comment 13

[Phabricator Automation]

firefox-beta Uplift Approval Request

• User impact if declined: Regression: skips and freezes when attempting to resume playback after pause, in some less common a/v playback designs.
• Code covered by automated testing: no
• Fix verified in Nightly: no
• Needs manual QE test: no
• Steps to reproduce for manual QE testing:
• Risk associated with taking this patch: low
• Explanation of risk level: Reverts to a previous state.
• String changes made/needed: None
• Is Android affected?: yes

Comment 14

[Karl Tomlinson (:karlt)]

The uplift request via Lando for D277683 fails with "source_revisions: Select a valid choice. 277683 is not one of the available choices."
moz-phab generated the uplift.

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/a8abdd1ff460
https://hg.mozilla.org/integration/autoland/rev/87b989c9180d
Revert DecodedStream position interpolation and dependent changesets r=media-playback-reviewers,webidl,smaug,aosmond

Comment 16

[Atila Butkovits]

https://hg.mozilla.org/mozilla-central/rev/87b989c9180d

Comment 17

[Pulsebot]

https://github.com/mozilla-firefox/firefox/commit/a29f5889051a
https://hg.mozilla.org/releases/mozilla-beta/rev/d192fff06714

Comment 18

[knoa]

Verified fixed in the latest Nightly 148.0a1 on macOS.
The video glitch/desynchronization issue is resolved.

Comment 19

[Monica Chiorean]

I was able to reproduce the issue on Mac12.6 using FF build 147.0b6.
Verified as fixed on Mac12.6/Win11 using FF builds 147.0 and 148.0a1(20260107160934).