1872354 - Choppy video playback when looping

Status: RESOLVED FIXED

(Core :: Audio/Video: Playback, defect)

Description

[vlastimir.arnost]

User Agent: Mozilla/5.0 (Android 10; Mobile; rv:122.0) Gecko/122.0 Firefox/122.0

Steps to reproduce:

Watch any video on Twitter or Tiktok.
Device: Galaxy S23

Actual results:

Video plays as expected on the first run, but on the second and succeeding loops, playback is choppy.

Expected results:

Videos should play in a loop without frame drops.

Comment 1

[vlastimir.arnost]

Attachment: screen recording

Comment 2

[Jonathan Almeida [:jonalmeida]]

We've seen other reports of this happening also on youtube when uBlock is enabled, but it seems like uBlock is not the cause here so there is something going on deeper.

vlastimir.arnost, if you can include a link to a performance profile with the 'Media' setting by following these instructions, that would greatly help the team triage and fix the bug. Thanks!

Proactively sending to the media team.

Comment 3

[BugBot [:suhaib / :marco/ :calixte]]

The severity field is not set for this bug.
:jimm, could you have a look please?

For more information, please visit BugBot documentation.

Comment 4

[vlastimir.arnost]

(In reply to Jonathan Almeida [:jonalmeida] from comment #2)

We've seen other reports of this happening also on youtube when uBlock is enabled, but it seems like uBlock is not the cause here so there is something going on deeper.

vlastimir.arnost, if you can include a link to a performance profile with the 'Media' setting by following these instructions, that would greatly help the team triage and fix the bug. Thanks!

Proactively sending to the media team.

https://profiler.firefox.com/public/tnk77x0pq1rk32xpc7bdw17xjpxnwwsqnc048wr

Comment 5

[Jim Mathies [:jimm]]

I'll do some testing on Tiktok. The profile shows regular cycles of gc, but not much cpu utilization. I'm not sure what's going here.

Comment 6

[vlastimir.arnost]

Here's another sample I took using Twitter/X. See attached file 'screen recording twitter'

https://profiler.firefox.com/public/dyh2vzfy0gyhnfwa9rhpx2h8na7mg79zdc34v70

Comment 7

[vlastimir.arnost]

Attachment: screen recording twitter

Comment 8

[Jeff Muizelaar [:jrmuizel]]

(In reply to vlastimir.arnost from comment #6)

Here's another sample I took using Twitter/X. See attached file 'screen recording twitter'

https://profiler.firefox.com/public/dyh2vzfy0gyhnfwa9rhpx2h8na7mg79zdc34v70

This profile shows a bunch VideoSinkDroppedFrame markers on the MediaDecoderStateMachine thread.

If you look at the compositor thread there's also a bunch of bad looking stuff there.

Comment 9

[Jeff Muizelaar [:jrmuizel]]

alwu, thoughts on what might be going on?

Comment 10

[Jonathan Almeida [:jonalmeida]]

Attachment: screen recording of youtube.mov

Alternative steps to reproduce

1. Open a video on youtube.com
2. Wait for an ad to play (or use uBlock to skip it; it's still reproducible in both cases).
3. Observe the choppy playback.
4. Use the video controls to skip through the video to get rid of the choppiness.
5. Observe the playback is now smooth.

See attached video.

Aside from TikTok that the OP has reported, I've also observed this bug on Instagram and Vimeo videos as well.

Comment 11

[Jonathan Almeida [:jonalmeida]]

This has happened before with reporters and myself were not able to reproduce it. This might have been also been because it was harder to reproduce for a time, in which case, this bug it at least 8 months old when first reported.

Comment 12

[John Lin [:jhlin][:jolin]]

The Twitter/X profile in comment 8 shows that after seeking (23.234s), 4 video frames were decoded and pushed and there was no more request for video data. It seems they were queued and didn't get processed until media sinks were restarted. By then the audio clock time was already > 100ms and all video frames were considered too late and got dropped. After that, the video decoding just didn't catch up and the video played at ~10hz. (keyframes only?)

According to Alastor's comment offline, the media sinks didn't start sooner because MDSM waits for audio data preloading. However, the 1st VideoSinkDroppedFrame marker after that shows the audio clock time is 115,216μs, which seems quite larger than the time from audio sink start (~23.291s) to the marker (23.307s)

Comment 14

[John Lin [:jhlin][:jolin]]

I added some profile markers locally to check the audio clock time and found that the frame# from the Android cubeb backend won't reset from zero when the video plays again and it causes audio clock to increase in chunk size (~100ms). This doesn't happen on other platforms.

After analyzing cubeb log, it seems the aaudio backend recycles stream but the old timing_info buffers will be used. Not sure why this only happens when looping video, though.

Matthew, what do you think? Should the buffers be cleared when destroying the stream?

Comment 15

[John Lin [:jhlin][:jolin]]

Patch and pull request created on GitHub cubeb repo: https://github.com/mozilla/cubeb/pull/784

Comment 16

[Matthew Gregan [:kinetik]]

Thanks! I've reviewed the patch on GitHub.

Comment 17

[Jeff Muizelaar [:jrmuizel]]

Matthew what's the next step for getting cubeb change into mozilla-central?

Comment 18

[Jeff Muizelaar [:jrmuizel]]

And can we uplift it to beta?

Comment 19

[Matthew Gregan [:kinetik]]

Update will land via bug 1894401. I'll prep a beta uplift patch now.

Comment 20

[Matthew Gregan [:kinetik]]

Attachment: Bug 1872354 - Uplift cubeb AAudio timing fix for reused streams. r?#cubeb-reviewers

Comment 21

[Matthew Gregan [:kinetik]]

Attachment: Bug 1872354 - Uplift cubeb AAudio timing fix for reused streams.

Original Revision: https://phabricator.services.mozilla.com/D209083

Comment 22

[Phabricator Automation]

beta Uplift Approval Request

• User impact if declined: Audio glitching or desync when looping videos
• Code covered by automated testing: yes
• Fix verified in Nightly: no
• Needs manual QE test: yes
• Steps to reproduce for manual QE testing: Per bug 1872354 comment 0
• Risk associated with taking this patch: Low risk
• Explanation of risk level: Small change, adds a missing timing reset when stream reused
• String changes made/needed: No
• Is Android affected?: yes

Comment 23

[Donal Meehan [:dmeehan - PTO - Back Oct 13]]

:kinetik can you confirm the fix for this landing is landing in central as tracked under Bug 1894401?
The patch attached here is for Beta only and you are not tracking anything else under this bug? Bug 1894401 will not require a beta uplift?
(This appears to be the case, but wanted to confirm)

We are in the final week of Beta. I would prefer Bug 1894401 to land and stick in central before taking this to beta.

Comment 24

[Donal Meehan [:dmeehan - PTO - Back Oct 13]]

Got some information on Slack.
Confirmed that this is only a cherry-pick of Bug 1894401, low risk, and nothing else will land under Bug 1872354.

Comment 25

[Pulsebot]

https://hg.mozilla.org/releases/mozilla-beta/rev/a45e2174649a

Comment 26

[Adina Petridean QA [:apetridean]]

Verified as fixed on Firefox 126.09 and Firefox 126 with: Lenovo tab M10 (Android 10)
Samsung Galaxy S23 Ultra (Android 14)
OPPO A58 ( Android 13)

Comment 27

[Jeff Muizelaar [:jrmuizel]]

Adina, should status-firefox127 be reset from 'verified' as this hasn't been fixed in 127 yet.

Comment 28

[Adina Petridean QA [:apetridean]]

(In reply to Jeff Muizelaar [:jrmuizel] from comment #27)

Adina, should status-firefox127 be reset from 'verified' as this hasn't been fixed in 127 yet.

Certainly! Thanks for catching that typo!

Comment 29

[Markus Stange [:mstange]]

Reopening because this is still unfixed in mozilla-central, until bug 1894401 lands. It is fixed in Beta (126). The bug status usually indicates the mozilla-central status.

Comment 30

[Donal Meehan [:dmeehan - PTO - Back Oct 13]]

Bug 1895471 is on autoland and is tracking fixing this in central

Comment 31

[Donal Meehan [:dmeehan - PTO - Back Oct 13]]

Bug 1895471 has landed in central.
It's not good practice to track the same fix across multiple bugs