2008641 - Video with createMediaElementSource becomes permanently sluggish / freezes after switching playbackRate a few times with zero video frame copy

Status: RESOLVED FIXED

(Core :: Audio/Video: Playback, defect, P3)

Description

[Karl Tomlinson (:karlt)]

Most of the changes for bug 1931328 were backed out to address bug 2006276.

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:132.0) Gecko/20100101 Firefox/132.0

Steps to reproduce:

1. Download https://download.blender.org/demo/movies/BBB/bbb_sunflower_1080p_30fps_normal.mp4.zip
2. Open it as a local file through Firefox and start playback
3. Execute the following code:

ctx = new AudioContext();
v = document.querySelector('video');
src = ctx.createMediaElementSource(v);
src.connect(ctx.destination);

switchesLeft = 20;
intervalId = setInterval(() => {
	v.playbackRate = 8;
 	// v.playbackRate = 8.25; // (a muted speed) also makes the bug appear.

	switchesLeft--;
	if (switchesLeft <= 0) {
		clearInterval(intervalId);
		v.playbackRate = 1;
		console.log('back to normal speed');
		return;
	}

	setTimeout(() => {
		v.playbackRate = 1.25;
	}, 100)
}, 300)

Actual results:

After a few playbackRate changes the video becomes super sluggish (FPS drops to about 0.1).

• The behavior seems to remain even after we go back to playbackRate = 1.
• The audio seems to be unaffected and plays normally.
• Pausing + unpausing the video restores normal FPS.
• Seeking also restores normal behavior.
• I tried the same with one YouTube video I was unable to reproduce this.
• Lower playback rates (below ~6) don't always make this bug appear.
• The CPU doesn't seem to be overloaded. None of the cores' usage goes above 50%.
• Sometimes switching playbackRate even once makes this bug show up. But sometimes the normal behavior restores after a while of playing back the video at the rate of 1.

I was able to reproduce this on the following Firefox versions: 134.0a1, 132.0.1, 130.0b9, 128.4.0esr, 107.0.1, 103.0.2, 103.0.
On 92.0b9, 97.0b9, 100.0.2, 101.0.1, 102.9.0esr I was unable to reproduce the bug at all.

So it looks like 103.0 is the first version with this bug.

This bug significantly affects an extension of mine called "Jump Cutter", where switching playback rate is the entire purpose of the extension. See original bug report: https://github.com/WofWca/jumpcutter/issues/186

Expected results:

Video doesn't freeze.
If it's impossible to maintain high FPS on high playbackRate, it at least should go back to normal after the playbackRate restores to normal.

Comment 1

[BugBot [:suhaib / :marco]]

Set release status flags based on info from the regressing bug 1899450

:sotaro, since you are the author of the regressor, bug 1899450, could you take a look?

For more information, please visit BugBot documentation.

Comment 2

[Sotaro Ikeda [:sotaro]]

It seems better to disable zero video frame copy with createMediaElementSource()

Comment 3

[Sotaro Ikeda [:sotaro]]

Going to look into the problem.

Comment 4

[Sotaro Ikeda [:sotaro]]

Attachment: temporal patch - Add log

Comment 5

[Sotaro Ikeda [:sotaro]]

Attachment: temporal patch - Add log

Comment 6

[Sotaro Ikeda [:sotaro]]

From Attachment 9537107 [details] [diff], MediaStream in content process seemed to hold a lot of video frames. And it seemed to cause the hardware decoder to run out of video frames for decoding during zero video frame copy.

Comment 7

[Sotaro Ikeda [:sotaro]]

Attachment: Bug 2008641 - Disable zero copy when too many video frames are used

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/df5ccf1d555e
https://hg.mozilla.org/integration/autoland/rev/0b33a95ffa0e
Disable zero copy when too many video frames are used r=gfx-reviewers,media-playback-reviewers,lsalzman,alwu

Comment 9

[Goloman Adrian[:agoloman]]

https://hg.mozilla.org/mozilla-central/rev/0b33a95ffa0e

Comment 10

[BugBot [:suhaib / :marco]]

The patch landed in nightly and beta is affected.
:sotaro, is this bug important enough to require an uplift?

• If yes, please nominate the patch for beta approval.
  • See https://wiki.mozilla.org/Release_Management/Requesting_an_Uplift for documentation on how to request an uplift.
• If no, please set status-firefox148 to wontfix.

For more information, please visit BugBot documentation.