1685163 - MediaStreamTrack freezes when played in opened popup (used to work if visiting fullscreen)

Status: RESOLVED WORKSFORME

(Core :: Audio/Video: Playback, defect, P3)

Description

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

The DOM object model allows manipulation of some other same-origin documents, which lets JS play media across documents in some cases:

popUpWindow.document.body.appendChild(videoElem).srcObject = stream;

Camera STRs (follow all 6 steps for full effect):

1. Open https://jsfiddle.net/jib1/1kqbphj0/9/

2. Click the Start capture button (Allow camera if asked for permission).

3. If popup is blocked, disable popup blocker ("Allow" in pageinfo dropdown), close any popup opened (it's wrong), refresh page & goto 2

4. Observe about:blank popup window opening

   Expected result (Chrome, Safari 14.0.2):	Live moving camera video in popup	✅
   Actual result (Firefox 84):	Still image from camera in popup	❌
   Actual result (Nightly 86):	Extremely dark (almost black) still image from camera in popup	❌

5. Workaround: Right-click on the video element in the popup and select Full Screen

   Expected result (Chrome + Safari 14.0.2):	Live moving camera video in fullscreen	✅
   Actual result (Firefox 84):	Live moving camera video in fullscreen	✅
   Actual result (Nightly 86):	Extremely dark (almost black) still image from camera in fullscreen	❌

6. Hit Escape to exit fullscreen

   Expected result (Chrome + Safari 14.0.2):	Live moving camera video in popup	✅
   Actual result (Firefox 84):	Live moving camera video in popup	✅
   Actual result (Nightly 86):	Extremely dark (almost black) still image from camera in popup	❌

Two differences are observed between 84 (release) and 86 (nightly):

1. The dimness of the still image appears timing related and unrelated to any regression range (more likely profile related)
2. The fullscreen workaround (step 5) appears broken by D96072 in bug 1635914.

To summarize: this never worked properly, but somehow got worse (though more consistent) with bug 1635914 in that the fullscreen workaround stopped working.

This summary assumes it's desirable for the video to play, based on this working in Chrome and Safari 14.0.2, for web compat. It should be noted this stopped working in Safari 14.1 (Tech Preview) which shows a black frame. There are perhaps some privacy concerns here, like the lack of privacy indicators in the popup (they are on the main page instead). Note the track ends once the page that requested it is closed, even in Chrome (i.e. popup video does not outlive the page that opened it; freezing video at that point).

Comment 1

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Attachment: STRs w/screen-capture instead of camera caused fullscreen garbage prior to D96072

This issue was discovered thanks to a StackOverflow question about screen-capture instead of camera. Results are similar, but different enough to document separately:

Screen-capture STRs (follow all 6 steps for full effect):

1. Open https://jsfiddle.net/jib1/1kqbphj0/8/

2. Click the Start capture button and share "Entire Desktop".

3. If popup is blocked, disable popup blocker ("Allow" in pageinfo dropdown), close any popup opened (it's wrong), refresh page & goto 2

4. Observe about:blank popup window opening

   Expected result (Chrome + Safari):	Live moving desktop capture in popup	✅
   Actual result (Firefox 84):	Still image from desktop in popup	❌
   Actual result (Nightly 86):	Still image from desktop in popup	❌

5. Workaround: Right-click on the video element in the popup and select Full Screen

   Expected result (Chrome + Safari):	Live moving desktop capture in fullscreen	✅
   Actual result (Firefox 84):	Live black or garbage video w/mouse droppings i fullscreen (see attached image)	❌😱
   Actual result (Nightly 86):	Still image from desktop in fullscreen	❌

6. Hit Escape to exit fullscreen

   Expected result (Chrome + Safari):	Live moving desktop capture in popup	✅
   Actual result (Firefox 84):	Live moving desktop capture in popup	✅
   Actual result (Nightly 86):	Still image from desktop in popup	❌

Good news: the live garbage appears "fixed" (replaced by a non-garbage still image) by D96072 in bug 1635914.
Bad news: the fullscreen "workaround" to get live video stopped working.

Note: this works fine in Safari 14.1 (Tech Preview) as well, suggesting the black frame on camera is a bug. I'll file a bug on webkit about that.

Comment 2

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

This use case isn't necessarily high priority. I'm filing it more to make people aware of how MediaStreamTracks may play outside of their responsible documents, in case it breaks any assumptions made in various layers of our code, from playback to permissions. I've cc'ed some people who might be interested.

Comment 3

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1635914

Comment 4

[Andreas Pehrson [:pehrsons]]

The displayed frame seems to simply be the first frame that reached the video element. As if the first-frame listener worked but not the regular one.

I'll get a pernosco recording.

Comment 5

[Andreas Pehrson [:pehrsons]]

https://pernos.co/debug/AzeIaKvctbZJOiqQhVjG8A/index.html

Comment 6

[Andreas Pehrson [:pehrsons]]

The video element in the popup is simply not autoplaying because of MediaPlaybackDelayPolicy::ShouldDelayPlayback(this) being true.

If I enable media.block-autoplay-until-in-foreground it works somewhat as described for 84 in comment 0. It's frozen on the first frame but going to fullscreen unfreezes it. On exiting fullscreen it remains unfrozen. Note that before going fullscreen I cannot unfreeze it by playing the video element manually, which should work for autoplay blocking.

Alastor, do you have insight into the intended behavior here?

Comment 7

[Andreas Pehrson [:pehrsons]]

(In reply to Jan-Ivar Bruaroey [:jib] (needinfo? me) from comment #1)

Actual result (Firefox 84): | Live black or garbage video w/mouse droppings i fullscreen (see attached image) | ❌😱

This is nothing to worry about. This is it working as intended.

Imagine that you do a format conversion of the same image a couple hundred times, and that format conversion involves a rounding error. That's basically what you have here, since you're capturing the entire screen and rendering that in full screen. Then rinse and repeat a couple hundred times.

Comment 8

[Andreas Pehrson [:pehrsons]]

FWIW this pernosco link pretty much summarizes the problem. Ready state is fine, autoplay waits on a delayed resume promise but it never resolves or rejects.

Comment 9

[Alastor Wu [:alwu]]

That issue is same as bug1611748, because the visibile state for the new pop-up document is not set in the same way as other normal document. I got some patches in that bug but encountered an issue that would broke the entire mechanism for delaying autoplay (see bug1611748 comment7)

I confirm that this issue would be solved by applying patches in bug1611748. I will go back to that bug to see if that issue still exists or not.

Comment 10

[Alastor Wu [:alwu]]

I found that the issue mentioned in bug 1611748 is no longer being able to be reproduced, so it'd be better to close that one and work on this bug.

Comment 11

[Emilio Cobos Álvarez (:emilio)]

Alwu, the patch in that bug looks really close to the one in bug 1572798, and I think the problem that got me backed out is the same. I can probably fix it, but TLDR I think some equivalent of https://phabricator.services.mozilla.com/D42329 would fix the foreground mechanism.

Comment 12

[Alastor Wu [:alwu]]

Yes, it seems your patch is exact the solution we want! I will look forward to landing that patch, and check if this issue still exists afterthen. Thank you so much!

Comment 13

[Alastor Wu [:alwu]]

I can confirm that this issue has been fixed by bug1572798.