Open Bug 1592539 Opened Last month Updated Last month

HTMLVideoElement cloned for PiP doesn't track the selected video track properly when playing a MediaStream

Categories

(Toolkit :: Video/Audio Controls, defect, P2)

defect

Tracking

()

People

(Reporter: pehrsons, Unassigned)

References

(Blocks 1 open bug)

Details

HTMLVideoElement::MaybeBeginCloningVisually sets up the target element's VideoFrameContainer with the selected video track when playing a MediaStream.

Then it forgets about things.

Problem #1:
HTMLVideoElement::EndCloningVisually removes the target element's VideoFrameContainer from the target element's selected video track. Is this ever non-null?

Problem #2:
Even with problem #1 fixed, the selected video track in HTMLVideoElement::MaybeBeginCloningVisually and HTMLVideoElement::EndCloningVisually might not be the same. That's a bummer for cleaning up. It's worse for the user if the application changed the selected track and the PiP player did not follow along. A simple test case that should trigger this bug follows below:

const gumStream = await navigator.mediaDevices.getUserMedia({audio: true, video: true});
const screenTrack = (await navigator.mediaDevices.getDisplayMedia()).getTracks()[0];
const video = document.createElement("video");
document.body.appendElement(video);
video.srcObject = new MediaStream(gumStream.getTracks());
video.play();
// Start picture-in-picture of `video`
video.srcObject.removeTrack(gumStream.getVideoTracks()[0]);
video.srcObject.addTrack(screenTrack);

Expected: PiP player displays the screen track
Actual: PiP player displays the camera track

I wrote a jsfiddle to demonstrate this.

STR:
Pre: Ensure PiP is enabled

  1. Open https://fiddle.jshell.net/pehrsons/xLor50td/show/light/
  2. (to work around bug 1590479) Right-click in the main iframe and click "This Frame"->"Show Only This Frame"
  3. Click Start. Approve the camera/mic capture in the first prompt, and select a screen or window to capture and approve the second prompt.
  4. Start PiP for the video element on the page.
  5. Click "Display Screen"

Expected: PiP player shows the screen capture
Actual: PiP player shows frames from camera and screen capture interleaved

I am not sure how the PiP player got a hold of the new track, but as expected from reading the code, there's a bug in that the PiP player is still displaying frames from the video track that is no longer selected.

According to the folks in #media, this sort of thing should probably be a release blocker.

Hey astevenson, just getting this on your radar. The solution that the folks in #media suggested was that we uplift a patch to Beta (71) that disables the PiP toggle and context menu item for videos that have srcObject !== null on it.

This means that for 71, PiP would not be available for the types of <video> elements that are receiving, for example, streams from the camera, the desktop, or any other type of MediaStream. Normal videos, like the ones on YouTube, Twitch, Amazon, Netflix, etc, will be unaffected.

We would then work to fix this bug, hopefully for 72, and re-enable the PiP toggle and context menus for that kind of <video> element.

This seemed like the least riskiest option for 71. Does that sound satisfactory?

Edit: Updated because I got the version numbers wrong.

Blocks: 1527926
No longer blocks: videopip
Flags: needinfo?(astevenson)

Mike, that makes sense to me. Just want to clarify that the patch to disable the PiP toggle + context menu for MediaStream would happen in 71 Beta (current Beta) and 72 would hopefully have the complete fix.

Flags: needinfo?(astevenson)
Blocks: 1532675
No longer blocks: 1527926
Priority: -- → P2

I filed bug 1592729 to land patches to disable the toggle and context menu items for MediaStream videos.

You need to log in before you can comment on or make changes to this bug.