1547349 - PiP Twitter/Facebook video stops when original video is scrolled out of view

Status: NEW

(Toolkit :: Picture-in-Picture, defect, P4)

Description

[Asif Youssuff]

STR:

1. Navigate to https://twitter.com/TheView/status/1121806349950537728
2. Pop out video
3. Scroll down past video in timeline

What happens:

Video pauses when the original video is scrolled out of view.

Expected result:

The video should keep playing in the PiP window.

Comment 1

[Kestrel]

Same for reddit videos when switching away from the owner tab.

Comment 2

[Timea Cernea [:tbabos][inactive]]

Reproducible for Facebook videos as well. Updating flags.

Comment 3

[Cristian Craciun]

Build ID 20191124230652
User Agent Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:72.0) Gecko/20100101 Firefox/72.0

Reproducible also on Nightly 72 on macOS 10.13 platform.

Comment 5

[Adam Stevenson [:adamopenweb]]

Just noting that Chrome has the same issue, when selecting PiP from the Media Hub.

Comment 6

[:Gijs (out for now; he/him)]

So presumably this is happening because twitter is pausing the original video, and fixing this would require overriding the decision from the website to pause the video? Or cloning the video into our own element, transferring the pip to that video, and continuing to play it in that clone, but doing that without any interruption to the video sounds... difficult, as we'd presumably have <20ms from the point where the website pauses the video...

We'd also have to make sure that if an explicit user interaction triggered the pause, we'd still "honour" that pause.

What do you think, Mike?

Comment 7

[Mike Conley (:mconley) (:⚙️)]

So presumably this is happening because twitter is pausing the original video

I actually think it depends. Scroll a little bit, and it pauses. Scroll a bit further, and eventually the player closes - probably because the <video> element was removed from the DOM.

There might be something clever we could do where if something attempts to remove the <video> from the DOM while it's being closed visually, we put it into some kind of special domain where it still exists in memory but isn't in the page anywhere.... but I suspect this might break expectations for a site that re-introduces a <video> back into the DOM afterwards, as it probably would expect the video to have fully unbound.

The pause thing also sounds like it'd be brittle...

I don't have a great solution to this. :(

Comment 9

[Ania]

Out of sight, out of mind (c) Twitter.
This is very unpleasant, but it sounds like there's no reasonable workaround for it (Chrome, Opera, and Vivaldi disable their PiP in feeds, which seems somewhat extreme to me).

However, on Twitter, there's a way to recover from this behavior: stop scrolling if PiP window is still visible, press play, the video continues. A user can then reopen the feed in the new tab and carry on with both activities.

In FB feed, the experience is more disruptive (macOS, Nightly 96.0a1 (2021-11-29) (64-bit)). Steps to reproduce:

1. Navigate to https://www.facebook.com/watch and log in with an FB account
2. Launch PiP
3. Continue scrolling
4. PiP video pauses (it does not disappear with further scrolling)
5. Stop Scrolling
6. Press Play on PiP video

AR:
Video plays for a fraction of a second, then pauses again.

ER
I feel like expected result here would be: video continues playing until the "source" is scrolled out of view.

Mike, Gijs, is this something we can reasonably control or do we need to consider some alternative flow to this?

Comment 10

[:Gijs (out for now; he/him)]

(In reply to amininkova from comment #9)

Mike, Gijs, is this something we can reasonably control or do we need to consider some alternative flow to this?

We previously discussed this a bit in comment #6 and comment #7. I'm not sure if anything has changed, and/or if the work to keep the video alive if the tab is closed would help with some of this, Mike?

Pausing could be fixable, though as Mike says brittle - but removing is the harder bit to fix...

Comment 11

[Mike Conley (:mconley) (:⚙️)]

Attachment: WIP: Bug 1547349 - Proof-of-concept: don't let sites pause videos without user input when in PiP.

I don't know if we'd want to try this, but what it does is it prevents site scripts from
pausing <video> elements when they're being used for Picture-in-Picture unless the pause
is requested when handling user input.

In my (very limited) testing, this addresses both the Twitter and the Facebook case in
the bug (except for the part where Twitter removes the <video> element from the DOM). There
is at least one potential drawback here: sites could use this as an ability to detect when
users are viewing a <video> in PiP in Firefox, by attempting to pause a video and then
checking the play/pause state and seeing that the pause attempt failed. I'm not certain
if that's really a problem in practice, but calling it out here.

Comment 12

[Mike Conley (:mconley) (:⚙️)]

(Chrome, Opera, and Vivaldi disable their PiP in feeds, which seems somewhat extreme to me).

These browsers aren't allowing for PiP because Twitter has put disablepictureinpicture="true" on the video. We (intentionally) do not support that (or any) part of the Picture-in-Picture WebAPI specification. See bug 1611831 for more context.

I just posted a patch that we might want to consider. What it does is it prevents site scripts from
pausing <video> elements when they're being used for Picture-in-Picture unless the pause
is requested when handling user input.

In my (very limited) testing, this addresses both the Twitter and the Facebook case in
the bug (except for the part where Twitter removes the <video> element from the DOM). There
is at least one potential drawback here: sites could use this as an ability to detect when
users are viewing a <video> in PiP in Firefox, by attempting to pause a video and then
checking the play/pause state and seeing that the pause attempt failed. I'm not certain
if that's really a problem in practice, but calling it out here.

I've pushed to try to generate some builds for Ania to test. We'd probably want to discuss
this with the DOM media folk to see if this is an okay solution regardless.

Comment 13

[Mike Conley (:mconley) (:⚙️)]

Try push is here: https://treeherder.mozilla.org/#/jobs?repo=try&revision=82a2250da7554976e81f4b77ca272f89d658abda

I'll put links to the generated builds when they're created.

Comment 14

[Ania]

Thank you, Mike! I'll take a look at the builds once they are ready.
Who is the right person to tag on the DOM media side?

Comment 15

[Mike Conley (:mconley) (:⚙️)]

Builds are here:

Windows: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/S4Q3CWgNQCiMlKgPu6u3YA/runs/0/artifacts/public/build/target.zip
macOS: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Cgn6NG_CReC9ywyYwcb0BA/runs/0/artifacts/public/build/target.dmg
Linux: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/fkZUB-E2SwGmPz-tjEqpGA/runs/1/artifacts/public/build/target.tar.bz2

For DOM media, we could try asking pehrsons to get his take.

Comment 16

[Ania]

Mike, thank you so much. I just got to look at the builds:

macOS Big Sur and Win 10 Pro:

Facebook:

1. Start PiP, scroll away from the "source" video, so it's out of view.
2. PiP stays on screen, and the video continues to play (hooray!)

However, PiP sound is muted as you scroll away (and the volume button changes its state to muted).
If you unmute and continue scrolling, the video gets muted again almost momentarily.

Twitter
I feel like behavior didn't change. Perhaps you can scroll away a tad more before PiP disappears.

Comment 18

[Mike Conley (:mconley) (:⚙️)]

I feel like behavior didn't change. Perhaps you can scroll away a tad more before PiP disappears.

Correct. To be clear, there are two issues here:

1. On Twitter, when the video exits the viewport, Twitter is automatically pausing the video. This is what the patch I posted attempts to address.
2. On Twitter, when the timeline scrolls sufficiently past the video, Twitter is removing the video from the DOM which closes the player window. The patch does not address this separate problem.

Comment 19

[Ania]

Ah, please forgive me for my lack of observation; now I see that on Twitter, the video continues to play on Twitter before it is removed from the DOM.

Let me file a separate issue for Twitter removing the video when a user scrolls away to keep it on our radar.

Comment 20

[Andreas Pehrson [:pehrsons]]

(In reply to Mike Conley (:mconley) (:⚙️) from comment #15)

For DOM media, we could try asking pehrsons to get his take.

So I hear multiple reasons playback could be stopped by an application:

• pause
• removing from the DOM (without really knowing this surprises me, audio should reasonably still play)
• changing the src attribute (probably what they do when also removing from the DOM?)

Re pause the spec doesn't seem to allow for ignoring the pause conditionally based on user input.

Re changing the src attribute the spec is very clear that we must effectively stop playback of the resource.
Also, the spec (non-normatively) has something to say about best practices for authors:

For this reason, it is a good practice to release resources held by media elements when they are done playing, either by being very careful about removing all references to the element and allowing it to be garbage collected, or, even better, by setting the element's src attribute to an empty string. In cases where srcObject was set, instead set the srcObject to null.

OTOH what the site's media element does and what our PiP window does are different things. The site's media element could be spec compliant even if the PiP window takes over playback of a resource. Though to look for flaws in this approach there are several, including:

• if the site decided to stop playback and we continue playing, we are effectively fetching data from the site's servers for playing this resource without their consent?
• if the site's media element was playing a MediaSource object (MSE) their collaboration is required in fetching media data.

So I suppose transferring ownership of the MediaDecoder instance to the PiP window's media element on pause/src-change is technically feasible, though it'd only work with file (well, non-MSE) resources. It seems like a better approach than breaking the spec (!) and ignoring certain calls from the site. But I guess it requires some re-architecturing of PiP.

But hey, I do mostly realtime (webrtc). I'll loop in Bryce for his playback take also.

Comment 21

[Bryce Seager van Dyk [:bryce] (he/him) - Not reading bugmail]

Echoing the above, from my perspective much of this comes down to how much effort we wish to make to make this work and how PiP fits into spec conformance. Andreas has already done a great analysis, so I'll largely defer to that!

Other thoughts, it makes sense for PiP to keep playing from a UX perspective. However, if sites think an element is paused I can see issues with

• Multiple elements playing at the same time.
• The MSE case mentioned above, where a site stops appending media data because it thinks an element isn't playing (may depend on the site how they handle this).
• Depending on how we handle interactions, PiP may suddenly end once the site decides to clear src or similar (as comment 18).

As well as the endless scroll case, single page applications are something that crossed my mind, depending on how we tinker.

Comment 22

[Alastor Wu [:alwu]]

I don't think to explicitly ignore pause calls for a media element is a good idea. Although I don't know how does Twitter decide to pause or remove a media element, if it's related with the element visibility, I think to address element's visibility state would help. See bug 1598654.

In addition, by checking the elements on Twitter, they explictly added an attribute disablepictureinpicture on their videos. So From this perspective, what we could do is to implement this attriute and honor their decision in order to not provide PIP on those videos.

Comment 23

[Mike Conley (:mconley) (:⚙️)]

(In reply to Alastor Wu [:alwu] from comment #22)

In addition, by checking the elements on Twitter, they explictly added an attribute disablepictureinpicture on their videos. So From this perspective, what we could do is to implement this attriute and honor their decision in order to not provide PIP on those videos.

To be clear, at this time, there is no intent to implement any part of the WCAG's Picture-in-Picture WebAPI, including the disablepictureinpicture attribute. Please see https://bugzilla.mozilla.org/show_bug.cgi?id=1611831 and https://github.com/mozilla/standards-positions/issues/72 for further details.

Comment 24

[Daniel Bodea [:danibodea]]

Setting severity S3 since this does not seem as any higher severity (the behavior is kind of expected since the video in the original tab would also stop playing when scrolling it out of view).

Comment 25

[BugBot [:suhaib / :marco/ :calixte]]

The severity field for this bug is relatively low, S3. However, the bug has 3 duplicates.
:mconley, could you consider increasing the bug severity?

For more information, please visit auto_nag documentation.

Comment 26

[Mike Conley (:mconley) (:⚙️)]

Let's keep the severity where it is.

Comment 27

[Daniel Bodea [:danibodea]]

The current behavior will be evaluated with the latest Nightly v100.0a1 on Windows 10, Ubuntu 20.04 LTS and Mac OS 11.6.4 based on these 4 steps:

1. Launching the PiP.
2. Scrolling the original video player out of the viewport.
3. PiP Video and controls functionality.
4. Continue scrolling the website's timeline.

A. Twitter: all OSes
1.1 When the user scrolls Twitter and (finally) finds a long enough video to activate PiP and clicks the toggle, the PiP is launched and playing.
1.2 If he scrolls down enough for the original video/player to move out of the viewport, the video in the PiP pauses.
1.3 The user can now replay the video and control the sound from the PiP.
1.4 However, when he scrolls further, the PiP will close automatically.

** The issues I see here (Twitter + any OS) are: **
** PiP pauses when scrolling down. (steps 2) ** -> will be addressed in this bug, if ever.
** PiP closes by itself when scrolling further. (step4) ** -> should I log a different bug for this?

B. Facebook: Unfortunately, it appears that Facebook behaves differently on Windows than it does on Mac OS and Ubuntu 20.

• Win 10:
  2.1 When the user scrolls Facebook and (finally) finds a long enough video to activate PiP and clicks the toggle, the PiP is launched but NOT playing as the PiP click also counted as video pause.
  2.2 If he scrolls down enough for the original video/player to move out of the viewport, the video in the PiP pauses.
  2.3 If he then clicks the Play button in the PiP, the video stutters, but does not play. If the user then clicks the "Resume video" in the main video/player, the video will play in PiP. If he clicks the Mute button in PiP, the video will also pause.
  2.4 When he scrolls further, the PiP will remain on screen in a paused state (not close).

** The issues I see here (Facebook + Win 10) are: **
** The clicking of the PiP toggle counts as a video click that will pause the video in the opened PiP or open the video's own page (step 1). ** -> should I log this separately?
** The fact that Facebook's own PiP does not take over, as it does with Mac OS and Ubuntu! (step 2) ** -> should I log this separately?
** The incorrect behavior of the PiP controls (step 3) ** -> this is addressed in this bug, if ever.
** The fact that the PiP does NOT close, considering that it can't be made to play while scrolling (step 4) ** -> should I log this separately?

• Mac OS 11 and Ubuntu 20:
  2.1 When the user scrolls Facebook and (finally) finds a long enough video to activate PiP and clicks the toggle, the PiP is launched and PLAYING (in contrary to behavior on Windows where the click counts as video pause).
  2.2 If he scrolls down enough for the original video/player to move out of the viewport, the PiP will close as Facebook's own PiP will take over and play the video further (as opposed to the behavior on Windows where Facebook does not open its own PiP).
  2.3 The user can now scroll further using Facebook's own PiP OR open a new PiP from it. With both of these PiPs, the user will be able to control play/pause and mute/unmute while scrolling Facebook.
  2.4 When he scrolls further, the PiP will remain on screen and properly working.

** I would say this is acceptable. ** -> don't you?

C. Reddit: Unfortunately, it appears that Reddit also behaves differently on Windows than it does on Mac OS and Ubuntu 20.

• Win 10:
  3.1 When the user scrolls Reddit and (finally) finds a long enough video to activate PiP and clicks the toggle, the PiP is launched but NOT playing as the PiP click also counted as video pause OR SOMETIMES the PiP click will also count as a video click, which will redirect to the video's page.
  3.2 If he scrolls down enough for the original video/player to move out of the viewport, the video in the PiP pauses.
  3.3 The user can now replay the video and control the sound from the PiP.
  2.4 When he scrolls further, the PiP will remain on-screen (not close).

** The issues I see here (Reddit + Win 10) are: **
** The clicking of the PiP toggle counts as a video click that will pause the video in the opened PiP or open the video's own page. (step 1) ** -> should I log this separately?
** PiP pauses when scrolling down. (steps 2) ** -> this will be addressed in this bug, if ever. right?

• Mac OS 11 and Ubuntu 20:
  3.1 When the user scrolls Reddit and (finally) finds a long enough video to activate PiP and clicks the toggle, the PiP is launched but and playing (as opposed to the behavior seen on Window where the PiP launches paused or even counts as video click).
  3.2 If he scrolls down enough for the original video/player to move out of the viewport, the video in the PiP pauses.
  3.3 The user can now replay the video and control the sound from the PiP.
  3.4 When he scrolls further, the PiP video will pause again, but will remain on-screen (not close) and can be re-player (again).

** The issues I see here (Reddit + MacOS/Ubuntu) are: **
** The PiP video pauses (twice) when scrolling down. (steps 2 and 4) ** -> this will be addressed in this bug, if ever. right?

These scenarios will be included in the PiP test cases and the test runs for Beta 100 and later.

@ania: I will start by apologizing for the "wall of text", but I don't see any other way to address these issues in detail.
Can you give me some feedback on my opinions relating to the different behaviors I consider valid/invalid? (the ** starred paragraphs **)

Comment 28

[Ania]

Daniel, thank you so much for investigating this problem in-depth.

I have a few thoughts:

• let's please not log the newly discovered issues separately,
• let's have this ticket as a source of truth for all the issues related to the PiP behavior on scrolling out on the feed-based websites.
• I'm not sure it's worthwhile to include these test cases from Fx 100 until we decide to fix them. Very early on, there was a hope that this could be addressed by May (hence, the whiteboard label), but at this point, there are no plans on when we can handle this. The priority of the corresponding ticket in JIRA is set to low.

I would like to hope that we can address all of these issues sooner rather than later, but:

• from the little information that is available on the average video length/watch time on social medio, a Facebook video's average watch time is 10 seconds. The number is probably a little higher for non-Live videos, but I'm going to use it as a benchmark.
• because we can't use URL-tracking, it is hard to quantify how many users are impacted by this problem on various platforms and which we should fix first.
• a (dumb) but simple workaround exists for these issues - leave the tab that plays a video open, and open the feed in the next one.

Perhaps, we can come up with some proxy metric to understand the scale of the problem in the future. Molly, I'm curious on your thoughts on whether this should have higher priority/severity, but for now I place this into a "low priority/might be addressed some time in the future bucket.

Comment 29

[Molly Howell (she/her) (mostly inactive)]

I agree with your prioritization there. I'll also echo the rest of the thread in that I do not know what to do about this. Trying to prevent the video from being either paused or deleted by the site seems way too risky and fragile to me.

Comment 30

[Ania]

Thank you, Molly. I further deprioritized this.

Comment 32

[Ina Popescu, Desktop QA]

This issue is noticeable on www.instagram.com when the video launched in PiP is paused while scrolling away from the origin video. Continuing to scroll away will eventually close the PiP window.

Comment 34

[Brindusa Tot, Desktop QA]

This issue is reproducible on www.linkedin.com - open a video on PIP, then scrolling down so that video is out of view - the video stops playing.

Comment 35

[Bogdan Maris, Desktop Test Engineering]

(In reply to Brindusa Tot[:brindusat] from comment #34)

This issue is reproducible on www.linkedin.com - open a video on PIP, then scrolling down so that video is out of view - the video stops playing.

I can also reproducible this on a video from linkedin.com if you double click on the pip window to enter fullscreen.

Comment 36

[Ina Popescu, Desktop QA]

Similar issue is noticeable on www.tiktok.com where the Pip window is closed when scrolling away from the origin video. Test was performed on Firefox 113.0.
Please let me know if a separate bug should be filed for this one.