1920097 - PiP should not be autotrigger in the case of short, silent embedded videos

Status: VERIFIED FIXED

(Toolkit :: Picture-in-Picture, defect, P3)

Description

[Daniel Bodea [:danibodea]]

Note

• When loading a web page with a short embedded autoplay video and focus a different page, the PiP triggers automatically.

Found in

• Nightly v132.0a1

Affected versions

• Nightly v132.0a1

Tested platforms

• Affected platforms: Windows 10, Ubuntu 22, MacOS 11
• Unaffected platforms: -

Steps to reproduce

1. Enable PiP auto-trigger:
   media.videocontrols.picture-in-picture.enable-when-switching-tabs.enabled = true
2. Load https://www.twicpics.com/blog/the-optimal-ways-to-embed-video-according-to-use-cases#short-autoplay-videos
3. Open a new tab.

Expected result

• A PiP should not auto-trigger for the short embedded video.

Actual result

• PiP will auto-trigger for the short embedded autoplay video.

Regression range

• not a regression, but an issue since the implementation of PiP auto-trigger.

Comment 1

[Mike Conley (:mconley) (:⚙️)]

Attachment: Bug 1920097 - Use toggle heuristics for autoPiP feature as well. r?#pip-reviewers!

Comment 2

[Mike Conley (:mconley) (:⚙️)]

Hey padenot,

You had mentioned that there were some heuristics that we could potentially use to more precisely target when it's appropriate to apply the autoPiP feature. In the patch I just posted, I call into the pre-existing logic that we use for offering the toggle.

Besides the following:

1. Checking that the length exceeds our minimum 45s threshold
2. Checking that the video width or height exceeds 140 pixels
3. Checking that the video has an audio track

Can you propose any additional heuristics that might be useful here?

Comment 3

[Paul Adenot (:padenot)]

Here are some ideas, including some things that are best answered by Gecko and not always surfaced easily. If we want to experiment, we can plumb the information through in a way that's easily consumed by the code making the decision about auto-pipping or not:

• The video has almost ended (switching tab during the credits etc.), we probably don't want to pop it out
• The video is looping -- same deal, there's high chance it's a gif-style video. There's probably something to explore around allowing looping once for longer videos that loop (e.g. tiktok / ig reel / yt short type content)
• The video was seemingly watched for some time, it's plausible that we want to pop it out: it was front and center on the focused page and there was no kb/mouse activity for that time. We can simply look at painted frames vs. decoded frames, etc., there, and we have extensive visibility checks and respective duration (invisible duration, visible duration, etc.) in the engine already. Layout can tell us if it was side content or main content but it is likely that this will require some tuning
• The video is heavily overlaid by other things, don't pop it out. This is probably best answered by layout as well, but we don't want to pop out "background video" type stuff. This might be caught by the 45s threshold but maybe not
• The video isn't playing regular media (webrtc, etc.), no automatic pop out
• The video has an audio track but is muted (this takes care of auto-playing muted videos and lots of ads), probably not pop it out, but I could also see users wanting the opposite, especially if there are subtitles. I see that sometimes on news websites, they have a video bottom right with a broadcast, muted + autoplay, but are otherwise a news site.

Comment 7

[Ina Popescu, Desktop QA]

While testing in the latest Firefox 132.0b4, I noticed that the ads and banner videos shorter than 45s are auto-triggered in Pip on the following websites: 9gag.com , www.iq.com/?lang=en_us , www.espn.com/espnplus , www.crunchyroll.com , edition.cnn.com, www.twitch.tv, www.youtube.com/channel/UC-9-kyTW8ZkZNDHQJ6FgpwQ ,https://www.youtube.com/ .

Comment 10

[Ina Popescu, Desktop QA]

Attachment: www.iq.comlang=en_us.mp4

Regarding ads being auto-triggered in Pip, the ads on https://www.iq.com/?lang=en_us are auto-played in Pip when switching to a new tab. However, once the first ad finishes, the Pip window closes preventing the actual video from playing in Pip. If a separate issue needs to be submitted for this, please let me know.

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/10ccea0bc011
Use toggle heuristics for autoPiP feature as well. r=pip-reviewers,niklas

Comment 12

[Serban Stanca [:SerbanS]]

https://hg.mozilla.org/mozilla-central/rev/10ccea0bc011

Comment 13

[BugBot [:suhaib / :marco/ :calixte]]

The patch landed in nightly and beta is affected.
:mconley, is this bug important enough to require an uplift?

• If yes, please nominate the patch for beta approval.
• If no, please set status-firefox132 to wontfix.

For more information, please visit BugBot documentation.

Comment 16

[Daniel Bodea [:danibodea]]

I can happily confirm that most of the scattered unwanted auto-triggers of PiP no longer occur in the latest Nightly v133.0a1. I've closed all the reports that were fixed as a duplicate of this bug and made sure that the remaining ones are still open (bug 1922757, bug 1922784). Thank you!

Comment 17

[Mike Conley (:mconley) (:⚙️)]

Release Note Request (optional, but appreciated)
[Why is this notable]:

Users that have opted in to using the "Picture-in-Picture: auto-open on tab switch" feature from Firefox Labs will find that the feature behaves better on more sites.

[Affects Firefox for Android]:

No.

[Suggested wording]:

If you're trying the "Picture-in-Picture: auto-open on tab switch" feature from Firefox Labs, you should find that it behaves more reliably on more sites - auto-opening more videos that you care about, and ignoring ones that you probably don't care about.

Comment 18

[Donal Meehan [:dmeehan]]

Thanks, added to the Fx133 beta release notes, please allow 30 minutes for the site to update.
Keeping the relnote-firefox flag as ? to keep it on the radar for inclusion in the final Fx133 release notes.

Comment 19

[Donal Meehan [:dmeehan]]

Added to the Fx133 release notes.
Available to preview on staging, please allow 30 minutes for the site to update.