Open Bug 1463402 (pip-api) Opened 8 years ago Updated 21 days ago

Implement Picture-in-Picture Web API on Desktop

Categories

(Core :: DOM: Core & HTML, enhancement, P2)

Product:
Core
Component:
DOM: Core & HTML
Type:
enhancement

Tracking

()

Status:
REOPENED
Milestone:
153 Branch
Project Flags:
a11y-review assigned
Tracking Flags:
Tracking Status
relnote-firefox --- 153+
firefox153 --- fixed

People

(Reporter: beaufort.francois, Assigned: sfarre, NeedInfo)

References

(Depends on 1 open bug, Blocks 1 open bug, Regressed 1 open bug)

Details

(5 keywords, Whiteboard: [geckoview:p3][webcompat], [wptsync upstream])

User Story

web-feature: picture-in-picture
platform-scheduled:2026-06-30
user-impact-score:900

Attachments

(12 files, 6 obsolete files)

Bug 1463402: Add pref for PIP API r=#dom-core!
48 bytes, text/x-phabricator-request
Details | Review
Bug 1463402: Implement pseudo class :picture-in-picture r=#dom-core!
48 bytes, text/x-phabricator-request
Details | Review
Bug 1463402: Implement Picture In Picture API r=#dom-core!
48 bytes, text/x-phabricator-request
Details | Review
Bug 1463402: Implement Desktop support for PIP API r=#dom-core!
48 bytes, text/x-phabricator-request
Details | Review
WIP: Bug 1463402 - Android PIP & PIP API
48 bytes, text/x-phabricator-request
Details | Review
WIP: Bug 1463402 - User agent triggered Android PIP via context menu
48 bytes, text/x-phabricator-request
Details | Review
WIP: Bug 1463402 - Open PIP from Media notification
48 bytes, text/x-phabricator-request
Details | Review
Bug 1463402: Add WPT for iframe PIP attribute testing r=#dom-core!
48 bytes, text/x-phabricator-request
Details | Review
Bug 1463402: Add WPT for exit-when-having-pip ordering is observed r=smaug!
48 bytes, text/x-phabricator-request
Details | Review
Bug 1463402: Add additional Shadow DOM tests r=smaug!
48 bytes, text/x-phabricator-request
Details | Review
Bug 1463402: Add WPT for unloading steps r=smaug!
48 bytes, text/x-phabricator-request
Details | Review
Bug 1463402: Flip pref for dom.media-pip.enabled r=#dom-core!
48 bytes, text/x-phabricator-request
Details | Review
User Agent: Mozilla/5.0 (X11; CrOS x86_64 10698.0.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3436.0 Safari/537.36 Steps to reproduce: Spec: https://wicg.github.io/picture-in-picture/ Explainer: https://github.com/WICG/picture-in-picture/blob/master/explainer.md
Note that https://bugzilla.mozilla.org/show_bug.cgi?id=1280418 is related to this feature request.
Severity: normal → enhancement
Component: Untriaged → Audio/Video: Playback
Product: Firefox → Core
Our stance on this is unclear at the moment. Marking this P3 while we figure it out.
Status: UNCONFIRMED → NEW
Rank: 25
Ever confirmed: true
Priority: -- → P3
FYI, here's the latest statement I got from Mozilla: https://github.com/mozilla/standards-positions/issues/72#issuecomment-393596352
Right, thanks. P3 seems appropriate then.
chrome has shipped this on 69 as flag options
and it was activated with 70 by default for the stable release. As a macOS user I hope to get this feature in the next couple months.
Here is some documentation available about Picture-in-Picture support in Chrome 70: - https://developers.google.com/web/updates/2018/10/watch-video-using-picture-in-picture - https://www.chromestatus.com/feature/5729206566649856
Alias: PiP
Depends on: 1280418
Whiteboard: [geckoview:p3]

BBC's site is now using this, as discovered in https://webcompat.com/issues/26256.

Flags: webcompat?
See Also: → https://webcompat.com/issues/26256
Whiteboard: [geckoview:p3] → [geckoview:p3][webcompat]

Migrating Webcompat whiteboard priorities to project flags. See bug 1547409.

Webcompat Priority: --- → ?

See bug 1547409. Migrating whiteboard priority tags to program flags.

Depends on: 1580095
Depends on: 1601560
Depends on: 1610522
No longer depends on: 1580095
No longer depends on: 1601560
Webcompat Priority: ? → ---
Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → DUPLICATE

why this bug is marked as duplicate of Bug 1611831?
Picture-in-Picture JavaScript API and disablePictureInPicuture attribute is different things.
e.g. sometimes developer want to add customized "Start PiP" button on their website.

Amininkova, I added the need-info flag for you in order to clarify the question in the previous comment.

Sebastian

Flags: needinfo?(amininkova)

Hi Sebastian and Rinsuki,

Mozilla has no current plans to support Picture-in-Picture Web API or its parts. I will remove the resolution to avoid further confusion but will close the ticket nevertheless.

Flags: needinfo?(amininkova)
Resolution: DUPLICATE → WONTFIX

But from what I heard in the #standard discussion, it seems our standard team recently changed their thought and we might start supporting this API in the future.

It seems incredibly strange to me that Firefox supports picture-in-picture web video but not document.pictureInPictureEnabled + video.requestPictureInPicture(). The end result is that a custom video player cannot show a pip control alongside other custom controls in Firefox, instead having the highly inconsistent floating icon.

Strong +1 towards supporting the PIP API, please.

Hi,
I don't really get why you wont at-all support the disablePictureInPicture attribute. We have built a WebRTC conference application and the usability is completely broken since every participant's video has this ridiculous floating PiP overlay, that completely interferes with the UI and just confuses our users.
The fact that we can not even disable it for video elements that display a WebRTC MediaStream brought us to the decision that we will no longer support Firefox and will recommend the usage of other browsers to our customers.

Hi Sven! Note that this is the meta-issue for the implementation of all the PiP API (which unfortunately also got closed because Mozilla obviously doesn't have any plans to implement this API, for the time being).
The specific bug for disablePictureInPicture is/was bug 1611831. See Mike Conley's comments there for Mozilla's reasoning not to implement this API. He also mentions there that an enterprise policy got introduced in bug 1619658 that allows system administrators to turn the the PiP overlay off. Users can also individually disable the button in the settings by unchecking the option "Enable picture-in-picture video controls".

Sebastian

Alias: PiP
Duplicate of this bug: 1826115
See Also: → 1858562
Flags: needinfo?(jmuizelaar)
Status: RESOLVED → REOPENED
Flags: needinfo?(jmuizelaar)
Resolution: WONTFIX → ---
User Story: (updated)
Severity: normal → S2
Component: Audio/Video: Playback → DOM: Core & HTML
OS: Unspecified → All
Priority: P3 → P2
Hardware: Unspecified → All
Alias: pip-api
Duplicate of this bug: 1984891
Assignee: nobody → jmuizelaar
User Story: (updated)
User Story: (updated)
Blocks: 1890774
Attachment #9509542 - Attachment description: Bug 1463402. Initial WIP implementation → Bug 1463402 - Implement PiP API
Attachment #9509542 - Attachment description: Bug 1463402 - Implement PiP API → WIP: Bug 1463402 - Implement PiP API
Attachment #9509542 - Attachment description: WIP: Bug 1463402 - Implement PiP API → WIP: Bug 1463402 - Part 1: Implement PiP API scaffolding

By introducing nsIPictureInPictureFunctions we can call PictureInPicture functions directly and have JS promises returned to us. With those we can build the parallel queue that we do work on in the spec, by wrapping the promises in our own type that will be a PromiseNativeHandler that on resolve can resume the remainder of the in-parallel steps.

Added a chrome-only way to notify our PictureInPictureWindow of a resize event, which it then can "forward" to web content.

The PictureInPictureService maintains the parallel queue, protecting the PiP window creation/destruction from races.

It wraps our chrome-js promises into PictureInPictureRequest, which on settling resumes the remainder of the in-parallel steps.

Co-authored-by: Simon Farre <sfarre@mozilla.com>

Assignee: jmuizelaar → sfarre
User Story: (updated)
Attachment #9509542 - Attachment description: WIP: Bug 1463402 - Part 1: Implement PiP API scaffolding → Bug 1463402 - Part 1: Implement PiP API scaffolding
Attachment #9509542 - Attachment description: Bug 1463402 - Part 1: Implement PiP API scaffolding → WIP: Bug 1463402 - Part 1: Implement PiP API scaffolding
Attachment #9543803 - Attachment description: WIP: Bug 1463402 - Part 4: Remove .ini's for tests → WIP: Bug 1463402 - Part 5: Remove .ini's for tests
No longer blocks: pip-api-use
See Also: → pip-api-use
User Story: (updated)
Attachment #9509542 - Attachment is obsolete: true
Attachment #9543801 - Attachment is obsolete: true
Attachment #9543802 - Attachment is obsolete: true
Attachment #9548621 - Attachment is obsolete: true
Attachment #9543803 - Attachment is obsolete: true

Set default false

Implements the gecko-parts of Picture In Picture API, to which the various platforms can add their implementation for opening a PIP Window to.

nsIPictureInPictureFunctions.idl shall be implemented by the different platforms. For desktop we already have a "native" PIP implementation and this will be re-used.

Platforms which does not implement nsIPictureInPictureFunctions.idl, will signal that they don't support it (and attempts to request via the API algorithms, will reject accordingly).

Adds implementation of nsIPictureInPictureFunctions.idl for desktop in browser components.

Makes changes to current native implementation by making it so that we can track the entirety of a request via chaining promises effectively by using sendQuery where we can.

Added the (made up) restriction of just having one PIP window per browsing context; technically our desktop implementation supports any number of windows. The spec "allows" for multiple windows, but it's own language actually makes it impossible, at least in current version and the idea here is to match an eventual Android implementation as close as possible and Android's native PIP support, supports just a single window (see https://source.android.com/docs/core/display/pip)

This is just to prototype the chrome js interface nsIPictureInPictureFunctions.idl to see if it's suitable to wrap all platforms.

The Android PIP implementation here involves creating a new activity, a gecko session and then we load a data URI with the minimal content we want in it (with a video element). This is a super hack, but I couldn't find any other way to make sure that the bc actually gets to live in the content process we want (where the originating video is from).

When the document is loaded, we send messages down using the js window actors, and start cloning.

No longer depends on: 1610522
Attachment #9550077 - Attachment description: WIP: Bug 1463402: Android PIP Minimal hack to enter PIP with video → WIP: Bug 1463402: Android PIP API Support with native PIP

Description:

This feature exposes the native PIP implementation to be triggered from Picture in picture API. To trigger the behavior, a website has to run the script someVideoElement.requestPictureInPicture(), potentially via clicking a button. I don't know if this is enough context, but if we have previous understanding of how Native PIP works, there should be parity here. It is up to the website to expose this feature in an accessible way.

How do we test this?

  • Set the preference dom.picture-in-picture.enabled to true either via ./mach run --setpref dom.picture-in-picture.enabled=true https://developer.mozilla.org/en-US/docs/Web/API/Picture-in-Picture_API or via about:config
  • If above command wasn't used to start, navigate to https://developer.mozilla.org/en-US/docs/Web/API/Picture-in-Picture_API
  • Scroll down to example and click button

When will this ship?

Fx 152

Tracking bug/issue:

1463402

Design documents (e.g. Product Requirements Document, UI spec):

Overview & android related exploration

Engineering lead:

[:sfarre] Simon Farre

Product manager:

[:hsinyi] Hsin-Yi Tsai

The accessibility team has developed the Mozilla Accessibility Release Guidelines which outline what is needed to make user interfaces accessible: https://wiki.mozilla.org/Accessibility/Guidelines. Please describe the accessibility guidelines you considered and what steps you've taken to address them:

I've not taken any particular steps to address them, but instead has relied on not regressing the native PIP feature on desktop, so if it's possible to defer to the a11y request made for Native PIP (if there is any that is), I will.

Describe any areas of concern to which you want the accessibility team to give special attention:

To ensure that behavior parity/expected behaviors remain the same as for Native PIP, with the exception of that this has to be triggered from web content. So if we have some a11y related review for Native PIP, I'd like to defer to that while main focus would be no regressions/difference beyond how it's triggered.

If there are issues with the request, I'll gladly address these, just let me know!

Edit:
Note: The android patch of this stack is just a prototype and it's possible it will be discarded entirely. So disregard that one.

a11y-review: --- → requested
User Story: (updated)
No longer blocks: 1890774
See Also: → 1890774
User Story: (updated)
a11y-review: requested → assigned
User Story: (updated)

As expected, this all looks pretty solid on desktop. Since the only browser UI part is the same as the existing PiP UI, and since the API is triggered by web content which the accessibility engine already makes accessible, there are no accessibility concerns regarding any UI specific to this implementation. This has led me to discover a number of accessibility bugs with the existing PiP UI, but those already apply to native PiP, so those don't block this work.

One open question/minor concern is how this should interact with native PiP. If you start native PiP (e.g. with the keyboard shortcut), document.pictureInPictureElement is null. If you then call video.requestPictureInPicture, this closes PiP. This impacts everyone and is not specific to accessibility. However, I think it disproportionately impacts users who primarily use the keyboard instead of the mouse (including screen reader users) because such users are more likely to use the native keyboard shortcut to open PiP, since many video players on the web don't have proper keyboard and semantic accessibility. Ideally, I think the state would be synchronised somehow, but I can understand that this might be tricky within the bounds of the API. If it's not possible within the current API, it does seem like an issue worth raising against the API, as surely this impacts any other browser with any form of native PiP as well.

:sfarre, is there anything we can do here?

Flags: needinfo?(sfarre)

(In reply to James Teh [:Jamie] from comment #36)

One open question/minor concern is how this should interact with native PiP. If you start native PiP (e.g. with the keyboard shortcut), document.pictureInPictureElement is null. If you then call video.requestPictureInPicture, this closes PiP. This impacts everyone and is not specific to accessibility. However, I think it disproportionately impacts users who primarily use the keyboard instead of the mouse (including screen reader users) because such users are more likely to use the native keyboard shortcut to open PiP, since many video players on the web don't have proper keyboard and semantic accessibility. Ideally, I think the state would be synchronised somehow, but I can understand that this might be tricky within the bounds of the API. If it's not possible within the current API, it does seem like an issue worth raising against the API, as surely this impacts any other browser with any form of native PiP as well.

So, the reason why this closes, is because of a minor check inside our current native picture in picture implementation. Like you're already saying; we can synchronize this somehow. This also only happens if the PIP API is triggered for a video element that is already in Native PIP. The check is there to protect (for our Native PIP) against racyness (see here: https://searchfox.org/firefox-main/source/toolkit/actors/PictureInPictureChild.sys.mjs#182). I believe I can maintain spec correctness, by just checking for that up front (i.e. the PIP API will resolve immediately, because native pip implementation already has "served us" with a window as it were).

Not as important, but a side note:
I'm not sure we can write spec with native PIP in mind, given that the browsers (and platforms) differ in support and behavior. For instance, chrome only supports 1 PIP window in total (though should be able to support more). Safari seems like it's actually constrained by the OS to not have multiple though, it's not clear to me if that assumption is true. For mobile (android) there the OS only supports 1 PIP window in total, and it comes with constraints on how it can receive inputs.

Flags: needinfo?(sfarre)
Attachment #9550072 - Attachment description: WIP: Bug 1463402: Add pref for PIP API → Bug 1463402: Add pref for PIP API r=#dom-core!
Attachment #9550073 - Attachment description: WIP: Bug 1463402: Implement pseudo class → Bug 1463402: Implement pseudo class r=#dom-core!
Attachment #9550074 - Attachment description: WIP: Bug 1463402: Implement Picture In Picture API → Bug 1463402: Implement Picture In Picture API r=#dom-core!
Attachment #9550076 - Attachment description: WIP: Bug 1463402: Implement Desktop support for PIP API → Bug 1463402: Implement Desktop support for PIP API r=#dom-core!

(In reply to Simon Farre [:sfarre] from comment #37)

I believe I can maintain spec correctness, by just checking for that up front (i.e. the PIP API will resolve immediately, because native pip implementation already has "served us" with a window as it were).

I see you've done this now; thank you. I also see that this sets document.pictureInPictureElement when this resolves, even if we just "reused" the native PIP window.

I'm not sure we can write spec with native PIP in mind, given that the browsers (and platforms) differ in support and behavior. For instance, chrome only supports 1 PIP window in total (though should be able to support more). Safari seems like it's actually constrained by the OS to not have multiple though, it's not clear to me if that assumption is true. For mobile (android) there the OS only supports 1 PIP window in total, and it comes with constraints on how it can receive inputs.

Understood. But is it non-compliant to set document.pictureInPictureElement if native PIP opens? That would at least mean that any JS code that relies on that being set to determine whether PIP is already open will also detect the native case. This might also be useful for the pseudo class. I can see why this might be a problem though; e.g. if there's no way for the browser to notify the page that PIP opened without the author provided trigger being used. Are there other problems with that I'm missing related to multiple PIP windows, etc.?

In any case, with your fix above, I think this decreases the importance of this issue even further.

Speaking of the pseudo class, can you give me some examples of how this might be used in practice? As I understand it, this will match on the element matching document.pictureInPictureElement; e.g. the video which is currently in PIP. I assume this might be used by the author to display some indicator that the video is in PIP, etc.? The reason I ask is that pseudo classes are often accompanied by some sort of state in DOM; e.g. :invalid matches invalid form fields, but invalid form fields also expose a semantic "invalid" state to accessibility. This is useful because these pseudo classes are often just styled using visual only indication (colour, icon, etc.). My hope is that authors would generally include some text indication with the PIP pseudo class as well, but if this isn't normally the case in the wild (i.e. if the indication is usually only visual in nature), we might need to figure out some other way to expose this to accessibility clients in future.

(In reply to James Teh [:Jamie] from comment #38)

Understood. But is it non-compliant to set document.pictureInPictureElement if native PIP opens? That would at least mean that any JS code that relies on that being set to determine whether PIP is already open will also detect the native case. This might also be useful for the pseudo class. I can see why this might be a problem though; e.g. if there's no way for the browser to notify the page that PIP opened without the author provided trigger being used. Are there other problems with that I'm missing related to multiple PIP windows, etc.?

I actually checked Google chrome's "native" approach, and in fact they have the same behavior your request here, which makes me believe I should probably move to also do that.

Speaking of the pseudo class, can you give me some examples of how this might be used in practice? As I understand it, this will match on the element matching document.pictureInPictureElement; e.g. the video which is currently in PIP. I assume this might be used by the author to display some indicator that the video is in PIP, etc.? The reason I ask is that pseudo classes are often accompanied by some sort of state in DOM; e.g. :invalid matches invalid form fields, but invalid form fields also expose a semantic "invalid" state to accessibility. This is useful because these pseudo classes are often just styled using visual only indication (colour, icon, etc.). My hope is that authors would generally include some text indication with the PIP pseudo class as well, but if this isn't normally the case in the wild (i.e. if the indication is usually only visual in nature), we might need to figure out some other way to expose this to accessibility clients in future.

I wouldn't know exactly how the pseudo class is intended to be used to be honest beyond what you already mention. Given that I'm also addressing the PIP spec in parallel, do you have any suggestions that would be useful, or any potential spec-language (for other things, similar or not) that potentially could be cargo culted (and also implemented, of course)? Does there exist some standards defined way to notify a11y technology or something?

(In reply to Simon Farre [:sfarre] from comment #39)

(In reply to James Teh [:Jamie] from comment #38)

Speaking of the pseudo class, can you give me some examples of how this might be used in practice? As I understand it, this will match on the element matching document.pictureInPictureElement; e.g. the video which is currently in PIP. I assume this might be used by the author to display some indicator that the video is in PIP, etc.? The reason I ask is that pseudo classes are often accompanied by some sort of state in DOM; e.g. :invalid matches invalid form fields, but invalid form fields also expose a semantic "invalid" state to accessibility. This is useful because these pseudo classes are often just styled using visual only indication (colour, icon, etc.). My hope is that authors would generally include some text indication with the PIP pseudo class as well, but if this isn't normally the case in the wild (i.e. if the indication is usually only visual in nature), we might need to figure out some other way to expose this to accessibility clients in future.

I wouldn't know exactly how the pseudo class is intended to be used to be honest beyond what you already mention. Given that I'm also addressing the PIP spec in parallel, do you have any suggestions that would be useful, or any potential spec-language (for other things, similar or not) that potentially could be cargo culted (and also implemented, of course)? Does there exist some standards defined way to notify a11y technology or something?

Just for reference, the :picture-in-picture pseudo-class is defined here:
https://drafts.csswg.org/selectors/#pip-state
https://w3c.github.io/picture-in-picture/#css-pseudo-class

Sebastian

(In reply to Simon Farre [:sfarre] from comment #39)

I actually checked Google chrome's "native" approach, and in fact they have the same behavior your request here, which makes me believe I should probably move to also do that.

I do think that would have a better chance of behaving well when a user invokes using native PIP, but for some reason dismisses using a PIP API trigger.

I wouldn't know exactly how the pseudo class is intended to be used to be honest beyond what you already mention.

Fair enough. Can you clarify for me whether the pseudo class applies to the PIP element in the opener or the PIP window? I thought it was the PIP element in the opener, but the CSS definition made me question that.

Given that I'm also addressing the PIP spec in parallel, do you have any suggestions that would be useful, or any potential spec-language (for other things, similar or not) that potentially could be cargo culted (and also implemented, of course)? Does there exist some standards defined way to notify a11y technology or something?

There are standardised ways to expose certain states to accessibility; e.g. required, invalid, checked, pressed. There is not a standard way to expose that something is picture-in-picture. For now, the best thing we can probably do is note in the spec that authors should be aware that there is no implicit exposure of this state to accessibility tools, so any indication should expose text or there should be another indication somewhere nearby in the document.

FWIW, there is a new spec being spun up by the ARIA Working Group for CSS Accessibility API Mappings. I think that spec should mention this as well. However, it isn't far enough along yet for this to be included yet.

Testing with Chrome on MacOS and checking the a11y tree in devtools and the AX tree in Accessibility Inspector, I notice these things:

  • When a video is PiP'd, visually the in-page video box has the text "Playing in picture-in-picture". This text is not exposed to a11y.
  • When a page has a PiP, the window's title (AX "Top Level UIElement") includes "Video playing in picture-in-picture mode".
  • When a video is PiP'd, the AX tree has Children for the play/pause button and the scrub slider, etc. Regardless of whether there is a controls attribute. I think this means screen reader users should be able to navigate to the controls in the PiP window as if they were children of the in-page video element.

If we want to standardize the third bullet point, https://w3c.github.io/html-aam/#el-video should refer to https://html.spec.whatwg.org/#expose-a-user-interface-to-the-user instead of "If the controls attribute is present", and then HTML can say that UAs should "expose a user interface to the user" if the video is PiP'd, regardless of the controls attribute.

(In reply to James Teh [:Jamie] from comment #41)

(In reply to Simon Farre [:sfarre] from comment #39)

I wouldn't know exactly how the pseudo class is intended to be used to be honest beyond what you already mention.

Fair enough. Can you clarify for me whether the pseudo class applies to the PIP element in the opener or the PIP window? I thought it was the PIP element in the opener, but the CSS definition made me question that.

It matches the element in the opener. The PiP window can't be styled, at the moment, as far as I know. I've created a request to clarify the CSS specification.

Sebastian

(In reply to James Teh [:Jamie] from comment #41)

I do think that would have a better chance of behaving well when a user invokes using native PIP, but for some reason dismisses using a PIP API trigger.

Yes that is reasonable I think.

Fair enough. Can you clarify for me whether the pseudo class applies to the PIP element in the opener or the PIP window? I thought it was the PIP element in the opener, but the CSS definition made me question that.

Like :sebo says, it's for the original video element, not the one in the picture in picture window. As far as I understand the feature, the difference between document pip and media pip, is that media pip window is in control of the user agent, so, web content is not actually seeing anything or controlling anything there, in any direct way at least (meaning, there's no access to the <video> element in that window).

There are standardised ways to expose certain states to accessibility; e.g. required, invalid, checked, pressed. There is not a standard way to expose that something is picture-in-picture. For now, the best thing we can probably do is note in the spec that authors should be aware that there is no implicit exposure of this state to accessibility tools, so any indication should expose text or there should be another indication somewhere nearby in the document.

FWIW, there is a new spec being spun up by the ARIA Working Group for CSS Accessibility API Mappings. I think that spec should mention this as well. However, it isn't far enough along yet for this to be included yet.

Ok, given this and what :zcorpan says it means there can be room for improvement, spec wise.

Ok, latest change makes "Native PIP" and "PIP API" be observed in the same way from web content. Entering PIP "natively" will now fire enter/leave events and this gets us webcompatible with chrome as well. It also means that that document.pictureInPictureElement is also non-null if native PIP was used (which also, is webcompatible with chrome).

(In reply to Simon Pieters [:zcorpan] from comment #42)

  • When a video is PiP'd, visually the in-page video box has the text "Playing in picture-in-picture". This text is not exposed to a11y.

FWIW, Firefox exposes the text "This video is playing in Picture-in-Picture mode." to accessibility. I assume it's also visible too.

  • When a video is PiP'd, the AX tree has Children for the play/pause button and the scrub slider, etc. Regardless of whether there is a controls attribute. I think this means screen reader users should be able to navigate to the controls in the PiP window as if they were children of the in-page video element.

This is also true in Firefox.

then HTML can say that UAs should "expose a user interface to the user" if the video is PiP'd, regardless of the controls attribute.

I do think that would be nice to standardise if there's no reason not to do so.

(In reply to Simon Farre [:sfarre] from comment #45)

Ok, latest change makes "Native PIP" and "PIP API" be observed in the same way from web content. Entering PIP "natively" will now fire enter/leave events and this gets us webcompatible with chrome as well. It also means that that document.pictureInPictureElement is also non-null if native PIP was used (which also, is webcompatible with chrome).

Excellent. Thank you.

I'm happy with this overall and from a screen reader and keyboard perspective. Natalie on my team will be doing a few visual spot checks before we close out the accessibility review, but aside from any existing issues with the native video PIP window, I don't anticipate any further issues here.

So one thing that was raised when discussing the PIP API feature with :pehrsons, is that, by having native PIP be web observable, web sites can effectively "block" our native PIP. I am not sure that's acceptable. The way they would do that, is effectively listen for enter picture in picture events, and in that case, exit immediately.

If we don't emit that event for native pip as a way to circumvent the first trivial block-approach, sites would still trivially block by periodically checking document.pictureInPictureElement and then exit that way.

cc'ing :mconley

Flags: needinfo?(mconley)

My concerns are essentially:

  • As a user, I don't want a site to care or know that I open native-pip.
  • Even if we sync native-pip with api-pip for a single pip-window; opening multiple native-pip windows will mis-align with the api-pip model anyway (document.pictureInPictureElement, document.exitPictureInPicture are enforcing at-most 1 pip per document).
  • As mentioned, exposing native-pip through api-pip events and attributes will allow sites to effectively block native-pip, which would be a regression.

I would have leaned towards the solution discussed in comment 38, comment 41, etc; to allow distinct native-pip and api-pip windows where api-pip doesn't learn of a native-pip window (by neither events nor attributes) until it requests pip for the same video element. The main concern seems to be that the page cannot collaborate with native-pip if it doesn't know of its state, so the UI might end up confusing. That is not different from today. Native-pip has been a differentiator for us, and it can still be with this solution. If we align with Chrome we lose (most of) that.

It would be good with some guidance from frontend here so we don't regress something we shouldn't.

Ah, that's a great point. If we can't do this for that reason, I think we can live with this from an accessibility perspective. If someone opens via native PIP (which I think will be more likely for keyboard/screen reader users given the generally poor accessibility of author-implemented video controls), it means the state won't be synchronised, but they can still close native PIP, so that's a minor quirk. I think the downside of a site potentially blocking native PIP outweighs the slightly quirky UX of native and API PIP being unsynchronised, for accessibility and otherwise.

We could effectively also implement PIP API to throw an exception if Native PIP is engaged for a video element and PIP API tries to open PIP for that same element.

The PR that I have waiting to be landed (and the one I am implementing against) has these steps added to be able to deal with whatever platform/OS errors one may run into when attempting to open a PIP window & configuring it. It was specifically added to leave some room for potential differences between browsers. We could simply abort there and what ultimately is seen is an InvalidStateError exception being thrown.

If we believe that PIP API in any way can hamper our Native PIP or have a11y-related stuff be less good, I think we're perfectly free to reject in that scenario.

(In reply to Simon Farre [:sfarre] from comment #50)

We could effectively also implement PIP API to throw an exception if Native PIP is engaged for a video element and PIP API tries to open PIP for that same element.
...
If we believe that PIP API in any way can hamper our Native PIP or have a11y-related stuff be less good, I think we're perfectly free to reject in that scenario.

In terms of accessibility, I don't think this change helps. My (minor) concern was opening with native and then having API out of sync (because accessibility users might be more likely to open using native), but as noted above, we can't really have them in sync without the risk of the site blocking our native PIP. That said, I do appreciate your efforts to make this as accessible as possible.

Attachment #9562675 - Attachment is obsolete: true
User Story: (updated)
Attachment #9550077 - Attachment description: WIP: Bug 1463402: Android PIP API Support with native PIP → WIP: Bug 1463402 - Android PIP & PIP API

Proof of concept on how to trigger PIP from a context menu.

Proof of concept on how to open PIP from a media notification.

Added test to verify that iframe attribute is honored.

Tests the following scenario
Steps:

  • video1 enters pip
  • video2 requests pip

Observe that video1 has its event handlers fire before video2 sees its enter event. Observe that when video2 enter handler runs, document.pictureInPictureElement is null (i.e. not video1).

At the moment, this requires a change to the exit algorithm, because it always queues a task to fire the exit event in step 3, which would mean that it would always come after the enter event for video2.

Tested on Chrome & Gecko. Both pass.

Previous Shadow DOM test were nowhere near enough covering. This tests
both current state of spec, but also
https://github.com/w3c/picture-in-picture/issues/248.

https://w3c.github.io/picture-in-picture/#exit-picture-in-picture-algorithm
Explains "As one of the unloading document cleanup steps, run the exit
Picture-in-Picture algorithm.".

This test verifies that browsers implement this.

Keywords: web-feature
Blocks: 2040930
Attachment #9550073 - Attachment description: Bug 1463402: Implement pseudo class r=#dom-core! → Bug 1463402: Implement pseudo class :picture-in-picture r=#dom-core!
See Also: → 2034415
Pushed by sfarre@mozilla.com: https://github.com/mozilla-firefox/firefox/commit/21de73ae0d6a https://hg.mozilla.org/integration/autoland/rev/8020b58921e3 Add pref for PIP API r=dom-core,smaug https://github.com/mozilla-firefox/firefox/commit/f28faf6aecd3 https://hg.mozilla.org/integration/autoland/rev/77896b3fa851 Implement pseudo class :picture-in-picture r=firefox-style-system-reviewers,emilio https://github.com/mozilla-firefox/firefox/commit/ae7d9fe07a1e https://hg.mozilla.org/integration/autoland/rev/2ef3734b6b65 Implement Picture In Picture API r=dom-core,webidl,emilio,smaug https://github.com/mozilla-firefox/firefox/commit/28caa069efc2 https://hg.mozilla.org/integration/autoland/rev/2f30f43b7894 Implement Desktop support for PIP API r=kpatenio,mconley https://github.com/mozilla-firefox/firefox/commit/8a0c3b75e0ec https://hg.mozilla.org/integration/autoland/rev/724101fc56c6 Add WPT for iframe PIP attribute testing r=dom-core,smaug https://github.com/mozilla-firefox/firefox/commit/41603be05b92 https://hg.mozilla.org/integration/autoland/rev/e48889dbf961 Add WPT for exit-when-having-pip ordering is observed r=smaug https://github.com/mozilla-firefox/firefox/commit/f91c00df98d6 https://hg.mozilla.org/integration/autoland/rev/d9f6b2678144 Add additional Shadow DOM tests r=smaug https://github.com/mozilla-firefox/firefox/commit/b3671a33e5d9 https://hg.mozilla.org/integration/autoland/rev/7e8cd410c9a3 Add WPT for unloading steps r=smaug https://github.com/mozilla-firefox/firefox/commit/63b2bf96a34a https://hg.mozilla.org/integration/autoland/rev/c4a12a7baadd Flip pref for dom.media-pip.enabled r=dom-core,smaug

Created web-platform-tests PR https://github.com/web-platform-tests/wpt/pull/60166 for changes under testing/web-platform/tests

Whiteboard: [geckoview:p3][webcompat] → [geckoview:p3][webcompat], [wptsync upstream]
Pushed by sstanca@mozilla.com: https://github.com/mozilla-firefox/firefox/commit/1643f0215ddb https://hg.mozilla.org/integration/autoland/rev/12f8b22269de Revert "Bug 1463402: Flip pref for dom.media-pip.enabled r=dom-core,smaug" for causing multiple failures.

Fixed issues and waiting for try run results at https://treeherder.mozilla.org/jobs?repo=try&revision=bcb013f13cc4d2d086f188ac8031c60fdcad9a9d before landing again.

Flags: needinfo?(sfarre)
Pushed by sfarre@mozilla.com: https://github.com/mozilla-firefox/firefox/commit/fdb187c0b7a5 https://hg.mozilla.org/integration/autoland/rev/6a4c766499e7 Add pref for PIP API r=dom-core,smaug https://github.com/mozilla-firefox/firefox/commit/81eed66bbb3d https://hg.mozilla.org/integration/autoland/rev/d4501818a928 Implement pseudo class :picture-in-picture r=firefox-style-system-reviewers,emilio https://github.com/mozilla-firefox/firefox/commit/70a87cf3ed48 https://hg.mozilla.org/integration/autoland/rev/ea966f94a0ed Implement Picture In Picture API r=dom-core,webidl,emilio,smaug https://github.com/mozilla-firefox/firefox/commit/317a79f02b9c https://hg.mozilla.org/integration/autoland/rev/6900a5708674 Implement Desktop support for PIP API r=kpatenio,mconley https://github.com/mozilla-firefox/firefox/commit/c37750feba29 https://hg.mozilla.org/integration/autoland/rev/07eef1d8d959 Add WPT for iframe PIP attribute testing r=dom-core,smaug https://github.com/mozilla-firefox/firefox/commit/21b4ab653875 https://hg.mozilla.org/integration/autoland/rev/20f434c8a54f Add WPT for exit-when-having-pip ordering is observed r=smaug https://github.com/mozilla-firefox/firefox/commit/23e4f81816b0 https://hg.mozilla.org/integration/autoland/rev/7845fbc3c823 Add additional Shadow DOM tests r=smaug https://github.com/mozilla-firefox/firefox/commit/21c5864802a4 https://hg.mozilla.org/integration/autoland/rev/096639ac2610 Add WPT for unloading steps r=smaug https://github.com/mozilla-firefox/firefox/commit/c59eb47e0f04 https://hg.mozilla.org/integration/autoland/rev/5d5511139d84 Flip pref for dom.media-pip.enabled r=dom-core,smaug,webidl

Upstream PR was closed without merging

Pushed by sstanca@mozilla.com: https://github.com/mozilla-firefox/firefox/commit/9128793aef11 https://hg.mozilla.org/integration/autoland/rev/c9f2ca85aa2c Revert "Bug 1463402: Flip pref for dom.media-pip.enabled r=dom-core,smaug,webidl" for causing multiple failures.
  • Had forgotten an interface test
  • Apparently macOS has WebDriver issues where trusted clicks are not going through?

Try run: https://treeherder.mozilla.org/jobs?repo=try&landoInstance=lando-prod-2025&landoCommitID=50287

Flags: needinfo?(sfarre)

Upstream PR was closed without merging

Pushed by sfarre@mozilla.com: https://github.com/mozilla-firefox/firefox/commit/ddfd358304ca https://hg.mozilla.org/integration/autoland/rev/3d57991dbfd9 Add pref for PIP API r=dom-core,smaug https://github.com/mozilla-firefox/firefox/commit/f28624cdaf1f https://hg.mozilla.org/integration/autoland/rev/8751a4584f41 Implement pseudo class :picture-in-picture r=firefox-style-system-reviewers,emilio https://github.com/mozilla-firefox/firefox/commit/f5d05b6f480e https://hg.mozilla.org/integration/autoland/rev/49b398ec2b5b Implement Picture In Picture API r=dom-core,webidl,emilio,smaug https://github.com/mozilla-firefox/firefox/commit/d8d8edc96cde https://hg.mozilla.org/integration/autoland/rev/ea283584914c Implement Desktop support for PIP API r=kpatenio,mconley https://github.com/mozilla-firefox/firefox/commit/9c82850502c4 https://hg.mozilla.org/integration/autoland/rev/6136a73417c6 Add WPT for iframe PIP attribute testing r=dom-core,smaug https://github.com/mozilla-firefox/firefox/commit/0801a3e7004b https://hg.mozilla.org/integration/autoland/rev/cdf58afc5c00 Add WPT for exit-when-having-pip ordering is observed r=smaug https://github.com/mozilla-firefox/firefox/commit/3192646985fd https://hg.mozilla.org/integration/autoland/rev/b49f8c11b14f Add additional Shadow DOM tests r=smaug https://github.com/mozilla-firefox/firefox/commit/8f4e3752fd4f https://hg.mozilla.org/integration/autoland/rev/cf25736f8bd5 Add WPT for unloading steps r=smaug https://github.com/mozilla-firefox/firefox/commit/d388c643aa56 https://hg.mozilla.org/integration/autoland/rev/bbca0c920a6c Flip pref for dom.media-pip.enabled r=dom-core,smaug,webidl
Pushed by imoraru@mozilla.com: https://github.com/mozilla-firefox/firefox/commit/ed06c06b94b2 https://hg.mozilla.org/integration/autoland/rev/ed0cef610c78 Revert "Bug 1463402: Flip pref for dom.media-pip.enabled r=dom-core,smaug,webidl" for causing dt failures on browser_webconsole_object_inspector_entries.js.

Revert for causing dt failures on browser_webconsole_object_inspector_entries.js.

Flags: needinfo?(sfarre)

That seems like an intermittent failure and not caused by this landing?

There are other failures related to WPT in that though.

Edit: Found the invocation UPDATE_SNAPSHOT=true ./mach test devtools/client/webconsole/test/browser/browser_webconsole_object_inspector_entries.js --headless to solve it.

Flags: needinfo?(sfarre)
Depends on: 2042970
Depends on: 2043046
Pushed by sfarre@mozilla.com: https://github.com/mozilla-firefox/firefox/commit/2beb01dcef10 https://hg.mozilla.org/integration/autoland/rev/e515978561d4 Add pref for PIP API r=dom-core,smaug https://github.com/mozilla-firefox/firefox/commit/2e3ab76d53d8 https://hg.mozilla.org/integration/autoland/rev/abc2d5db3571 Implement pseudo class :picture-in-picture r=firefox-style-system-reviewers,emilio https://github.com/mozilla-firefox/firefox/commit/3842c6faa40a https://hg.mozilla.org/integration/autoland/rev/54f426721981 Implement Picture In Picture API r=dom-core,webidl,emilio,smaug,devtools-reviewers,ochameau https://github.com/mozilla-firefox/firefox/commit/c9cd940b88c5 https://hg.mozilla.org/integration/autoland/rev/9428a6ee5ec6 Implement Desktop support for PIP API r=kpatenio,mconley https://github.com/mozilla-firefox/firefox/commit/dbf90cfe0d11 https://hg.mozilla.org/integration/autoland/rev/5a7730602065 Add WPT for iframe PIP attribute testing r=dom-core,smaug https://github.com/mozilla-firefox/firefox/commit/160de9227f38 https://hg.mozilla.org/integration/autoland/rev/25bfb6334637 Add WPT for exit-when-having-pip ordering is observed r=smaug https://github.com/mozilla-firefox/firefox/commit/7df07e6a220a https://hg.mozilla.org/integration/autoland/rev/ec56a068b2fe Add additional Shadow DOM tests r=smaug https://github.com/mozilla-firefox/firefox/commit/958e255d90c0 https://hg.mozilla.org/integration/autoland/rev/361c22c773cd Add WPT for unloading steps r=smaug https://github.com/mozilla-firefox/firefox/commit/a958b57333ee https://hg.mozilla.org/integration/autoland/rev/6cb0b7fb6d8b Flip pref for dom.media-pip.enabled r=dom-core,smaug,webidl

Upstream PR was closed without merging

Pushed by sstanca@mozilla.com: https://github.com/mozilla-firefox/firefox/commit/4f93cc444071 https://hg.mozilla.org/integration/autoland/rev/7071fdeb0c25 Revert "Bug 1463402: Flip pref for dom.media-pip.enabled r=dom-core,smaug,webidl" for causing Picture-in-Picture related wpt failures.

Reverted this because it was causing Picture-in-Picture related wpt failures.

Status: RESOLVED → REOPENED
status-firefox153: fixed → ---
Flags: needinfo?(sfarre)
Resolution: FIXED → ---
Target Milestone: 153 Branch → ---

Upstream PR merged by moz-wptsync-bot

It is clear that macOS has some defect with the tests. Will be disabling the tests entirely for the failing platform due to its seemingly intermittent failure (see try runs that don't contain the failures the build had).

Flags: needinfo?(sfarre)
Pushed by amarc@mozilla.com: https://github.com/mozilla-firefox/firefox/commit/dce91ccb32b0 https://hg.mozilla.org/integration/autoland/rev/c65f3a1e9edb Add pref for PIP API r=dom-core,smaug https://github.com/mozilla-firefox/firefox/commit/228705d6fd1d https://hg.mozilla.org/integration/autoland/rev/030812a8714f Implement pseudo class :picture-in-picture r=firefox-style-system-reviewers,emilio https://github.com/mozilla-firefox/firefox/commit/5107077c7d6c https://hg.mozilla.org/integration/autoland/rev/46886bcf86e8 Implement Picture In Picture API r=dom-core,webidl,emilio,smaug,devtools-reviewers,ochameau https://github.com/mozilla-firefox/firefox/commit/30d6e6586c44 https://hg.mozilla.org/integration/autoland/rev/c9a57addabe6 Implement Desktop support for PIP API r=kpatenio,mconley https://github.com/mozilla-firefox/firefox/commit/cc1092a99eb4 https://hg.mozilla.org/integration/autoland/rev/306d09e4126e Add WPT for iframe PIP attribute testing r=dom-core,smaug https://github.com/mozilla-firefox/firefox/commit/804fb88d9b7f https://hg.mozilla.org/integration/autoland/rev/560c147d61e8 Add WPT for exit-when-having-pip ordering is observed r=smaug https://github.com/mozilla-firefox/firefox/commit/a7c2f26ff4f6 https://hg.mozilla.org/integration/autoland/rev/51b69798195c Add additional Shadow DOM tests r=smaug https://github.com/mozilla-firefox/firefox/commit/3d496341e152 https://hg.mozilla.org/integration/autoland/rev/5c20d9f55f50 Add WPT for unloading steps r=smaug https://github.com/mozilla-firefox/firefox/commit/fda5f34e9137 https://hg.mozilla.org/integration/autoland/rev/545b5ff878db Flip pref for dom.media-pip.enabled r=dom-core,smaug,webidl https://github.com/mozilla-firefox/firefox/commit/20c4b6ebf1b2 https://hg.mozilla.org/integration/autoland/rev/1c60e4500c8f Adjust metadata for picture-in-picture tests on macos

Created web-platform-tests PR https://github.com/web-platform-tests/wpt/pull/60273 for changes under testing/web-platform/tests

Regressions: 2043554

Upstream PR merged by moz-wptsync-bot

Simon, is that something we should include in the Web platform section of our general release notes?
https://wiki.mozilla.org/Release_Management/Release_Notes_Nomination

Flags: needinfo?(sfarre)
No longer depends on: 2042970
See Also: → 2042970

Release Note Request
[Why is this notable]: Release of PIP API support.
[Affects Firefox for Android]: No, Android needs a PIP implementation before deciding on whether it should expose that implementation to the PIP API.
[Suggested wording]:
Picture In Picture API exposes a way for web developers to make use of user agent Picture-In-Picture functionality.
[Links (documentation, blog post, etc)]:
MDN: https://developer.mozilla.org/en-US/docs/Web/API/Picture-in-Picture_API

relnote-firefox: --- → ?
Flags: needinfo?(sfarre)

Thanks, note added to our 153 nightly release notes in the Web Platform section with this wording:

The Picture In Picture API exposes a way for web developers to make use of user agent Picture-In-Picture functionality.

I am keeping the relnote-firefox? flag set until we include this note in our final 153 release notes, thanks.

Regressions: 2044672
No longer blocks: 1852768
Summary: Implement Picture-in-Picture Web API → Implement Picture-in-Picture Web API on Desktop

I am keeping the relnote-firefox? flag set until we include this note in our final 153 release notes, thanks.

(Change of internal process)

relnote-firefox: ? → 153+
Keywords: relnote-cms
QA Whiteboard: [qa-triage-done-c154/b153]
Regressions: 2048580

FF153 MDN work for this can be tracked in https://github.com/mdn/content/issues/44464 (Done, in review)

Keywords: dev-doc-needed → dev-doc-complete

Bug was resolved, but the web-feature picture-in-picture (definition file) is not yet marked as supported in Firefox.

Feature bugs are usually automatically closed once the corresponding web-features are marked as supported; this typically happens after the feature reaches release.

Status: RESOLVED → REOPENED
Keywords: parity-chrome, parity-safari
Resolution: FIXED → ---
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: