1551058 - support subtitles/captions in picture-in-picture mode (PiP)

Status: RESOLVED DUPLICATE of bug 1751505

(Toolkit :: Picture-in-Picture, enhancement, P5)

Description

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Support for subtitles in picture-in-picture mode has been requested https://discourse.mozilla.org/t/picture-in-picture-not-bringing-subtitles-in/40231

That makes especially sense if the video is not in the native language of the viewer.

Steps to reproduce:

1. Find a video with closed captions. Used https://www.youtube.com/watch?v=6zsGHba61Sk
2. Turn on subtitles ("CC" button).
3. Right click video. Youtube's menu opens.
4. Right click somewhere else on the video. Firefox's menu for the video opens.
5. Select Picture-in-Picture.

Actual result:
Mini-player opens but subtitles are missing.
Expected result:
Subtitles shown.

Comment 1

[pablo.r.arevalo]

Even more frustratingly, the subtitles open in the original video frame tab on sites such as Crunchyroll. Otherwise, the picture in picture mode works well. I can drag the pop up window to other monitors, scale it to whatever size I want, but the subtitles remain with the original tab, over the placeholder that tells me the video is playing in Picture-in-picture mode...

Comment 2

[jose.eduardo.jr]

Have the same problem with Netflix. Subtitles are not displayed at PiP screen (they continue to be displayed at original tab)

Comment 3

[weiluntong.inbox]

Figured I'd chime in and add another test case: https://crunchyroll.com

Comment 5

[Marcela (Please NI request to Brindusa Tot)]

This behavior is reproducible on Nightly version 75 on macOS 10.14. Changed flags accordingly.

Comment 6

[Sagar]

Have this same issue on macOS 10.15.3 on Nightly v76 on videos from YT/GDrive

Comment 7

[Seba S.]

Hi, same issue happening on Windows 10 Nightly 76.0a1 (2020-03-17) (64-bit)

Regards,

Comment 8

[Jaden Curley]

Hello, I have a couple concerns to add to the table. Specifically around feature accessibility.

Right now, for those who are hearing-impaired or deaf, the PiP solution provided by all builds(as of the time of writing) is very nice but does not have subtitle support. For those in the deaf-community, it is inconvenient to not have access to this feature in the same way that those with full hearing do.

To compound that, even for those with full hearing, access to this feature would allow them to essentially watch videos without having to have sound enabled. If you were working on something in a public place and had a video tutorial open in PiP, you must have sound on in order to get through the video. Implementation of this enhancement would assist

Overall, by definition, this provides a hurdle for some to use this feature of this product. Since it isn't the most devastating of losses to not be able to use PiP fully if you are deaf, it does provide, if not an accessibility challenge, then at the very least a consideration for accessibility going forward.

Comment 9

[Chua Chee Wee]

Attachment: Subtitles showing in original frame, not in the Picture-in-Picture frame.

Subtitles when enabled on Youtube, shows in the original frame, not the Picture-in-Picture mode.
Using Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:76.0) Gecko/20100101 Firefox/76.0 Build ID: 20200429185419

Comment 10

[Chua Chee Wee]

Comment on attachment 9146086 [details]
Subtitles showing in original frame, not in the Picture-in-Picture frame.

Subtitles showing in original frame, not in the Picture-in-Picture frame.

Comment 11

[Marcela (Please NI request to Brindusa Tot)]

This issue is reproducible on latest Nightly 78.0a1 (2020-05-12) on macOS 10.14. Changed flags accordingly.

Comment 12

[BugBot [:suhaib / :marco/ :calixte]]

Change the status for beta to have the same as nightly and release.
For more information, please visit auto_nag documentation.

Comment 13

[jhgorse]

Confirmed as an issue on a fresh build from version control.

Can someone point me in the direction of the relevant code sections for 1) Closed Captioning and 2) Picture-in-Picture? The code-base is massive. That being said, I am pleased to report that the build instructions "just worked" for me on macOS. Cheers!

Comment 14

[scram]

although the best-known platform is youtube and netflix, this issue affects other video-on-demand platforms.

this also applies to windows

Comment 15

[nathanwoodcodes]

Netflix's subtitles are formatted in the DOM like this:

<div>
    <video></video>
    <div class="player-timedtext"><div class="player-timedtext-text-container">
        <span>...</span>
        <span>...</span>
    </div></div>
</div>

Each span contains a line of subtitles.

Youtube's are more complex:

<div class="html5-video-container">
    <video></video>
</div>
<div class="caption-window ytp-caption-window-bottom ytp-caption-window-rollup" id="caption-window-1">
    <span class="caption-visual-line"><span class="ytp-caption-segment">...</span></span>
    <span class="caption-visual-line"><span class="ytp-caption-segment">...</span></span>
</div>

Each span/span contains a line of subtitles, however when using automatic subtitles, it appears that they each contain a list of text nodes, whereas in properly subtitled videos it appears to all be contained in one text node.

Regardless, this will require reading the DOM and watching for mutations. The popout video player will either need the ability to render text directly onto the drawing, or put HTML on top.

Comment 16

[Vincent Bernat]

When using HTML5 video, the subtitles don't appear at all: not in the popout, not in the original location.

Comment 17

[Seba S.]

Hello, confirmed on Ubuntu 20.04 LTS Nightly 79.0a1 (2020-06-22)(64-bit).

Regards,

Comment 18

[Mike Conley (:mconley) (:⚙️)]

(In reply to jhgorse from comment #13)

Can someone point me in the direction of the relevant code sections for 1) Closed Captioning and 2) Picture-in-Picture? The code-base is massive. That being said, I am pleased to report that the build instructions "just worked" for me on macOS. Cheers!

The Picture-in-Picture code is hosted mainly in the following files:

• https://searchfox.org/mozilla-central/source/toolkit/components/pictureinpicture/PictureInPicture.jsm is where most of the parent process orchestration occurs
• https://searchfox.org/mozilla-central/source/toolkit/actors/PictureInPictureChild.jsm is where most of the content process orchestration occurs
• https://searchfox.org/mozilla-central/source/toolkit/components/pictureinpicture/content/player.js is the script that controls the player window

I'm actually not 100% certain where the subtitles and captioning code that we use for HTML video is. The contents of dom/media/webvtt seem to handle the parsing of VTT files and the construction of the actual text boxes that appear over videos - but it looks like there are some hooks into native code here to inject them into an anonymous content area in DOM. I'm not 100% familiar with how that works.

Comment 19

[Mike Conley (:mconley) (:⚙️)]

I had some in-tree documentation for Picture-in-Picture in-flight before I got pulled off onto other things that you might find useful: https://bugzilla.mozilla.org/show_bug.cgi?id=1589158

Comment 20

[victortorrealba.es01]

spam

Comment 21

[Negritas Sergiu, Desktop QA]

This issue is reproducible on latest Nightly 80.0a1 (2020-07-07) on macOS 10.15.

Comment 22

[Luke]

I don't even think that's using it loosely! Sounds fair.
https://www.njmc.website/

Comment 23

[Brindusa Tot, Desktop QA]

This issue is reproducible on the latest Nightly 81.0a1 (2020-08-04)

Comment 25

[Alexandra Martin]

Issue reproduced on latest Nightly 82.0a1 (20200902095359) using a MacOS 10.14.6 device.

Comment 26

[Koala Yeung]

I understand that things are complicated if the video vendors who do their own caption display implementations (i.e. They'd probably need some new custom API to expose captions to picture-in-picture video popup). But things don't even work if the video vendor using HTML5's own VTT caption with <track> element.

<video id="wistia_simple_video_178" crossorigin="anonymous" style="background: transparent none repeat scroll 0% 0%; display: block; height: 100%; max-height: none; max-width: none; position: static; visibility: visible; width: 100%; object-fit: fill;" poster="https://fast.wistia.com/assets/images/blank.gif" aria-label="Video" defaultplaybackrate="1" src="blob:https://codewithmosh.com/some-video-uuid" controlslist="nodownload" playsinline="" preload="metadata" type="video/m3u8" x-webkit-airplay="allow">
    <track kind="captions" label="English" srclang="eng" src="https://fast.wistia.net/embed/captions/some-hash.vtt?language=eng">
</video>

Behavior in Firefox 81.0.1:
https://imgur.com/a/ow9zavK

Comment 27

[Phil Chan]

It would seem that the current Video PiP implementation works as intended, displaying an arbitrary <video> in a floating window. For websites like YouTube and Netflix that draws their own elements over the video, one way to support them would be implementing a windowed fullscreen feature that puts whatever custom fullscreen UI a website has implemented into a window.

Comment 28

[steinmb]

Hi
Two years out and Firefox 86 is out. We got multiple PiP support in the browser but no subtitle support. What is blocking this from getting addressed?

Comment 29

[fahrenheight]

I have seen this implemented in iOS due to standardization so I'm wondering what the chances are of this happening are with Firefox. I regularly watch shows that have scenes that require subtitles so it's quite annoying to have to find the tab and go back to whatever I missed.

Comment 30

[Matt Cosentino]

Would it be possible for there to be a defined video container element that it moved to the PiP window instead of the <video> element itself? This would allow custom player UIs to be used and for captions to be displayed. It would have to be an opt-in feature for the developers since not all custom UIs would work in the small window. An alternative would be to let developers disable the native PiP and design their own.

Comment 32

[myaseen98]

Any updates on this?

Comment 33

[Diana]

I really love PiP, but losing subtitles makes it basically impossible for me to use. So I started looking into this.

I think having a defined element that a site must support for it to work properly in Firefox would put the responsibility on the site owners (Google, Netflix, every blog and tutorial site) to implement, which is not currently how PiP works. Since it only looks for a video element, it can support every site. It's not that it'd be impossible, just that there'd have to be an effort made to get everyone to support Firefox's specific fix for this problem, when it wasn't a problem of their making.

I posit that this is an issue to be taken to w3c itself, not with Firefox. We need standardized captioning elements, and they need to be tied to the PiP project: https://github.com/w3c/picture-in-picture

What I would ask of Firefox is maybe making it easy for us to make our own extensions in the meantime? Maybe it's already possible! I'd love to know, if so.

It's going to mean keeping up with the HTML changes for every site I want to use, which is why Firefox hasn't even tried to do this... that's wild :D But it's also something I'm personally willing to do for myself, and release as an extension for the greater community. So if you have any info on whether that'd be possible, I'd be grateful!

Comment 34

[Diana]

Hi Mike! I saw your gist about this when I went searching further (this one: https://gist.github.com/mikeconley/fd8bcbed5837bd7f849efa847fbad67d)

So no response needed on the extension thing, unless of course you have new exciting details :D

Comment 35

[Micah [:mtigley] (she/her)]

(In reply to Diana from comment #34)

Hi Mike! I saw your gist about this when I went searching further (this one: https://gist.github.com/mikeconley/fd8bcbed5837bd7f849efa847fbad67d)

So no response needed on the extension thing, unless of course you have new exciting details :D

Hi Diana! Not sure if you've seen this already, but you can also follow along with site-specific adapter work in bug 1670108 to get an idea where it's at for now :)

Comment 37

[Daniel Bodea [:danibodea]]

Subtitles inside the PiP window are now supported in Nightly 100 for a few webapps/streaming services: Youtube, Amazon Prime Video, Netflix, Coursera, CBS Sport and other videos with "pure WebVTT" subtitles/captions.

Other streaming services are probably to be implemented in the future, but it is important to know that the steps in comment 0 are now invalid.
Ania, don't we have this covered in some other bug/task?

Comment 39

[Ania]

I added the stories/acceptance criteria to the metabug that this issue duplicates.

Comment 40

[vee w]

Hello,
I'm not sure can I comment about this here but When those website - for example Netflix, using subtitle as SVG. The subtitle will not working with picture in picture mode.

I try to inspect elements and found that some language like "Thai", they use something look like this... <svg><img href="blob:https://www.netflix.com/70cd5002-7c13-4fa1-b3cc-88f29276c305"></svg>.
But in many languages (most languages) they use something look like this... <span style="">好</span> or <span style="">Good.</span>.

I don't know that SVG is included in this case as subtitle or not, if not that must be Netflix's problem now.

Comment 41

[NellySteven]

spam

Comment 42

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Please create a new bug here.

Comment 43

[AngelAngle112]

The feature is available in Firefox 100. You should be using the beta, or wait for the release.
(In reply to Micah [:mtigley] (she/her) from comment #35)

(In reply to Diana from comment #34)

Hi Mike! I saw your gist about this when I went searching further (this one: https://gist.github.com/mikeconley/fd8bcbed5837bd7f849efa847fbad67d slope unblocked

So no response needed on the extension thing, unless of course you have new exciting details :D

Hi Diana! Not sure if you've seen this already, but you can also follow along with site-specific adapter work in bug 1670108 to get an idea where it's at for now :)

The feature is available in Firefox 100. You should be using the beta, or wait for the release.