1751505 - [meta] Caption support in PiP

Status: NEW

(Toolkit :: Picture-in-Picture, task)

Description

[Ania]

Meta bug for tasks enabling subtitles/caption support in Picture-in-Picture and related defects.

User Stories

1. As a user with a hearing impairment, I want to be able to watch PiP videos with captions, as it’s my only way to follow along.
2. As a user, I want to be able to watch PiP videos with subtitles, so I can follow along with videos in unfamiliar languages or with poor sound quality.
3. As a user, I want to be able to watch PiP videos in situations when sound is distracting, so that I don’t have to interrupt my flow and return to the in-tab video playback with subtitles.

Acceptance Criteria

1. Subtitles implementation can read WebVTT subtitles to support video hosting websites that are using the standard.
2. Subtitles in a user-chosen language are displayed in PiP if the user turned them on in the source tab (where WebVTT or site-specific adapters have been supported)
3. Users can turn subtitles on or off from the source tab, but not from the PiP window.
4. Subtitles in non-Latin scripts, if chosen by a user in a source tab, are successfully displayed in PiP.
5. The subtitles layer in PiP does not overlap with the existing playback or window controls.

Comment 1

[Ania]

My apologies, I have requested the accessibility review prematurely. Will submit the request tomorrow after filling out all the necessary information.

Comment 2

[Ania]

Description:

• Even though captions and subtitles are not the same, we use the term interchangeably for our purposes.
  User Stories

1. As a user with a hearing impairment, I want to be able to watch PiP videos with captions, as it’s my only way to follow along.
2. As a user, I want to be able to watch PiP videos with subtitles, so I can follow along with videos in unfamiliar languages or with poor sound quality.
3. As a user, I want to be able to watch PiP videos in situations when sound is distracting, so that I don’t have to interrupt my flow and return to the in-tab video playback with subtitles.

Acceptance Criteria

1. Subtitles implementation can read WebVTT subtitles and supports subtitles display on all websites that are using the WebVTT API.
2. Websites that do not support WebVTT require site-specific adapters for subtitles to work in PiP. In this stage of the project, we will work on site-specific adapters for YouTube, Prime Video, and Netflix.
3. Subtitles in a user-chosen language are displayed in PiP if the user turned them on in the source tab. Users can turn subtitles on or off from the source tab, but not from the PiP window.
4. Subtitles in non-Latin scripts, if chosen by a user in a source tab, are successfully displayed in PiP.
5. The subtitles layer in PiP does not overlap with the existing playback or window controls.

How do we test this?
If there is an implementation to test, please provide instructions for testing it; e.g. setting preferences, other preparation, how to trigger the UI, etc.
Support of the WebVTT captions and captions on websites requiring site-specific adapters is currently behind a media.videocontrols.picture-in-picture.display-text-tracks.enabled pref and defaults to off.
In Nightly 99, if you flip the pref, you will be able to see captions on all websites that support WebVTT spec (e.g., coursera.org, CBC) and YouTube due to this site-specific adapter patch.
Please note: the responsive styling patch is still in progress, but we hoped to review the existing behavior and design guidelines with the accessibility team.
Please note: we have chosen to use our own PiP Subtitles Styling for a more consistent user experience and to limit initial investment into a risky bet.

When will this ship?
Firefox 100 - full support for WebVTT captions; captions support on YouTube, Prime Video, and Netflix via site-specific adapters.

Tracking bug/issue:
Metabug for all captions support
WebVTT captions support
Design documents (e.g. Product Requirements Document, UI spec):
Captions/Subtitles Support Story in the Picture-in-Picture Product Spec
PiP Subtitles Styling
Figma captions behavior guidelines
Initial engineering scoping
Video demos of the in-progress responsive captions styling

Engineering lead: [e-mail]
Product manager: [e-mail]

The accessibility team has developed the Mozilla Accessibility Release Guidelines which outline what is needed to make user interfaces accessible:
https://wiki.mozilla.org/Accessibility/Guidelines
Please describe the accessibility guidelines you considered and what steps you've taken to address them:

Describe any areas of concern to which you want the accessibility team to give special attention:
Minimum font size readability
Font styling readability

Comment 3

[Ania]

Please see the comment above

Comment 5

[Sylvestre Ledru [:Sylvestre]]

Release Note Request (optional, but appreciated)
[Why is this notable]: New great feature
[Affects Firefox for Android]: No, right?
[Suggested wording]: Support of subtitles in Picture-in-Picture mode
[Links (documentation, blog post, etc)]:

Comment 6

[Dianna Smith [:diannaS]]

Release note added, please see https://www.mozilla.org/firefox/100.0a1/releasenotes/