Closed Bug 1748884 Opened 11 months ago Closed 10 months ago

Add support for displaying subtitles/captions if WebVTT is supported

Categories

(Toolkit :: Picture-in-Picture, task, P1)

task
Points:
5

Tracking

()

RESOLVED FIXED
99 Branch
Tracking Status
firefox99 --- fixed

People

(Reporter: mtigley, Assigned: kpatenio)

References

(Blocks 1 open bug)

Details

(Whiteboard: [fidefe-MR1-2022])

Attachments

(1 file)

We can take advantage of videos using the <track> element to display subtitles/captions, which are formatted using WebVTT.

This should be put behind a pref, maybe something like media.videocontrols.picture-in-picture.display_tracks.

Assignee: nobody → kpatenio
Status: NEW → ASSIGNED
Severity: -- → N/A
Priority: -- → P1
Whiteboard: [fidefe-MR1-2022]
Type: defect → task

This patch integrates WebVTT support for Picture in Picture that can be enabled/disabled via a pref.

Points: --- → 5
Blocks: 1751231
Attachment #9258430 - Attachment description: WIP: Bug 1748884 - add PIP support for WebVTT supported videos → Bug 1748884 - add PIP support for WebVTT supported videos
Blocks: 1751505
Attachment #9258430 - Attachment description: Bug 1748884 - add PIP support for WebVTT supported videos → WIP: Bug 1748884 - add PIP support for WebVTT supported videos
Attachment #9258430 - Attachment description: WIP: Bug 1748884 - add PIP support for WebVTT supported videos → Bug 1748884 - add PIP support for WebVTT supported videos
Pushed by kpatenio@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/951a331c6511
add PIP support for WebVTT supported videos r=mtigley,mhowell,niklas

Backed out for causing failures on browser_text_tracks_webvtt_2.js

[task 2022-02-02T21:36:01.566Z] 21:36:01     INFO - TEST-PASS | toolkit/components/pictureinpicture/tests/browser_text_tracks_webvtt_2.js | Video is paused - true == true - 
[task 2022-02-02T21:36:01.566Z] 21:36:01     INFO - Checking second cue in pip window
[task 2022-02-02T21:36:01.566Z] 21:36:01     INFO - Buffered messages finished
[task 2022-02-02T21:36:01.566Z] 21:36:01     INFO - TEST-UNEXPECTED-FAIL | toolkit/components/pictureinpicture/tests/browser_text_tracks_webvtt_2.js | Text track second cue should be visible on the pip window - false == true - got false, expected true (operator ==)
[task 2022-02-02T21:36:01.567Z] 21:36:01     INFO - Stack trace:
[task 2022-02-02T21:36:01.567Z] 21:36:01     INFO - ok@resource://specialpowers/SpecialPowersSandbox.jsm:87:21
[task 2022-02-02T21:36:01.567Z] 21:36:01     INFO - @chrome://mochitests/content/browser/toolkit/components/pictureinpicture/tests/browser_text_tracks_webvtt_2.js:336:11
[task 2022-02-02T21:36:01.567Z] 21:36:01     INFO - execute@resource://specialpowers/SpecialPowersSandbox.jsm:141:12
[task 2022-02-02T21:36:01.567Z] 21:36:01     INFO - _spawnTask@resource://specialpowers/SpecialPowersChild.jsm:1656:15
[task 2022-02-02T21:36:01.567Z] 21:36:01     INFO - receiveMessage@resource://specialpowers/SpecialPowersChild.jsm:310:21
[task 2022-02-02T21:36:01.567Z] 21:36:01     INFO - GECKO(9921) | [Parent 9921, Main Thread] WARNING: 'aOwner->IsDiscarded()', file /builds/worker/workspace/obj-build/dist/include/mozilla/dom/SyncedContextInlines.h:94
[task 2022-02-02T21:36:01.567Z] 21:36:01     INFO - Leaving test bound test_text_tracks_existing_window_cues
[task 2022-02-02T21:36:01.567Z] 21:36:01     INFO - Entering test bound test_text_tracks_existing_window_no_track
[task 2022-02-02T21:36:01.567Z] 21:36:01     INFO - Running test: existing window - no track
[task 2022-02-02T21:36:01.567Z] 21:36:01     INFO - GECKO(9921) | [Parent 9921, Compositor] WARNING: Possibly dropping task posted to updater thread: file /builds/worker/checkouts/gecko/gfx/layers/apz/src/APZUpdater.cpp:362
[task 2022-02-02T21:36:01.567Z] 21:36:01     INFO - GECKO(9921) | [Child 10006: Main Thread]: I/DocShellAndDOMWindowLeak ++DOCSHELL 7faed6fbf000 == 1 [pid = 10006] [id = 57]
[task 2022-02-02T21:36:01.568Z] 21:36:01     INFO - GECKO(9921) | [Child 10006: Main Thread]: I/DocShellAndDOMWindowLeak ++DOMWINDOW == 5 (7faed6710200) [pid = 10006] [serial = 134] [outer = 0]
[task 2022-02-02T21:36:01.568Z] 21:36:01     INFO - GECKO(9921) | [Child 10006: Main Thread]: I/DocShellAndDOMWindowLeak ++DOMWINDOW == 6 (7faed6fbf800) [pid = 10006] [serial = 135] [outer = 7faed6710200]
[task 2022-02-02T21:36:01.747Z] 21:36:01     INFO - GECKO(9921) | [Child 10006: Main Thread]: I/DocShellAndDOMWindowLeak ++DOMWINDOW == 7 (7faed6fc0800) [pid = 10006] [serial = 136] [outer = 7faed6710200]
[task 2022-02-02T21:36:01.885Z] 21:36:01     INFO - GECKO(9921) | [Parent 9921: Main Thread]: I/DocShellAndDOMWindowLeak --DOMWINDOW == 17 (7fdf5cbdd580) [pid = 9921] [serial = 134] [outer = 0] [url = chrome://global/content/pictureinpicture/player.xhtml]
[task 2022-02-02T21:36:01.959Z] 21:36:01     INFO - GECKO(9921) | [Parent 9921: Main Thread]: I/DocShellAndDOMWindowLeak --DOMWINDOW == 16 (7fdf238bc900) [pid = 9921] [serial = 138] [outer = 0] [url = chrome://global/content/pictureinpicture/player.xhtml]
[task 2022-02-02T21:36:01.961Z] 21:36:01     INFO - GECKO(9921) | [Parent 9921: Main Thread]: I/DocShellAndDOMWindowLeak --DOMWINDOW == 15 (7fdf5dfb3200) [pid = 9921] [serial = 136] [outer = 0] [url = chrome://global/content/pictureinpicture/player.xhtml]
[task 2022-02-02T21:36:02.107Z] 21:36:02     INFO - Preparing video and initial text tracks
[task 2022-02-02T21:36:02.109Z] 21:36:02     INFO - Waiting for videos to be ready
[task 2022-02-02T21:36:02.127Z] 21:36:02     INFO - GECKO(9921) | [Child 10161: Main Thread]: I/DocShellAndDOMWindowLeak --DOCSHELL 7f477f05f400 == 1 [pid = 10161] [id = 67] [url = about:blank]
[task 2022-02-02T21:36:02.192Z] 21:36:02     INFO - TEST-PASS | toolkit/components/pictureinpicture/tests/browser_text_tracks_webvtt_2.js | Number of tracks loaded should be 3 - 3 == 3 - 
Flags: needinfo?(kpatenio)

The test expects textTracks.textContent to have a substring "cue 2", but fails. I'm thinking this may be due to the fact that despite waiting for oncuechange to trigger, the textContent itself is not yet updated when we read the assertion (so basically we are running includes() on an empty string).

I think adding another waitForCondition to wait until textContent updates will help here.

Flags: needinfo?(kpatenio)

(In reply to kpatenio from comment #4)

The test expects textTracks.textContent to have a substring "cue 2", but fails. I'm thinking this may be due to the fact that despite waiting for oncuechange to trigger, the textContent itself is not yet updated when we read the assertion (so basically we are running includes() on an empty string).

I think adding another waitForCondition to wait until textContent updates will help here.

To follow up, adding waitForCondition helps, but ultimately the second cue duration for test vtt files also had to be adjusted to ensure automated tests (test verify) for linux 18.04 x64 webrender debug builds passed.

Will queue to land the patch after merge date next week.

Pushed by kpatenio@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/2b62322206fd
add PIP support for WebVTT supported videos r=mtigley,mhowell,niklas
Status: ASSIGNED → RESOLVED
Closed: 10 months ago
Resolution: --- → FIXED
Target Milestone: --- → 99 Branch
Depends on: 1755012
Depends on: 1755020
Depends on: 1755028
Depends on: 1755029
Depends on: 1756717
Depends on: 1756718
Depends on: 1757814
You need to log in before you can comment on or make changes to this bug.