1748884 - Add support for displaying subtitles/captions if WebVTT is supported

Status: RESOLVED FIXED

(Toolkit :: Picture-in-Picture, task, P1)

Description

[Micah [:mtigley] (she/her)]

We can take advantage of videos using the <track> element to display subtitles/captions, which are formatted using WebVTT.

This should be put behind a pref, maybe something like media.videocontrols.picture-in-picture.display_tracks.

Comment 1

[kpatenio]

Attachment: Bug 1748884 - add PIP support for WebVTT supported videos

This patch integrates WebVTT support for Picture in Picture that can be enabled/disabled via a pref.

Comment 2

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/951a331c6511
add PIP support for WebVTT supported videos r=mtigley,mhowell,niklas

Comment 3

[Natalia Csoregi [:nataliaCs]]

Backed out for causing failures on browser_text_tracks_webvtt_2.js

• backout: https://hg.mozilla.org/integration/autoland/rev/c34aa4c3b8a18a3c03027e2d1eeaed146a2cd1e5
• push: https://treeherder.mozilla.org/jobs?repo=autoland&group_state=expanded&revision=951a331c6511bd6160970987b7ac6d5edde0d9b0&selectedTaskRun=PsYshBXeTrO0Egh-XEzrRg.0
• push with tier 1 failure: https://treeherder.mozilla.org/jobs?repo=autoland&group_state=expanded&revision=6123fcc0963099422bbffd76d5e8316a56df6b14&selectedTaskRun=Z_1DEueaSouniBuIzH5U8w.0
• failure log: https://treeherder.mozilla.org/logviewer?job_id=366504211&repo=autoland&lineNumber=32833

[task 2022-02-02T21:36:01.566Z] 21:36:01     INFO - TEST-PASS | toolkit/components/pictureinpicture/tests/browser_text_tracks_webvtt_2.js | Video is paused - true == true - 
[task 2022-02-02T21:36:01.566Z] 21:36:01     INFO - Checking second cue in pip window
[task 2022-02-02T21:36:01.566Z] 21:36:01     INFO - Buffered messages finished
[task 2022-02-02T21:36:01.566Z] 21:36:01     INFO - TEST-UNEXPECTED-FAIL | toolkit/components/pictureinpicture/tests/browser_text_tracks_webvtt_2.js | Text track second cue should be visible on the pip window - false == true - got false, expected true (operator ==)
[task 2022-02-02T21:36:01.567Z] 21:36:01     INFO - Stack trace:
[task 2022-02-02T21:36:01.567Z] 21:36:01     INFO - ok@resource://specialpowers/SpecialPowersSandbox.jsm:87:21
[task 2022-02-02T21:36:01.567Z] 21:36:01     INFO - @chrome://mochitests/content/browser/toolkit/components/pictureinpicture/tests/browser_text_tracks_webvtt_2.js:336:11
[task 2022-02-02T21:36:01.567Z] 21:36:01     INFO - execute@resource://specialpowers/SpecialPowersSandbox.jsm:141:12
[task 2022-02-02T21:36:01.567Z] 21:36:01     INFO - _spawnTask@resource://specialpowers/SpecialPowersChild.jsm:1656:15
[task 2022-02-02T21:36:01.567Z] 21:36:01     INFO - receiveMessage@resource://specialpowers/SpecialPowersChild.jsm:310:21
[task 2022-02-02T21:36:01.567Z] 21:36:01     INFO - GECKO(9921) | [Parent 9921, Main Thread] WARNING: 'aOwner->IsDiscarded()', file /builds/worker/workspace/obj-build/dist/include/mozilla/dom/SyncedContextInlines.h:94
[task 2022-02-02T21:36:01.567Z] 21:36:01     INFO - Leaving test bound test_text_tracks_existing_window_cues
[task 2022-02-02T21:36:01.567Z] 21:36:01     INFO - Entering test bound test_text_tracks_existing_window_no_track
[task 2022-02-02T21:36:01.567Z] 21:36:01     INFO - Running test: existing window - no track
[task 2022-02-02T21:36:01.567Z] 21:36:01     INFO - GECKO(9921) | [Parent 9921, Compositor] WARNING: Possibly dropping task posted to updater thread: file /builds/worker/checkouts/gecko/gfx/layers/apz/src/APZUpdater.cpp:362
[task 2022-02-02T21:36:01.567Z] 21:36:01     INFO - GECKO(9921) | [Child 10006: Main Thread]: I/DocShellAndDOMWindowLeak ++DOCSHELL 7faed6fbf000 == 1 [pid = 10006] [id = 57]
[task 2022-02-02T21:36:01.568Z] 21:36:01     INFO - GECKO(9921) | [Child 10006: Main Thread]: I/DocShellAndDOMWindowLeak ++DOMWINDOW == 5 (7faed6710200) [pid = 10006] [serial = 134] [outer = 0]
[task 2022-02-02T21:36:01.568Z] 21:36:01     INFO - GECKO(9921) | [Child 10006: Main Thread]: I/DocShellAndDOMWindowLeak ++DOMWINDOW == 6 (7faed6fbf800) [pid = 10006] [serial = 135] [outer = 7faed6710200]
[task 2022-02-02T21:36:01.747Z] 21:36:01     INFO - GECKO(9921) | [Child 10006: Main Thread]: I/DocShellAndDOMWindowLeak ++DOMWINDOW == 7 (7faed6fc0800) [pid = 10006] [serial = 136] [outer = 7faed6710200]
[task 2022-02-02T21:36:01.885Z] 21:36:01     INFO - GECKO(9921) | [Parent 9921: Main Thread]: I/DocShellAndDOMWindowLeak --DOMWINDOW == 17 (7fdf5cbdd580) [pid = 9921] [serial = 134] [outer = 0] [url = chrome://global/content/pictureinpicture/player.xhtml]
[task 2022-02-02T21:36:01.959Z] 21:36:01     INFO - GECKO(9921) | [Parent 9921: Main Thread]: I/DocShellAndDOMWindowLeak --DOMWINDOW == 16 (7fdf238bc900) [pid = 9921] [serial = 138] [outer = 0] [url = chrome://global/content/pictureinpicture/player.xhtml]
[task 2022-02-02T21:36:01.961Z] 21:36:01     INFO - GECKO(9921) | [Parent 9921: Main Thread]: I/DocShellAndDOMWindowLeak --DOMWINDOW == 15 (7fdf5dfb3200) [pid = 9921] [serial = 136] [outer = 0] [url = chrome://global/content/pictureinpicture/player.xhtml]
[task 2022-02-02T21:36:02.107Z] 21:36:02     INFO - Preparing video and initial text tracks
[task 2022-02-02T21:36:02.109Z] 21:36:02     INFO - Waiting for videos to be ready
[task 2022-02-02T21:36:02.127Z] 21:36:02     INFO - GECKO(9921) | [Child 10161: Main Thread]: I/DocShellAndDOMWindowLeak --DOCSHELL 7f477f05f400 == 1 [pid = 10161] [id = 67] [url = about:blank]
[task 2022-02-02T21:36:02.192Z] 21:36:02     INFO - TEST-PASS | toolkit/components/pictureinpicture/tests/browser_text_tracks_webvtt_2.js | Number of tracks loaded should be 3 - 3 == 3 - 

Comment 4

[kpatenio]

The test expects textTracks.textContent to have a substring "cue 2", but fails. I'm thinking this may be due to the fact that despite waiting for oncuechange to trigger, the textContent itself is not yet updated when we read the assertion (so basically we are running includes() on an empty string).

I think adding another waitForCondition to wait until textContent updates will help here.

Comment 5

[kpatenio]

(In reply to kpatenio from comment #4)

The test expects textTracks.textContent to have a substring "cue 2", but fails. I'm thinking this may be due to the fact that despite waiting for oncuechange to trigger, the textContent itself is not yet updated when we read the assertion (so basically we are running includes() on an empty string).

I think adding another waitForCondition to wait until textContent updates will help here.

To follow up, adding waitForCondition helps, but ultimately the second cue duration for test vtt files also had to be adjusted to ensure automated tests (test verify) for linux 18.04 x64 webrender debug builds passed.

Will queue to land the patch after merge date next week.

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/2b62322206fd
add PIP support for WebVTT supported videos r=mtigley,mhowell,niklas

Comment 7

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/2b62322206fd