1214027 - Can't dynamically change subtitle tracks on a <video> element

Status: RESOLVED INVALID

(Core :: Audio/Video: Playback, defect, P1)

Description

[David Flanagan [:djf]]

I'm trying to modify the FirefoxOS Video app to display subtitles when there is a .vtt file on the device sdcard along side the video, and have run into a gecko bug, or maybe two gecko bugs.

1) A <track> element inside a <video> element works if I add the track before playing the video, or if I add the track element even after the video is already playing.  And if I remove the track element, the video element stops displaying subtitles. So far, so good. But now if I add a <track> element back to the <video> element, it does not display any subtitles.  It appears that once a <track> has been removed from a <video> element, that video will never again display captions or subtitles.  The FirefoxOS Video app reuses a single long-lived <video> element for all of its videos. So this bug means that I can display captions for one video. But then if I play a video with no captions (which causes the track to be removed) captions will never again work unti the user kills the app and restarts it.

2) I've tried working around this by leaving the <track> element in the video element, but just removing the src attribute and setting track.default = false.  This appears to have no effect at all... the video just keeps playing and displaying subtitles.

I'll attach files to demonstrate these bugs.  bug.html and bug2.html demonstrate bugs 1 and 2.  They use upc.ogv and upc.vtt as the test movie and captions.

This blocks my work in bug 1213390, so getting it fixed (or finding a workaround) is high-priority for me.

Comment 1

[David Flanagan [:djf]]

Attachment: bug.html: test file that demonstrates the first bug

Comment 2

[David Flanagan [:djf]]

Attachment: bug2.html: demonstrates the second bug

Comment 3

[David Flanagan [:djf]]

Attachment: upc.vtt: captions for the movie

Comment 4

[David Flanagan [:djf]]

Attachment: upc.ogv: video used by the test files

Comment 5

[David Flanagan [:djf]]

This bug does not appear to be specific to .ogv videos. I've seen it with .mp4 as well.

Comment 6

[Gerald Squelart (he/him) (not at Mozilla since 2022-09-15)]

Could it be the same issue as experienced in bug 1205644 and/or bug 1206304?

Comment 7

[David Flanagan [:djf]]

I should add that this bug is affecting me on FirefoxOS but that the demos I've attached demonstrate the bug on MacOS running Firefox Developer Edition (gecko 43)

Comment 8

[David Flanagan [:djf]]

(In reply to Gerald Squelart [:gerald] from comment #6)
> Could it be the same issue as experienced in bug 1205644 and/or bug 1206304?

I just added a console.log(JSON.stringify(player.textTracks)); to my bug.html file. textTracks is not updated immediately after I add the <track> element, but it does get update asynchronously, so that if I do a setTimeout(function() {console.log(...)}); I do see textTracks having the right number of elements.

So I think that this is not the same issue, unless there is some code in gecko that is also being tripped up by the async textTracks update.

Comment 9

[Ralph Giles (:rillian)]

Thanks for the excellent bug report, David. Sounds like a real bug, but no one at moco is working on <track> at the moment. Might have to fix this yourself. :/

Gerald, would you be interested in this one.

Comment 10

[Gerald Squelart (he/him) (not at Mozilla since 2022-09-15)]

Working on it now.

Comment 11

[Gerald Squelart (he/him) (not at Mozilla since 2022-09-15)]

Attachment: bug-fixed.html bug.html with correct track mode setting

The first "bug" is due to the added tracks' mode staying "disabled".

According to the HTML specs, when the very first track is added to the video element, an algorithm is performed to select the relevant track(s) to be displayed. But after that, whether tracks are added or removed, this algorithm is not run anymore.
Search for "did-perform-automatic-track-selection" in the following page to see how this is specified:
http://www.w3.org/TR/html5/embedded-content-0.html#did-perform-automatic-track-selection

So our implementation actually follows the specs correctly!

As an exercise (and before finding out about the specs), I was able to "fix" the first bug by removing the did-perform-automatic-track-selection check:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=6b44e56165a7

Attached is a slightly modified version of bug.html, which sets the track mode to "showing" *after* attaching the track to the video element; this one works fine, and even simultaneously displays all copies of the added tracks!

Comment 12

[Gerald Squelart (he/him) (not at Mozilla since 2022-09-15)]

As for the 2nd bug:

The specs describe the src attribute: "The [src] value must be a valid non-empty URL potentially surrounded by spaces. This attribute must be present."
http://www.w3.org/TR/html5/embedded-content-0.html#attr-track-src

So if I interpret this correctly, setting src to an empty string, or removing the attribute, are not valid operations.

Clearing the src attribute of the track reverts it to the html page URI, but somehow internally keeps a reference to the VTT cues (I didn't investigate further).

The <video> element probably handles this undefined behavior better: It produces an error, and modifies the video contents.
We could modify <track> to also produce an error and remove the captions (as probably intended by the webpage author), but I would argue that a separate low-priority bug should be created for that.

Comment 13

[Gerald Squelart (he/him) (not at Mozilla since 2022-09-15)]

David: Based on my analysis above, I'm tempted to close this bug as Invalid, but please let me know if you think I've missed something.

Hopefully my proposed solution in attachment 8674660 [details] should help you continue with bug 1213390 -- or as you've already experimented in the meantime, you may use a different <video> element for each video instead.

Comment 14

[David Flanagan [:djf]]

Thank you Gerald!  I had not ever read the spec, and didn't even know about track.mode. Sounds like that is my bad (though maybe MDN needs more docs, too.)

I agree that this is probably an invalid bug.

Comment 15

[Gerald Squelart (he/him) (not at Mozilla since 2022-09-15)]

Thank you David.
Indeed the "TextTrack" type (referred to in https://developer.mozilla.org/en-US/docs/Web/API/HTMLTrackElement) is not documented, I'll open a bug for that.