515898 - implement UI for subtitles on html5 video/audio

Status: RESOLVED DUPLICATE of bug 887934

(Toolkit :: Video/Audio Controls, enhancement)

Description

[Felipe Corrêa da Silva Sanches]

User-Agent:       Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2
Build Identifier: 

This is related to bug "481529 - Support for Kate overlay streams in <video> tag"

Video content that contains embedded subtitles need a user interface to select which subtitle track to display.

I am submitting a patch that implements an initial draft of such user interface.

Reproducible: Always

Comment 1

[Felipe Corrêa da Silva Sanches]

Attachment: patch to video XBL that implements subtitling UI

Comment 2

[Felipe Corrêa da Silva Sanches]

Attachment: user interface icon

icon to be placed in src/toolkit/themes/{w,p}instripe/global/media

Comment 3

[Felipe Corrêa da Silva Sanches]

Attachment: Screenshot

The screenshot did not capture the mouse cursor. It was on top of the "English" option in the menu. That's why it is a bit darker than the "Português" option.

These two subtitles are hardcoded in the test_subtitles function. Please remove it before commiting. This function is there only to showcase the feature.

Comment 4

[Silvia Pfeiffer]

I would also suggest adding the list of available subtitle tracks in the right-click menu. Further, the same should be available for multitrack audio or video files, i.e. all available audio and video tracks to select from.

Comment 5

[ogg.k.ogg.k]

I've posted two new patches that hook into your menu to fill it with the subtitle track from an Ogg stream, and select one of them.
See https://bugzilla.mozilla.org/show_bug.cgi?id=481529

Proof of concept for now, some test code is left, and displays raw language codes, etc.

Comment 6

[Silvia Pfeiffer]

You can get the mapping to real languages from this file: http://mxr.mozilla.org/mozilla/source/xpfe/global/resources/locale/en-US/languageNames.properties and http://mxr.mozilla.org/mozilla/source/xpfe/global/resources/locale/en-US/regionNames.properties and GetLocalName in http://mxr.mozilla.org/mozilla/source/cck/globals/globals.cpp uses them to do the pretty language mapping.

(In reply to comment #5)
> I've posted two new patches that hook into your menu to fill it with the
> subtitle track from an Ogg stream, and select one of them.
> See https://bugzilla.mozilla.org/show_bug.cgi?id=481529
> 
> Proof of concept for now, some test code is left, and displays raw language
> codes, etc.

Comment 7

[Justin Dolske [:Dolske]]

Comment on attachment 399976 [details] [diff] [review]
patch to video XBL that implements subtitling UI 

General comment: before we can consider taking this into the tree, someone needs to decide if this (and/or bug 481529) is the spec Mozilla wants to support and deal with standards issues. I don't really follow those topics, so that's not me. :)

One high-level code comment, though...

>@@ -512,6 +522,8 @@
>                         case "timeupdate":
>                             var currentTime = Math.round(this.video.currentTime * 1000); // in ms
>                             var duration = Math.round(this.video.duration * 1000); // in ms
>+
>+                            this.displaySubtitles(currentTime);

This is terribly inefficient. The timeupdate event is dispatched every frame, so running code in each event should be avoided as much as possible. Your displaySubtitles() is also very inefficient, as it does a linear search though the subtitles each time (!) it's called.

I'd suggest doing something like caching the min/max time for the current subtitle, and bailing out early if .currentTime is between those two values (because you then know the current subtitle doesn't need changed).

And the subtitles button shouldn't be displayed in the the controls if there are no subtitles available.

Comment 8

[Silvia Pfeiffer]

(In reply to comment #7)
> (From update of attachment 399976 [details] [diff] [review])
> General comment: before we can consider taking this into the tree, someone
> needs to decide if this (and/or bug 481529) is the spec Mozilla wants to
> support and deal with standards issues. I don't really follow those topics, so
> that's not me. :)

I am the one involved with W3C and WHATWG on video accessibility issues.

I regard this all as experiments for how to get video (and audio) accessibility into HTML5 and thus into browsers.

Since there is no standardisation on the HTML5 side yet, I don't think this should go into any releases as yet.

But it is important to have these experiments and show how it can be done, so that we can move forward with the specifications in the standard.

How is this kind of thing generally handled in Firefox? Are there alpha releases that contain new feature demos? Or would it just be a set of patches that stay in bugzilla until the correct specification has come to an agreement in the standards body?

Comment 9

[cajbir (:cajbir)]

(In reply to comment #8)
> How is this kind of thing generally handled in Firefox? Are there alpha
> releases that contain new feature demos? Or would it just be a set of patches
> that stay in bugzilla until the correct specification has come to an agreement
> in the standards body?

The latter seems to be the best approach. We can generate builds from the patches for people to try if needed.

Comment 10

[Justin Dolske [:Dolske]]

(In reply to comment #8)

> But it is important to have these experiments and show how it can be done, so
> that we can move forward with the specifications in the standard.
> 
> How is this kind of thing generally handled in Firefox?

Well, it depends on the circumstances and goals.

In this case, I believe Felipe has already implemented things as an extension, which is an excellent way to test out an experimental feature. I'd continue with that, until the things settle enough in HTML5 such that it makes sense ship with the browser.

I hope I'm not emitting stop energy here; just trying to make sure Felipe isn't wasting effort before things are ready.

Comment 11

[Silvia Pfeiffer]

(In reply to comment #10)
> (In reply to comment #8)
> 
> > But it is important to have these experiments and show how it can be done, so
> > that we can move forward with the specifications in the standard.
> > 
> > How is this kind of thing generally handled in Firefox?
> 
> Well, it depends on the circumstances and goals.
> 
> In this case, I believe Felipe has already implemented things as an extension,
> which is an excellent way to test out an experimental feature. I'd continue
> with that, until the things settle enough in HTML5 such that it makes sense
> ship with the browser.
> 
> I hope I'm not emitting stop energy here; just trying to make sure Felipe isn't
> wasting effort before things are ready.


The last thing I want is stop energy. I think it's awesome what Felipe and OggK have done and continue to do. I just wanted to make sure it's understood that there is a standard that needs to be updated around all the awesome work being done here and that that may take much longer than the actual software development. We should not ship interfaces that don't exist in the standard yet or are in huge flux. As the first addition of itext happens into the HTML5 standard (or whatever it will be called by then), I think we can safely start applying these things to trunk. I am hoping to have some achievements on this front by the end of the year.

Comment 12

[Po-chiang Chao [:bobchao]]

Just saw that Timed Text Markup Language (TTML) 1.0 becoming Candidate Recommendation in Feb 2010: http://www.w3.org/TR/2010/CR-ttaf1-dfxp-20100223/

Maybe we should go for this approach?

Comment 13

[Felipe Corrêa da Silva Sanches]

This is the WHATWG wiki page where the current draft is being worked on:
http://wiki.whatwg.org/wiki/Timed_tracks

It seem to me that they are gradually reaching some consensus on it.

Comment 14

[Silvia Pfeiffer]

Right now, the W3C and WHATWG are working on getting something into the spec to support subtitles for video, see http://www.whatwg.org/specs/web-apps/current-work/multipage/video.html#the-track-element . The format that Ian is proposing that browsers should use as caption files is a new one that he has developed by extending SRT with further features. TTML has been rejected by many on a broad basis of it relying on XSL-FO and xml namespaces (e.g. for<ruby> support) and other issues (readability). It is as yet unclear which format should be supported by browsers - that discussion still has to be had in both groups, WHATWG and W3C. 

I guess implementing support for SRT (the plain vanilla one) is probably a safe bid at this stage. Anything else more feature-complete needs a more thorough analysis and possibly javascript demos first that compare them.

Comment 15

[Silvia Pfeiffer]

Incidentally, I am about to start on another Mozilla contract to analyse this stuff and come up with a path forward for Mozilla.

Comment 16

[Felipe Corrêa da Silva Sanches]

So, what is the current status of this bug? What are the plans for landing that patch I sent 2 years ago? Does it need refinement to get applied to the current Firefox codebase?

Is there anybody implementing in Firefox the WHATWG specs for subtitles support? What about the other major browsers?

It seems to me that the specs (of the 'track' element) are ready for implementation, right? 

I think it would be good to have an overall update on this issue so that we can try to get back to work on it again.

Comment 17

[Felipe Corrêa da Silva Sanches]

Silvia, can you update us on the status of this issue? Are you still involved in it?

Comment 18

[Silvia Pfeiffer]

Felipe, I am not contracting to Mozilla on this any more, but I continue to work at the W3C on video accessibility.

Ralph Giles is now working for Mozilla on implementing WebVTT support the way that the HTML5 spec has specified it. IE have already implemented it and have a test release. WebKit are working on it.

I would suggest focusing on external WebVTT file support first. WebM is also currently discussing how to get WebVTT encapsulated into it. So, in-band support can happen at a later stage. As for Ogg: my suggestion would be to find a way to encapsulate WebVTT into Ogg Kate (similar to how it already encapsulated SRT) and then also support that through the TextTrack API of HTML5.

Comment 19

[Alexander Farkas]

I have developed a very simple track/textrack polyfill (http://jsfiddle.net/trixta/QZJTM/) and also a script for styleable controls (https://github.com/aFarkas/jMediaelement).

My problem with the current Track spec and implementations as a webdeveloper are the following:

1. We need a way to shrink the rectangel in which the cues are displayed using CSS. 
The webvtt features are not suitable here. As soon as we develope custom styleable controls, which are placed over the video element, we need a way to "reserve" this space for those controls. Due to the fact, that this is depending on the style of our webpage and not related to the contetnt of the vtt file, it has to be defined using CSS. For example:

::cuedisplay {
    top: 0;
    left: 0;
    right: 0;
    bottom: 40px; /* space is needed for overlaying custom styleable controls at the bottom of the video */
}

2. There is no 'trackmode' change event.
While we have a lot of special events for adding/removing tracks and cuechange/cueexit/cueenter. We do not have an event for the case, where a user changes the mode of a track. Again, this is for example needed for custom styleable controls. If the user changes the mode using the context menue from showing/disabled to disabled/showing we need an event for this to update for example the visual state of our controls.

Currently all custom styleable mediaplayers with track support, removedo not relay on the native implementations and handle the "texttrack" display with script. This is a shame. :-(

I know this is mainly spec related, but you guys are bringing the web forward :-D

Comment 20

[Silvia Pfeiffer]

Alexander: you should indeed make these proposals either at the WHATWG or the HTML WG.

Comment 21

[Florian Bender]

Is this a dupe of Bug 887934?

Comment 22

[Jared Wein [:jaws] (please needinfo? me)]

Yes, duping forward as 887934 has more recent comments.