Bug 1165180 (jsdec)

Provide the ability for HTML video elements to have their decoders written in JavaScript

ASSIGNED
Assigned to

Status

()

Core
Audio/Video: Playback
P5
normal
ASSIGNED
2 years ago
2 years ago

People

(Reporter: cajbir, Assigned: cajbir)

Tracking

40 Branch
Points:
---

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(2 attachments)

(Assignee)

Description

2 years ago
Video decoders for the HTML media elements are implemented internally in the browser in C++. We can provide the ability for the decoder itself to be implemented in JavaScript and provided by the website. This allows any website to use any codec they want as long as they do it in JS.

The intent here is to provide away to implement the PlatformDecoderModule interface in JavaScript.

Comment 1

2 years ago
We'll want off-main-threadness and process isolaton. Preemptively adding a dependency on bug 558184 for that stuff.
Depends on: 558184

Comment 2

2 years ago
I'm interested in this for Wikipedia where licensing concerns have kept us off MPEGLA formats - we havr to jump through hoops for successful Ogv or WebM playback on Safari and IE/Edge. Hopefully will be able to see this API supported in Edge and Safari someday, so an API that integrates well with MSE which they already support would be ideal for adoption.

(I've already got emscripten-powered Ogg playback using canvas & web audio, but would prefer to offload more logic to the browser to make playback and interop smoother. Planning to work on VP8 for WebM playback as well.)

Would be happy to help with a JS shim or sample codecs for the API as well; I can adapt my emscriptened codecs to be more isolated.
(Assignee)

Updated

2 years ago
Assignee: nobody → cajbir.bugzilla
Status: NEW → ASSIGNED
(Assignee)

Comment 3

2 years ago
Created attachment 8623322 [details] [diff] [review]
Adds ability to have JS based decoders

This adds two methods to an HTML video element. They are:

* registerVideoDataDecoder
* registerAudioDataDecoder

They take a JS object that can have the following properties:

* initVideo
* initAudio
* input

There must be at least one of the first two and the third. For a video decoder you'd provide 'initVideo' and 'input'. For an audio decoder 'initAudio' and 'input'. These properties should be functions and map to our internal MediaDataDecoder 'init' and 'input' methods.

The functions have the following arguments:

* initVideo(frameWidth, frameHeight)
* initAudio(channelCount, sampleRate)
* input(offset, time, timecode, duration, keyframe, discontinuity, buffer)

These are explained in the webidl:

callback interface JSMediaDataDecoder {
  void initVideo(long frameWidth, long frameHeight);
  void initAudio(long channelCount, long sampleRate);
  object? input(long offset,           /* Approximate byte offset where this data was demuxed from its media */
                long time,             /* Start time of sample, in microseconds */
                long timeCode,         /* Codec specific internal time code. For Ogg based codecs this is the granule pos */
                long duration,         /* Duration of sample, in microseconds */
                boolean keyframe,      /* true if this is a keyframe */
                boolean discontinuity, /* true if this is the first sample after a gap or discontinuity in the stream */
                Int8Array v);          /* Array of sample/frame data */
};

I'll attach an example implementation of a decoder next.
(Assignee)

Comment 4

2 years ago
Created attachment 8623324 [details]
Example JS decoder

The zip file contains index.html and a test.mp4 file. The video element has two javascript decoders registered on it. They are functionality the same as the 'BlankDecoderModule' internal C++ implementation.

The video decoder just presents a green frame. The audio decoder produces a tone. The data from the MP4 file is not used at all in this example, although I console.log the length of the sample data as an example of accessing the typed array of raw data.

The example is also available online as http://cd.pn/jsdecoder.

A next test would be to plug in an enscriptmen ffmpeg implementation, Broadway.js or similar to actually decoded a file. I haven't done this myself yet though.
Alias: jsdec

Comment 5

2 years ago
(In reply to cajbir (:cajbir) from comment #4)
> A next test would be to plug in an enscriptmen ffmpeg implementation,
> Broadway.js or similar to actually decoded a file. I haven't done this
> myself yet though.

I whipped up a quick variant of the example using OpenH264 built with emscripten -- it works! (Video only of course; the audio is still the sine wave.)

Source: https://github.com/brion/jsdec-openh264
Sample: https://brionv.com/misc/jsdec-openh264/

A couple notes (testing on Mac OS X 10.10):
* Sometimes the JS decoder stops engaging and the native H.264 playback seems to happen? Have to restart Nightly to get it working again.
* The first video input packet reports back from OpenH264 that it's not a complete frame; to keep the decoder from shutting down I had to report back a fake frame for that case, so you see green until you hit play. The second packet and on work as expected.
(Assignee)

Comment 6

2 years ago
(In reply to Brion Vibber from comment #5)
> I whipped up a quick variant of the example using OpenH264 built with
> emscripten -- it works! (Video only of course; the audio is still the sine
> wave.)

That's awesome!
Component: Audio/Video → Audio/Video: Playback
Priority: -- → P5
You need to log in before you can comment on or make changes to this bug.