1988447 - Support Matroska in MSE

Status: NEW

(Core :: Audio/Video: Playback, enhancement, P4)

Description

[Jeroen Vreeken]

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:138.0) Gecko/20100101 Firefox/138.0

Steps to reproduce:

None of the matroska mimetypes work with MediaSource.isTypeSupported()
Even though they work with the video element.
Trying to play a file (ignoring the istypesupported result) also does not work

Actual results:

Support function returned false

Expected results:

Video should have played. Just like with the video element.

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

The Bugbug bot thinks this bug should belong to the 'Core::Audio/Video: Playback' component, and is moving the bug to that component. Please correct in case you think the bot is wrong.

Comment 2

[Alastor Wu [:alwu]]

The MSE spec explicitly defines the supported formats, and Matroska is not among them. Therefore, it’s not part of our plan.

Comment 3

[Alastor Wu [:alwu]]

Do other browsers support Matroska on MSE despite it not being included in the spec? If so, we could revisit this bug if usage is high enough to justify adding an exception.

Comment 4

[Jeroen Vreeken]

Supporting a format in the video tag, but not in mse leads to weird inconsistencies.

Comment 5

[Alastor Wu [:alwu]]

That is a good point, let's reopen this bug for more discussion. Thanks!

Comment 6

[Alastor Wu [:alwu]]

Could you help me confirm if your MSE test case works on other browsers? Thanks!

Comment 7

[Jeroen Vreeken]

It works for palemoon.

Comment 8

[guest271314]

Could you help me confirm if your MSE test case works on other browsers? Thanks!

Chromium Version 142.0.7415.0 (Developer Build) (64-bit)

MediaSource.isTypeSupported("video/x-matroska")
false
MediaSource.isTypeSupported("video/x-matroska;codecs=opus")
false
MediaSource.isTypeSupported("audio/x-matroska;codecs=opus")
false

Once you start getting in to what other browsers support and what Firefox doesn't you're gonna have to address lack of MediaStreamTrackProcessor, MediaStreamTrackGenerator https://github.com/guest271314/webcodecs/blob/main/mediastreamtrackgenerator-audio.js#L107-L113, and WebCodecs EncodedAudioChunk support in MediaSource https://github.com/guest271314/WebCodecsOpusRecorder/blob/fec3ee9cbea35f76c94484ded883356659363ec9/WebCodecsOpusRecorder.js#L177-L200, too, et al. I know the Opus in Matroska plays on Chromium

    document.body.appendChild(this.audio);
    if (this.type === "mediaSource") {
      this.ms = new MediaSource();
      this.ms.addEventListener("sourceopen", async (e) => {
        console.log(e.type);
        URL.revokeObjectURL(this.audio.src);
        const sourceBuffer = this.ms.addSourceBuffer({
          audioConfig: this.config.decoderConfig,
        });
        console.log(this.ms.activeSourceBuffers);
        sourceBuffer.onupdate = (e) => console.log(e.type);
        sourceBuffer.mode = "sequence";
        for (const offset of this.config.offsets) {
          const eac = new EncodedAudioChunk({
            type: "key",
            timestamp: this.timestamp,
            duration: !this.index ? 53500 : this.duration,
            data: this.data.subarray(this.index, this.index + offset),
          });
          await sourceBuffer.appendEncodedChunks(eac);
          this.timestamp += eac.duration;
          this.index += offset;
        }
      });

Comment 9

[guest271314]

Firefox Nightly 144 implementation of BaseAudioContext: decodeAudioData() does not decode Opus in video/x-matroska, Chromium 142 does.

Blob { size: 13065, type: "video/x-matroska" }
Uncaught DOMException: The buffer passed to decodeAudioData contains invalid content which cannot be decoded successfully.

Blob size: 13065 type: "video/x-matroska" [[Prototype]]: Blob
AudioBuffer duration: 1.5698412698412698 length: 69230 numberOfChannels: 1 sampleRate: 44100 [[Prototype]]: AudioBuffer

Comment 10

[guest271314]

Chromium 142 does not play "video/x-matroska" (MIME type) with Opus audio codec in MediaSource.

Chromium 142 does play "video/x-matroska" (MIME type) with Opus audio codec in HTMLMediaElement.

Firefox has an opportunity to make this so. Specifications are not completely conformed to in a number of areas in different browsers; and browsers go beyond specifications for use cases that pop up internally, all of the time.

Chromium's implementation of MediaStreamTrack of kind audio is not conformant with W3C Media Capture and Streams, it doesn't render silence. They know that, and have not fix the implementation to be specification conformant, etc..

Comment 11

[Sebastian Zartner [:sebo]]

(In reply to Alastor Wu [:alwu] from comment #2)

The MSE spec explicitly defines the supported formats, and Matroska is not among them. Therefore, it’s not part of our plan.

The MSE spec. itself explicitly notes to

Not require support for any particular media format or codec.

but instead refers to the Media Source Extensions Byte Stream Format Registry as the authorative source for supported byte streams.

Though the registry spec. claims contradicts that, as it claims in its abstract that it is non-normative.

From its purpose section:

This registry is intended to enhance interoperability among implementations and users of SourceBuffer objects described in the Media Source Extensions™ specification.

So I guess, somebody (with a deeper understanding of this topic than me) should ask the Media Working Group to add the related MIME types to the registry at https://github.com/w3c/mse-byte-stream-format-registry/issues and maybe another one for clarifying whether the Media Source Extensions Byte Stream Format Registry is actually normative.

Sebastian

Comment 12

[guest271314]

Specifications are just a guide.

Firefox calls its implementation "Picture in Picture", though doesn'e even try to implement W3C Picture In Picture specification, the last time I checked.

There's nothing stopping Firefox from implementing this, then wraggling over minutea in specification discussions.

The demand is here. Specifications not infrequently try to catch up to whatever is already deployed in the field. For compatibility and interoperability. Sometimes that works, on paper; sometime that doesn't translate to practice and policy and actual capabilities in the field.

Comment 13

[Alastor Wu [:alwu]]

Currently, we don’t have plans to support Matroska in MSE, as our preference is still for people to use WebM over Matroska. According to this test page, Chromium-based browsers and Safari also do not support Matroska in MSE. I’ll keep this bug open for further discussion.

Comment 14

[guest271314]

Currently, we don’t have plans to support Matroska in MSE, as our preference is still for people to use WebM over Matroska.

What's up with the your preferences thing? We know that already. The point of these bugs is to maybe get Firefox out of the ice age, based on user-defined preferences - the people in the field who are still using Firefox.

There was a time when Firefox might actually take the lead in bringing technologies to user space. That can happen here.

If Firefox supported Matroska in MediaSource, I'd wager the Chromium Project would hear about that support and make it happen in CHromium quite fast, in response to Firefox not waiting on Chromium Project to do stuff.

Safari is a lost cause...