1496497 - mimetype provided by the server should be used after sniffing fail

Status: RESOLVED FIXED

(Core :: Audio/Video: Playback, enhancement, P3)

Description

[Jean-Yves Avenard [:jya]]

Currently, when opening a resource we run the nsMediaSniffer to determine the type of content.
We ignore the mimetype provided by the server.

We did so to prevent false positive a while back.

However, looking at the spec, we see:

https://html.spec.whatwg.org/#loading-the-media-resource:
 When the load() method on a media element is invoked, the user agent must run the media element load algorithm.. 


The load algorithm:
 https://html.spec.whatwg.org/#media-element-load-algorithm -> 

if the source is local:
 If the current media resource is a raw data stream (e.g. from a File object), then to determine the format of the media resource, the user agent must use the rules for sniffing audio and video specifically. Otherwise, if the data stream is pre-decoded, then the format is the format given by the relevant specification.

https://mimesniff.spec.whatwg.org/#rules-for-sniffing-audio-and-video-

In step 4 we read at the end of the steps: "The computed MIME type is the supplied MIME type. "

supplied mimetype is defined as " A supplied MIME type, the MIME type determined by the supplied MIME type detection algorithm."

"If the resource is retrieved via another protocol (such as FTP), set supplied-type to the MIME type as determined by that protocol, if any. "


As such, if sniffing failed to determine the type, we should be using the server's resource provided mimetype.

Comment 1

[Jean-Yves Avenard [:jya]]

This may be a nsHttp component as all of that logic is handled there.

Comment 2

[Jean-Yves Avenard [:jya]]

https://www.w3.org/Bugs/Public/show_bug.cgi?id=11984 was the discussion that made sniffing the default.

Comment 3

[Jean-Yves Avenard [:jya]]

As per comment 2, and following the W3C discussion:
"Rather than continuing to maintain this mess, we should jump directly to the most sane of the likely outcomes, which is to always ignore Content-Type for <video>."

I extend this to include <audio> too.

Comment 4

[Jean-Yves Avenard [:jya]]

After discussion with :padenot

https://mimesniff.spec.whatwg.org/#sniffing-in-an-audio-or-video-context


    1. If the supplied MIME type is an XML MIME type, the computed MIME type is the supplied MIME type.

    Abort these steps.
    2. Let audio-or-video-type-matched be the result of executing the audio or video type pattern matching algorithm with the resource header as the byte sequence to be matched.
    3. If audio-or-video-type-matched is not undefined, the computed MIME type is audio-or-video-type-matched.

    Abort these steps.
    4. The computed MIME type is the supplied MIME type. 

So if sniffing failed, we fallback to the supplied MIMEtype which is the value provided by content-type

Comment 5

[Sammy]

Just checking in on Bug 1496497 (open for 6 years and the last comment was 2 years ago). It's also marked as S3 currently. This issue may have already driven people away from using Firefox as it renders certain media unable to be played in the browser.

Comment 6

[Paul Adenot (:padenot)]

Sammy, do you have any specific case that fails?

Comment 7

[Sammy]

(In reply to Paul Adenot (:padenot) from comment #6)

Sammy, do you have any specific case that fails?

I do have a case, thank you for following up.

audio/flac @ 24 bit.

Audio Media info:
Format : FLAC
Format/Info : Free Lossless Audio Codec
Duration : 2 min 54 s
Bit rate mode : Variable
Bit rate : 20.7 Mb/s
Channel(s) : 2 channels
Channel layout : L R
Sampling rate : 655 kHz
Bit depth : 24 bits
Compression mode : Lossless
Replay gain : 2.58 dB
Replay gain peak : 0.376740
Stream size : 432 MiB (100%)
Writing library : GStreamer encoded vorbiscomment
MD5 of the unencoded content : E9C7D6F9644FED235FB7AB070E7460B4

(Plays on Chromium)

Comment 8

[Sammy]

(In reply to Sammy from comment #7)

(In reply to Paul Adenot (:padenot) from comment #6)

Sammy, do you have any specific case that fails?

I do have a case, thank you for following up.

audio/flac @ 24 bit.

Audio Media info:
Format : FLAC
Format/Info : Free Lossless Audio Codec
Duration : 2 min 54 s
Bit rate mode : Variable
Bit rate : 20.7 Mb/s
Channel(s) : 2 channels
Channel layout : L R
Sampling rate : 655 kHz
Bit depth : 24 bits
Compression mode : Lossless
Replay gain : 2.58 dB
Replay gain peak : 0.376740
Stream size : 432 MiB (100%)
Writing library : GStreamer encoded vorbiscomment
MD5 of the unencoded content : E9C7D6F9644FED235FB7AB070E7460B4

(Plays on Chromium)

here is a temporary example file to use: http://203.12.2.28:4533/rest/stream?u=admin&t=1c2b65244f09b2589da78d18ce066e1f&s=692d90&f=json&v=1.8.0&c=NavidromeUI&id=9870c7d4168ae571a1acd497ea1dd423&_=1711424028848 (I'll leave this up for two weeks).

Comment 9

[Paul Adenot (:padenot)]

Thanks, I've grabbed a fragment of this stream for local debugging, you can take the URL down now if you want.

The problem with this file is that it has a sample-rate of 655kHz, which is unusual, and it's unrelated to sniffing. Listening to the file, it sounds slowed down significantly. I'll check why Firefox chokes on it, tomorrow.

Comment 10

[Paul Adenot (:padenot)]

Yeah, we limit (somewhat arbitrarily) to 640kHz. Sammy, does this file correspond to a valid use-case? It's likely that we can just raise the limit to decode it.

It's not going to be particularly useful for direct playback, but maybe it's for some special processing.

In any case, this happens here: https://searchfox.org/mozilla-central/source/dom/media/MediaInfo.h#565-571

Comment 11

[Sammy]

Well, in my case this file is streamed from a server. I'm not sure I understand the justification for the limit; is it to protect against crashing the browser? If so could it maybe be changed to the technical limit rather than an arbitrary one?

But to answer more about valid use case: I could opt for a chromium based electron app instead to stream this content or another app. But generally anyway, I'd think if the media exists it would make sense to be allowed to play it on Firefox (even if for performance or other reasons it has to be a configuration flag to enable higher fidelity audio/video).

tl;dr I'm not sure why you wouldn't go for it, especially if it doesn't negatively impact anything. From what I understand, all it would mean is a more capable and competitive browser.

Comment 12

[Paul Adenot (:padenot)]

The file linked has a sample-rate of 655kHz. 655kHz sample-sample can reproduce frequencies about 1.5 times higher than the hearing range of a dolphin, which can hear about 10 times higher pitch than a human. There's only a few digital to analog converter that can handle that, and it's completely pointeless in any case. Extremely high sample rate have their place, but generally not for listening.

We can make it work, but that sounds like a buggy file in the first place, hence my question. I'm attaching a patch to make it work in any case, because it should play, but the entire thing is weird in the first place.

Comment 13

[Paul Adenot (:padenot)]

Attachment: Bug 1496497 - Consider 768kHz files valid, but resample down to 384kHz for playback. r?#media-playback-reviewers

Comment 14

[Paul Adenot (:padenot)]

This also needs https://github.com/mozilla/cubeb/pull/782 on the audio IO layer to make it work. The extract I got from the stream linked above plays fine with those two patches.

Comment 15

[Sammy]

(In reply to Paul Adenot (:padenot) from comment #12)

The file linked has a sample-rate of 655kHz. 655kHz sample-sample can reproduce frequencies about 1.5 times higher than the hearing range of a dolphin, which can hear about 10 times higher pitch than a human. There's only a few digital to analog converter that can handle that, and it's completely pointeless in any case. Extremely high sample rate have their place, but generally not for listening.

We can make it work, but that sounds like a buggy file in the first place, hence my question. I'm attaching a patch to make it work in any case, because it should play, but the entire thing is weird in the first place.

We agree that it should have the facility. It's not so weird. It seems like you know that to have it would not be for discriminating detail.

If I really have to provide justification then, more than anything, I have the files because I value close to source (or source) for some albums sentimentally. The only reason I'd want to have a resampled copy is to avoid issues with real-time resampling (which is also a non-issue largely).

I don't think it makes sense to store two copies, unless for some reason software is arbitrarily deciding that I shouldn't be able to play this file because it's "unnecessary ?". This is my confusion, is it really decided on that, rather than the details around technical capacity (and if so, why 24 bit playback at all)? I think I'd refer to it as a bit of a heavy handed approach to it.

I'm fairly sure Chromium also resamples in this situation.

Anyway, even if you were to assume I think I perceive an audible difference, would I be the only user in the world with this placebo. Also, if I had that placebo, wouldn't I abandon Firefox for Chromium because of it anyway? (a not good thing)

To me this comes down to, would you really make an error that explains "We don't see a valid reason to play this, so we chose to disable it". Hopefully that clears up any confusion/sense of weirdness, and provides some more purpose. But, I can't say needing this answer and justification makes sense to me, it seems unrelated to the issue. Maybe I'd see it as related if it was about building infrastructure to perfectly playback those higher sample rates (but even then, there are audio discrimination test websites for example, that could and would use that capability ~ even if you might see it as superfluous personally). But to be super clear and candid, it's not really about that for me, or maybe even most people. As far as I'm concerned this is just about Firefox not cutting short its own abilities. I hope there aren't other cases where it's decided to put an arbitrary limit on capabilities, because the idea people shouldn't or it would be "weird".

Comment 16

[Paul Adenot (:padenot)]

This file should play, regardless of if it's useless or not, I was simply wondering if it was an error or if it was intended, since it's so far outside what I've seen in my decades of working on and with digital audio.

Before raising the sample-rate limit in this patch, the existing limit was in place to catch potential problems early, with bogus files (that are not infrequent on the Web, be it malicious to crash or exploit bugs, or simply bugs in an encoder, etc.). We had put a limit that was extremely high, thinking it was high enough for all purposes, but that isn't the case in 2024 apparently, so we need to raise it to match what exists, and also, as you note, match other software.

Firefox 126 should play the type of files you helpfully linked above, released mid-May this year. Firefox Beta will have this patch mid-April, and Nightly in the next few days: https://www.mozilla.org/fr/firefox/channel/desktop/. Nightly installs fine alongside a regular Firefox (by default won't share the user profile).

Comment 17

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/1eb7fb5dab3b
Consider 768kHz files valid, but resample down to 384kHz for playback. r=media-playback-reviewers,alwu

Comment 18

[pstanciu]

https://hg.mozilla.org/mozilla-central/rev/1eb7fb5dab3b