Closed Bug 489071 Opened 15 years ago Closed 14 years ago

Incorrect HTTP Accept header when requesting <audio> and <video> elements

Categories

(Core :: Audio/Video, defect)

1.9.1 Branch
x86
Linux
defect
Not set
normal

Tracking

()

RESOLVED FIXED

People

(Reporter: jim, Assigned: cajbir)

References

(Depends on 1 open bug)

Details

Attachments

(2 files, 3 obsolete files)

User-Agent:       Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.1b4pre) Gecko/20090418 Shiretoko/3.5b4pre
Build Identifier: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.1b4pre) Gecko/20090418 Shiretoko/3.5b4pre

Firefox 3.5 introduces the HTML5 <audio> and <video> elements.  However, when requesting the content of one of those elements, Firefox sends a misleading HTTP Accept header, claiming that it prefers HTML content over audio content.

Reproducible: Always

Steps to Reproduce:
1. <audio src="http://my-server/audio.ogg"></audio>
2. <video src="http://my-server/video.ogg"></audio>
3. View the HTTP headers the server receives

Actual Results:  
The client's request includes the following header:

Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8

Expected Results:  
At the very minimum, these headers should not specify html, xhtml, and xml types.  Here is a proposed ordering for the <audio> tag:

Accept: audio/ogg,audio/*;q=0.9,*/*;q=0.5

And for the <video> tag:

Accept: video/ogg,video/*;q=0.9,*/*;q=0.5
moving this to core:video/auto because this is Gecko and not Firefox
Component: General → Video/Audio
Product: Firefox → Core
QA Contact: general → video.audio
Version: unspecified → 1.9.1 Branch
This bug still exists in 3.5rc2, but it sits UNCONFIRMED and is assigned to no one.  I don't think this bug has been appropriately triaged.

Since <audio> and <video> are new features in Firefox 3.5, it would be really nice to get them right.  If Firefox 3.5 final sends the wrong headers, web developers will for a very long time not be able to rely on the headers sent by web browsers requesting video and audio.
To reproduce, run this server on your local machine and create a test HTML file that contains the line:

    <audio src="http://localhost:8888/audio.ogg"></audio>

The HTTP headers will be sent to stdout.

On my machine, the incorrect header reads:

    Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
We won't be able to fix it for 3.5, but we should fix it for the next release.
Status: UNCONFIRMED → NEW
Ever confirmed: true
Flags: wanted1.9.2?
Flags: wanted1.9.2? → wanted1.9.2+
I wonder if we should stop sending |Accept-Encoding: gzip,deflate| for Ogg Theora/Vorbis, too.  It makes sense for Wave files, but almost nothing in an Ogg container is going to benefit from gzip encoding.

It turns out that there's a case where we are unable to calculate the duration of a file or seek because of us sending that Accept-Encoding header: http://www.fsf.org/video/?u=important_sm

In the above example URL, the server ends up sending the video compressed with gzip, so we don't get a Content-Length header, and can't seek inside the media.
Yes, it would be good to not send Accept-Encoding there.
I've changed my mind about Accept-Encoding.  I think it should be left to the server administrator to decide whether compression is appropriate for a particular response.  This is what we do for images right now, too.
This bug still exists in the latest webm build of Firefox available at <http://nightly.mozilla.org/webm/>.

Also, I just tested Chromium and noticed that its behavior is simply not to include the HTTP Accept header for requests for <audio> or <video> content.  Not sure if this is the solution most appropriate for Firefox, but it is certainly preferable as opposed to claiming text/html,application/xhtml+xml,application/xml are the preferred formats for audio and video.
I've implemented this by sending the following Accept header:
 
  video/*,audio/*;q=0.9,*/*;q=0.5

I didn't send specific MIME types (video/ogg, video/webm, etc) to keep things simple in the presence of other backends being added in the future.
Assignee: nobody → chris.double
Status: NEW → ASSIGNED
Attachment #459229 - Flags: review?(roc)
I propose the following accept header:

    video/webm,video/ogg;q=0.9,video/*,audio/*;q=0.8,image/png,image/gif,image/*;q=0.7,*/*;q=0.5

I'm not sure that animated PNG and GIF are supported in <video> tags yet, so you might want to exclude them. Also, speaking of image formats, would it be reasonable to treat static images as single-frame videos?
png, gif and image/* are not supported so we'd want to exclude them.
We should have a single method somewhere, maybe a static method of nsMediaStream, that sets the header on a channel, so we don't have to repeat this header string.

I don't want to bikeshed the string, but I think it does make sense to list video/webm for video and audio/ogg for audio as preferred formats if they've been built. So something like
#ifdef MOZ_WEBM
  "video/webm,"
#endif
#ifdef MOZ_OGG
  "audio/ogg,"
#endif
  "video/*,audio/*;q=0.9,*/*;q=0.5"

?
Surely, you'd want to prefer them more than video/* then. video/* should then be pushed down to something like q=0.8.
Attached patch Send correct Accept header (obsolete) — Splinter Review
Adjusted patch as per comments. The string is now obtained from a static method of nsMediaStream. The string returned is:

"video/webm;q=0.9,video/ogg,audio/ogg;q=0.9,video/*,audio/*;q=0.8,*/*;q=0.5"

The presence of "video/web" and "video/ogg,audio/ogg" depends on those backends being configured.

As suggested in comment 14 I made the video/* and audio/* have a slightly lower q.
Attachment #459229 - Attachment is obsolete: true
Attachment #459297 - Flags: review?(roc)
Attachment #459229 - Flags: review?(roc)
(In reply to comment #15)
> As suggested in comment 14 I made the video/* and audio/* have a slightly lower
> q.

Actually, I don't think this is what the proposed string implies.  With the proposed string, video/* actually has a priority of 1.0 because there is no ";q=0.8" immediately after it.  According to the HTTP spec:

> A more elaborate example is
>
>       Accept: text/plain; q=0.5, text/html,
>               text/x-dvi; q=0.8, text/x-c
>
> Verbally, this would be interpreted as "text/html and text/x-c
> are the preferred media types, but if they do not exist, then
> send the text/x-dvi entity, and if that does not exist, send
> the text/plain entity."
Oops, I guess I got the header format wrong. Sorry about that. I think maybe we should prioritize audio formats more for <audio> and video formats more for <video>. Here are my revised suggestions: 

<video> (assuming WebM and OGG are on) would send:

    video/webm,video/ogg,audio/ogg;q=0.9,video/*;q=0.8,audio/*;q=0.7,*/*;q=0.5

I'm not sure if <audio> can play audio streams from video formats, so I have kept the video formats in:

<audio> (assuming WebM and OGG are on) would send:

    audio/ogg,video/webm;q=0.9,video/ogg;q=0.9,audio/*;q=0.8,video/*;q=0.7,*/*;q=0.5
I just did some searching and I noticed that audio/webm is also a format, so that'd be worked in as such:

<video>:

video/webm,video/ogg,audio/webm;q=0.9,audio/ogg;q=0.9,video/*;q=0.8,audio/*;q=0.7,*/*;q=0.5

<audio>:

audio/webm,audio/ogg,video/webm;q=0.9,video/ogg;q=0.9,audio/*;q=0.8,video/*;q=0.7,*/*;q=0.5

What do you guys think about these accept strings?
For the <video> tag, does audio/ogg really deserve a higher priority than video/*?  What is the justification for having audio/ogg be part of the accept string at all for the <video> tag?  (And similarly, what is the justification for having video/ogg be part of the accept string for the <audio> tag?)
We should use the same accept string whether it is coming from an audio element or a video element. People often use an audio element to play a file served as video/ogg to only play the audio stream. Likewise people use a video element to play an audio only file served as audio/ogg as they use generic 'video player' javascript wrappers to play things.
Comment on attachment 459297 [details] [diff] [review]
Send correct Accept header

Removed review request until I address comment 16.
Attachment #459297 - Flags: review?(roc)
(In reply to comment #19)

We don't integrate MIME types from plugins yet (do we need a bug for that), so it may be best to prefer formats that you know you support until we integrate. Maybe a good compromise might be just assigning (in the case of <video>) audio formats the same priority as video/*, and placing them after it. For <audio>, you would do the opposite.

(In reply to comment #20)

> People often use an audio element to play a file served as
video/ogg to only play the audio stream.

Which is precisely why I think we should use different accept headers for each. Assume http://example.com/some-cool-video respected the Accept header and served the normal video usually, but if audio is preferred, it would serve the audio instead.
(In reply to comment #22)
> We don't integrate MIME types from plugins yet (do we need a bug for that), so
> it may be best to prefer formats that you know you support until we integrate.

I don't understand what you mean by this. What does 'integrate MIME types from plugins' mean in the context of HTML 5 audio/video?
(In reply to comment #20)
> We should use the same accept string whether it is coming from an audio element
> or a video element. People often use an audio element to play a file served as
> video/ogg to only play the audio stream. Likewise people use a video element to
> play an audio only file served as audio/ogg as they use generic 'video player'
> javascript wrappers to play things.

The point of the Accept header is to tell the web server what mime type we would prefer for a resource, so using the phrase "a file served as video/ogg" is misleading because a URL refers only to a resource; it is up to the web server, with help of the HTTP Accept and other headers, to provide a file that matches the description as requested by the client.

If the browser only wants to play the audio stream (i.e. the request is based on the URL being referenced in an <audio> tag), then the client should surely tell the browser that it is looking for audio, so that it doesn't wastefully send a full video file instead.
(In reply to comment #23)

I was thinking of a plugin implementing a decoder for a codec that could be used by HTML5 <audio> and <video> and registering the MIME types it supports. For example, someone might make a plugin for video/mp4;codecs="avc1.42E01E" (H.264) and audio/mpeg (MP3). One would hope that their plugin's supported codecs would be added to the appropriate media Accept headers.

Also, can plugins even associate with MIME types that include a codecs MIME parameter?
(In reply to comment #24)
> then the client should surely
> tell the browser that it is looking for audio, so that it doesn't wastefully
> send a full video file instead.

s/browser/server/
(In reply to comment #24)
> If the browser only wants to play the audio stream (i.e. the request is based
> on the URL being referenced in an <audio> tag), then the client should surely
> tell the browser that it is looking for audio, so that it doesn't wastefully
> send a full video file instead.

Yes, you're right. I'll change it to serve based on the element type.
(In reply to comment #25)
> I was thinking of a plugin implementing a decoder for a codec that could be
> used by HTML5 <audio> and <video> and registering the MIME types it supports.

As far as I know we have know plans for HTML 5 media backends to be implementable by plugins. Is that what you mean?
s/know/no/
(In reply to comment #28)

Yes, that's exactly what I meant. I was suggesting that plugins should be able to add types to the Accept headers for media elements, and I guess that would be part of the HTML5 media backends for plugins.
(In reply to comment #25)
> I was thinking of a plugin implementing a decoder for a codec that could be
> used by HTML5 <audio> and <video> and registering the MIME types it supports.
> For example, someone might make a plugin for video/mp4;codecs="avc1.42E01E"
> (H.264) and audio/mpeg (MP3). One would hope that their plugin's supported
> codecs would be added to the appropriate media Accept headers.

There's probably no reason to enumerate every single format that is provided by plugins (assuming plugins are one day even able to provide HTML5 media backends).  In my opinion, this is why the <video> tag should accept video/* as a lower priority than our truly preferred formats, but as a higher priority than audio/ogg, audio/webm, and audio/*.  To compare, the Accept tag for resources fetched based on the <img> tag doesn't contain all possible formats accepted.  Rather, it just says Firefox prefers png images over others, if a resource is available in multiple formats:

> Accept: image/png,image/*;q=0.8,*/*;q=0.5
(In reply to comment #31)

My suggestion is intended more for right now, where we don't yet have HTML5 plugin backends, so you can safely assume that video/* is less likely to be supported than audio/ogg or audio/webm. I see what you mean, and for the sake of consistency with future versions that do support HTML5 plugin backends, I guess it may be best to prefer video/* more.
So how does this look:

<video>:

video/webm;q=0.9,video/ogg;q=0.9,video/*;q=0.8,audio/webm;q=0.7,audio/ogg;q=0.7,audio/x-wav;q=0.7,audio/*;q=0.6,*/*;q=0.5

<audio>:

audio/webm;q=0.9,audio/ogg;q=0.9,audio/*;q=0.8,video/webm;q=0.7,video/ogg;q=0.7,video/*;q=0.6,*/*;q=0.5
(In reply to comment #33)
Looks good, but why are video/webm and video/ogg 0.9 instead of an implicit 1 (and the same for audio/... in <audio>)?
(In reply to comment #33)
> So how does this look:
> 
> <video>:
> 
> video/webm;q=0.9,video/ogg;q=0.9,video/*;q=0.8,audio/webm;q=0.7,audio/ogg;q=0.7,audio/x-wav;q=0.7,audio/*;q=0.6,*/*;q=0.5
> 
> <audio>:
> 
> audio/webm;q=0.9,audio/ogg;q=0.9,audio/*;q=0.8,video/webm;q=0.7,video/ogg;q=0.7,video/*;q=0.6,*/*;q=0.5

There's probably no reason to specify q=0.9 on the highest-priority mime types, since the default is 1.0.

Also, if the <video> tag is going to include audio/ogg in the Accept header, I think application/ogg should be accepted with the same priority, based on the notes at <https://developer.mozilla.org/En/Media_formats_supported_by_the_audio_and_video_elements#Recognized_MIME.c2.a0types_for_Ogg_media>.  (And the <audio> tag should accept application/ogg with the same priority as video/ogg.)

So how about:

<video>:

video/webm,video/ogg,video/*;q=0.9,audio/webm;q=0.7,audio/ogg;q=0.7,audio/x-wav;q=0.7,application/ogg;q=0.7,audio/*;q=0.6,*/*;q=0.5

<audio>:

audio/webm,audio/ogg,audio/*;q=0.9,video/webm;q=0.7,video/ogg;q=0.7,application/ogg;q=0.7,video/*;q=0.6,*/*;q=0.5

Also, why is audio/x-wav included for <video> but not <audio> ?
(In reply to comment #35)
> Also, why is audio/x-wav included for <video> but not <audio> ?

Accidental cut/paste error, sorry. Patch coming shortly with your suggested strings.
I basically like where things stand, but I'm still unsure whether anything representing audio should be included in the <video> accept tag, and whether anything representing video should be included in <audio>'s accept tag.  If the server chooses to send audio in response to something in a <video> tag, that's fine either way, as it's covered by the */* with the lowest priority.
(In reply to comment #33)

audio/wav is officially registered, so why use audio/x-wav? Also, I think that audio/x-wav should be lower than the other audio formats by .1 due to being anything but a format for the web.
(In reply to comment #37)

If I were an video site host, I'd serve text/html when the most preferred type in an Accept header is */*, so it's best to include the video and audio formats in both headers.
(In reply to comment #39)
> (In reply to comment #37)
> 
> If I were an video site host, I'd serve text/html when the most preferred type
> in an Accept header is */*, so it's best to include the video and audio formats
> in both headers.

Good point.  I think it might be worth consulting the WHATWG list to see what others think of this proposed behavior of explicitly accepting audio for a <video> tag and vice versa.
Attached patch Send correct Accept header (obsolete) — Splinter Review
Adjust Accept header text as per comment 35, with change to audio/web suggested in comment 38.
Attachment #459297 - Attachment is obsolete: true
Attachment #459320 - Flags: review?(roc)
static ns(C)Strings will report as a leak.

Let's make the virtual methods take the nsIChannel as a parameter and call SetRequestHeader themselves. Use NS_LITERAL_CSTRING to create the literal string.

The Accept header text seems like a bit of overkill. For audio I would just have "video/*" for all video types, and likewise for video I would just have "audio/*" for all audio types.
I removed the specific audio types from the video request and the video types from the audio request. I couldn't use NS_LITERAL_CSTRING due to macro-in-macro errors so I went for a temporary nsCAutoString as discussed.
Attachment #459320 - Attachment is obsolete: true
Attachment #459697 - Flags: review?(roc)
Attachment #459320 - Flags: review?(roc)
Comment on attachment 459697 [details] [diff] [review]
Send correct Accept header

Requesting approval for landing. This patch is low risk, changes the existing incorrect Accept header string during video/audio requests only to the correct value.
Attachment #459697 - Flags: approval2.0?
Comment on attachment 459697 [details] [diff] [review]
Send correct Accept header

Approved for 2.0 -- let's get this in early enough in the betas that we have time to be surprised by someone depending on the broken behaviour. :)
Attachment #459697 - Flags: approval2.0? → approval2.0+
Keywords: checkin-needed
Pushed:
http://hg.mozilla.org/mozilla-central/rev/3ddffdd3d809
Status: ASSIGNED → RESOLVED
Closed: 14 years ago
Resolution: --- → FIXED
Keywords: checkin-needed
What's the reasoning for not having these as prefs like network.http.accept.default and image.http.accept? It would be useful to have these for testing servers.
(In reply to comment #47)
> What's the reasoning for not having these as prefs like
> network.http.accept.default and image.http.accept?

No one asked for it and I didn't know about the existing prefs. If you raise a bug for it I'll get on to it.
Depends on: 583139
(In reply to comment #48)
> (In reply to comment #47)
> > What's the reasoning for not having these as prefs like
> > network.http.accept.default and image.http.accept?
> 
> No one asked for it and I didn't know about the existing prefs. If you raise a
> bug for it I'll get on to it.

Filed bug 583139.
Hasn't this stuff pretty much been made irrelevant now that HTML 5 has been announced as standard or am I completely off the mark?

http://www.remark-group.co.uk/audio-visual/
(In reply to Kelly Webber from comment #50)

> Hasn't this stuff pretty much been made irrelevant now that HTML 5 has been
> announced as standard or am I completely off the mark?

Please don't post unrelated comments and links to the tracker. There are some guidelines at https://bugzilla.mozilla.org/page.cgi?id=etiquette.html if you're unsure what would be relevant.
My apologies.

I have just seen the last time someone commented also. Over 2 years ago. Which makes me assume these errors will now have been fixed.
(In reply to Kelly Webber from comment #52)

> I have just seen the last time someone commented also. Over 2 years ago.
> Which makes me assume these errors will now have been fixed.

They have. I've just verified with Firefox Nightly 17.0 (2012-07-21). Are you seeing incorrect accept headers for media element sources?
Am I the only one still getting questionable behavior? Maybe my expectations are wrong here, but I believe I'm having the same issue noted here for .zip files. Any time I monitor Firefox's requests for .zip files, I see an Accept field of:

text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8

and yet rarely do I ever have an issue with .zip files showing up as gibberish in my browser window or other such nonsense. Those files tend to prompt me for a download. Do mainstream web servers even bother with the Accept field, or are they just checking file extensions, seeing that they're .zip, and sending back a Content-Type of application/zip anyway?
You do not have the same issue or do you use ".zip" files in audio/video tags ?
The posted header is absolutely ok for a general request for a document.
I hope you noticed the */* wildcard content type in the Accept header ?
I understand the wildcard, I guess I was just wondering how mainstream servers (i.e. Apache) handle it. I assumed that the web server looked at the file extension to make its determination, and wanted to know if this was correct. Either way, this is probably not a great place to ask this question, it just seemed semi-relevant.

Thanks for your quick response, but don't worry about it.
(In reply to TheBlackDeath333 from comment #56)
> I understand the wildcard, I guess I was just wondering how mainstream
> servers (i.e. Apache) handle it. I assumed that the web server looked at the
> file extension to make its determination, and wanted to know if this was
> correct. Either way, this is probably not a great place to ask this
> question, it just seemed semi-relevant.
> 
> Thanks for your quick response, but don't worry about it.

IIUC, the q= value is a preference: the browser "prefers" HTML, XHTML, or XML, but it will "accept" anything else. So an HTML file, for instance, should be served as text/html rather than text/plain; but anything which cannot be regarded as HTML, XHTML or XML will be sent as whatever the server thinks it is (a properly configured server ought for instance not to serve a zipfile as any of these three), and the browser will try to display it, or download it, as best it can according to the filetype declared by the server.

How the server builds the Content-Type header may depend on the file name, and I imagine that if the filename gives an ambiguous or indeterminate answer the server might have a look at some part of the file's contents.

Developers: sorry for the bugspam.
You need to log in before you can comment on or make changes to this bug.