User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.3a4webm) Gecko/20100518 MozillaDeveloperPreview/3.7a4webm
Build Identifier: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.3a4webm) Gecko/20100518 MozillaDeveloperPreview/3.7a4webm
I am testing a pay-per-minute video script. On the initial request for a video the script returns a 206 status and only returns 0-X bytes of the video, along with the full content-length and range headers. After Firefox plays this content is does not attempt to make additional requests to get the data that it does not have.
I have tested this with a direct video url (play.php?file=x - video/webm) directly and also using a video.html page that includes a video tag with "autobuffer=false preload=metadata".
Chrome behaves differently because it reads the first 1k and the last few kilobytes of the file (to get seek ranges / metadata?) and it performs ranged http requests (seeks) when additional video data is needed or when the video control is used to seek ahead or in reverse.
Created attachment 449901 [details]
a PHP play script to demonstrate how the browser requests videos
I'm attaching a sample PHP script that is very basic but demonstrates (in a log file) the way Firefox loads video files. Compare these logs to the way Chrome requests the video file.
The PHP script must be able to create "log" in the directory it is run from - otherwise change the path.
Whoops - replace "http://xdev01a/~mjohansson/" with "" in the add time link.
Sorry for the spamminess of my comments.. You may also want to change the $max_length setting to 102400; -- This will cause it to seek more regularly and better illustrates the point. If it stops playing then you ran out of time - hit play.php?addtime=1 again.
Can you put the script on a server somewhere we can access it? That'll be a bit easier than setting up PHP.
I'm not sure I understand what the issue is. Currently we download the entire video (up to a configured maximum size) if autobuffer is true. What you want in this bug is us to change that so we use byte range requests to only receive small portions of the file as we play it. Is that correct?
Looking at the PHP code I think I understand a bit better. So what you are doing is sending a Content-Range header back with a range different than that what we asked for. We're ignoring that. What is the expected behaviour in this case? Should we stop playback after we reach the end of that range? Or should we make another request to get the rest of the video? Is it the latter that you are expecting?
I'm not sure what should be done in this situation for video playback per spec.
BTW, unrelated to the bug, I note in your script you have:
I might be wrong by I don't think Accept-Ranges takes a byte range.
@roc I don't have a host available at the moment. I'll see what I can do if needed.
When the browser requests the full document I want to tell it that this isn't possible, that I am only willing to give it a range.
I return the first X many bytes with a 206 status. I tried some other headers in the code, included with an Accept-Ranges header, but none of them caused the browser to try again. The one that I thought would have worked is "416 Range Not Satisfiable".
I've changed Accept-Ranges to bytes - it doesn't seem to make any difference in Chrome or Firefox. You're right that I was using it incorrectly. Since fixing the Accept-Ranges I've retried these other status codes and they still don't have the desired effect of getting the browser to start making ranged requests.
> A server sending a response with status code 416 (Requested range not satisfiable) SHOULD include a Content-Range field with a byte-range- resp-spec of "*".
It seemed to me that if I returned a 416 with the full range in response to an un-ranged GET, the client would try the GET again with a range (similar to how a 401 is handled). That didn't happen.
I'm also curious if a 301 or 307 redirect with an Accept-Ranges header would have the desired affect. Since the redirect target could be elsewhere it wouldn't make sense for my range suggestion to apply to the redirect target.
Since 2xx's are successful, Firefox is processing the 206 data that I return, but it isn't making successive requests and that is the problem (and where Firefox differs from Chrome).
In the case of video, I would expect Firefox to continue making 206 requests when it reaches the end of the data that it initially received based on the range it was given.
The log from Chrome looks like this. While playing I seeked forward and reverse in the video. All requests are handled with a 206 response. The first request had no range. It then seems that they request meta data before starting to play the video. Also notice how they rerequest the same blocks. Because this is a php script the no-cache headers are being returned by default. This is what I want to have happen - the browser is not allowed to cache this content.
[ filename] [range requested or ""] [first byte returned] [last byte returned] [seeks/time remaining counter]
big_buck_bunny.webm 0 102400 60
big_buck_bunny.webm bytes=0-1024 0 1024 59
big_buck_bunny.webm bytes=1024- 1024 103424 58
big_buck_bunny.webm bytes=5068966- 5068966 5171366 57
big_buck_bunny.webm bytes=3970- 3970 106370 56
big_buck_bunny.webm bytes=106370- 106370 208770 55
big_buck_bunny.webm bytes=208770- 208770 311170 54
big_buck_bunny.webm bytes=311170- 311170 413570 53
big_buck_bunny.webm bytes=413570- 413570 515970 52
big_buck_bunny.webm bytes=1183338- 1183338 1285738 51
big_buck_bunny.webm bytes=1285738- 1285738 1388138 50
big_buck_bunny.webm bytes=1388138- 1388138 1490538 49
big_buck_bunny.webm bytes=2495452- 2495452 2597852 48
big_buck_bunny.webm bytes=2597852- 2597852 2700252 47
big_buck_bunny.webm bytes=2280672- 2280672 2383072 46
big_buck_bunny.webm bytes=2383072- 2383072 2485472 45
big_buck_bunny.webm bytes=2485472- 2485472 2587872 44
big_buck_bunny.webm bytes=2587872- 2587872 2690272 43
big_buck_bunny.webm bytes=464050- 464050 566450 42
big_buck_bunny.webm bytes=566450- 566450 668850 41
big_buck_bunny.webm bytes=668850- 668850 771250 40
big_buck_bunny.webm bytes=771250- 771250 873650 39
And here is what happens when Firefox attempts the same thing:
big_buck_bunny.webm 0 102400 60
I returned a 206 status with 100K of data to the un-ranged request. The full size of the file is about 5M, so the video only plays for about a second and then stops.
(In reply to comment #7)
> I'm also curious if a 301 or 307 redirect with an Accept-Ranges header would
> have the desired affect. Since the redirect target could be elsewhere it
> wouldn't make sense for my range suggestion to apply to the redirect target.
Range headers are passed along an HTTP redirect. See bug 560806 which implemented this. This is not in Firefox 3.6 but is in trunk builds.
Created attachment 450118 [details]
PHP file to illustrate GET behavior by browsers accessing video content
I removed the log file used by attachment 449901 [details] in favor of dumping debug data from the session into a separate info frame. made the status code setting more obvious and added hooks for other types of status codes that can be used to attempt to get the browser to use ranged requests. As this code is, only 206 is successful in getting a video to play but Firefox does not continue to fetch past the end of the first 206 response.
Are the right people seeing this bug to decide if the browser should begin using ranged-requests when an initial un-ranged request is handled by the server with a 206 response?
Is this handled differently for video - or would the rest of the browser work the same way for other content types?
If there is a better HTTP way of switching to ranged requests I will use it. What separates my request from bug 560806 is that I am never willing to send a 200 because I will never offer the user the full content in one go - I need to meter it.
Marques, Chris is the right person for video.
> Is this handled differently for video
Right now I think this is just our bug. Marques, from what you've described of your setup, I think it should work. At least, once we've finished reading the first range you've sent, we should ask for more.
After discussing with Chris D, I take that back. I think we *could* handle this fairly easily. However, it's unclear whether this server behaviour is allowed by the HTTP spec. Marques, can you convince me that it is?
I doubt it is allowed by the HTTP spec, because a simple HTTP client that just issues the simplest possible GET would not get the whole resource. The spec does not require clients to retry the way you want us to.
I've been scouring specs looking for justification, but the best I can find seems to point against this behavior (damn):
The server has fulfilled the partial GET request for the resource.
The request MUST have included a Range header field (section 14.35)
indicating the desired range, and MAY have included an If-Range
header field (section 14.27) to make the request conditional.
Emphasis on the "The request MUST have included a Range".
As I stated before Chrome doesn't seem to care about this - Opera behaves as Firefox does.
Perhaps all video requests could be sent with a range of '0-' to give a server the opportunity to reply with a 206?
I see the potential for a 302 status to include accept-ranges, but that doesn't seem to assure the server that the client will use the ranges when it follows the redirect.
A 416 seems like it wouldn't be appropriate if a range wasn't requested.
Are there any other strategies for a server to request a client to do something?
If I returned 402 Payment Required - that would seem to make sense, because if you are willing to buy the entire video I could return it on a 200 status to an unranged request. The spec is vague on that (reserved for future use... indefinitely?), and then what? Would I also include Accept-Ranges? Would the browser retry with the range?
Under 406 ( http://tools.ietf.org/html/rfc2616#page-66 ):
Note: HTTP/1.1 servers are allowed to return responses which are
not acceptable according to the accept headers sent in the
request. In some cases, this may even be preferable to sending a
406 response. User agents are encouraged to inspect the headers of
an incoming response to determine if it is acceptable.
We aren't dealing with "Accept:" headers, but does this set a precedent for optimistic behavior? Let's see the rest of the Content Negotiation section...
12 Content Negotiation
Most HTTP responses include an entity which contains information for
interpretation by a human user. Naturally, it is desirable to supply
the user with the "best available" entity corresponding to the
request. Unfortunately for servers and caches, not all users have the
same preferences for what is "best," and not all user agents are
equally capable of rendering all entity types. For that reason, HTTP
has provisions for several mechanisms for "content negotiation" --
the process of selecting the best representation for a given response
when there are multiple representations available.
Note: This is not called "format negotiation" because the
alternate representations may be of the same media type, but use
different capabilities of that type, be in different languages,
We are negotiating Ranges and range support? Seems valid.
12.2 Agent-driven Negotiation
With agent-driven negotiation, selection of the best representation
for a response is performed by the user agent after receiving an
initial response from the origin server. Selection is based on a list
of the available representations of the response included within the
header fields or entity-body of the initial response, with each
representation identified by its own URI. Selection from among the
representations may be performed automatically (if the user agent is
capable of doing so) or manually by the user selecting from a
generated (possibly hypertext) menu.
Agent-driven negotiation is advantageous when the response would vary
over commonly-used dimensions (such as type, language, or encoding),
when the origin server is unable to determine a user agent's
capabilities from examining the request, and generally when public
caches are used to distribute server load and reduce network usage.
That last line says to me that since this is for the betterment of network utilization and server load the client should be nice and address this bug ;-)
The next block about Transparent Negotiation seems to indicate that this sort of behavior would be okay for a cache agent.
12.3 Transparent Negotiation
Transparent negotiation is a combination of both server-driven and
agent-driven negotiation. When a cache is supplied with a form of the
list of available representations of the response (as in agent-driven
negotiation) and the dimensions of variance are completely understood
by the cache, then the cache becomes capable of performing server-
driven negotiation on behalf of the origin server for subsequent
requests on that resource.
Transparent negotiation has the advantage of distributing the
negotiation work that would otherwise be required of the origin
server and also removing the second request delay of agent-driven
negotiation when the cache is able to correctly guess the right
Although the server isn't supposed to return a 206 unless a Range: was requested, it seems like the rest of the spec wants the client to negotiate - optimally without a second request. In this case, I think that implies that the client should handle the response as though it had sent a "Range: bytes 0-", which should probably be sent for any request since it seems to be the only way to give the server the opportunity to follow spec and respond with pieces.
The simplest client would be using http/1.0. I should see how curl and wget handle thi
And yet, if the browser requests range 0-*, I still may choose to respond with a smaller partial response. I don't know if FireFox or the spec deal with this situation either. Chrome handles it the way I expect - by taking what it is given and going back for more.
As a side note - If browsers are going to accept smaller ranges than they asked for (seems like all browsers do this) - should there be a limit to how small and how many requests a server can make a browser request?
What if the server returns 1 byte at a time? What if the server keeps returning the same fragment, not the one requested - will the browser retry for ever?
I changed my test script to use VIDEO elements instead of just loading the video content directly in the frame - using this method Opera will play the OGG video - which means it treats the 206s the way I describe. Their webm handler does not do this however. Using the VIDEO element didn't change any of Firefox's behaviors.
Back to comment 8 - If my initial header is not a 206 (like a 307) I can get Firefox to do partial GETs, but it doesn't continue to fetch when it's requested range isn't satisfied (this is starting to sound like my initial bug description).
Server: 307 (Accept-Ranges: bytes)
Firefox: GET (Range: bytes=0-)
Server: 206 (Content-Range: bytes 0-99999/1000000) (Content-Length: 100000)
Firefox: * has metadata - sets play time on the control bar, plays the piece of the video when play is pressed, no further requests to continue playing *
User: * Seeks somewhere in the video control bar *
Firefox: GET (Range: bytes=[x]-) (x is the correct byte for the segment the user requested)
Server: 206 (Content-Range: bytes [x]-[x+99999]/1000000) (Content-Length: 100000)
Firefox: * plays the piece of the video, no further requests to continue playing *
Aside from continuing to play the video without additional user interaction, Chrome has a second initial request to fetch a block of data at the end of the video - I assume for meta data purposes.
(in reply to my comment 16)
with initial 206:
- curl and lftpget will save the first range that the server responded with and stop there.
- wget will continually retry the fetch - not satisfied with the initial 206 response.
with initial 307:
- curl, lftpget, wget do not follow up with partial GET requests, so result as above
I've taken this to the IETF-http working group:
and to WHAT working group:
I hope to get some official way to handle this since I think it should be possible based on the existing spec. It seems to me a matter of clarification to get other browsers to behave the way some browsers already do (sometimes).
*** Bug 820340 has been marked as a duplicate of this bug. ***