245447 - response to HEAD-request containing content results in corrupt download when using HTTP/1.1 and keep-alive

Status: VERIFIED INVALID

(Core :: Networking: HTTP, defect)

Description

[Bruno De Fraine]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8a1) Gecko/20040520
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8a1) Gecko/20040520

When using Mozilla to access the OTRS application, downloads made using "Save
Link Target as..." often are corrupted and contain part of the HTTP-response
header in the middle. Not saving the link, but just displaying it in the browser
shows correct content.
After some investigation, I suspect that this is caused by OTRS replying to
Mozilla's HEAD-request with a response that doesn't contain just headers, but
also the content (see additional information below).

Reproducible: Sometimes
Steps to Reproduce:
1. Fire up Mozilla and open the URL of the OTRS demo site
2. Click on "Login" to login with the default username and password
3. In the message view that shows up next, locate the "attachment" section,
right-click the link "license.txt" and select "Save link Target as..."
4. When the dialog shows up to specify the download file name, quickly click "Save"
5. Wait for the download to complete

Actual Results:  
The downloaded file is either correct or corrupted in a very structured way (the
latter is what happens most often). When the resulting file is corrupted, it is
much bigger (about 58.205 bytes in size), and has a structure like this:
[ second part of the Mozilla license text ]
[ HTTP response-header ]
[ a line containing the text "7895" ]
[ Mozilla license text ]
[ a line containing the text "0" ]

With "HTTP response header" I mean for example the following text:

HTTP/1.1 200 OK
Date: Thu, 03 Jun 2004 08:38:40 GMT
Server: Apache/1.3.27 (Linux/SuSE) PHP/4.3.1 mod_perl/1.27
Content-Disposition: filename=license.txt
Keep-Alive: timeout=10, max=98
Connection: Keep-Alive
Transfer-Encoding: chunked
Content-Type: text/plain; charset="us-ascii"


Expected Results:  
The downloaded file should always be the correct file. By this I mean the file
that is also displayed in the browser when you simply click the link, not when
you "Save link target as...". This file is 30.869 bytes in size and just
contains the Mozilla license text.

When using the "Save link target as..." function, Mozilla first issues a
HEAD-request (to discover the filename, I guess), then shows the dialog where
you can specify where you want to save, and then issues the GET-request to make
the actual download. I suspect that this is caused by OTRS replying to the first
HEAD-request with a response that doesn't contain just headers, but also the
content (i.e. replying as if it was a GET-request). Under certain circumstances
Mozilla mixes this unexpected response-content from the HEAD-request with the
response from the GET-request.

I managed to isolate the problem by writing a simple Perl HTTP daemon that sends
a HTTP response with content, no matter what type of request. When you make a
link to this server in a separate page, and use the "Save link target as..." on
that link, this always produces the problem mentioned above, only slightly
different. What happens is the following:
1. Mozilla asks where to download and proposes the filename reported by the server.
2. When you specify a filename, Mozilla starts to download in the download
manager, but after it has received all the information, it keeps waiting for
several minutes for the download to end (i.e. the download is still marked as
"Saving").
3. When I make the daemon close its connection by killing it, Mozilla marks the
download as completed; however the downloaded file is corrupt in the same
structural way as described above.

It does work correctly however when I change the HTTP networking settings and
disable either HTTP/1.1 or "Keep-Alive".

Comment 1

[Bruno De Fraine]

Attachment: Sample HTTP server in Perl using HTTP::Daemon

Comment 2

[Christian :Biesinger (don't email me, ping me on IRC)]

>replying to
>Mozilla's HEAD-request with a response that doesn't contain just headers, but
>also the content (see additional information below).

that is not allowed. fix the server.

Comment 3

[Jo Hermans]

Verified.

See RFC2616 :

9.4 HEAD

   The HEAD method is identical to GET except that the server MUST NOT
   return a message-body in the response. The metainformation contained
   in the HTTP headers in response to a HEAD request SHOULD be identical
   to the information sent in response to a GET request. This method can
   be used for obtaining metainformation about the entity implied by the
   request without transferring the entity-body itself. This method is
   often used for testing hypertext links for validity, accessibility,
   and recent modification.

Comment 4

[geoffrey young]

unfortunately, not everyone is RFC compliant with respect to HEAD requests.  and
it's not just the small players:

$ telnet www.yahoo.com 80
Trying 216.109.117.207...
Connected to www.yahoo.com (216.109.117.207).
Escape character is '^]'.
HEAD / HTTP/1.1
host: www.yahoo.com
 
HTTP/1.1 200 OK
Date: Sun, 06 Jun 2004 08:15:39 GMT
P3P: policyref="http://p3p.yahoo.com/w3c/p3p.xml", CP="CAO DSP COR CUR ADM DEV
TAI PSA PSD IVAi IVDi CONi TELo OTPi OUR DELi SAMi OTRi UNRi PUBi IND PHY ONL
UNI PUR FIN COM NAV INT DEM CNT STA POL HEA PRE GOV"
Cache-Control: private
Vary: User-Agent
Connection: close
Content-Type: text/html
 
<html><head>
 
<title>Yahoo!</title>...

in the interests of playing nice in the sandbox and following the 'be strict in
what you send, lenient in what you receive' mantra, properly handling invalid
responses to HEAD requests probably ought to be addressed.  at least if there is
an interest in making mozilla a better client.

Comment 5

[Christian :Biesinger (don't email me, ping me on IRC)]

(In reply to comment #4)
> in the interests of playing nice in the sandbox and following the 'be strict in
> what you send, lenient in what you receive' mantra, properly handling invalid
> responses to HEAD requests probably ought to be addressed.  at least if there is
> an interest in making mozilla a better client.

unfortunately that is not exactly possible. there is a bug to stop sending HEAD
requests, which would avoid this issue, another workaround would be to stop
using keepalive w/ HEAD, but it is not possible to find out whether this body
was a HTTP/0.9 response to the following GET request or whether it was to HEAD;
at least not reliably.

Comment 6

[Bruno De Fraine]

(In reply to comment #5)
> there is a bug to stop sending HEAD
> requests, 

That would be bug 160454

> which would avoid this issue,

Then perhaps this bug should be apropriately linked to that one?