92915 - Ignore Transfer-Encoding header for 304 response (rfc2616, 4.4)

Status: RESOLVED FIXED

(Core :: Networking: HTTP, defect, P2)

Description

[Bob McElrath]

When using an HTTP/1.1 proxy, and hitting reload, mozilla fails to completely
reload the page and subsequently is unable to issue any HTTP requests.

I can reproduce it by:
  1) load a page (I use http://www.mozillazine.org for this test)
  2) wait for it to complete loading
  3) hit reload.
When using a http/1.1 proxy, the HTML page loads fine, then mozilla sends
requests for (most of) the images.  On the url above, the proxy returns 304 (not
modified) for all images.  There are a handful of images on the page though that
mozilla does *not* send new requests for.  Each of these is *not* redrawn, and
mozilla hangs (the throbber never stops spinning).  The set of images it
"forgets" to request is the same each time (tile.jpg, grass.gif, a few
others...)  I then hit 'esc' or the stop button to stop the throbber.
Subsequently, mozilla won't load any pages.  Hitting bookmarks or typing
something into the url bar doesn't generate new requests to the proxy.  (I'm
watching the network traffic with ngrep -- this can't be a proxy bug)  Mozilla
has to be killed and restarted to get back its functionality.

This doesn't happen if the proxy is not enabled.

This bug has appeared in the last ~1-2 months and isn't present in older
releases.

Comment 1

[benc]

bob: Who's the vendor of your proxy?

bradley: what's next, packet trace?

Comment 2

[Bob McElrath]

The vendor is me.  http://draal.physics.wisc.edu/FilterProxy/

Can someone verify this with *any* other HTTP/1.1 proxy?  (sometimes methinks
they don't exist!)

This shouldn't be a proxy bug though, since the requests for the images it hangs
on are not sent (watching with ngrep).  Also, this bug existed before the
keep-alive fix went in (bug 84264), so it shouldn't be related to connections
being left open or anything.  I just checked that it existed with a nightly from
7/2.

I have seen this happen without hitting reload (but it's much harder to
trigger)...if mozilla is left alone for a while, and all connections to the
proxy close.  Again, it hangs and is unable to load anything.

Comment 3

[Bradley Baetz (:bbaetz)]

I can't try filterproxy, since it locks up my machine, and attempting to
downgrade perl is probably just going to introduce more problems.

I don't know of any 1.1 proxies that are in general use, although squid supports
most of the 1.1 caching headers

Lets try packet traces between mozilla and the proxy first, and see if it shows
anything.

This could be bug 77072, which is probably some form of race condition.

Comment 4

[coen]

I can confirm this problem, although it's very hard to reproduce.
I'm using Squid 2.3S4. The testcase above didn't show the problem for me.

I see it sometimes when I'm posting messages to a message board, or when I'm
generating al lot of requests in a short time. (i.e. opening pages with allot
off images)

Comment 5

[Bob McElrath]

Well, since I can easily reproduce this bug, I will try to get some debugging
information.  Can some of the experts point me in the right direction?  I'm
using a talkback build right now, but I don't know how to get it to dump info
after it's "hung".  Should I produce a tcpdump or ngrep trace of the connection?
 Bradley mentioned a "debug build" before...where can I get that, and once I
have it how can I get useful info out of it?

Comment 6

[Bob McElrath]

Some notes on the above testcase:
1) Hitting shift-reload does not cause it to hang.  It reloads the main page and
all images.  It receives [200 OK] for all.
2) Clicking on the URL bar, and hitting enter does not cause it to hang. It
reloads the main page only and receives [200 OK].
3) Hitting my bookmark for this page does not hang it.  It reloads the main page
only and received [200 OK].
4) Hitting reload (or ctrl-R) hangs it.  It reloads main page, style.css, and 7 
images (abdefghk) -- receives [304 Not Modified] for all except the main page.

The page has 11 images:
a) new/title.gif
b) new/blimp.gif
c) grass.gif
d) transparent.gif
e) build_good.gif
f) buildbar_help.gif
g) transparent.gif
h) article_icon.gif
i) poll_navhead.gif
j) oldernews.gif
k) separator.gif
l) new/tile.jpg

What does "reload" do differently than shift-reload and the URL bar?  Can
someone suggest where to set a breakpoint to catch "reload" when debugging
mozilla, so I can trace it?

BTW, shift-reload loads images once for each time they appear in a page.  (bug
79020)

All combinations of the option Advanced->Cache: "Compare the page in the cache
to the page on the network" have exactly the same effect.  In each case it
reloads its 7 images anyway and gets 304 responses -- these options clearly are
having no effect (anyone know if this is filed as another bug?)

Comment 7

[Bradley Baetz (:bbaetz)]

I cannot reproduce this, using squid (I'll try filterproxy on monday).

Can you use ethereal (www.ethereal.com) to get a packet trace? tpcdump or ngrep
is also fine, as log as you make sure that you capture the entire packet. I
prefer ethereal; the output is easier to read/filter.

You have to build a debug build yourself. I should be turning on http logging in
the default build at some point soon, hopefully.

With reloading, I'm not sure if that is the expected behaviour, but there have
een reports that the cache is broken in recent builds.

-> darin

Comment 8

[Darin Fisher]

-> moz 0.9.4

Comment 9

[Bob McElrath]

Attachment: ethereal (lipcap) dump of successful http://www.mozillazine.org request

Comment 10

[Bob McElrath]

Attachment: Ethereal (libpcap) dump of http://www.mozillazine.org hang case.

Comment 11

[Bob McElrath]

One last note: explicitly setting http version to 1.0 makes this hang not happen
(Preferences->Debug->Networking).

Comment 12

[coen]

"explicitly setting http version to 1.0 makes this hang not happen"

I'll try this, and report when it helps for me.

Comment 13

[Bradley Baetz (:bbaetz)]

coen: With that description, I think that you're seeing bug 77072.

On the 304 response, the proxy is sending a Transfer-Encoding: chunked header.
According to rfc2616, section 4.4:

"   1.Any response message which "MUST NOT" include a message-body (such
     as the 1xx, 204, and 304 responses and any response to a HEAD
     request) is always terminated by the first empty line after the
     header fields, regardless of the entity-header fields present in
     the message.
"

We need to do this. The best place for this (for 304's) would be
nsHttpResponseHead::UpdateHeaders (and remove the comment about "wacky servers";
they're technically following the spec. That won't help 204s, 100s, or HEAD
requests. I suspect that we get the content-length stuff wrong for those, too.

Updating summary, and confirming

Comment 14

[coen]

Well, actually I can't easily reproduce this. It occurs after browsing some
time. What I see, perfectly matches the description given by the reporter of
this bug.

I've set http to 1.0 now. The only thing i can do, is wait if it happen now. I
hope I can report this later this evening.

Comment 15

[coen]

With http set 1.0, the problem didn't happened

Comment 16

[Darin Fisher]

the fix for this should be pretty simple... patch to come.

Comment 17

[Darin Fisher]

my patch for bug 89113 will fix this as well.

Comment 18

[Darin Fisher]

fixed-on-trunk by my patch for bug 89113

Comment 19

[Bradley Baetz (:bbaetz)]

*** Bug 96628 has been marked as a duplicate of this bug. ***