137155 - POST request sent as two small packets (IIS 5 sometimes chokes)

Status: RESOLVED FIXED

(Core :: Networking: HTTP, enhancement, P3)

Description

[Jeffrey Baker]

I noticed that in Mozilla 0.9.9 on Linux, POST requests are sent as two small
when they would fit in one packet.  The first packet on a particular request I
observed had most of the headers, up to and including Referer, in 861 bytes. 
The second packet included only the Content-Type header in 117.  It would be
more efficient if these could be sent in one packet.

Comment 1

[Darin Fisher]

true, but we're talking about a very small inefficiency here.

Comment 2

[Brant Gurganus]

[RFE] is deprecated in favor of severity: enhancement.  They have the same meaning.

Comment 3

[Nicola Fankhauser]

this is in fact not merely a matter of efficiency, but actually blocks POST
requests to Microsoft IIS 5 webservers.

after the first packet IIS 5 immediately responds with a "400 Bad Request" and
sliently discards the second one.

I attached an ethereal libcap dump, as well as a human-readable hexdump of the
libcap dump, showing a POST request to a Microsoft IIS 5.0 webserver resulting
in a "400 Bad Request".

To "prove" that it's mozilla, I ran the following:
# nc mica.swissonline.ch 80 < mozilla_request.txt

which worked (mozilla_request.txt contains the two assembled packets from mozilla).

Comment 4

[Nicola Fankhauser]

Attachment: Ethereal dump showing a failing POST to a IIS 5.0

Comment 5

[Nicola Fankhauser]

Attachment: Ethereal human-readable hex-dump showing a failing POST to a IIS 5.0

Comment 6

[Michael Newton]

I suggest changing the severity from an RFE so it gets the attention it needs.

There is no way to submit a POST form to an IIS 5.0 server.  This just came up
on the MZ forums - http://forums.mozillazine.org/viewtopic.php?p=996972

Comment 7

[Darin Fisher]

If IIS 5.0 has problems with this request, then it is a major bug in that
application.  I recommend that you bring it up with Microsoft.  Afterall, it is
always possible, given the way TCP/IP works, that a HTTP request may be
arbitrarily split into multiple packets.  No matter what we do here, this
problem will never go away 100%.  Of course, by buffering the request before
sending it, we can probably avoid the problem most of the time.  But, that would
be far from a perfect solution.

Given that this causes problems in the wild, I'll elevate the severity, and try
to get it fixed for Mozilla 1.8.  But, that doesn't help Firefox 1.0 or other
existing version of Mozilla.

The server should be fixed.

Comment 8

[Jeffrey Baker]

I could not give a flying handshake about IIS 5, but on Linux at least you can
make POST more efficient by setting TCP_CORK on the outbound socket.

Comment 9

[Jeffrey Baker]

The crux of the problem seems to be that the headers Content-Length and
Content-Type for an HTTP POST are attached to the request body, instead of the
request headers.  The headers and the body are attached in that order to a
multiplex input stream, and then read in order.  The channel emits a packet for
each element in the multiplex stream, which is almost certainly another bug by
itself.

The fix may be a subtle tweak in nsMultiplexInputStream.cpp.  Otherwise
something intrusive may need to be done in netwerk/protocol/http.

Comment 10

[Darin Fisher]

It's not a bug that the request is split into multiple TCP segments.  That can
happen no matter what.  At most this is a performance problem or an opportunity
to improve compatability with broken web servers.

Comment 11

[Michael Newton]

(In reply to comment #9)
> The crux of the problem seems to be that the headers Content-Length and
> Content-Type for an HTTP POST are attached to the request body, instead of the
> request headers.

Actually, if you have a look at attachment 136281 [details] in Ethereal, you'll see that
the Content-Length and Content-Type are sent as headers, they just get dropped
into the second packet.  I expect this is what's causing IIS problems.

I agree that this is more a bug in IIS than Gecko, but just as we support
rendering bugs found in other browsers, we should try to work around bugs in
servers.  Especially when all that's needed is to stop the splitting of one packet.

(Disclaimer: I don't know C++ and understand that 'all that's needed' could mean
9 hours of work!)

Comment 12

[Jeffrey Baker]

If you don't understand C++ then it's not going to help you much.  But if you look in netwerk/protocol/
http you will see what I meant about the headers.  There's two datastructures, one is a list of headers, 
and the other is a stream for the request body.  The headers Content-Type and -Length are not in the 
headers data structure where you would expect, they are prepended to the body stream.

There's a boolean in the request object that indicates the presence of these extra headers.

Comment 13

[Michael Newton]

Yes I saw that earlier, for anyone else who might be watching this, I think it's
at 
http://lxr.mozilla.org/seamonkey/source/netwerk/protocol/http/src/nsHttpChannel.cpp#3373
[I do program PHP and Perl, so I can make /some/ sense of this stuff :)]  If
only someone else saw it, who could fix it!  However, it doesn't look like there
are resources available at MoFo right now for this little bug.

Comment 14

[Darin Fisher]

*** Bug 325327 has been marked as a duplicate of this bug. ***

Comment 15

[Darin Fisher]

Attachment: v1 patch (causes regressions)

Long overdue patch.  This is not on the critical performance path, so there's not much reason to worry about the extra buffer copies.  Indeed, this seems well worth it considering the number of times this bug has come up.

Comment 16

[Jeffrey Baker]

Looks good from here.  This seems worth committing now that <a ping> will be firing of POST requests constantly.

Comment 17

[Darin Fisher]

fixed-on-trunk

Comment 18

[Darin Fisher]

fixed1.8.1

Comment 19

[Daniel Veditz [:dveditz]]

Comment on attachment 210652 [details] [diff] [review]
v1 patch (causes regressions)

approved for 1.8.0 branch, a=dveditz for drivers

Comment 20

[Darin Fisher]

fixed1.8.0.2

Comment 21

[Dave Liebreich [:davel]]

to verify: look at network trace of small POST requests - should be one packet, not two.

Comment 22

[:Gavin Sharp [email: [e-mail]]]

Could this have caused bug 329460?

Comment 23

[Darin Fisher]

This patch was reverted in an attempt to fix bug 334142.

Comment 24

[Boris Zbarsky [:bzbarsky]]

This is actually causing people issues with XMLHttpRequest performance, since it doubles the time each call takes, more or less.

The problem is that we'd need to figure out exactly what was going on with bug 334142... :(

Comment 26

[Benjamin Smedberg]

Not going to block, but we'd like to get this fixed.

Comment 28

[Boris Zbarsky [:bzbarsky]]

Once bug 378629 and bug 383976 land, it might be worth retrying this...  Probably trunk-only and then ask reporter of bug 334142 to retest.

Comment 29

[Darin Fisher]

-> reassign to default owner

Comment 30

[Boris Zbarsky [:bzbarsky]]

Attachment: Merged to tip

Per discussion with biesi, we'll try this again and hope the NSS changes in the interim will prevent bug 334142.

Comment 31

[Boris Zbarsky [:bzbarsky]]

Comment on attachment 277957 [details] [diff] [review]
Merged to tip

a=me

Comment 32

[Boris Zbarsky [:bzbarsky]]

Re-checked in.

Comment 33

[Michal Majchrowicz]

The problem still occurs. I checked recent nightly. The only difference is that the packet has to be long enough for instance if site has 800 bytes of post data firefox will still split it into two parts without any noticable pattern :( It (in example) doesn't allow to configure Micronet switches on firefox.

Comment 34

[Boris Zbarsky [:bzbarsky]]

Michal, do you have a testcase showing the problem?

In general, we limit our packet size to 4KB.  If the headers are very big (e.g. lots of cookies), I can see the first packet not having space for all the data.

Also in general, if you have more data, it'll get split into more packets.

What this bug was about was us sending small packets.  At this point all the packets except the last one should be 4KB.  If that's not happening, I'd like to know.

Comment 35

[Michal Majchrowicz]

Well it's trange thing. It happens if packet is longer than 730 bytes (whole packet not only post data). I noticed one strange thing current trunk works with my switch (and sends correct packets) on windoze but not on linux. This would explain why I wasn't able to access router with ies4linux. So this is something general that affects all "linux" browser but the question remains whether it can be fixed in any way and if then how ? :(

Comment 36

[Boris Zbarsky [:bzbarsky]]

It might be that your OS further breaks up TCP packets (e.g your TCP window is small or some such).  Worth checking with someone familiar with the Linux networking stack.

Comment 37

[Nelson Bolyard (seldom reads bugmail)]

Boris, this may be getting OT here, but why do we still have a 4KB limit
in 2007?  4KB may have been reasonable in the era of 28kb modems and CPUs 
with 16MB of RAM, but it's so small that it triggers TCP Nagle delays, 
because it's much smaller than typical TCP window sizes, and doesn't keep 
the output queue from going empty in one round trip time on today's faster 
networks.  Could we raise it to, say, 32KB or 64KB?

Comment 38

[Boris Zbarsky [:bzbarsky]]

I have no idea why this code is reading things in 4KB chunks.  You'd have to ask Darin.  Necko seems to pretty consistently use 4KB buffers for all sorts of things.

That's a discussion for a separate bug, though.  And that discussion will need some hard data on how different packet sizes affect users.

Comment 39

[Michal Majchrowicz]

The problem is that the packet doesn't have even 1KB :( I don't know why it happens this way. Does anyone know any way of controling it ?

Comment 40

[Michal Majchrowicz]

I have created a simple test case that shows the problem
http://xfree.republika.pl/test.html
Can someone with linux and wireshark installed confirm the existance of the bug?

Comment 41

[Jeffrey Baker]

I tried your test case with Firefox 2.0.0.8, where the POST is sent as two packets, and with a Firefox trunk nightly, where the POST is sent as a single packet.  I suggest that any problems you are seeing are related to your specific computer or network.

Tested with Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9a9pre) Gecko/2007102904 Minefield/3.0a9pre

I recommend leaving this RESOLVED/FIXED

Comment 42

[Michal Majchrowicz]

Did you check the packet size of with nightly? I noticed that wireshark not always correctly displays packets. Moreover I confirmed this bug on few machines not only mine.

Comment 43

[Jeffrey Baker]

Yes, I confirmed with wireshark, tshark, and plain tcpdump.  In your test, on the trunk nightly, there are 1514 bytes in the first frame and 841 in the second.  That is the expected result.

Comment 44

[Michal Majchrowicz]

So why do you thing the bug is fixed? And why do you say it is expected result. You first say it was one single packet and then that there were two frames (one 1514 and second 841). Why not one single frame? That is what happens when you run nightly on windoze.

Comment 45

[Jeffrey Baker]

I don't think we should continue this conversation on this bug, as many people are receiving unnecessary bug spam.  1514 bytes is the expected frame size of ethernet.  No larger packets are expected normally, unless jumbo frames are in use.