702941 - Disable HTTP pipelining Winstone when talking to Servlet Engine

Status: RESOLVED FIXED

(Core :: Networking: HTTP, defect)

Description

[Luca Tettamanti]

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:8.0) Gecko/20100101 Firefox/8.0 Iceweasel/8.0
Build ID: 20111109081537

Steps to reproduce:

I'm using Firefox 8 and manually enabled HTTP pipelining.
At work we have a self-hosted Jenkins (CI system) and the web frontend does not work properly with Firefox.


Actual results:

The JS debugger will log errors in apparently random locations, further debugging with tcpdump and firebug net panel showed that resources (mostly JS files) were being sent truncated.
After disabling pipelining the issue disappeared.
The HTTP engine is "Winstone Servlet Engine v0.9.10" and unfortunately it insists that it's HTTP/1.1 capable:

HTTP/1.1 200 OK
Server: Winstone Servlet Engine v0.9.10
Last-Modified: Thu, 10 Nov 2011 14:18:13 GMT
Expires: Thu, 15 Nov 2012 13:14:56 GMT
Accept-Ranges: bytes
Content-Type: application/javascript
Content-Encoding: gzip
Connection: Close
Date: Wed, 16 Nov 2011 13:14:56 GMT
X-Powered-By: Servlet/2.5 (Winstone/0.9.10)
Set-Cookie: etc.

I'm attaching a patch against mozilla-central to blacklist Winstone versions 0.*

Comment 1

[Luca Tettamanti]

Attachment: pipelining: blacklist Winstone Servlet Engine version v0.*

Comment 2

[Patrick McManus [:mcmanus]]

Luca, I'll probably want to take this patch - thanks for the contribution!

I need to ask a few questions first:

* some piece of infrastructure clearly has a pipelining problem - did you do any work to ensure that it is the server instead of a load balancer or something like that? It would be bad to blacklist the server if it wasn't its fault.

* It looks like winstone is on sourceforge, though it hasn't been updated since 2008. Can you file a bug there that points to this bugzilla entry? I didn't see any pipeline bugs in its database.

* Do you know of any public url running winstone that reproduces this problem? I would like to analyze the failure mode in detail. If you don't know of such a url, would you be able to form a packet capture of a failed load using wireshark so I could look at that? The idea is to make sure we can make firefox robust to these kinds of failures in other servers too without blacklisting them.

Comment 3

[Robert Hancock]

We have seen the same problem on our Jenkins installation. I can verify that it must be the server itself causing the problem as there is no intervening load balancer or anything like that.

I'm not sure if there is a public URL that would reproduce the problem. I only noticed the problem on the project configuration page, which isn't (or at least shouldn't) be accessible to the public.

Comment 4

[Patrick McManus [:mcmanus]]

I filed the sf bug

https://sourceforge.net/tracker/?func=detail&aid=3438879&group_id=98922&atid=622497

Comment 5

[Luca Tettamanti]

(In reply to Patrick McManus from comment #2)
> Luca, I'll probably want to take this patch - thanks for the contribution!
> 
> I need to ask a few questions first:
> 
> * some piece of infrastructure clearly has a pipelining problem - did you do
> any work to ensure that it is the server instead of a load balancer or
> something like that? It would be bad to blacklist the server if it wasn't
> its fault.

The server is sitting on our LAN, no proxy or balancer in between. I can also reproduce on localhost.

> * It looks like winstone is on sourceforge, though it hasn't been updated
> since 2008. Can you file a bug there that points to this bugzilla entry? I
> didn't see any pipeline bugs in its database.

You beat me to it :) I actually intended to follow this up with Jenkins devs... they probably need to stick with HTTP/1.0 (or fix pipelining).

> * Do you know of any public url running winstone that reproduces this
> problem? I would like to analyze the failure mode in detail. If you don't
> know of such a url, would you be able to form a packet capture of a failed
> load using wireshark so I could look at that? The idea is to make sure we
> can make firefox robust to these kinds of failures in other servers too
> without blacklisting them.

You can actually try it locally, not setup required. Download this war:
http://mirrors.jenkins-ci.org/war/latest/jenkins.war

And run it standalone:
java -jar jenkins.war --httpPort=8181

(to cleanup just rf -rf ~/.jenkins)

As Robert pointed out the problem is more noticeable in the configuration pages (e.g. Manage Jenkins -> Configure System) since JS errors will leave those pages unusable, but the truncation happens even in the homepage (again, looking for JS errors you'll see that the corresponding file is incomplete)

If you still want a packet dump I'll capture it tomorrow.

Following an intuition I prevented Firefox from accepting gzip encoded resources, and the files were sent correctly... so maybe there's just a flush of the compressed stream missing (or the different timing is enough to mask the issue).

Comment 6

[Rick Knowles]

Hi,

Original winstone developer here.

You're right - this is an unfortunate leftover from 8 years ago when I misunderstood the HTTP/1.1 spec regarding pipelining conditions. 

Winstone is expecting a content-length to be set on the servlet response in order for it to be considered acceptable to pipeline. Winstone will actively force a Connection:close header on the response if there is no content-length set on the response by the servlet. Probably not ideal, but it was the best I could come up with at the time.

(to be honest I'm not sure what the correct servlet container behaviour should be here, but I can see everyone else is able to handle this case so I'm obviously missing something).

Hudson/jenkins have forked from the primary winstone tree (which went into hibernation around 2008 IIRC), so it would be up to them to fix in any case.

Easiest workaround I can think of is to set content length headers on the hudson/jenkins responses ... or alternatively change to tomcat/jetty which I believe is being discussed. Given that Winstone is very long in the tooth, I'm quite okay with it being retired if they choose it to be.

Thanks,

Rick

Comment 7

[Robert Hancock]

This issue was reported in the Jenkins bug tracker here:

https://issues.jenkins-ci.org/browse/JENKINS-11163

Comment 8

[Patrick McManus [:mcmanus]]

> You can actually try it locally, not setup required. Download this war:
> http://mirrors.jenkins-ci.org/war/latest/jenkins.war

Luca, thanks for this - it will help firefox generally!

(In reply to Rick Knowles from comment #6)

> Original winstone developer here.
> 
> You're right - this is an unfortunate leftover 

Rick, thank you too. Your involvement here makes this a lot easier to apply.

Comment 9

[Patrick McManus [:mcmanus]]

Attachment: pcap of a pipelined session showing failure mode

One interesting flow in here is on port 38180.

That port has 5 pipelined requests, the 2nd is for codemirror.js.

There are 2 responses for the first 2 requests (correct behavior). The second is labeled connection close, has no content-type, and is gzip'd. EOF follows.

requests for the aborted 3 requests do appear as successful retries on other connections.

The problem is if you ungzip codemirror.js you'll see it is truncated.

Comment 10

[Patrick McManus [:mcmanus]]

(In reply to Rick Knowles from comment #6)
>
> 
> Winstone is expecting a content-length to be set on the servlet response in
> order for it to be considered acceptable to pipeline. Winstone will actively
> force a Connection:close header on the response if there is no
> content-length set on the response by the servlet. Probably not ideal, but
> it was the best I could come up with at the time.
> 

I wanted to investigate this further, because the way it is described here it should be bad for pipelining performance but it shouldn't actually break anything - the unanswered members of the pipeline ought to be retried.

And basically that was what was happening.. but when you inserted the close a RST is generated on your end because you closed the pipe without consuming the whole request stream.. that rst resulted in aborting the connection and truncating one of the js items. Since that js was only framed with EOF (always a bad idea) it was used as-is and generated the js errors.

The future pipeline code will dynamically detect that and auto-blacklist the host.

The patch is working its way through our test service now.. if it passes I will push the change. Thanks all!

Comment 11

[Patrick McManus [:mcmanus]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/4d6ac4042f25

Comment 12

[Ed Morley [:emorley]]

https://hg.mozilla.org/mozilla-central/rev/4d6ac4042f25