1118700 - Firebug + firefox net console dont reliably report response headers

Status: RESOLVED INCOMPLETE

(DevTools :: Netmonitor, defect)

Description

[c3172354]

User Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:34.0) Gecko/20100101 Firefox/34.0
Build ID: 20141125180439

Steps to reproduce:

I spent a large amount of time thinking that my tomcat 7 setups weren't compressing things correctly

http://stackoverflow.com/questions/27816958/problems-getting-tomcat-7-to-use-gzip-compression

I was trying to check that gzip compression was in effect on our servers using the different firefox tools available (the normal net console, firebug's net console and yslow).  I had firebug installed just because yslow requires it, but both consoles were failing in a similar way.


Actual results:

It appeared that gzip compression was not working when in fact it was.  This was also intermittent and led to me spending somewhere in the region of 2 days of googling about tomcat gzip compression, diffing and changing config options, and generally being baffled about what could be causing what I was seeing.

When looking at a compressed response firefox was intermittently misreporting what was going on and showing a type of bytes and a content length with a number, rather than 'chunked' and type gzip.

Some of the time it was reporting the right thing which really confused things more as it wasn't giving me a clear picture of the effect (if any) that the changes I was making were having to the state of the compression.

After going round in circles I accidentally used chrome on a computer which didn't have firefox which correctly reported that compression was indeed switched on in the place where it didn't seem to be working at all according to firefox (see the stack overflow post for more details).  After seeing that I then began checking what was going on with curl and wireshark.

Curl was consistently reporting compression as being on and then wireshark confirmed that compression was indeed on for a browser request that firefox reported it as not being.

I could see this in two ways - the headers shown in wireshark were different (showing gzip and chunked as you'd expect) and because you could look at the content and see that it was compressed as it looked like gibberish.

In firefox you can't see the raw content (which is fine) but the headers that were reported were incorrect - they did not say gzip and specified an exact length (which is what you would see if the file was not compressed).  


Expected results:

Firefox network console should report the response headers from the server correctly.  It clearly doesn't do this as I observed from the discrepancy between the sniffed packets in wireshark and what was reported by the developer tools.

I dont know what is going on there - possibly those values have been calculated after decompression in an effort to be more helpful.  If so it is definitely not more helpful!!  Please report exactly the headers that the server sends back!

Comment 1

[c3172354]

If you want to contact me further about this then please post on the stackoverflow answer I've linked to.  I signed up for this with a temporary email address as I dont want my email address to be public + receive a load of spam so wont receive any email notifications or whatever you might normally get

Comment 2

[:Gijs (he/him)]

Victor, can you suggest what to do here to figure out what's going on?

Comment 3

[Victor Porof [:vporof][:vp]]

Should probably be fixed by bug 731318. Check out Firefox Nightly.

Comment 5

[c3172354]

Attachment: firefox headers.PNG

Here is what the firefox debugger shows...

Comment 6

[c3172354]

Attachment: firefox raw headers view.PNG

Here is what the raw headers view shows (I believe this is the new bit?)...

Comment 7

[c3172354]

Here is the request/response as observed by wireshark

GET /Veolia/concat.js?r=20150106-1145 HTTP/1.1
Host: trac-leeds-11
User-Agent: Mozilla/5.0 (Windows NT 6.3; Win64; x64; rv:37.0) Gecko/20100101 Firefox/37.0
Accept: */*
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
DNT: 1
Referer: http://trac-leeds-11/Veolia/
Cookie: JSESSIONID=68C458112C46D64A0DF13A27FEBFC564; ext-mainEventsGridState=o%3Acolumns%3Da%253Ao%25253Aid%25253Ds%2525253Ah1%255Eo%25253Aid%25253Ds%2525253Ah2%255Eo%25253Aid%25253Ds%2525253Ah3%255Eo%25253Aid%25253Ds%2525253Ah4%255Eo%25253Aid%25253Ds%2525253Ah5%255Eo%25253Aid%25253Ds%2525253Ah6%255Eo%25253Aid%25253Ds%2525253Ah7%255Eo%25253Aid%25253Ds%2525253Ah8%255Eo%25253Aid%25253Ds%2525253Ah9%255Eo%25253Aid%25253Ds%2525253Ah10%255Eo%25253Aid%25253Ds%2525253Ah11%255Eo%25253Aid%25253Ds%2525253Ah12%255Eo%25253Aid%25253Ds%2525253Ah13%255Eo%25253Aid%25253Ds%2525253Ah14%255Eo%25253Aid%25253Ds%2525253Ah15%255Eo%25253Aid%25253Ds%2525253Ah16%255Eo%25253Aid%25253Ds%2525253Ah17%255Eo%25253Aid%25253Ds%2525253Ah18%255Eo%25253Aid%25253Ds%2525253Ah19
Connection: keep-alive
Pragma: no-cache
Cache-Control: no-cache

HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Accept-Ranges: bytes
ETag: W/"29625-1420633264500"
Last-Modified: Wed, 07 Jan 2015 12:21:04 GMT
Content-Type: application/javascript;charset=UTF-8
Transfer-Encoding: chunked
Content-Encoding: gzip
Vary: Accept-Encoding
Date: Thu, 08 Jan 2015 12:17:58 GMT

a
..........
200
.}.w....W.yw].Fd....s..>..7n.8..t.\..X.$6....m]Y.}g. ......g..IE....0..?.?..


As you can see it is actually compressed and firefox is still getting the headers completely wrong.

Comment 8

[c3172354]

The last 3 comments are having just tried it on the nightly build 37.0a1 (2015-01-07).

It's still not working so I've added a couple of screen shots of what firefox says along with what wireshark says.

I guess the new bit is the raw headers?  I've not noticed that before anyway, either way its still getting it wrong.  

I don't know the code at all but surely nothing should be messing around with the values in here - it should just be a matter of reading them off the wire and then just displaying them straight. It certainly seems like this isn't happening and that they are tampered with on the way through.

Comment 9

[J. Ryan Stinnett [:jryans] (Use needinfo, replies may be slow)]

Are you trying Firefox Nightly in e10s mode?  Please try disabling e10s for now (in the Preferences), as not all parts of Net Monitor work correctly in e10s.  See bug 982319 for more info on this.

The part that is new is the "Transferred" column, which correctly matches in size on the wire.  In non-e10s mode, the Transferred and Size columns give different values, where Transferred is size on wire, and Size is the uncompressed size.

However, I see what you mean that some headers have shifted:

* Net Monitor is missing Transfer-Encoding
* Net Monitor is missing Content-Encoding
* Net Monitor added Content-Length

Can you re-test in Nightly with e10s disabled to see if you get different results?

Comment 10

[c3172354]

Hi,

I've just checked with it definitely not in e10s mode and see the same thing.

I dont think I was in e10s mode before anyway - I dont remember clicking to enable it.

Cheers

Comment 11

[J. Ryan Stinnett [:jryans] (Use needinfo, replies may be slow)]

Okay, I am not able to reproduce this behavior on just any site using these headers, so something must be specific to your site.

Is there a public URL where I can test this same request?  Or if not, could you attach a Wireshark / Fiddler / something capture that can easily be replayed?

Comment 12

[c3172354]

Hi, Sorry didn't realise there had been more activity here.  So there's no public url + I'm not on the project anymore.
I'm not sure what else I can do to help.  Thanks for taking an interest though!