1419401 - HTTP status code is missing in raw response headers display

Status: VERIFIED FIXED

(DevTools :: Netmonitor, defect, P3)

Description

[Maarten Bruins]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.36

Steps to reproduce:

See: https://stackoverflow.com/questions/47412949/where-is-the-logic-behind-mozilla-network-monitor-in-firefox-developer-tools-in

Summery see bold part in that post: "1. Firefox forgot about the "Status code" in the raw response headers."


Actual results:

Missing status code in raw headers. In Developer Tools -> Network tab.


Expected results:

Status code must be also part of the raw headers shown.

Comment 1

[Maarten Bruins]

They banned me on Stack Overflow ;). I posted a link to Stack Overflow to report the bug (which was not that clear anyway), so maybe they will remove that link, so i'll post the bug one more time (a bit more clear) here.

In Firefox you can check the "raw headers" via "Firefox Developer Tools" > "Network".

Example of the raw response headers:

    Date: Thu, 23 Nov 2017 12:43:21 GMT
    Server: Apache/2.4.17 (Unix) OpenSSL/1.0.1e-fips PHP/5.6.16
    Connection: Keep-Alive
    Keep-Alive: timeout=1, max=100
    Cache-Control: max-age=9, public
    Vary: User-Agent

But I miss (for example): "HTTP/1.1 304 Not Modified". Firefox shows somewhere else "304 Not Modified", but not RAW.

So they let me think I have the response headers in raw form, but actually it's only a part of the response headers, excluding the status code. This can be really confusing for people.
In my opinion it would make much more sense (and more correct) to add the "status code" too at that place. Now it's not really logical.

Comment 2

[[e-mail]]

Maarten, thanks for the report.
Do you use recent nightly? or is this issue also happens in other version?

Comment 3

[Maarten Bruins]

I'm not using nightly. I've Firefox on my computer just like how non developers would have it (and up-to-date). So I only have 1 version installed. If you meant something different, let me know.

Comment 4

[[e-mail]]

I saw even for 200 OK, the raw header does not contain the status code as well.
And you can easily see the status code above the raw header.

From code perspective, raw responseHeaders.headers does not contain the status code
https://searchfox.org/mozilla-central/source/devtools/client/netmonitor/src/components/HeadersPanel.js#252

Honza, do you think we need to include the status code as part of the raw headers?

Comment 5

[Jan Honza Odvarko [:Honza] (always need-info? me)]

(In reply to Fred Lin [:gasolin] from comment #4)
> Honza, do you think we need to include the status code as part of the raw
> headers?
We discussed this at our meeting today and I agree let's include the
status code as part of the raw headers.

Honza

Comment 6

[Maarten Bruins]

Good to hear, thanks! When Firefox gives you the amount of bytes of the Response headers, the status code is also included in the calculation. So that would also be a reason to include it. At least keep it consistent. So or in both cases no status code included, or in both cases the status code included. The last option makes more sense in my opinion.

Comment 7

[[e-mail]]

I made more research on this and found wikipedia said `Normal HTTP responses will return a separate `Status-Line` instead of include "status" in responseHeader`

```
status: CGI header field specifying the status of the HTTP response. Normal HTTP responses use a separate "Status-Line" instead, defined by RFC 7230```

https://en.wikipedia.org/wiki/List_of_HTTP_header_fields#Common_non-standard_response_fields

I test with 3 tools to get response headers and none of them include status code field in response header.

1. Curl

$ curl -i https://ichef.bbci.co.uk/news/490/cpsprodpb/16C41/productpmay.jpg94239_trumpmay.jpg
HTTP/1.1 200 OK
Server: AmazonS3
x-amz-id-2: iccQhvuNw9IWXwLUE2rBdCJNf8blBOrKwC6a4H1zQcXmg8U4dmQG5EEJuqsULb6drsnsjXmE60k=
x-amz-request-id: 38EA84F7BF9A443C
Last-Modified: Thu, 30 Nov 2017 01:19:08 GMT
ETag: "f3318344c7d44775f5905f8687616610"
Cache-Control: max-age=31536000, max-age=31536000
x-amz-version-id: v8C5TWjnqbB2X09vWKe51ryt2jMgi40c
Accept-Ranges: bytes
Content-Type: image/jpeg
Content-Length: 17179
Expires: Fri, 30 Nov 2018 01:23:23 GMT
Date: Thu, 30 Nov 2017 06:51:39 GMT
Connection: keep-alive


2. wget

$ wget -S https://ichef.bbci.co.uk/news/490/cpsprodpb/16C41/production/_98994239_trumpmay.jpg

  HTTP/1.1 200 OK
  Server: AmazonS3
  x-amz-id-2: iccQhvuNw9IWXwLUE2rBdCJNf8blBOrKwC6a4H1zQcXmg8U4dmQG5EEJuqsULb6drsnsjXmE60k=
  x-amz-request-id: 38EA84F7BF9A443C
  Last-Modified: Thu, 30 Nov 2017 01:19:08 GMT
  ETag: "f3318344c7d44775f5905f8687616610"
  Cache-Control: max-age=31536000, max-age=31536000
  x-amz-version-id: v8C5TWjnqbB2X09vWKe51ryt2jMgi40c
  Accept-Ranges: bytes
  Content-Type: image/jpeg
  Content-Length: 17179
  Expires: Fri, 30 Nov 2018 01:23:23 GMT
  Date: Thu, 30 Nov 2017 06:52:50 GMT
  Connection: keep-alive

3 httpie

$ http https://ichef.bbci.co.uk/news/490/cpsprodpb/16C41/production/_98994239_trumpmay.jpg --headers
HTTP/1.1 200 OK
Accept-Ranges: bytes
Cache-Control: max-age=31536000, max-age=31536000
Connection: keep-alive
Content-Length: 17179
Content-Type: image/jpeg
Date: Thu, 30 Nov 2017 06:56:59 GMT
ETag: "f3318344c7d44775f5905f8687616610"
Expires: Fri, 30 Nov 2018 01:23:23 GMT
Last-Modified: Thu, 30 Nov 2017 01:19:08 GMT
Server: AmazonS3
x-amz-id-2: iccQhvuNw9IWXwLUE2rBdCJNf8blBOrKwC6a4H1zQcXmg8U4dmQG5EEJuqsULb6drsnsjXmE60k=
x-amz-request-id: 38EA84F7BF9A443C
x-amz-version-id: v8C5TWjnqbB2X09vWKe51ryt2jMgi40c


So I think we already did the right implementation and no need to change.

Comment 8

[[e-mail]]

(In reply to Maarten Bruins from comment #6)
> Good to hear, thanks! When Firefox gives you the amount of bytes of the
> Response headers, the status code is also included in the calculation. So
> that would also be a reason to include it. At least keep it consistent. So
> or in both cases no status code included, or in both cases the status code
> included. The last option makes more sense in my opinion.

Maarten, could you share how you calc the amount of bytes of the Response headers?

Comment 9

[Sebastian Zartner [:sebo]]

(In reply to Fred Lin [:gasolin] from comment #7)
> I test with 3 tools to get response headers and none of them include status
> code field in response header.
> 
> 1. Curl
> 
> $ curl -i
> https://ichef.bbci.co.uk/news/490/cpsprodpb/16C41/productpmay.
> jpg94239_trumpmay.jpg
> HTTP/1.1 200 OK

See ↑. That's what Maarten is talking about. So, obviously all three tools you tested output the HTTP status code as part of the headers.

> I made more research on this and found wikipedia said `Normal HTTP responses
> will return a separate `Status-Line` instead of include "status" in
> responseHeader`
> 
> ```
> status: CGI header field specifying the status of the HTTP response. Normal
> HTTP responses use a separate "Status-Line" instead, defined by RFC 7230```
> 
> https://en.wikipedia.org/wiki/List_of_HTTP_header_fields#Common_non-
> standard_response_fields

The headline there says "Common non-standard response fields". So, that's not what is asked for in this request.
It's about including "HTTP/1.1 200 OK", "HTTP/1.1 304 Not Modified", etc. in the raw headers section.

Sebastian

Comment 10

[[e-mail]]

> > HTTP/1.1 200 OK
> 
> See ↑. That's what Maarten is talking about. So, obviously all three tools
> you tested output the HTTP status code as part of the headers.

It's a different format from the rest of headers like `Accept-Ranges: bytes`, So I guess its what wikipedia said `the separate Status-Line`

> 
> The headline there says "Common non-standard response fields". So, that's
> not what is asked for in this request.
> It's about including "HTTP/1.1 200 OK", "HTTP/1.1 304 Not Modified", etc. in
> the raw headers section.
> 

Chrome adds `status: 200` in response Headers. We could do the same if it looks more friendly.

Comment 11

[Sebastian Zartner [:sebo]]

Attachment: HTTP status code in Chrome DevTools

(In reply to Fred Lin [:gasolin] from comment #10)
> > > HTTP/1.1 200 OK
> > 
> > See ↑. That's what Maarten is talking about. So, obviously all three tools
> > you tested output the HTTP status code as part of the headers.
> 
> It's a different format from the rest of headers like `Accept-Ranges:
> bytes`

Right.

> So I guess its what wikipedia said `the separate Status-Line`

Ok, maybe I misunderstood the Wikipedia page. What's meant is obviously https://tools.ietf.org/html/rfc7230#section-3.1.2.

> > The headline there says "Common non-standard response fields". So, that's
> > not what is asked for in this request.
> > It's about including "HTTP/1.1 200 OK", "HTTP/1.1 304 Not Modified", etc. in
> > the raw headers section.
> > 
> 
> Chrome adds `status: 200` in response Headers. We could do the same if it
> looks more friendly.

Chrome shows `HTTP/1.1 200 OK` as expected. See the attached screenshot. Strangly, the option to show the raw headers is not always there in the Chrome DevTools. No idea why.
Anyway, the Firefox DevTools should show the status code as it was received.

Sebastian

Comment 12

[Maarten Bruins]

Sebastian actually already said exactly what I meant, thanks a lot! 

And Fred Lin [:gasolin], you were asking: "Maarten, could you share how you calc the amount of bytes of the Response headers?"

It's not that easy, but I'll share with you.

Calculating and checking the bytes of the response headers was pretty difficult for me, because I only wanted to trust the test results and not theory or source code or whatever.

So I did this:

TEST 1:

RESPONSE HEADERS (325 B)

HTTP/1.1 200 OK
Date: Fri, 01 Dec 2017 10:49:33 GMT
Server: Apache/2.4.17 (Unix) OpenSSL/1.0.1e-fips PHP/5.6.16
Last-Modified: Fri, 01 Dec 2017 10:36:49 GMT
Accept-Ranges: bytes
Content-Length: 143
Cache-Control: max-age=99999, public
Keep-Alive: timeout=1, max=100
Connection: Keep-Alive
Content-Type: text/html

So Firefox shows 325 bytes for the Response headers. 

Now i did exactly the same test, but without the line "Cache-Control: max-age=99999, public".

RESPONSE HEADERS (287 B)

HTTP/1.1 200 OK
Date: Fri, 01 Dec 2017 10:50:12 GMT
Server: Apache/2.4.17 (Unix) OpenSSL/1.0.1e-fips PHP/5.6.16
Last-Modified: Fri, 01 Dec 2017 10:36:49 GMT
Accept-Ranges: bytes
Content-Length: 143
Keep-Alive: timeout=1, max=100
Connection: Keep-Alive
Content-Type: text/html

So the difference between those 2 tests are: 325 - 287 = 38 bytes

The difference is actually: "Cache-Control: max-age=99999, public". I copy / pasted that line in a tool like this:

https://lingojam.com/ByteCounter

That will give 36 bytes. Probably there are 2 bytes at the end of it (enter). So that makes indeed 36 + 2 = 38 bytes.

So thinking the same way as above, I concluded:

HTTP/1.1 200 OK			-> will give 17 (15 + 2) extra bytes in the Response headers
HTTP/1.1 304 Not Modified	-> will give 27 (25 + 2) extra bytes in the Response headers

So the difference is: 10 bytes

Now I make the same request, but with a "304 Not Modified" response.

RESPONSE HEADERS (183 B)

HTTP/1.1 304 Not Modified
Date: Fri, 01 Dec 2017 11:07:45 GMT
Server: Apache/2.4.17 (Unix) OpenSSL/1.0.1e-fips PHP/5.6.16
Connection: Keep-Alive
Keep-Alive: timeout=1, max=100

So the difference with the previous test is:

RESPONSE HEADERS (287 B)
RESPONSE HEADERS (183 B)
------------------------ -
104 bytes

The difference in lines (without looking at the status code) is:

Last-Modified: Fri, 01 Dec 2017 10:36:49 GMT
Accept-Ranges: bytes
Content-Length: 143
Content-Type: text/html

So this is:
(44 + 2) + (20 + 2) + (19 + 2) + (23 + 2) = 114 bytes

But the difference Firefox is showing, is 104 bytes! So there is a difference of 10 bytes. That's exactly the 10 bytes difference between:

"HTTP/1.1 304 Not Modified" and "HTTP/1.1 200 OK"

That means Firefox is also looking at the status code in the Reponse headers, when calculating the amount of bytes of the Response headers.

So it would be really strange if Firefox would not show the status code in the raw Response headers, while Firefox is using them in that calculation.

Comment 13

[[e-mail]]

Thanks you Maarten for the through calculation.

The result shows `HTTP/1.1 200 OK` should be part of the response header panel

Comment 14

[[e-mail]]

Attachment: Bug 1419401 - show HTTP/2.0 200 OK in raw header;

Review commit: https://reviewboard.mozilla.org/r/206186/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/206186/

Comment 15

[[e-mail]]

Comment on attachment 8935286 [details]
Bug 1419401 - show HTTP/2.0 200 OK in raw header;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/206186/diff/1-2/

Comment 16

[[e-mail]]

The patch show Status:200 in responseHeader, and show `HTTP/2.0 200 OK` in raw header

Here's how it looks like https://i.imgur.com/jnr1wQz.gif

Comment 17

[Maarten Bruins]

Thanks! You're talking only about 200, but I assume that, that are just examples and you're talking about the status code in general (304 for example)?

Comment 18

[Jan Honza Odvarko [:Honza] (always need-info? me)]

Comment on attachment 8935286 [details]
Bug 1419401 - show HTTP/2.0 200 OK in raw header;

https://reviewboard.mozilla.org/r/206186/#review211798

Thanks Fred for the patch!

Please see my inline comments.

Honza

::: devtools/client/netmonitor/src/components/HeadersPanel.js:107
(Diff revision 2)
>            headers.headers.reduce((acc, { name, value }) =>
>              name ? Object.assign(acc, { [name]: value }) : acc
>            , {})
>        };
>  
> +      // Bug 1419401 display status code in responseHeader tree view

The Status code should be displayed only at the first line of the raw data preview. Not as a parsed header in the tree (since it isn't a header)

Also, the comment should not use bug number.

::: devtools/client/netmonitor/src/components/HeadersPanel.js:265
(Diff revision 2)
>        );
>      }
>  
>      let summaryVersion = httpVersion ?
>        this.renderSummary(SUMMARY_VERSION, httpVersion) : null;
> +    // Bug 1419401 display Status-Line above other response headers

Please remove bug number from the comment.

Comment 19

[[e-mail]]

Comment on attachment 8935286 [details]
Bug 1419401 - show HTTP/2.0 200 OK in raw header;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/206186/diff/2-3/

Comment 20

[Jan Honza Odvarko [:Honza] (always need-info? me)]

Comment on attachment 8935286 [details]
Bug 1419401 - show HTTP/2.0 200 OK in raw header;

https://reviewboard.mozilla.org/r/206186/#review212326

R+ assuming try is green

Thanks Fred!

Honza

Comment 21

[[e-mail]]

try green, thanks!

Comment 22

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/b30476062daf
show HTTP/2.0 200 OK in raw header;r=Honza

Comment 23

[Andrei Ciure[:aciure]]

https://hg.mozilla.org/mozilla-central/rev/b30476062daf

Comment 24

[Sebastian Zartner [:sebo]]

Tested on Nightly 59.0a1 (2017-12-11) and it works as expected for me. Thank you for the patch!

Sebastian

Comment 25

[Maarten Bruins]

Thanks! Now at least the size is correct, if you compare it with the raw data. But actually you also have to change the words: "response headers" to "reponse message". 

See: https://tools.ietf.org/html/rfc7230#page-22 (3.1.2. and 3.2)

So:

REPONSE MESSAGE = STATUS-LINE + RESPONSE HEADERS

Now it's actually still incorrect (but at least the size is correct). But Firefox is not the only browser, which is not consistent in using the term "reponse headers".

Chrome: https://developers.google.com/web/tools/chrome-devtools/network-performance/reference#requests

They are saying:

"Size. The combined size of the response headers plus the response body, as delivered by the server."

I did some tests and "Size" also includes the size of the status-line and they are also saying "response headers" and not "response message", but that's just incorrect, according to the definition. So I have no idea why everyone is using those terms the wrong way. It's not that difficult, right? Just say / write "reponse message" instead of "response headers".

Comment 26

[[e-mail]]

Thanks Maarten for point this out! Created bug 1425036 for the followup string change.

Comment 27

[Maarten Bruins]

Thanks! Maybe I was a bit too fast to post it, because:

https://tools.ietf.org/html/rfc7230#page-8

   A server responds to a client's request by sending one or more HTTP
   <b>response messages</b>, each beginning with a status line that includes
   the protocol version, a success or error code, and textual reason
   phrase (Section 3.1.2), possibly followed by header fields containing
   server information, resource metadata, and representation metadata
   (Section 3.2), an empty line to indicate the end of the header
   section, and finally a message body containing the payload body (if
   any, Section 3.3).

So actually I had to say:

REPONSE MESSAGE = STATUS-LINE + RESPONSE HEADERS + EMPTY LINE + MESSAGE BODY

So anyway it's still wrong how it is now, but I think we have to make a new definition or word for:

STATUS-LINE + RESPONSE HEADERS + EMPTY LINE = SOMETHING

That's what the Size is about it Firefox and Chrome. 

Or they must not say it's the size of the response headers (but size of status-line + response headers + empty line). But then if you're gonna change the name of the "raw response headers" to (status-line + response headers + empty line), then there must also come an empty line at the end.

But anyway I think you guys understand my point. How it is now, it's actually incorrect, so there must come some changes. There are a couple of solutions, so it does not really matter which one. It only has to be correct and consistent with other parts (naming, size, raw content et cetera).

Comment 28

[Maarten Bruins]

You could call it for example:

response message, without message body (or maybe something shorter and 1 word)

But then when showing the "raw response message, without message body", there must be an extra empty line at the end.

Firefox is also showing something like:

Response headers (842 B)

Then that must become something like:

response message without message body (842 B)

Comment 29

[Chris Mills [:cmills]]

I have documented this change:

* I've added a note about the Raw headers button to the Network monitor page: https://developer.mozilla.org/en-US/docs/Tools/Network_Monitor#Headers
* I've added a note to the Fx59 rel notes: https://developer.mozilla.org/en-US/Firefox/Releases/59#Changes_for_web_developers

Let me know if this is OK. Thanks!

Comment 30

[[e-mail]]

looks good, thanks

Comment 31

[Adam]

Was the status code text supposed to be removed in the "Status code:" line for 61.0? https://imgur.com/a/FEGzXtj

Comment 32

[Sebastian Zartner [:sebo]]

(In reply to Adam from comment #31)
> Was the status code text supposed to be removed in the "Status code:" line
> for 61.0? https://imgur.com/a/FEGzXtj

This bug is unrelated to that change. It's bug 1435232 you're looking for.
Also note that you should rather file a new bug when a bug is already closed and shipped. In the new bug you can then refer to the bug where the change got introduced.

Sebastian

Comment 33

[Jolly Smith]

I'm facing this issue while visiting this: https://easynamesgenerator.com/on Mozilla. I've added a note about the Raw headers button to the Network monitor page.

Comment 34

[Sebastian Zartner [:sebo]]

This bug is closed for several years already. Please open a new bug report and provide some steps to reproduce your issue. Thank you!

Sebastian