833774 - The displayed HTTPS response status / remote address comes from proxy, not from web server

Status: ASSIGNED

(DevTools :: Netmonitor, defect, P2)

Description

[Frederik Braun [:freddy]]

I am POSTing to an HTTPS website using XHR and the Web Console shows me "200 Connection Esablished", which is my proxy's response, not the response from the web server.
The web server has in fact responded with a 403 Forbidden.

Comment 1

[Frederik Braun [:freddy]]

Also, clicking on the XHR gives me the typical request/response popup, but it has only a request in it. The response is not shown. This is probably fixed in one, since you confuse the proxy response with the HTTP response.

Comment 2

[Rob Campbell [:rc] (:robcee)]

Do you have "Enable Request and Response Bodies" checked in the Network filter panel?

Comment 3

[Frederik Braun [:freddy]]

Nope

Comment 4

[Mihai Sucan [:msucan]]

Thanks for the report!

Can you please provide a STR? I do not have an available proxy to test with...

Comment 5

[Frederik Braun [:freddy]]

Sorry, what is an STR? :)

Comment 6

[J. Ryan Stinnett [:jryans] (Use needinfo, replies may be slow)]

(In reply to Frederik Braun [:freddyb] from comment #5)
> Sorry, what is an STR? :)

Steps to Reproduce. For this bug, as part of the exact steps, it would be good to have the particular site and proxy used (assuming they are publicly accessible).

Comment 7

[Frederik Braun [:freddy]]

I am running "ZAP Proxy", a proxy to man-in-the-middle HTTP(S) requests for security testing (https://www.owasp.org/index.php/ZAP). Any other proxy would do as well. The fact that it intercepts SSL requests and sends you a wrong certificate (to do the interception) is not part of this bug.

Steps to Reproduce:
1) Download and run ZAP (listens on port 8080 by default, I think)
2) Set Firefox to use localhost:8080 as a proxy, tick "use this for all protocols".
3) Open the Web Console
4) Visit an HTTPS site, like https://mozilla.org
5) ZAP will intercept this connection and you will get a certificate warning. This is expected and not really part of the bug.
6) The HTTP response is shown is "GET https://www.mozilla.org/en-US/ [HTTP/1.1 200 Connection established 1014ms]"
7) It should show the server response, which might be HTTP/1.1 200 OK or HTTP/1.1 404 Not Found ... and not the proxy response

What really happens:
1) Visiting an HTTPS site means that the browser sends a CONNECT request (HTTP connections do a "GET http://site ..." which is fully transparent to the browser, but HTTPS connections do a CONNECT request, because it wants to talk to the server directly for mutual SSL authentication).
4) The proxy server responds with a "HTTP/1.1 200 Connection established"
5) The Web Console is happy and shows this response
6) Now the browser does its actual request to the http server and starts the SSL handshake.
7) After that the real request (e.g. GET / HTTP/1.1\r\nHost: mozilla.org...) is sent

Comment 8

[Mihai Sucan [:msucan]]

Thank you for the STR.

I tested and I'm not certain this is a bug.

For non-HTTPS sites the web console shows the network information correctly. The proxy doesn't affect network logging in that case.

For HTTPS you get the certificate error and the page doesn't load, so there's not actual HTTP response to show. You have a transport/connection error - the SSL certificate is invalid/not accepted.

If you properly configure ZAP to use a valid certificate, can you visit HTTPS web sites? Is network logging working as expected in the web console after you do that?

I'm not sure what we can do here: we could probably show the SSL error code: ssl_error_rx_record_too_long (this is the error I got). Beyond that, is there anything else we can show? Certainly we cannot show the server response headers, 200 OK or anything like that - afaik we do not get it.

Comment 9

[Frederik Braun [:freddy]]

(In reply to Mihai Sucan [:msucan] from comment #8)
> Thank you for the STR.
> 
> I tested and I'm not certain this is a bug.
> 
> For non-HTTPS sites the web console shows the network information correctly.
> The proxy doesn't affect network logging in that case.

This is because proxying for HTTP works differently than with HTTPS.

> 
> For HTTPS you get the certificate error and the page doesn't load, so
> there's not actual HTTP response to show. You have a transport/connection
> error - the SSL certificate is invalid/not accepted.

I think my STR were a bit misworded.

My step 5 (ZAP will intercept this connection and you will get a certificate warning. This is expected and not really part of the bug.) should have included that you have to *click through* the warning. Please visit the website despite the certificate error (it's not part of the bug, it's because you use ZAP - any other proxy would not produce this error).

> 
> If you properly configure ZAP to use a valid certificate, can you visit
> HTTPS web sites? Is network logging working as expected in the web console
> after you do that?

No. It says "200 Connection established" (the response to the browser's HTTP CONNECT request, not the response to the following HTTP GET request).

I can highly recommend using Wireshark to see what actual traffic goes through the wire. If your proxy runs on localhost, you will have to pick your loopback interface (lo), not your ehternet device (eth0, wlan0)

> 
> I'm not sure what we can do here: we could probably show the SSL error code:
> ssl_error_rx_record_too_long (this is the error I got). Beyond that, is
> there anything else we can show? Certainly we cannot show the server
> response headers, 200 OK or anything like that - afaik we do not get it.

Please disregard the ssl error for this. It has nothing to do with the bug.

Comment 10

[Mihai Sucan [:msucan]]

Issue confirmed. We get the http status code and text wrongly. The actual request and response headers are determined correctly.

Thanks for the bug report. Moving to the netmonitor component for better tracking.

Comment 11

[Evgeny Kapun]

Are there any plans to fix this bug? Or is it not old enough?

Comment 12

[J. Ryan Stinnett [:jryans] (Use needinfo, replies may be slow)]

@ericlaw mentioned this issue on Twitter recently:

https://twitter.com/ericlaw/status/662243516265840640

Comment 13

[J. Ryan Stinnett [:jryans] (Use needinfo, replies may be slow)]

From @ericlaw's screenshot, the remote address field is also incorrect, as it shows the proxy's address, not the destination server.

Comment 15

[Ricardo Iramar dos Santos]

A "4 years ago" bug. :(
I'm using FF with Burp and this bug is not helping with my tests.

Comment 16

[R4yKane]

I just encountered this issue still in Firefox Developer Edition 64.0b5 (64-Bit, MacOSX).
This bug confuses quite much in developing / testing. I finally checked with Chrome and Internet Explorer whats going on. Those browsers show the status correctly from the destination server. :-((

Comment 17

[Frederik Braun [:freddy]]

Honza, do you think we could revisit prioritization here?

Comment 18

[Jan Honza Odvarko [:Honza] (always need-info? me)]

Attachment: Bug 833774 - Fix response status for proxies

Comment 19

[Jan Honza Odvarko [:Honza] (always need-info? me)]

Frederik, I attached a test patch that seems to be fixing the problem on my machine.

I also pushed it to try:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=35c7f4fa7d2b5e4bc8282dcc4d034cbc5a16508c&selectedJob=254907619
(win, linux, mac builds available)

Can you please test it and let me know if the patch fixes the issue for you?

Thanks,
Honza

Comment 20

[Frederik Braun [:freddy]]

Attachment: Bildschirmfoto vom 2019-07-05 15-16-52.png

Thank you so much for taking a stab at this!
Unfortunately, instead of showing "200 Connection Established" it now shows "200 200", see attached screenshot

Comment 21

[Jan Honza Odvarko [:Honza] (always need-info? me)]

(In reply to Frederik Braun [:freddyb] (PTO July 8th to 29th) from comment #20)

Unfortunately, instead of showing "200 Connection Established" it now shows "200 200", see attached screenshot

I fixed a typo in the patch, can you please try again?

Honza

Comment 22

[Frederik Braun [:freddy]]

Sorry. I won't find the time today and I'm heading out for a few weeks. Can you find someone else to test please? I've been using https://github.com/zaproxy/zaproxy/releases/tag/v2.8.0 but any proxy would work :-)

Comment 23

[Jan Honza Odvarko [:Honza] (always need-info? me)]

Nicolas, would you have time to test my patch here?
Thanks,
Honza

Comment 24

[Nicolas Chevobbe [:nchevobbe]]

sure!

Comment 25

[Jan Honza Odvarko [:Honza] (always need-info? me)]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=143b7c6abdc6a828433dd2f5df13b25865b8c16a

Comment 26

[Nicolas Chevobbe [:nchevobbe]]

Attachment: Screenshot at Jul 08 17-21-46.png

So I was able to reproduce the issue using OWASP ZAP (doing a fetch to a non-existent endpoint would show a 200 status)
I'm seeing a fix with your patch, but only for the netmonitor (the console still displays a 200, I guess we are not re-rendering something?)

Comment 27

[Jan Honza Odvarko [:Honza] (always need-info? me)]

Thanks for testing this Nicolas!
Patch updated, the Console panel should be fixed now.
Honza

Comment 28

[Nicolas Chevobbe [:nchevobbe]]

yes, works fine in the console as well with the latest patch :) Thanks!

Comment 29

[Jan Honza Odvarko [:Honza] (always need-info? me)]

Some notes:

• HAR export needs to be tested (the true status needs to be exported)
• Console panel doesn't show the status+text within the expanded body
• There should be a test

Honza

Comment 31

[yaozg]

May we fix it?

Comment 32

[Jan Honza Odvarko [:Honza] (always need-info? me)]

(In reply to yaozg from comment #31)

May we fix it?

Sorry, this bug is already assigned.
But, you might try to check some of our good-first-bugs
https://bugzilla.mozilla.org/buglist.cgi?keywords=good-first-bug&keywords_type=allwords&resolution=---&query_format=advanced&bug_status=NEW&component=Netmonitor&product=DevTools&list_id=15230961

Honza

Comment 34

[Hubert Boma Manilla (:bomsy)]

This should be fixed by Bug 1734573. i'll confirm and close

Comment 35

[BugBot [:suhaib / :marco/ :calixte]]

The severity field for this bug is relatively low, S3. However, the bug has 3 duplicates.
:bomsy, could you consider increasing the bug severity?

For more information, please visit auto_nag documentation.

Comment 36

[BugBot (nomail) [:suhaib / :marco/ :calixte]]

The last needinfo from me was triggered in error by recent activity on the bug. I'm clearing the needinfo since this is a very old bug and I don't know if it's still relevant.