1484947 - SSL errors completely freeze any requests in all tabs

Status: VERIFIED FIXED

(Core :: Networking, defect, P2)

Description

[shark0der]

User Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36

Steps to reproduce:

When using a proxy, either by specifying a PAC file or using a VPN extension, navigating to a web-page that has a bad certificate (self-signed, expired, etc) Firefox will completely freeze all network requests in all tabs (including Firefox update checks). I've tested this using several Firefox extensions but also using a PAC script with a fresh squid proxy install with a valid certificate. The same proxy works perfectly in Chrome and displays the corresponding certificate error page.

This is especially frustrating since navigating through sites, if any site has an ajax request or a script or an ad that's on a domain with a bad certificate - the entire browsing experience is ruined as no other page works.

Tested with Firefox 61.0.2 on Windows 8.1 and Firefox 61.0.1 and 62.0b18 on Ubuntu 18.04.

Steps to reproduce:
1. Install Browsec extension (free) https://addons.mozilla.org/en-US/firefox/addon/browsec/
You can test with other extensions and proxies as well, this is not a problem with the extension itself.
2. Turn on VPN
3. Navigate to https://badssl.com/
4. Click on "expired" link (https://expired.badssl.com/)



Actual results:

Browser hangs: pages don't load. Opening and closing tabs works, the browser itself is not frozen but no network requests go through (not even the Firefox update checks). During this there's a 10-15% load on the CPU (Firefox process). Hitting ESC stops the hanged request and everything goes back to normal. 


Expected results:

Firefox should display the appropriate certificate error message (or just fail to load the request if it was done via ajax or a script/css).

Comment 1

[Damira Akh]

I was not able to reproduce this issue.
Tested on Ubuntu 18.04 and Windows 7x64
Build ID 	20180830143136
User Agent 	Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:62.0) Gecko/20100101 Firefox/62.0

Browser displaying the following error message.

"
expired.badssl.com uses an invalid security certificate. 
The certificate expired on Sunday, April 12, 2015, 5:59:59 PM. The current time is September 7, 2018, 7:38 PM. 
Error code: SEC_ERROR_EXPIRED_CERTIFICATE
"

Thank you.

Comment 2

[shark0der]

Me and a friend of mine have just retested this and the issue is still present with the latest firefox. I've made a video to demo it: https://youtu.be/Pzh7vuyghGw . Please make sure that you've followed all the steps carefully.

Comment 3

[Johann Hofmann [:johannh]]

Sounds more like a Networking issue to me.

Comment 4

[u408661]

I can see this behaviour, but it appears to be a proxy-side issue to me. Here's the steps I used to reproduce:

1. Set up a squid proxy on localhost
2. Set firefox to use said proxy for all connections
3. Visit expired.badssl.com
4. In a new tab, go to google.com
5. Note that the google tab appears to hang

Here's why I say this is a proxy-side issue - I ran wireshark during the above steps, and when the google load hung (trying to connect to www.gstatic.com), I saw in wireshark, firefox appropriately sends the CONNECT for www.gstatic.com:443, then squid just... doesn't respond. Eventually firefox timed out the connection (sent a FIN), and approximately a second and a half later, squid sent the HTTP 200. I'll attach the pcap.

Reporter, do you know what proxy software is being used in your case?

Comment 5

[u408661]

Attachment: proxy-hang.pcapng

Comment 6

[shark0der]

Hello Nicholas,

It is not a server side issue. First of all, the same servers work fine on other browsers or in command-line using cURL.
In order to prove that this is not a server side issue I've unpacked the Browsec extension (sorry Browsec...) and extracted the free server host and port: nl1.postls.com:443. Next, I've tested that the server indeed works fine:


## Simple curl request:

$ time curl https://expired.badssl.com -x https://nl1.postls.com:443
curl: (60) SSL certificate problem: certificate has expired
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

real	0m1.055s
user	0m0.121s
sys	0m0.109s


## With `--insecure` flag:

$ time curl https://expired.badssl.com -x https://nl1.postls.com:443 --insecure
<!DOCTYPE html>
<html>
<head>
  <meta name="viewport" content="width=device-width, initial-scale=1">
  <link rel="shortcut icon" href="/icons/favicon-red.ico"/>
  <link rel="apple-touch-icon" href="/icons/icon-red.png"/>
  <title>expired.badssl.com</title>
  <link rel="stylesheet" href="/style.css">
  <style>body { background: red; }</style>
</head>
<body>
<div id="content">
  <h1 style="font-size: 12vw;">
    expired.<br>badssl.com
  </h1>
</div>

</body>
</html>

real	0m2.505s
user	0m0.240s
sys	0m0.284s


As you can see I am able to perform requests and the replies come in a timely manner.

Next, in order to set this proxy in Firefox I've created a simple pac file, proxy.pac:

$ cat ~/tmp/proxy.pac 
function FindProxyForURL(url, host) {
 return "HTTPS nl1.postls.com:443;";
}


Next I've started a HTTP server using `python -m SimpleHTTPServer 8080` and have set the automatic proxy configuration URL in firefox to `http://localhost:8080/proxy.pac` like in this screenshot: https://i.imgur.com/wx9fkGM.png .

I've navigated to a couple of sites and have checked my IP in order to make sure that the proxy is being used and then navigated to expired.badssl.com and it hanged just like in the video.

Comment 7

[shark0der]

I've initially mentioned that there's a 10-15% load on the CPU (Firefox process) but I have to make a correction - the firefox process consumes 100% CPU.

Comment 8

[Honza Bambas (:mayhemer)]

(In reply to shark0der from comment #6)
> function FindProxyForURL(url, host) {
>  return "HTTPS nl1.postls.com:443;";
> }

Please retry with:

return "HTTPS nl1.postls.com:443";

(remove the trailing semicolon) and let us know.  I know you can specify multiple proxies but there may be a bug when the semicolon is not followed by one.

Comment 9

[shark0der]

> Please retry with:
> 
> return "HTTPS nl1.postls.com:443";

I have removed the trailing semicolon, it made no difference - invalid certificates still freeze all the network requests and a process spikes to 100% CPU as previously mentioned.

Comment 10

[Honza Bambas (:mayhemer)]

OK, thanks.  Then please provide a log, it may tell us something:

https://developer.mozilla.org/en-US/docs/Mozilla/Debugging/HTTP_logging

please add few more modules to the MOZ_LOG list: pipnss:5,proxy:5

thanks.

Comment 11

[Kershaw Chang [:kershaw]]

I can reproduce this.

Please see the link for the log below.
https://drive.google.com/file/d/1KgveEHxHlENntPyyk-4Pq7SzLEu8EU8g/view?usp=sharing

Note that this log is produced with MOZ_LOG=timestamp,nsHttp:5,nsSocketTransport:5,pipnss:5

Comment 12

[Kershaw Chang [:kershaw]]

(In reply to Kershaw Chang [:kershaw] from comment #11)
> I can reproduce this.
> 
> Please see the link for the log below.
> https://drive.google.com/file/d/1KgveEHxHlENntPyyk-4Pq7SzLEu8EU8g/
> view?usp=sharing
> 
> Note that this log is produced with
> MOZ_LOG=timestamp,nsHttp:5,nsSocketTransport:5,pipnss:5

The log shows that STS thread is stuck in this loop.
https://searchfox.org/mozilla-central/rev/72b1e834f384a2ffec6eb4ce405fbd4b5e881109/netwerk/protocol/http/nsHttpConnection.cpp#2049-2091

Comment 13

[Kershaw Chang [:kershaw]]

Attachment: Bug 1484947 - Close the transaction if PR_Read/PR_Write failed, r=nwgh

When PR_Read/PR_White returns -1, we have to use ErrorAccordingToNSPR to get the error code. We need to close the transaction if the real error happens.

Comment 15

[Kershaw Chang [:kershaw]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=29fb7b35c74fa5bc188f6ee808d96aecfcd6ea4b&selectedJob=208673041

Comment 17

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/e90247632fd8
Close the transaction if PR_Read/PR_Write failed, r=nwgh

Comment 18

[Arthur Iakab [arthur_iakab]]

https://hg.mozilla.org/mozilla-central/rev/e90247632fd8

Comment 19

[Daniel Cicas [:dcicas], Release QA]

I reproduced this issue using Fx 63.0a1, build ID:20180821100053, on Ubuntu 18.04 x64.

I can confirm this issue is fixed, I verified using Fx 65.0b8, build ID: 20190103150357, on Ubuntu 18.04 LTS, Windows 10 x64 and macOS X 10.13.6.