1427334 - cannot open github.com while other browsers can

Status: RESOLVED WONTFIX

(Core :: Security: PSM, defect)

Description

[Lu Wei]

Attachment: wireshark captured network flow log

User Agent: Mozilla/5.0 (Windows NT 5.1; rv:52.0) Gecko/20100101 Firefox/52.0
Build ID: 20171206101620

Steps to reproduce:

In my company, type github.com in address bar


Actual results:

address redirected to https://github.com, then shows:

Secure Connection Failed

The connection to the server was reset while the page was loading.

    The page you are trying to view cannot be shown because the authenticity of the received data could not be verified.
    Please contact the website owners to inform them of this problem.


Expected results:

IE and Chrome on the same computer can open github.com without problem. 
At my home Firefox can open github without problem.

I run Wireshark and captured the network flow as attachment.
use filter ip.addr == 192.30.253.112 to show github.com related records. The attempt from 11:47:20 is by Firefox and got RST immediately, while the attempt from 11:47:44 is by Chrome and succeeded.

Comment 1

[Lu Wei]

The RST behavior seems similar to the domain blocking by the great firewall, but Github is not blocked for the time being, and my company does not have the intention to block it either. And the blocking should not be specified to Firefox, I think.

I could provide teamviewer access for troubleshooting further if needed.

Comment 2

[Flore Allemandou [:flore]]

What happens in safe mode or with a fresh profile?
https://support.mozilla.org/en-US/kb/troubleshoot-firefox-issues-using-safe-mode
https://support.mozilla.org/en-US/kb/profile-manager-create-and-remove-firefox-profiles

Comment 3

[Lu Wei]

(In reply to Flore Allemandou [:flore] from comment #2)
> What happens in safe mode or with a fresh profile?
> https://support.mozilla.org/en-US/kb/troubleshoot-firefox-issues-using-safe-
> mode
> https://support.mozilla.org/en-US/kb/profile-manager-create-and-remove-
> firefox-profiles

The report is done on a fresh installed Firefox 52 ESR with a new profile. And safe mode tried too.

Comment 4

[Kershaw Chang [:kershaw]]

Hi,

Could you try to capture the HTTP log [1] and them submit to this bug?

Thanks.

[1] https://developer.mozilla.org/en-US/docs/Mozilla/Debugging/HTTP_logging

Comment 5

[Lu Wei]

Attachment: log.txt-main.5228

the log of a failed attempt to open http://github.com

Comment 6

[Kershaw Chang [:kershaw]]

2018-01-04 11:30:30.890000 UTC - [Socket Thread]: V/nsHttp nsHttpConnection::Activate [this=b2889d0 trans=16ce8400 caps=21]
2018-01-04 11:30:30.890000 UTC - [Socket Thread]: V/nsHttp nsHttpConnection::SetupSSL b2889d0 caps=0x21 .S.....github.com:443
2018-01-04 11:30:30.890000 UTC - [Socket Thread]: V/nsHttp nsHttpConnection::StartShortLivedTCPKeepalives[b2889d0] idle time[10s].
2018-01-04 11:30:30.890000 UTC - [Socket Thread]: D/nsSocketTransport nsSocketTransport::SetKeepaliveVals [15fc6400] keepalive disabled, idle time[10s] retry interval[1s] packet count[10]
2018-01-04 11:30:30.890000 UTC - [Socket Thread]: D/nsSocketTransport JIMB: ReleaseFD_Locked: mFDref = 2
2018-01-04 11:30:30.890000 UTC - [Socket Thread]: D/nsSocketTransport nsSocketTransport::SetKeepaliveEnabled [15fc6400] enabled, idle time[10s] retry interval[1s] packet count[10]: globally enabled.
2018-01-04 11:30:30.891000 UTC - [Socket Thread]: D/nsSocketTransport JIMB: ReleaseFD_Locked: mFDref = 2
2018-01-04 11:30:30.891000 UTC - [Socket Thread]: V/nsHttp nsHttpConnection::OnSocketWritable [this=b2889d0] host=github.com
2018-01-04 11:30:30.891000 UTC - [Socket Thread]: V/nsHttp nsHttpConnection::GetSecurityInfo trans=16ce8400 tlsfilter=0 socket=15fc6410
2018-01-04 11:30:30.891000 UTC - [Socket Thread]: V/nsHttp nsHttpConnection::EnsureNPNComplete b2889d0 - early selected alpn not available, we will try one more time.
2018-01-04 11:30:30.891000 UTC - [Socket Thread]: V/nsHttp nsHttpConnection::EnsureNPNComplete setting complete to true
2018-01-04 11:30:30.891000 UTC - [Socket Thread]: V/nsHttp   writing transaction request stream
2018-01-04 11:30:30.891000 UTC - [Socket Thread]: V/nsHttp nsHttpConnection::GetSecurityInfo trans=16ce8400 tlsfilter=0 socket=15fc6410
2018-01-04 11:30:30.891000 UTC - [Socket Thread]: D/nsSocketTransport nsSocketOutputStream::Write [this=15fc65e8 count=311]
2018-01-04 11:30:30.891000 UTC - [Socket Thread]: D/nsSocketTransport   calling PR_Write [count=311]
2018-01-04 11:30:30.891000 UTC - [Socket Thread]: D/nsSocketTransport   PR_Write returned [n=-1]
2018-01-04 11:30:30.891000 UTC - [Socket Thread]: D/nsSocketTransport JIMB: ReleaseFD_Locked: mFDref = 2
2018-01-04 11:30:30.891000 UTC - [Socket Thread]: D/nsSocketTransport ErrorAccordingToNSPR [in=-5961 out=804b0014]
2018-01-04 11:30:30.891000 UTC - [Socket Thread]: D/nsSocketTransport nsSocketTransport::OnMsgOutputClosed [this=15fc6400 reason=804b0014]
2018-01-04 11:30:30.891000 UTC - [Socket Thread]: V/nsHttp nsHttpConnection::OnSocketWritable b2889d0 ReadSegments returned [rv=0 read=0 sock-cond=804b0014]
2018-01-04 11:30:30.891000 UTC - [Socket Thread]: V/nsHttp nsHttpConnection::CloseTransaction[this=b2889d0 trans=16ce8400 reason=804b0014]

The log shows that the transaction has been closed multiple times due to the same error: NS_ERROR_NET_RESET.

Dragana, do you have any idea here? Thanks.

Comment 7

[Dragana Damjanovic [:dragana]]

The server resets the connection after receiving a ClientHello.
David, can you take a look at the pcap. There is a Firefox and a Chrome connection in the pcap.

Comment 8

[Dana Keeler (she/her) (use needinfo) [:keeler]]

Running ESR 52 on my machine, I get the same client hello as in the packet trace and it succeeds, indicating this may be a problem with a middlebox. As a wild guess, the Firefox ESR 52 client hello is a bit smaller than the Chrome client hello, so maybe that's the problem? (I have a vague recollection of some issue with small client hellos, but I don't remember the details...) More recent versions of Firefox seem to have a larger client hello - if you try with one of those, does it work?

Comment 9

[Lu Wei]

Attachment: captured using Firefox57

I use Firefox52 because I am mainly on a WindowsXP system. Downloaded from mozilla.org and installed Firefox57.0.4 on a Windows7 system, still failed. Got a new wireshark captured log.

The operation was:
open Chrome;
type github.com in address bar of Chrome and enter;
wait the page loaded and open Firefox;
type github.com in address bar of Firefox and enter;
Failed page shown and stop capturing.

Comment 10

[Lu Wei]

Attachment: log.txt-main.7736 of Firefox57

Firefox57.0.4 log of opening github.com

Comment 11

[Lu Wei]

I am afraid changing the tracking version to 57 will make the bug fixed in 57 or later but not 52ESR. So I'd rather changed it back.

Comment 12

[Dana Keeler (she/her) (use needinfo) [:keeler]]

Looks like 57 still has the small client hellos. I guess I was using Firefox 58 - would you mind trying with that?

Comment 13

[Lu Wei]

Attachment: captured using Firefox58

Downloaded latest Firefox58 from releases.mozilla.org. Still not work. This time only captured a failed Firefox operation. The network packets seems a little different though: there are two different length client hello packets, both failed.

Comment 14

[Dana Keeler (she/her) (use needinfo) [:keeler]]

Is there a version of Firefox that does work for you? That might help narrow down the incompatibility. Also, do you know what the network appliance is that's sending the resets? Presumably it's some sort of firewall or packet inspection device.

Comment 15

[Lu Wei]

Firefox 24/12/6/3:
Secure Connection Failed

An error occurred during a connection to github.com. Cannot communicate securely with peer: no common encryption algorithm(s). (Error code: ssl_error_no_cypher_overlap)

Firfox 2.0:
pops up an alert: Firefox can't connect securely to github.com because the site uses a security protocol which isn't enabled. Then a blank page is shown and cannot continue.

Firfox 1.5/1.0.8:
After two security warning (I think it's because "DigiCert High Assurance EV Root CA" is not in the CA list of version 1.0.8), github.com loaded. 

So the problem occurs after version 1.5. Do you need any pcap log of above?

As to who is sending the resets, I have no idea but only suspect great firewall, though why only the combination of Firefox+Github is blocked is a mystery.

Comment 16

[Dana Keeler (she/her) (use needinfo) [:keeler]]

So presumably at some point after version 24 the error changes from "no cipher overlap" to just a RST - do you have time to figure out which versions that is? (binary search might help, or just going backwards from 52). Thanks!
Also, the reason I suspect a network appliance at your company is that you said Firefox works from home for you, so it's a reasonable guess that the issue is due to a difference between the two networks.

Comment 17

[Lu Wei]

Attachment: captured using Firefox45.9

So versions below 24 is wrong direction. I tried versions between 24~52, and good result: version 45.9.0ESR can open github.com, while 46.0 failed like 52. 

No problem at home. The problem need 3 factors present: office network, firfox above version 46, github.

Comment 18

[Lu Wei]

I use half-split method to narrow the searching range without testing every version so I did not know the version point at which error "no cipher overlap" disappear. Is it needed?

Comment 19

[Flore Allemandou [:flore]]

Mozregression would be more efficient to do this, I believe : https://blog.nightly.mozilla.org/2017/06/07/using-mozregression-on-windows/

Comment 20

[Lu Wei]

(In reply to Flore Allemandou [:flore] from comment #19)
> Mozregression would be more efficient to do this, I believe :
> https://blog.nightly.mozilla.org/2017/06/07/using-mozregression-on-windows/

Thanks. Might try it next time encountering regression.

Comment 21

[Dana Keeler (she/her) (use needinfo) [:keeler]]

Thanks! It seems this may be related to the 'extended master secret' extension. It was enabled in Firefox in 46 (and as far as I can tell, that's the only change in the client hello from 45). Confusingly, the Chrome (or IE? I guess I just assumed the others were Chrome) handshakes in attachment 8939052 [details] also have the extended master secret extension, but the order is a bit different (for example, the successful handshake from the other browser has the 'renegotiation info' extension before the extended master secret extension). Maybe we need to re-order some extensions again? (see e.g. bug 1243641)

I made a build that should reorder the client hello extensions to be a bit more like chrome: https://treeherder.mozilla.org/#/jobs?repo=try&revision=eb6eed7a01dab9ad6de71b70ac8805ed54ff6c3e
It's not actually done yet, but once it is (and if the build succeeds), if you're willing to keep debugging this, you should be able to click on the appropriate green "B" (x64 if you're 64-bit, the other one if you're not) and download/run either "setup.exe" or "target.installer.exe" (maybe? I'm not actually sure what it should be for Windows).

(As an aside, it looks like at least one vendor has had problems with this extension: https://support.f5.com/csp/article/K34019109 ).

Comment 22

[Lu Wei]

Attachment: comparison of IE, Chrome, and test build

The "other browsers" include IE and Chrome, yet I only captured Chrome for comparison in previous attachment 8939052 [details]. 

I clicked "B" of eb6eed7a01da for I am on win32. It appears success, and from the build log below I found the download link of "setup.exe" and "target.installer.exe". I downloaded target.installer.exe, installed and got fail result.

The new comparison capture of IE, Chrome, and target.installer.exe:
Connection beginning at 4s to 192.30.255.112 is from IE;
Connection beginning at 20s to 192.30.255.113 is from Chrome;
Connection beginning at 50s to 192.30.255.112 is from test build.

Comment 23

[Dana Keeler (she/her) (use needinfo) [:keeler]]

Ok - how about this one: https://treeherder.mozilla.org/#/jobs?repo=try&revision=09a67635270f48ec40a5f776f024c3b43a6911b3
I reordered the extensions to be as close to Chrome as I could get without more invasive changes.

Comment 24

[Lu Wei]

Attachment: client hellos comparison

Still failed. I saved all "client hello" packet of IE, Chrome, v45.9 and nightly for you to compare. Maybe it not because of order, but something else? I noticed all success packets' protocol is labeled by Wireshark "TLSv1.2", while nightly's protocol is labeled "SSL". Could you make a precisely clone of the packets of v45.9?

Comment 25

[Dana Keeler (she/her) (use needinfo) [:keeler]]

This is as close as I could get to the 45 TLS handshake for now: https://treeherder.mozilla.org/#/jobs?repo=try&revision=9e13d565f84fde25c230e433369e56007661f656

Comment 26

[Lu Wei]

Success! This time the packet is recognized as "TLSv1.2". Would you experiment further which bit exactly caused this bug? You could build a bunch of test builds for me to test.

Comment 27

[Dana Keeler (she/her) (use needinfo) [:keeler]]

Great! Unfortunately, our infrastructure is having some problems right now, so I probably won't be able to prepare more builds until at least early next week. One thing you could try in the meantime is take the build in comment 25 and set the about:config preferences "security.ssl3.ecdhe_ecdsa_chacha20_poly1305_sha256", "security.ssl3.ecdhe_rsa_chacha20_poly1305_sha256", "security.ssl3.ecdhe_ecdsa_aes_256_gcm_sha384", and "security.ssl3.ecdhe_rsa_aes_256_gcm_sha384" to true (basically, re-enable chacha and gcm-sha384). That should tell us if it's the extensions that are the problem.

Also, if you take a normal Nightly build and set those preferences to false, as well as setting "security.tls.version.max" to 3, we might be able to tell if it's the cipher suites that are the problem.

Comment 28

[Lu Wei]

The build in comment 25, re-enable chacha and gcm-sha384, even set "security.tls.version.max" to 4 still works.
In normal Firfox58, disable chacha and gcm-sha384 and set "security.tls.version.maxstill" to 3 still fails.

Comment 29

[Dana Keeler (she/her) (use needinfo) [:keeler]]

So it seems like it's either the extended master secret extension (which doesn't make a whole lot of sense because the chrome handshakes have it, and they succeed) or curve 25519 (which doesn't make a whole lot of sense because the chrome handshakes have it, and they succeed).

Here's a build without extended master secret: https://treeherder.mozilla.org/#/jobs?repo=try&revision=30bc33e63a2efec6e0a30468024248a0eca05744
Here's a build without curve 25519: https://treeherder.mozilla.org/#/jobs?repo=try&revision=b0e7bfae3de26ca1e9ced2f614b0657a359f9c45

Comment 30

[Lu Wei]

(In reply to David Keeler [:keeler] (use needinfo) from comment #29)

> Here's a build without extended master secret:
> https://treeherder.mozilla.org/#/
> jobs?repo=try&revision=30bc33e63a2efec6e0a30468024248a0eca05744

This version works.

> Here's a build without curve 25519:
> https://treeherder.mozilla.org/#/
> jobs?repo=try&revision=b0e7bfae3de26ca1e9ced2f614b0657a359f9c45

This one failed.

Comment 31

[Dana Keeler (she/her) (use needinfo) [:keeler]]

So I think this means some traffic monitoring device at your work has a problem with the extended master secret extension. It's strange that Chrome works with EMS enabled while Firefox doesn't, though.

Comment 32

[Lu Wei]

(In reply to David Keeler [:keeler] (use needinfo) from comment #31)
> So I think this means some traffic monitoring device at your work has a
> problem with the extended master secret extension. It's strange that Chrome
> works with EMS enabled while Firefox doesn't, though.

You could make a version simulating Chrome and see if it works.

Comment 33

[Dana Keeler (she/her) (use needinfo) [:keeler]]

Hi Lu Wei - I thought I was going to have time to keep working on this, but it's clear I just don't. Ultimately I still think the problem is with a network device messing with your traffic, and this isn't something Firefox is going to be able to solve. Sorry.

Comment 34

[Lu Wei]

(In reply to David Keeler [:keeler] (use needinfo) from comment #33)
> Hi Lu Wei - I thought I was going to have time to keep working on this, but
> it's clear I just don't. Ultimately I still think the problem is with a
> network device messing with your traffic, and this isn't something Firefox
> is going to be able to solve. Sorry.

I somewhat agree with that, though how it works is a mystery. Thanks for your effort.

Comment 35

[Lu Wei]

Happened to find that the latest Chrome 67 cannot visit github.com in that machine either:

This site can’t be reached
github.com is currently unreachable.
Try:

Checking the connection
Checking the proxy and the firewall
ERR_SSL_VERSION_INTERFERENCE

Now only IE works. And the corp IT man says "as long as you can open it, don't bother". Oh well.