1121706 - Firefox 35.0 security.ssl3 Problem - Unable to connect to Google

Status: RESOLVED FIXED

(Core :: Networking: HTTP, defect)

Description

[Joao Pedro]

User Agent: Mozilla/5.0 (Windows NT 6.3; rv:34.0) Gecko/20100101 Firefox/34.0
Build ID: 20141126041045

Steps to reproduce:

If both security.ssl3.ecdhe_ecdsa_aes_128_gcm_sha256 and security.ssl3.ecdhe_ecdsa_aes_128_gcm_sha256 are set to false, firefox 35 is unable to connect to google even if all other cipher suites remain enabled!



Actual results:

This results in all connections to google being aborted: Search doesn't work, manually typed google addresses don't work, etc...

Firefox 34 and earlier versions don't show this behavior, as they can connect to google with any of the other cipher suites (e.g., any of the aes_256).





Expected results:

Firefox 35 should be able to connect to google with any of the other cipher supported by google.

Bug 1109368 also had this issue, but the user made a reset to his profile, thereby re-enabling the offending cipher suites. The Ticket was closed.

Bug 1121330 also had this issue, but it was marked invalid because: "Firefox does not in any way block google or make it unavailable. This sounds like an issue with either your internet connection or your computer (add-ons or malware-related)."

Please check this issue before more users come storming in with this...

Comment 1

[Joao Pedro]

Forgot to add:

This happens in both x86 and x64 version of windows 8.1. Can't test in other OS at the moment.

Comment 2

[:Gijs (he/him)]

David/Brian, do you know what's going on here?

Comment 3

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

(In reply to Joao Pedro from comment #0)
> If both security.ssl3.ecdhe_ecdsa_aes_128_gcm_sha256 and
> security.ssl3.ecdhe_ecdsa_aes_128_gcm_sha256 are set to false, firefox 35 is
> unable to connect to google even if all other cipher suites remain enabled!

Those two cipher suites are the same, unless I'm misreading them. Are there any others you have disabled?

Comment 4

[:Gijs (he/him)]

[Tracking Requested - why for this release]:

Not sure how widespread this is, but 3 reports at this time after the release is worrying.


Joao, why are these prefs set to false for you? (and/or, if you can guess, these other users?)

Also,

(In reply to Joao Pedro from comment #0)
> security.ssl3.ecdhe_ecdsa_aes_128_gcm_sha256 and
> security.ssl3.ecdhe_ecdsa_aes_128_gcm_sha256

Copy/paste fail? Which other cipher did you mean?

Comment 5

[:Gijs (he/him)]

(In reply to :Gijs Kruitbosch from comment #4)
> (In reply to Joao Pedro from comment #0)
> > security.ssl3.ecdhe_ecdsa_aes_128_gcm_sha256 and
> > security.ssl3.ecdhe_ecdsa_aes_128_gcm_sha256
> 
> Copy/paste fail? Which other cipher did you mean?

I can reproduce on win8 if I also disable security.ssl3.ecdhe_rsa_aes_128_gcm_sha256

Comment 8

[:Gijs (he/him)]

Breaks on OS X as well.

Comment 9

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

(In reply to Joao Pedro from comment #0)
> If both security.ssl3.ecdhe_ecdsa_aes_128_gcm_sha256 and
> security.ssl3.ecdhe_ecdsa_aes_128_gcm_sha256 are set to false, firefox 35 is
> unable to connect to google even if all other cipher suites remain enabled!

Maybe that is a policy decision that Google made?

I don't think that changing the values of the security.ssl3.* prefs should be considered "supported". Leave those prefs set at their default values.

Comment 11

[:Gijs (he/him)]

(In reply to Brian Smith (:briansmith, :bsmith, use NEEDINFO?) from comment #9)
> (In reply to Joao Pedro from comment #0)
> > If both security.ssl3.ecdhe_ecdsa_aes_128_gcm_sha256 and
> > security.ssl3.ecdhe_ecdsa_aes_128_gcm_sha256 are set to false, firefox 35 is
> > unable to connect to google even if all other cipher suites remain enabled!
> 
> Maybe that is a policy decision that Google made?

Between two different browser versions? That seems unlikely...

> I don't think that changing the values of the security.ssl3.* prefs should
> be considered "supported". Leave those prefs set at their default values.

Even if that was workable (and though I agree that I have no idea why people mess with these things), we should still be giving a proper error page. As it is, the browser briefly tries to connect and then just stops - but the currently-loaded page (about:newtab, or whatever was in the tab when you tried to navigate to google.com) just stays. Why isn't this showing about:certerror with an error?

Comment 12

[Hauke]

I already had this settings in the previous version of Firefox and only after the update to 35.0 the problem occurred.
The reason why I had security.ssl3.ecdhe_rsa_aes_128_gcm_sha256 set to false is that I simply don't trust the elliptic curves of NIST (here https://www.schneier.com/blog/archives/2013/09/the_nsa_is_brea.html#c1675929 you can read a comment of Bruce Schneider about this topic) and prefer other encryption schemes ((dhe)rsa_aes). 

When you look at https://www.ssllabs.com/ssltest/analyze.html?d=google.com you can see that google offers lots of other encryption schemes (in a new Firefox profile even RC4 is activated by default so it should definitely find a common encryption algorithm besides ecdhe_rsa_aes_128_gcm_sha256) 

And even if google would have decided to only allow  ecdhe_rsa_aes_128_gcm_sha256 for Firefox 35 (which I would find strange) then it should display the ssl_error_no_cypher_overlap error and not just do nothing.

Comment 13

[Shawn Heisey]

I will check my ciphers when I reach that machine.  As far as I know, all of those settings are default, but I will find out.

Comment 14

[Joao Pedro]

Update:

I misspelled the second cipher, sorry. The ones are:

security.ssl3.ecdhe_ecdsa_aes_128_gcm_sha256
security.ssl3.ecdhe_rsa_aes_128_gcm_sha256

The issue remains whether all other ciphers are set to their defaults or not.

Why had I these disabled? To force firefox to only use ciphers with AES 256 bits keys.

Comment 15

[Rob Stradling]

Hi.  This is a wild guess, so please just ignore me if I'm completely wrong.

I wonder if [1] and [2] show the effect of this regression on the server side.

In [1], Matt Caswell wrote:
"It means that an attempt is being made to resume a session, however the 
list of ciphers that the client is sending in the ClientHello does not 
include the cipher that was negotiated in the original session."

Does FF35 have a different set of default ciphers than FF34?
When resuming a session, does FF remember which cipher it negotiated in the original session?
Might there be some sessions that were established by FF34...and then the user upgraded to FF35...and now FF35 is failing to resume those sessions?


[1] http://openssl.6102.n7.nabble.com/openssl-users-SSL3-GET-CLIENT-HELLO-required-cipher-missing-td55894.html

[2] http://serverfault.com/questions/659241/in-nginx-logs-ssl3-get-client-hellorequired-cipher-missing

Comment 16

[Shawn Heisey]

Based on what I saw, I think I was disabling ciphers so that I could decrypt a session in wireshark with the private key, and forgot to re-enable them.  After setting all the ciphers back to defaults, google sites now work.

Comment 17

[Shawn Heisey]

I'm curious why the browser does nothing in this situation, though.  In the past I've seen an error message when an SSL negotiation fails, why doesn't it appear for this?

Comment 18

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

What I'm seeing is the Http2 AEAD check is failing at https://hg.mozilla.org/mozilla-central/annotate/c1f6345f2803/netwerk/protocol/http/Http2Session.cpp#l3352 (presumably because all AEAD suites in common with Google have been disabled). After that, there doesn't seem to be any kind of notification to the front-end that something has gone wrong. Patrick, thoughts?

Comment 19

[Lukas Blakk [:lsblakk] use ?needinfo]

Triage drive-by: waiting to see more information here about how widespread this is, if it's affecting users with default settings.

Comment 20

[Patrick McManus [:mcmanus]]

it sounds like by config someone disabled all our AEAD suites? or did they just disable the ones that overlap with google?

here's what probably happened - 

1] we offered a tls 1.2 client hello with an alpn offer list that included h2. 

2] goog responded with a 1.2 server hello, an alpn selection of h2 and a non aead cipher suite.

#2 is non compliant http2 so we fail the connection with a protocol error.

goog should definitely not be doing #2. I'll point one of their GFE folk at this bug - the server does ALPN selection so the burden is on that side to make sure its negotiating something sane.

however, if we aren't sending any AEAD suites at all, then its not nice or sensible to be offering h2. We could fix that - though I don't think its illegal. otoh if we are sending an AEAD choice that just happens to not overlap with goog then our behavior shouldn't change.

The workaround would be to disable h2 by config - and if you're already changing ciphersuites in your config that seems reasonable. network.http.spdy.enabled.http2 and network.http.spdy.enabled.http2draft to false.

Comment 21

[Hauke]

(In reply to Patrick McManus [:mcmanus] from comment #20)
> it sounds like by config someone disabled all our AEAD suites? or did they just disable the ones that overlap with google?

I only set security.ssl3.ecdhe_rsa_aes_128_gcm_sha256 to false, the rest I left at the defaults. I'm not sure if then other "AEAD suites" are still active in Firefox 35.

(In reply to Patrick McManus [:mcmanus] from comment #20)
> #2 is non compliant http2 so we fail the connection with a protocol error.

If it depends on the server side, than google might not be the only one doing #2 and there a proper error message should be displayed for the Firefox user (at the moment it just does nothing). Or maybe an automatic retry without h2 (I'm not an expert and therefore don't know how this would affect security).

(In reply to Patrick McManus [:mcmanus] from comment #20)
> The workaround would be to disable h2 by config - and if you're already
> changing ciphersuites in your config that seems reasonable.
> network.http.spdy.enabled.http2 and network.http.spdy.enabled.http2draft to
> false.

network.http.spdy.enabled.http2 seems to be set to false by default. Setting also  network.http.spdy.enabled.http2draft to false seems to solve the problem. But of course this is only a workaround.

Comment 22

[Masatoshi Kimura [:emk]]

(In reply to Patrick McManus [:mcmanus] from comment #20)
> however, if we aren't sending any AEAD suites at all, then its not nice or
> sensible to be offering h2. We could fix that - though I don't think its
> illegal. otoh if we are sending an AEAD choice that just happens to not
> overlap with goog then our behavior shouldn't change.

If both client and server support h2 and are compliant with the spec, it will never happen because h2 requires supporting TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256.

That said, I'm not sure we need to save people shooting their foot, except that we should display an error if the connection failed for whatever reason.

Comment 23

[Patrick McManus [:mcmanus]]

:emk's comment 22 is a good one. Its easy to not offer h2 in alpn if TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 is disabled.

Comment 24

[Patrick McManus [:mcmanus]]

Attachment: don't offer h2 in alpn if w/out mandatory suite

Comment 25

[Patrick McManus [:mcmanus]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=6d3a762a61e9

Comment 26

[u408661]

Comment on attachment 8550083 [details] [diff] [review]
don't offer h2 in alpn if w/out mandatory suite

Review of attachment 8550083 [details] [diff] [review]:
-----------------------------------------------------------------

::: netwerk/protocol/http/Http2Session.cpp
@@ +3283,5 @@
>  bool // static
>  Http2Session::ALPNCallback(nsISupports *securityInfo)
>  {
> +  if (!gHttpHandler->IsH2MandatorySuiteEnabled()) {
> +    LOG3(("Http2Session::ALPNCallback Mandatory Cipher Suite Unavailablex\n"));

nit: you've got an extraneous x before the \n

Comment 27

[Patrick McManus [:mcmanus]]

Attachment: don't offer h2 in alpn if w/out mandatory suite

Comment 28

[Patrick McManus [:mcmanus]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/8ef8e6f97a58

Comment 29

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

I have a remaining concern: even if we only offer h2 if the necessary suite is enabled, a misconfigured server could see that h2 is offered and try to use it while still picking a non-AEAD ciphersuite, right? In those cases, Firefox needs to do better than just mysteriously stopping the page load and not notifying the user in any way (like :emk said in comment 22).

Comment 30

[Patrick McManus [:mcmanus]]

"you can have a better error message" is always a legitimate issue. Its also a never ending process :)

in this case this is determined as a mid stream protocol error like any other piece of peer-badness and how that is rendered is always confusing. (its not a handshake error because the defined error response is actually an h2 frame.) falling back in particular is probably not what we want here - though messaging is always good.

I'll make a pass at an error screen - though I'm glad they'll be in different commits for backport purposes.

Comment 31

[u408661]

@mcmanus & keeler - https://bugzilla.mozilla.org/show_bug.cgi?id=1122642

Comment 32

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

Great - thanks :)

Comment 33

[Lukas Blakk [:lsblakk] use ?needinfo]

We have a workaround in comment 21 so let's get this uplifted to beta and ship the fix with 36.

Comment 34

[Hauke]

Thanks for the fast fix.

Just to side questions (maybe somebody has time to answer them if they are not to difficult):

1) Why was Firefox 34.X not affected? I checked the release notes of Firefox 35 when I experienced the problem but as a non expert I couldn't see a point that would explain the change in behaviour.

2) Is there some telemetry data available to tell how wide spread the use of H2 is? I'm wondering why only google seems to have caused problems (of course there might be other sides that much less people look at, but at least the other few I tested worked fine).

Comment 35

[Masatoshi Kimura [:emk]]

Release Note Request (optional, but appreciated)
[Why is this notable]:
[Suggested wording]:
[Links (documentation, blog post, etc)]:

(In reply to Hauke from comment #34)
> 1) Why was Firefox 34.X not affected? I checked the release notes of Firefox
> 35 when I experienced the problem but as a non expert I couldn't see a point
> that would explain the change in behaviour.

Because h2 was disabled until Firefox 35 [1]. Although Firefox 34 release note and developers doc claim Firefox 34 enabled h2 [2][3], it is wrong (it does not reflect the late backout).

Flagging relnote-firefox and dev-doc-needed to request the fix.

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=1047594#c29
[2] https://www.mozilla.org/en-US/firefox/34.0/releasenotes/
[3] https://developer.mozilla.org/en-US/Firefox/Releases/34

Comment 36

[Masatoshi Kimura [:emk]]

Oh, I forgot to fill the template.

Release Note Request (optional, but appreciated)
[Why is this notable]: It was already relnoted for Firefox 34.
[Suggested wording]: N/A (Move the relnote item from Firefox 34 to 35.)
[Links (documentation, blog post, etc)]: N/A

Comment 37

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/8ef8e6f97a58

Comment 38

[Jean-Yves Perrier [:teoli]]

I've moved the Fx for Devs note from 34 to 35. (Unfortunately nobody put a ddn on bug 1088910 so it got unnoticed). Thanks.

Comment 39

[Patrick McManus [:mcmanus]]

Comment on attachment 8550359 [details] [diff] [review]
don't offer h2 in alpn if w/out mandatory suite

Approval Request Comment
See comment 33

[Feature/regressing bug #]:enable http/2 (gecko 35)
[User impact if declined]: only impacts users with modified about:config cipher suites - creates connection errors. workaround in about:config available.
[Describe test coverage new/current, TBPL]: new xpcshell test
[Risks and why]: very low - disables http/2 negotiation and falls back to http/1 if required cipher suite not avail
[String/UUID change made/needed]:none

Comment 40

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-aurora/rev/d453cb1ae664
https://hg.mozilla.org/releases/mozilla-beta/rev/131919c0babd

Comment 42

[Sylvestre Ledru [:Sylvestre]]

I have this item in the release note "Support for the full HTTP/2 protocol. HTTP/2 enables a faster, more scalable, and more responsive web."
and I removed "Implementation of HTTP/2 (draft14) and ALPN" from the 34 release notes.