669 bytes, text/plain
7 years ago
2.23 KB, text/plain
6.43 KB, patch
|Details | Diff | Splinter Review|
http://www.enbridge.com/mypage The above link caused a malformed SSL handshake in both Minefield nightly and Beta 2. The page loaded properly in Beta 1 and 3.6.8
https://bugzilla.mozilla.org/buglist.cgi?resolution=FIXED,VERIFIED&chfieldto=2010-07-20%2015%3A00&chfield=resolution&query_format=advanced&chfieldfrom=2010-06-30%2011%3A00&product=Core&product=Firefox&product=NSPR&product=NSS&product=Toolkit is the beta1-beta2 regression range - we need to narrow that down.
Regression range: http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=42249b1f40b3&tochange=6b58d2615145 I suspect: bug 575620.
Direct link to the problematic server: https://portal-plumprod.cgc.enbridge.com
Need a fix for this regression on 1.9.2 since we took bug 575620 on that branch. Either that or a backout of bug 575620 on branch. (Martell: you just saved us a regression on a security update - way to go!)
Wait! That server proposes to the browser that 256-bit (32-byte) Diffie-Hellman keys be used. That key size is too small to be secure. That's why NSS 3.12.7 aborts the SSL handshake. The new Diffie-Hellman key size check is here: http://bonsai.mozilla.org/cvsblame.cgi?file=mozilla/security/nss/lib/ssl/ssl3con.c&rev=1.142&mark=5296,5305-5306#5296 Users can communicate with this website securely by turning off all ephemeral Diffie-Hellman (DHE) SSL cipher suites as follows: 1. Type "about:config" in the location bar. 2. In the "Filter" field, type "ssl3.dhe". 3. Change the values of all the ssl3.dhe cipher suites to "false".
Are other servers likely to do that same thing? The way we discovered this was that someone was trying to see their gas bill, and Firefox stopped letting them do that. :/
Created attachment 461710 [details] [diff] [review] Workaround patch for mozilla-1.9.2 It seems fine for a security update to become more strict in checking cryptographic key sizes. The issue is that Firefox can communicate securely with this server if it does not offer the ephemeral Diffie-Hellman (DHE) SSL cipher suites. After some experiments, I found a solution that I think should satisfy everyone. Please use this patch for the mozilla-1.9.2 security update only. For mozilla-2.0, please insist that such servers are broken. Details: I found that the server picks the DHE cipher suite TLS_DHE_RSA_WITH_AES_256_CBC_SHA when Firefox uses unpatched NSS. If I apply this NSS patch, the server picks the non-DHE cipher suite TLS_RSA_WITH_AES_256_CBC_SHA. This patch changes NSS to list TLS_DHE_RSA_WITH_AES_256_CBC_SHA after TLS_RSA_WITH_AES_256_CBC_SHA in its SSL ClientHello message, thus affecting the server's selection (most servers simply picks the first cipher suite it supports from the client's list).
That server is absolutely behaving insecurely. Firefox is right to not deceive its user into thinking that it has securely delivered the page to them. The server is wrong and its admin should fix it, just the same as with any other server misconfiguration. I believe we should not modify NSS to weaken Firefox's security for this.
I remember this same problem happened numerous times years ago when we imposed minimum lengths on RSA modulus and public exponent sizes. We began to get reports of servers that stopped working. In one case, we found a server whose public key exponent was 1, which means that SSL to their server wasn't doing any RSA encryption AT ALL. Yet their admin wanted us to "fix the browser". Instead, we did what was in the best interest of our users and their security. I see no reason this case should be any different.
Comment on attachment 461710 [details] [diff] [review] Workaround patch for mozilla-1.9.2 Wan-Teh, your patch will have the effect that, all over the internet, servers everywhere will stop selecting that cipher suite. Most servers who use it do so correctly. In certain countries where the state has the right to demand that server operators give up their private keys, DHE is the only thing that protects the users from retroactive decryption of their prior SSL sessions by the state. IMO, it is GROSSLY wrong of us to deprive all Firefox users everywhere of this DHE cipher suite for the sake of a few users of one misconfigured server. So, r- for this patch for the NSS source trees in CVS. Now, Mozilla's HG copy of NSS is a "Downstream" copy, and they're free to do what they want with it. If they think it's best to take this DHE cipher suite away from the world to benefit a few users of some gas utility, then so be it.
Nelson, you are very right. The difference this time is that it *is* possible to communicate securely with this server. A server that prefers a DHE cipher suite should look for one in the ClientHello's cipher suite list, rather than just selecting the first supported cipher suite from the list. I proposed this patch for Mozilla's HG copy of NSS for Firefox 3.6.x only.
Server admins in the U.K. _could_ disable all but the DHE cipher suites, I suppose. But unless they do that, with this patch, instead of getting a weaker DHE cipher suite, they'd get TLS_RSA_WITH_AES_256_CBC_SHA, which is the non-DHE version of the same strong cipher suite they'd get without this patch.
Comment on attachment 461710 [details] [diff] [review] Workaround patch for mozilla-1.9.2 Nelson: I'm not sure I understand your latest comments. Do you now concur with wtc that we should take this to address the compatibility regression on our stable shipping 1.9.2 branch? It seems to me like the reordering only applies to the specific server cipher suite configuration we encountered here. What are the odds that the newly introduced DHE checks will affect other similar (but not necessarily identical) configurations?
Another alternative would be to introduce an NSS compile-time option to disable the additional DHE checks, which we can use on the 1.9.2 branch. I'm not very familiar with the severity of the problems the checks are supposed to prevent, though, so I don't know whether you would find that acceptable. Other browsers (Chrome/Safari) seem to deal with this server just fine. Are they insecure? It's somewhat difficult to take the lead on making sites incompatible for security reasons, since it gives users an incentive to switch to those other browsers (where the site they want to use works just fine). If we decide that we do want to take this hard line stance, we need to at the very least attempt to coordinate with other browsers.
No, Gavin, I do not concur with the patch as given. The chance that this patch will affect ALL users of DHE are 100%. One site's misconfiguration doesn't constitute a crisis for Mozilla. Defeating DHE security for users the world over just might! Be clear, the proposal is to hurt the security of all FF users for the sake of a few users who use a single misconfigured server.
Created attachment 462134 [details] Cipher suite list of IE 8.0 on Windows 7 gavin: even I will say no to disabling the additional DHE checks. The current minimum of 512 bits is already very conservative. This attachment shows that IE 8.0 on Windows 7 supports only DHE cipher suites that require certificates with DSA keys (very rarely used), and lists these DHE cipher suites at the end. This is why in practice no servers will select a DHE cipher suite with IE.
I suggest that Mozilla ask portal-plumprod.cgc.enbridge.com to fix this server configuration problem. The simplest fix is probably to disable all DHE cipher suites. It is bad to implement a workaround for just a single server. We can consider the workaround patch (attachment 461710 [details] [diff] [review]) if many servers are broken this way. It would also be nice to add a new NSS SSL error code for this error condition. Right now NSS reports "a malformed ServerKeyExchange message", which doesn't say the server's key is too weak.
> It would also be nice to add a new NSS SSL error code for this error > condition. Right now NSS reports "a malformed ServerKeyExchange > message", which doesn't say the server's key is too weak. Agreed. We should do that for all of our "server key too weak" detections.
> I suggest that Mozilla ask portal-plumprod.cgc.enbridge.com In that case we need a TE bug on that, right? Blocking the next branch release?
bz: I filed TE bug 583914. Could you please set the appropriate blocking flags on the bug? Thanks!
Ok, clearing the blocking flag for the branches
If other browsers give all DHE cipher suites lower priority (or don't support them at all), why can't we do the same? This is just a problem with one site for now, but we haven't even released a beta with this change in it yet, so there's plenty of potential for more bustage to be discovered - particularly since we seem to be the only browser that behaves this way.
I am having issues with other sites too e.g. https://www.tescodiets.com. Every other browser is working for this website except the Firefox 4 beta. Firefox seems to be aborting the connection with the alert "Illegal parameter", when the connection is viewed using wireshark, even though the key exchange data being passed is valid. The cipher that is being selected is the "TLS_DHE_RSA_WITH_AES_256_CBC_SHA" cipher as above. I suspect there may be many other websites out there that will experience this issue too. Is this definitely something that will not be addressed?
In reply to comment 25, This is something that will be addressed by server administrators fixing their web sites to deliver real security rather than pretend security based on tiny keys. In reply to comment 15, Gavin, There is a mailing list (owned by Opera) on which representatives of browsers and crypto libraries discuss issues such as this. It would be great if MoCo had an active participant in that list. But alas...
I'm not happy with WONTFIX, given the compatibility impact. Rejecting the weak ciphers is fine, but if we're going to do that we need to match other browsers in giving them the lowest priority to reduce the compatibility impact.
Just to be clear, my understanding is this: 1) NSS 3.12.7 rejects a certain class of "DHE" cipher suites (because they are insecure) that previous NSS versions didn't 2) NSS 3.12.7 lists these cipher suites first in its SSL ClientHello message, which means that servers are more likely to pick them I am OK with 1). But I don't think it should have been done without changing 2) to match other browsers, because the result is that we are now apparently the only browser that will refuse to load pages that we used to load securely, and that are still possible to load securely.
More precisely, we used to load them insecurely, since older NSS also had the DHE cipher suites early in the list. But yes, it is still possible to load them securely.
https://www.tescodiets.com also picks the cipher suite TLS_DHE_RSA_WITH_AES_256_CBC_SHA and uses a 256-bit (32-byte) ephemeral Diffie-Hellman key. The generator element (DH_g) is 2 for both servers. https://www.tescodiets.com identifies itself as: Server: Microsoft-IIS/7.5 X-AspNet-Version: 2.0.50727 X-Powered-By: ASP.NET https://portal-plumprod.cgc.enbridge.com identifies itself as: Server: Apache My workaround also allows Firefox to connect to https://www.tescodiets.com/.
(In reply to comment #25) noel.mulryan: given your tescodiets.com email address, you're in a perfect position to have the administrator of your company's website fix the server configuration error. Bug 583914 has the instructions for the server administrator. I'd appreciate if you could list the other websites that you've encounter this problem with. This will help us understand if the problem with these websites comes from a common piece of software. Thanks!
Opera detects these weak keys, and refuses to use them. Instead, it disables DHE for that server only and renegotiates. That's secure, and doesn't deprive any one of correct use of DHE. It's also a PSM feature, not an NSS feature. If you want it, reassign this bug to PSM.
Another impacted site: https://www.citylink.com.au
Thanks. https://www.citylink.com.au behaves exactly like the other two problematic servers. It doesn't send a Server response header.
Note that this bug should likely be blocking 2.0 final+ as 584138 was blocking it and is now invalid.
It should certainly block. The question is what. I feel we need to at least ship a beta with any changes we make here. Beta5 is probably the right milestone to block, given the feature freeze aspect of things...
FWIW, please note the NSS change also impacts chromium/google chrome.
Chrome's developers support the NSS change.
(In reply to comment #35) Christian: I am not familiar with how Mozilla does technical evangelism. In comment 22, I filed TE bug 583914 as bz suggested, and closed this bug WONTFIX. Then gavin reopened this bug in comment 27. My TE bug can't block a release, but the dummy shadow bug 584138 got closed as invalid. It seems that nothing will be done soon about my TE bug 583914 because it doesn't block any release.
Right, I don't expect the TE bug 583914 to be fixed or to block. It is my understanding this bug has engineering work that can be done from comment 32. We would like to get a patch for this so that we can ship it in 3.6.9. From a FF user's standpoint, this bug will be a regression once we ship next month (because we upgraded to this behavior, bug 575620). I would rather not revert back to an older version of NSS, but code freeze is tomorrow. I would be willing to push the freeze back for this bug if it gets some traction in the next day. Please let me know if this is not going anywhere and instead I should roll back nss.
Created attachment 465229 [details] [diff] [review] Workaround patch for mozilla-1.9.2, v2 For mozilla-1.9.2 only, I propose this patch. To review the NSS patch, verify that the two arrays of SSL cipher suites in ssl3con.c and sslenum.c are kept in the same order, as this comment notes: http://bonsai.mozilla.org/cvsblame.cgi?file=mozilla/security/nss/lib/ssl/ssl3con.c&rev=1.143&mark=103-104#101
Comment on attachment 465229 [details] [diff] [review] Workaround patch for mozilla-1.9.2, v2 I guess this is fine, though Nelson seemed to have some objections to this.... Were they weak enough that shipping a security release (or possibly all future 3.6.x releases) with this patch is ok?
I object to this because it takes DHE away from ALL Firefox users for the sake of few users who visit broken servers. A better fix is to enhance PSM's TLS intolerance fallback to also disable the DHE cipher suites. That's a very small change to PSM. I won't block this change if it is made ONLY in Mozilla's downstream Hg repository. I don't want to see it in CVS. SR- for the CVS tree!
Comment on attachment 465229 [details] [diff] [review] Workaround patch for mozilla-1.9.2, v2 a=LegNeato for 188.8.131.52. Please land this workaround on mozilla-1.9.2 default.
Comment on attachment 465229 [details] [diff] [review] Workaround patch for mozilla-1.9.2, v2 Pushed to mozilla-1.9.2 default: http://hg.mozilla.org/releases/mozilla-1.9.2/rev/68a4d8f0a18f
This was a Firefox 4 blocker as well, and it's not fixed there as far as I can see, only on the branch. So if this bug is getting marked fixed, I guess a new bug should be filed for doing the "better fix" for Firefox 4?
For Firefox 4 I won't work around the broken servers. I filed TE bugs for the broken servers, which are listed in the "Depends on" field. I propose that those TE bugs be Firefox 4 blockers.
We still need a bug for the better PSM fallback, right?
(In reply to comment #47) > For Firefox 4 I won't work around the broken servers. > I filed TE bugs for the broken servers, which are > listed in the "Depends on" field. I propose that > those TE bugs be Firefox 4 blockers. I'm not sure that's right. Doesn't the previous discussion indicate that the broken servers are likely to be a common pattern, and we should instead interpret the safest possible action? I agree that we should get a Firefox 4 bug on file to ensure we track this.
I filed bug 587407 on PSM.
What is the STR here? I note that http://www.enbridge.com/mypage loads without any errors in the nightly 184.108.40.206pre build and redirects to https://portal-plumprod.cgc.enbridge.com/portal/server.pt?space=CommunityPage&control=SetCommunity&cached=true&CommunityID=203&PageID=0 with no warning messages. I assume that there is probably more to test here or is there? This was tested using Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:220.127.116.11pre) Gecko/20100817 Namoroka/3.6.9pre ( .NET CLR 3.5.30729).
Al: the STR is: 1. Load http://www.enbridge.com/mypage in Firefox 4 Beta 3. You should get a "Secure Connection Failed" error page with the error code: ssl_error_rx_malformed_server_key_exch 2. Now load http://www.enbridge.com/mypage in the nightly 18.104.22.168pre build. You should not get this error page.
Yes, I already did that. I wanted to be sure that there is nothing more to do here. Marking verified for 1.9.2.
Comment on attachment 465229 [details] [diff] [review] Workaround patch for mozilla-1.9.2, v2 Firefox 3.5.12 also needs this workaround.
You missed the train for 3.5.12. We built today and are going to beta on Thursday barring rebuilds.
If we need to spin again for a regression I'll want this in, otherwise it'll get in .13
Why does 3.5 need this? Bug 575620 didn't land on 1.9.1, did it?
Comment on attachment 465229 [details] [diff] [review] Workaround patch for mozilla-1.9.2, v2 bz: Firefox 3.5 doesn't check ephemeral Diffie-Hellman key sizes. This patch prevents Firefox 3.5 from using ephemeral Diffie-Hellman cipher suites with the broken servers. Ideally we should also backport the key size checks: http://bonsai.mozilla.org/cvsview2.cgi?diff_mode=context&whitespace_mode=show&file=ssl3con.c&branch=&root=/cvsroot&subdir=mozilla/security/nss/lib/ssl&command=DIFF_FRAMESET&rev1=1.137&rev2=1.138 But this patch alone is good enough.
It's a bit confusing to be proposing that we take the patch on 1.9.1 in this bug - it's "necessary to ensure security" (because 1.9.1 doesn't check key sizes), which is different than the 1.9.2 patch, which is "necessary to avoid breaking sites due to extra security checks." But I agree that we should take it - perhaps file a new 1.9.1-only bug?
Yeah, it sounds like the 1.9.1 patch is fixing a different bug and should be tracked separately.
Yeah, talking with abillings today we realized this particular bug can't happen on 1.9.1 as it was a behavior change with the newer nspr, which didn't land on 1.9.1.
Sigh. One needs a strong will to persevere in this environment. Fact: 1.9.1 accepts tiny 256-bit Diffie-Hellman keys of the broken servers. Fact: 1.9.1 has been doing that since day one, so there was no behavior change. If you think this is worth fixing, I will follow the proper process and open a new 1.9.1-only bug.
Yes, it is worth fixing. Please do file the new bug (and CC me?).
We'll need this on the 1.9.1 branch since we'd like to upgrade to nss 3.12.8 there.
Comment on attachment 465229 [details] [diff] [review] Workaround patch for mozilla-1.9.2, v2 Approved for 22.214.171.124, a=dveditz for release-drivers (to be landed after upgrading to NSS 3.12.8)
Comment on attachment 465229 [details] [diff] [review] Workaround patch for mozilla-1.9.2, v2 Pushed to mozilla-1.9.1 in changeset 8ee042940966: http://hg.mozilla.org/releases/mozilla-1.9.1/rev/8ee042940966
Please make sure that this is seen as a kludge and a quick compatibility hack. This bug is definitely not fixed and the patch committed introduces behaviour that isn't desirable in the slightest and is regressive. It degrades security for Firefox users to workaround broken servers. Can this bug be reopened, therefore? The correct compatibility approach is to renegotiate on failure, precluding the use of a DHE based cipher suite on the second attempt. This as said in comment 32: https://bugzilla.mozilla.org/show_bug.cgi?id=583337#c32 DHE should be used in preference as it prevents retrospective decryption of intercepted and captured encrypted traffic. This decision, therefore, tangibly weakens the security of Firefox. That's bad. This was also said in comment 11: https://bugzilla.mozilla.org/show_bug.cgi?id=583337#c11 The fact that other browsers do, or might, not get this right does not make this the correct implementation decision. That's merely blind deference to the appeal to authority fallacy in any case; it isn't based in evidence or reason. Firefox should hold itself to a higher standard and lead the way, not tag along. The same arguments were made back in the day against introducing the use of TLS extensions as many servers broke with it. Finally, Microsoft forced the issue with the advent of IE7 and Vista and the world moved on and fixed its servers. http://blogs.msdn.com/b/wndp/archive/2006/04/12/tls-enabled-by-default.aspx http://blogs.msdn.com/b/ie/archive/2005/10/22/483795.aspx In abstract, I would certainly venture that security is really the wrong thing to be compromising against in such a way for compatibility purposes. That's moot in this case anyway as there is a correct approach here, yet it hasn't been implemented!?... Cheers, Nick
It's unlikely we are going to actually change FF 3.6's PSM. The discussion on the pros and cons of the different fix options are in bug 587407 .