259229 - TLS handshakes fail when server DHE params exceed NSS size limits

Status: RESOLVED FIXED

(NSS :: Libraries, defect, P2)

Description

[Andrey Jivsov]

User-Agent:       Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.6) Gecko/20040510
Build Identifier: Thunderbird version 0.7.3 (20040803)

Probably because of #define DH_MAX_P_BITS 1024, any client SSL/TSL handshake
that requires an ephemeral DH key generation will fail with error
SEC_ERROR_KEYGEN_FAIL (-8092).

By some estimates the strength of 1024 bit DH corresponds to 80 bit symmetrical
encryption. It is reasonable for the server to offer stronger DH in ciphersuites
such as TLS_DHE_RSA_WITH_AES_256_CBC_SHA. 

Client, such as Thunderbird 0.7.3, will select any DHE ciphersuite just to fail
later.

Reproducible: Always
Steps to Reproduce:
# On the same host where you run Thunderbird, do the following:

   openssl dhparam -out dhparam-2048 2048

# create file server.pem with proper content
# run the following: 

   openssl s_server -debug -accept 1993 -cipher EDH-RSA-DES-CBC3-SHA 

# Change IMAP server settings in Thunderbird to the localhost:1993, secure IMAP.
# Click on any IMAP folder, such as Inbox.   
# The Thunderbird will display certificate warnings and hang.
# openssl will wait after displaying ACCEPT line.
# This is a normal behaviour, everything is setup for the test


Actual Results:  
# now exit the openssl and run it again with the following addition: 

   openssl s_server -debug -accept 1993 \
      -cipher EDH-RSA-DES-CBC3-SHA -dhparam dhparam-2048

# Click on the IMAP folder in Thunderbird again. 
# This time the Thunderbird will display error -8092


Expected Results:  
The NSS should support at least 2048 bit DH key generation, consistent with the
strength of other algirithms it implements.

The fix is probably as easy as changing 
   #define DH_MAX_P_BITS 1024 
to 2048. 

If not, something else must be done to improve interoperability with correct
IMAPs/POPs (HTTPs too?) servers that are broken because of this issue

NSS seems the be the easiest place to fix this problem. Other workarounds
include scaling back support by the NSS client of stronger ciphersuites, such as
TLS_DHE_RSA_WITH_AES_256_CBC_SHA, to allow servers to limit 2048 EDH to these
ciphersuites.

Tested on Linux Redhat Fedora Core 2 and Win32 with Thunderbird and
openssl-0.9.7a-35. Also used another SSL server to verify the issue.

Comment 1

[Nelson Bolyard (seldom reads bugmail)]

The submittor wrote:
> By some estimates the strength of 1024 bit DH corresponds to 80 bit 
> symmetrical encryption.

I'd like to see those estimates, and the basis for them.  

The present limits were imposed in NSS 3.9.  NSS must limit the sizes of 
the keys and parameters it will accept, in order to limit its vulnerability 
to peers that use escessive key sizes.  This is particularly true for servers.  
There will always be some (few) servers in the wild that use keys that 
exceed those limits, and NSS-based products will not interoperate with them.
The question is whether those excessive key/prime sizes actually provide
extra security, or merely appeal to "mine is bigger than yours" mentalities.

Note that similar limits are found in DSS/DSA.  No-one has suggested that
1024-bit primes are insufficient for DSA, AFAIK, and indeed DSA does not 
require "strong" (e.g. sophie-germain) primes.

If a consensus can be shown among leading cryptopgrahers that 1k bit primes
are insufficient for DHE (whose purpose is PFS), then I'm happy to raise
the limit.  Otherwise, I do not think it is "major" that NSS fails to 
interoperate with a few servers that use unusually large DHE params.

Comment 2

[Andrey Jivsov]

Nelson, you are right that many authentication certificates in use today use 
relatively small key sizes. However, note that one of main purposes of EDH is 
to protect from passive attacker (i.e. just eavesdropper), and the strength of 
DSS or RSA for server authentication is irrelevant here. Active attack is less 
likely than passive and so EDH should (or at least can) be stronger than DSS 
or RSA authentication. 

Regarding relative key sizes, check out number of drafts, such as ECC TLS 
[http://www.ietf.org/internet-drafts/draft-ietf-tls-ecc-06.txt], which refers 
to "Selecting Cryptographic Key Sizes" in Journal of Cryptology 2001 
[http://security.ece.orst.edu/koc/ece575/papers/cryptosizes.pdf]. 

The problem is that current NSS implementation is breaking interoperability 
with correct (IMHO) SSL servers. It maybe because Thunderbird doesn't 
correctly handles limitations of NSS, but some module is not "tolerant in what 
you receive". 

Essentially, if this bug will be discarded, it will mean that that no TLS 
server is allowed to have anything stronger than 1024 EDH to communicate with 
NSS clients. It is up to the server implementation to choose to use EDH in the 
vicinity of 2048 bits with AES256 (let's say they have read this paper above). 
This will greatly limit the acceptance of Thunderbird or other NSS clients.

Comment 3

[Andrey Jivsov]

http://www.faqs.org/rfcs/rfc3526.html defines IKE groups from 1536 bit. Please 
check Introduction for the discussion on relative strength. I am not arguing 
that we should use 15400 bit DH with AES 256. Rather, I am saying that it is 
perfectly fine to use EDH 2048 in this case. 

I hope that that the max DH size in NSS should be raised. If needed, 
additional mechanisms can be developed to limit acceptable DH key size for 
some of symmetrical algorithms. 

Comment 4

[Nelson Bolyard (seldom reads bugmail)]

Thanks for reminding me of Lenstra's paper.  I had not read it in about 
4 years, and it obviously slipped my mind.  :-(  I find it more persuasive
than the higher key size estimates appearing in some of the other IETF
documents you cited.  However, I also observe that it appears that NIST
is using different estimates than any of the ones cited above.

Regarding the limits we imposed in NSS 3.9, I observe that the limits we 
imposed on DH were relatively much lower than the limits we imposed on RSA.  
We imposed limits that were effectively 1x currect practice for DH, and 4x
current practice on RSA (IIRC).  so I think it is not unreasonable to raise 
the limit for DH prime size.

According to Lenstra's table, a prime size of 2236 bits matches the
current discrete log size limit of 160 bits, so I would consider
raising DH_MAX_P_BITS to perhaps as much as 2236, but not more. 

Here's a chance to contribute to mozilla.  If you will setup a server,
as you suggested above, with a DH param P of 2236 bits, and email me
a URL for it (or put the URL in this bug), I will test against it.  

Comment 5

[Nelson Bolyard (seldom reads bugmail)]

Julien, do you have any input on the maximum size of DH params acceptable to 
the server products you support (including when they are acting as clients)?

Comment 6

[Nelson Bolyard (seldom reads bugmail)]

patch tested.  patch forthcoming.

Comment 7

[Nelson Bolyard (seldom reads bugmail)]

Attachment: patch v1

As the submittor expected, and as I had hoped, changing that one #define
to a higher number solved the problem.

Comment 8

[Nelson Bolyard (seldom reads bugmail)]

Comment on attachment 159064 [details] [diff] [review]
patch v1

Wan-Teh, please review for NSS 3.9.3

Comment 9

[Wan-Teh Chang]

Comment on attachment 159064 [details] [diff] [review]
patch v1

r=wtc.

Comment 10

[Nelson Bolyard (seldom reads bugmail)]

Checked in on 3.9 branch
Checking in freebl/blapit.h;  new revision: 1.12.16.1; previous revision: 1.12

Comment 11

[Nelson Bolyard (seldom reads bugmail)]

Checked in on trunk.
Checking in blapit.h;  new revision: 1.15; previous revision: 1.14

Andrey,  Thanks very much for reporting this bug, and for running a test
server to facilitate interoperability testing of the fix.  That's a very
useful contribution to NSS!  

Comment 12

[Asa Dotzler [:asa]]

Nelson, is this going to be fixed by the new NSS we just got for Seamonkey and
will get shortly for the aviary and 1.7.x branches?

Comment 13

[Wan-Teh Chang]

Asa, yes, this will be fixed by the new NSS 3.9.3.

Comment 14

[Peter van der Woude [:Peter6]]

1. shouldn't this be marked RESOLVED/WORKSFORME ?
2. shouldn't the ?1.0 flag be removed ?

Comment 15

[Nelson Bolyard (seldom reads bugmail)]

A bug was found and fixed, therefore IMO resolved/fixed is more appropriate
than resolved/worksforme.

The blocking 1.0 flag is there to document a problem with the way these
flags are being handled, or rather, ignored.  We discovered that the 
drivers are not noticing bugs like this one that are clearly flagged as
blocking aviary 1.0.  That needs to get fixed.  I want to leave this flag
there until some driver notices it.

Comment 16

[Jean-Marc Desperrier]

Also, NSS bugs are marked as fixed not when they are checked in NSS, but when
they can be verified in Mozilla. And Mozilla does not use the NSS head, but a
tag that's a bit older : NSS_CLIENT_TAG (which does include this patch as of now).

Comment 17

[Nelson Bolyard (seldom reads bugmail)]

Actually, Jean Marc, NSS bugs ARE marked fixed when they are checked into 
the trunk on NSS.  

Comment 18

[Benjamin Smedberg]

NSS 3.9.3 is on aviary/1.7