297327 - Can't interoperate with FIPS-compliant server

Status: RESOLVED WONTFIX

(Core :: Security: PSM, defect)

Description

[Ben Laurie]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.8) Gecko/20050511 Firefox/1.0.4
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.8) Gecko/20050511 Firefox/1.0.4

I am running an Apache 2 server with OpenSSL's experimental FIPS mode enabled.
As a result, MD5 is disabled. If I attempt to connect with Firefox, I get an
error -8182 (certificate invalid or corrupted). The server end shows:

[Fri Jun 10 16:23:09 2005] [info] SSL Library Error: 705052772
error:2A064064:FIPS routines:HASH_FINAL:non fips method
[Fri Jun 10 16:23:09 2005] [info] SSL Library Error: 336151568
error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure

Also, if I try to _just_ use TLS, I am told that SSL is not enabled (not on
other websites, only this one).

OpenSSL's command-line client can connect, no problem (the non-FIPS version of
OpenSSL).

Reproducible: Always

Comment 1

[Kai Engert (:KaiE:)]

Did you enable fips mode in the browser?
Not sure if that would be required to connect to a FIPS server.

Comment 2

[bugzilborn]

I am experiencing the "flip side" of this issue. I have not been able to find any SSL/TLS web server on the 'net that will allow an https connection with Mozilla/Firefox client when only the fips cipher suites are enabled in the client.  The client is in FIPS mode with SSL2 disabled.  For example, the server will complete the SSL handshake when the only enabled cipher suite is ssl3.rsa_des_ede3_sha,  fails when only ssl3.rsa_fips_des_ede3_sha is enabled.

I am not sure if this is a client bug, or I just have not found a properly configured server.  However, since TLS does not specify these two client selections as different ciphers, it is not clear why it should fail for the fips selection, unless it is a client bug.

Comment 3

[Nelson Bolyard (seldom reads bugmail)]

I think this bug should be resolved WONTFIX.  
Kai, if you concur, please mark this bug appropriately.

Apparently, OpenSSL had a FIPS mode that disabled MD5 alltogether,
including making it unavailable for use in TLS 1.0's PRF.
Such a mode is completely non-interoperable with TLS 1.0, and 
consequently, NSS will not interoperate with it.

The TLS Working Group is now working on defining a new standard version 
of TLS that will be able to be completely FIPS compliant, having both 
cipher suites AND KDFs that do not rely on MD5 at all.  When that standard emerges, you can expect that NSS will support it soon thereafter, I believe.

NIST has published a document about TLS and FIPS compliance.  It is
http://csrc.nist.gov/publications/nistpubs/800-52/SP800-52.pdf 
It contains a table 3, that lists all the FIPS compliant TLS cipher suites.
Persons wanting to be FIPS compliant should configure their software to:
a) Enable only TLS, disabling SSL 2 and SSL 3.0
b) Enable only cipher suites that are named in table 3 of SP800-52. 
Here is the list of cipher suites named in that table:

TLS_DHE_DSS_WITH_AES_256_CBC_SHA
TLS_DHE_RSA_WITH_AES_256_CBC_SHA
TLS_RSA_WITH_AES_256_CBC_SHA
TLS_DH_DSS_WITH_AES_256_CBC_SHA
TLS_DH_RSA_WITH_AES_256_CBC_SHA
TLS_DHE_DSS_WITH_AES_128_CBC_SHA
TLS_DHE_RSA_WITH_AES_128_CBC_SHA
TLS_RSA_WITH_AES_128_CBC_SHA
TLS_DH_DSS_WITH_AES_128_CBC_SHA
TLS_DH_RSA_WITH_AES_128_CBC_SHA
TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA
TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA
TLS_RSA_WITH_3DES_EDE_CBC_SHA
TLS_DH_DSS_WITH_3DES_EDE_CBC_SHA
TLS_DH_RSA_WITH_3DES_EDE_CBC_SHA

Note that NSS may not support all of the above-named cipher suites.

Comment 4

[Ben Laurie]

OpenSSL has never disabled MD5 altogether - as you point out, that breaks TLS.

In FIPS mode there is an exception for TLS allowing MD5 to be used.

So, in short, I don't agree this should be WONTFIX.

Comment 5

[Nelson Bolyard (seldom reads bugmail)]

Ben, Have you got output from ssldump or ssltap?
DO you have a test server setup running the FIPS-mode OpenSSL?
Did you perhaps use a server cert with an RSA-MD5 signature?
Did the server perhaps negotiate a non-FIPS cipher suite?
(e.g. one with RC4 or MD5, or some export cipher suite)?

Comment 6

[Ben Laurie]

I do not have any dumps, nor do I have test servers running anymore, and nor do I know what the cert was, sorry.

This was reported 18 months ago, after all!

Comment 7

[Nelson Bolyard (seldom reads bugmail)]

Ben, Is www.openssl.org operating in OpenSSL's FIPS mode?

This bug report claims interoperability problems that occur when an OpenSSL 
server is placed into OpenSSL's FIPS mode.  I was able to reproduce errors
very similar to those described in comment zero by visiting https://www.openssl.org/ with yesterday's trunk browser build, with SSL 3.0
disabled, and only TLS enabled.

I found two problems, which I have reported in bug 368126 and bug 368130.
However, neither bug has anything to do with any FIPS mode or with the use
of FIPS-compatible cipher suites.  The problems there were a result of the
site's self-issued cert.  Apparently the browser times out the connection
while the user is looking at bad cert dialogs.  I found that if I dismiss
(OK) the bad cert dialogs QUICKLY (within ~5 seconds) the first time that
they appear for that server (after the browser is started), the page then 
loads and displays OK after that.  

I cound not find any interoperability problem between NSS and 
www.openssl.org using TLS and AES cipher suites with tstclnt, NSS's test 
TLS client program.  

So, I think it unlikely that there are any problems here whose cause is 
that the OpenSSL server is in FIPS mode.  If there were, I would say that
the problem must be in the OpenSSL server.  After all, the TLS protocol 
does not have a FIPS mode.  As seen over the wire, a server in "FIPS mode" 
behaves no differently than an ordinary TLS server configured to use only
TLS version 1.0 and only the 3DES and AES cipher suites.  A remote client
cannot tell the two apart, and has no reason to treat either one differently
than the other.  

Comment 8

[Bob Lord]

Is this report still valid, or should it be closed?

Comment 9

[Kai Engert (:KaiE:)]

Mass change owner of unconfirmed "Core:Security UI/PSM/SMime" bugs to nobody.
Search for kaie-20100607-unconfirmed-nobody

Comment 10

[Kai Engert (:KaiE:)]

I apologize that we haven't been able to work on a resolution.
The issue seems to be outdate and impossible to test.
Closing as wontfix as suggested by Nelson in an earlier comment.