1155922 - NSS should offer SHA512 as a supported signature_algorithm in TLS client hello

Status: RESOLVED FIXED

(NSS :: Libraries, defect, P1)

Description

[Kai Engert (:KaiE:)]

We're seeing a failure to connect using NSS with TLS 1.2 to a MS server.
After sending the client hello, the server immediately disconnects.

The server's certificate is signed using sha512WithRSAEncryption.

After some testing, I experimented by adding 
        tls_hash_sha512, tls_sig_rsa,
        tls_hash_sha512, tls_sig_ecdsa,
to the array in function ssl3_ClientSendSigAlgsXtn.

With that tweak, the server was willing to continue the connection.
I believe the addition of tls_sig_rsa was sufficient, because the server selected a RSA cipher suite.

I have already discussed with Wan-Teh.

He said, either Adam Langley or Wan-Teh may have omitted the sha512 entries from signature_algorithms to make ClientHello shorter, and they expected CA certificates to use SHA384 instead of SHA512 signatures.

The NSS code might already support this data, and in my testing, the handshake with the server succeeds.

Comment 1

[Kai Engert (:KaiE:)]

External references:
https://bugzilla.redhat.com/show_bug.cgi?id=1212286
https://code.google.com/p/chromium/issues/detail?id=461391

Comment 2

[Kai Engert (:KaiE:)]

For the record:

I see that both OpenSSL and GnuTLS offer:
SHA512+DSA, SHA384+DSA, SHA224+RSA, SHA224+DSA, SHA224+ECDSA
in their handshake, too.

On top of that, only OpenSSL offers SHA384+DSA, SHA512+DSA, MD5+RSA.

Comment 3

[Kai Engert (:KaiE:)]

Attachment: patch v1

If you still would like to keep it as small as possible, we could use this patch.

Let me know if you think we should avoid running into this error again with other combinations, and if you think should add more.

Comment 4

[Wan-Teh Chang]

Comment on attachment 8594312 [details] [diff] [review]
patch v1

r=wtc. Thanks.

Comment 5

[Kai Engert (:KaiE:)]

https://hg.mozilla.org/projects/nss/rev/342f7e837802