1119983 - NSS TLS 1.2 server fails to interoperate with LibreSSL client

Status: RESOLVED FIXED

(NSS :: Libraries, defect)

Description

[Chris Newman]

When NSS is used as a TLS 1.2 server, it does not interoperate with the LibreSSL client due to two bugs in NSS TLS 1.2 implementation of the signature_algorithms extension. First, the matching for signature_algorithms uses the wrong number space as the SSL3SignatureAndHashAlgorithm structure uses a SECOidTag and the hashPreference array is using TLS hash identifier namespace in ssl3con.c. Second, the parser for signature_algorithms in ssl3_ServerHandleSigAlgsXtn handles the SEC_OID_UNKNOWN case incorrectly leaving uninitialized variables in the list.

Because SEC_OID_SHA1 is 4 and tls_hash_sha256 is also 4, the first bug is usually not noticed, but LibreSSL uses a private-use hash identifier and puts SHA1 at the end, so the second bug causes the first bug to manifest and no matching hash algorithm is found. This results in a bogus SSL_ERROR_UNSUPPORTED_HASH_ALGORITHM error from NSS server.

Comment 1

[Chris Newman]

Attachment: proposed patch to fix the problem

Comment 2

[Chris Newman]

Thanks for the review.

Comment 3

[Chris Newman]

Kai, can you help get this fix landed? This is my first time trying to fix a bug in NSS so feel free to point me to RTFM if I did something wrong. Thanks!

Comment 4

[Kai Engert [:KaiE:]]

https://hg.mozilla.org/projects/nss/rev/72b9d7234fa8

Comment 5

[Kai Engert [:KaiE:]]

mass change target milestone to 3.17.4