1481873 - NSS server fails DHE-DSA handshake when DSA algorithm is late in signature algorithms extension

Status: RESOLVED FIXED

(NSS :: Libraries, defect)

Description

[Alicja Kario]

When the DSA signature algorithm is at the 16th position or later, the connection is rejected by selfserv with handshake_failure alert and following message:

selfserv: HDX PR_Read returned error -12153:
The peer used an unsupported combination of signature and hash algorithm.

Reproducer:
openssl dsaparam -out dsaparam.pem 2048
openssl req -x509 -newkey dsa:dsaparam.pem -keyout /tmp/localhost.key -out /tmp/localhost.crt -subj /CN=localhost -nodes -batch
openssl pkcs12 -export -passout pass: -out /tmp/localhost.p12 -inkey /tmp/localhost.key -in /tmp/localhost.crt -name localhost
mkdir /tmp/nssdb
certutil -N -d sql:/tmp/nssdb --empty-password
pk12util -i /tmp/localhost.p12 -d sql:/tmp/nssdb -W ''

selfserv -d sql:/tmp/nssdb -p 4433 -V tls1.0: -H 1 -S localhost -u -c :0032


Run current master openssl with following command:
openssl s_client -connect localhost:4433 -no_tls1_3 -cipher DHE-DSS-AES128-SHA -servername localhost -sigalgs SHA1+RSA:SHA224+RSA:SHA384+RSA:SHA512+RSA:SHA1+ECDSA:SHA224+ECDSA:SHA256+ECDSA:SHA384+ECDSA:SHA512+ECDSA:ed25519:ed448:rsa_pss_pss_sha256:rsa_pss_pss_sha384:rsa_pss_pss_sha512:rsa_pss_rsae_sha256:SHA256+DSA

it fails.

Run the following command, and the connection is successful:
openssl s_client -connect localhost:4433 -no_tls1_3 -cipher DHE-DSS-AES128-SHA -servername localhost -sigalgs SHA1+RSA:SHA224+RSA:SHA384+RSA:SHA512+RSA:SHA1+ECDSA:SHA224+ECDSA:SHA256+ECDSA:SHA384+ECDSA:SHA512+ECDSA:ed25519:ed448:rsa_pss_pss_sha256:rsa_pss_pss_sha384:rsa_pss_pss_sha512:SHA256+DSA:rsa_pss_rsae_sha256

Comment 1

[Daiki Ueno [:ueno]]

Attachment: Bug 1481873, correct signature_algorithms extension handling

This fixes a couple of issues in signature_algorithms extension handling:
- MAX_SIGNATURE_SCHEMES is out of sync with ssl_IsSupportedSignatureScheme()
- when the extension consists of many bogus/duplicate entries followed by a valid signature scheme, ssl_ParseSignatureSchemes() gives up too early

Comment 2

[Alicja Kario]

verified with tlsfuzzer test-tls13-signature-algorithms.py and test-signature-algorithms.py that indeed long lists are now accepted by server

but there two few minor issues:
 * when the length of extension is 0, it is not rejected by server with correct alert (decode_error)
 * when the extension has only legacy (pkcs#1) signature algorithms, the server does not abort the connection

should I file a new bug for this?

Comment 3

[Martin Thomson [:mt:]]

Comment on attachment 8998834 [details]
Bug 1481873, correct signature_algorithms extension handling

Martin Thomson [:mt:] has approved the revision.

Comment 4

[Daiki Ueno [:ueno]]

Pushed as:
https://hg.mozilla.org/projects/nss/rev/560be4656a89

(In reply to Hubert Kario from comment #2)

> should I file a new bug for this?

Sure.

Comment 5

[Alicja Kario]

(In reply to Daiki Ueno [:ueno] from comment #4)
> (In reply to Hubert Kario from comment #2)
> > should I file a new bug for this?
> Sure.

Filed bug 1482386