1535630 - Wrong alert sent when client uses PKCS#1 signatures in TLS 1.3

Status: NEW

(NSS :: Libraries, defect, P5)

Description

[Alicja Kario]

When the connection negotiates TLS 1.3 but the client signs the Certificate Verify message with sha256, sha384 or sha512 hash with PKCS#1 v1.5 padding, the message is rejected with "decrypt_error" alert instead of the "illegal_parameter".
The correct alert is used for md5, sha1 or sha224 hashes.

This is with 3.44-dev tree (03d7bcade60a)

Reproducer:

1. set up server with client certificate support and RSA key:
   selfserv -d sql:/tmp/nssdb -p 4433 -V tls1.0: -H 1 -n server -u -r
2. run tlsfuzzer test:
   PYTHONPATH=. python scripts/test-tls13-certificate-verify.py -k /tmp/client/key.pem -c /tmp/client/cert.pem -n 10 -s "sha256+ecdsa sha384+ecdsa sha512+ecdsa sha1+ecdsa rsa_pss_rsae_sha256 rsa_pss_rsae_sha384 rsa_pss_rsae_sha512 rsa_pkcs1_sha256 rsa_pkcs1_sha384 rsa_pkcs1_sha512 rsa_pkcs1_sha1 sha256+dsa sha384+dsa sha512+dsa sha1+dsa"

Result:
3 tests failed:

'check rsa_pkcs1_sha256 signature is refused'
'check rsa_pkcs1_sha384 signature is refused'
'check rsa_pkcs1_sha512 signature is refused'

tlsfuzzer output:

check rsa_pkcs1_sha256 signature is refused ...
Error encountered while processing node <tlsfuzzer.expect.ExpectAlert object at 0x7f3050059d90> (child: <tlsfuzzer.expect.ExpectClose object at 0x7f3050059dd0>) with last message being: <tlslite.messages.Message object at 0x7f304ef95f50>
Error while processing
Traceback (most recent call last):
  File "scripts/test-tls13-certificate-verify.py", line 583, in main
    runner.run()
  File "/home/hkario/dev/tlsfuzzer/tlsfuzzer/runner.py", line 227, in run
    node.process(self.state, msg)
  File "/home/hkario/dev/tlsfuzzer/tlsfuzzer/expect.py", line 1412, in process
    raise AssertionError(problem_desc)
AssertionError: Expected alert description "illegal_parameter" does not match received "decrypt_error"

selfserv output:

selfserv: HDX PR_Read returned error -8182:
Peer's certificate has an invalid signature.