1482386 - Incorrect handling of malformed and invalid signature_algorithms extension

Status: NEW

(NSS :: Libraries, enhancement, P3)

Description

[Hubert Kario]

When NSS receives a ClientHello with malformed signature_algorithms extension, it  does not handle it correctly.
In TLS 1.3:
 * it does not abort the connection when the ClientHello includes only rsa_pkcs1 signature algorithms
 * It does abort the connection with a wrong alert (handshake_failure instead of decode_error) if the extension or the array inside it has length 0

in TLS 1.2 only the second issue is present.

Reproducer:
git clone https://github.com/tomato42/tlsfuzzer.git
pushd tlsfuzzer
git clone https://github.com/warner/python-ecdsa .python-ecdsa
ln -s .python-ecdsa/ecdsa ecdsa
git clone https://github.com/tomato42/tlslite-ng.git .tlslite-ng
ln -s .tlslite-ng/tlslite tlslite
popd

openssl req -x509 -newkey rsa -keyout localhost.key -out localhost.crt -nodes -batch -subj /CN=localhost
openssl pkcs12 -export -passout pass:  -out localhost.p12 -inkey localhost.key -in localhost.crt
mkdir nssdb
certutil -N -d sql:nssdb --empty-password
pk12util -i localhost.p12 -d sql:nssdb -W ''
selfserv -n localhost -p 4433 -d sql:./nssdb -V tls1.0: -H 1 -n localhost -u

# in another terminal, same directory
PYTHONPATH=. python scripts/test-tls13-signature-algorithms.py
PYTHONPATH=. python scripts/test-signature-algorithms.py


Result:
(...)
Signature Algorithms in TLS 1.3
Check if valid signature algorithm extensions are accepted and
invalid properly rejected by the TLS 1.3 server.

version: 1

Test end
successful: 268
failed: 3
  'empty list of signature methods'
  'fuzz length inside extension to 0'
  'only legacy sigalgs'
(...)
Test end
successful: 267
failed: 2
  'empty list of signature methods'
  'fuzz length inside extension to 0'

Comment 1

[Hubert Kario]

sorry, the above is with version 560be4656a89 (3.39 in devel)