1835425 - RSA overflow check fails with really small modulus

Status: RESOLVED FIXED

(NSS :: Libraries, defect)

Description

[Iaroslav Gridin]

Steps to reproduce:

call RSA_Sign with modulus size less than (3 + RSA_BLOCK_MIN_PAD_LEN)

Actual results:

heap overflow and memory overwrite. Security threat is not significant I think since that modulus is not normal.

Expected results:

check in rsa_FormatBlock worked and returned SECFailure

Comment 1

[Iaroslav Gridin]

Attachment: Bug 1835425: Improve the length check of RSA input to avoid heap overflow

In its current state, signed modulusLen overflows when it is less than
RSA_BLOCK_MIN_PAD_LEN + 3, the check fails and this leads to an
overflow in rsa_FormatOneBlock().

Comment 2

[John Schanck [:jschanck]]

I'm flagging this as security sensitive until we have a chance to review it in detail.

Comment 3

[Daniel Veditz [:dveditz]]

John and Anne said this shouldn't affect Firefox because we impose a minimum key length for RSA keys

Comment 4

[Anna Weine]

I am still leaving the security sensitive as installed for now.

Comment 5

[Anna Weine]

https://hg.mozilla.org/projects/nss/rev/639b0d5172bb934fd020e8241dbfa8942a4e5794

Comment 6

[Daniel Veditz [:dveditz]]

Sorry for the burst of bugspam: filter on tinkling-glitter-filtrate
Adding reporter-external keyword to security bugs found by non-employees for accounting reasons