1268235 - SEC_ERROR_BAD_SIGNATURE when a RSA certificate has a very large public exponent

Status: NEW

(NSS :: Libraries, defect, P3)

Description

[me]

Attachment: A completely valid certificate for the domain wiener.martincernac.cz, which gets dismissed by the NSS (and by extension Firefox) as a certificate wih "an invalid signature"

User Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0
Build ID: 20160414065514

Steps to reproduce:

As a part of my bachelor thesis, I crafted a RSA certificate (which I got signed by StarCom) susceptible to Wiener's Continued fractions attack. The only outstanding thing about the certificate is a very large public exponent (almost the size of the modulus). The NSS fails to verify the signature of said certificate.

Steps to reproduce:
* either use Firefox to browse to https://wiener.martincernac.cz
OR
* ./vfyserv wiener.martincernac.cz

I'm attaching the certificate for the sake of completion.


Actual results:

Firefox fails at "SEC_ERROR_BAD_SIGNATURE"

vfyserv wiener.martincernac.cz outputs the following:
Connecting to host wiener.martincernac.cz (addr 185.8.236.32) on port 443
Error in function PR_Write: -8182
 - Peer's certificate has an invalid signature.

Although, if vfyserv is run with -c option and vfychain is run on the dumped certificates, all passes OK.


Expected results:

NSS should be able to work with a non-standard (!=65537), large public exponent, or at least inform the user of some internal limitation.

Firefox shouldn't completely bail out on the user, the certificate is completely valid. (Said certificate works fine in Google Chrome, Android browser, the openssl s_client utility, wget and many others)