325495 - Crash verifying secp521r1 ECDSA signature

Status: VERIFIED DUPLICATE of bug 319252

(NSS :: Libraries, defect)

Description

[Andreas Sterbenz]

I am using NSS 3.11 plus ECC patch from https://bugzilla.mozilla.org/show_bug.cgi?id=236245 .

All other curves work fine, but when verifying CKM_ECDSA_SHA1 signatures using secp521r1 (aka NIST P-521) or generating such keypairs, I get a crash once every 3 runs or so. It appears to be caused by memory corruption. pstack of core below:

===
 feee0717 _lwp_kill (2, 6) + 7
 fee8ced3 raise    (6) + 1f
 fef85f79 umem_do_abort (2, fefa5000, fe5ca8b4, fef87b8a, fef92f3c, fefa9a00) + 25
 fef860a9 umem_err_recoverable (fef92f3c, fefa9a00, fe5cacbc, fef92f48) + 46
 fef87b8a process_free (fe5cacbc, 1, 0, fe5ca8e4, d6778f17, fe5cacbc) + 82
 fef87c2f free     (fe5cacbc) + 14
 d6778f17 s_mp_free (fe5cacbc) + 4f
 d6778c8c s_mp_grow (fe5cb4bc, 40) + c4
 d6778d1a s_mp_pad (fe5cb4bc, 23) + 52
 d6779282 s_mp_mul_2d (fe5cb4bc, 220) + ca
 d67713f1 mpl_lsh  (fe5cb780, fe5cb4bc, 220) + b1
 d6789e23 ec_GFp_enc_mont (fe5cb780, fe5cb4bc, 823a9d0) + 4b
 d67802f8 ec_pts_mul_simul_w2 (fe5cb864, fe5cb854, fe5cb780, fe5cb770, fe5cb760, fe5cb750) + 3b8
 d6780eb3 ECPoints_mul (846fd48, fe5cb864, fe5cb854, fe5cb780, fe5cb770, fe5cb760) + 1f3
 d6761278 ec_points_mul (83d75a0, fe5cb864, fe5cb854, 83d7624, fe5cb7c4) + 3b0
 d6763335 ECDSA_VerifyDigest (83d75a0, fe5cb908, fe5cb8fc) + 66d
 d68e9184 ECDSA_VerifyDigest (83d75a0, fe5cb908, fe5cb8fc) + 74
 d68cf369 nsc_ECDSAVerifyStub (83d7598, 846fb68, 84, d692c9d0, 14) + 59
 d68d135b NSC_Verify (2, d692c9d0, 14, 846fb68, 84) + 7b
 d68d321f sftk_PairwiseConsistencyCheck (2, 81a0008, 83d5008, 3) + 4bf
 d68d4649 NSC_GenerateKeyPair (2, fe5cbaf4, 80db0c8, 2, 8178388, 3) + 1369
 d6b1616b Java_sun_security_pkcs11_wrapper_PKCS11_C_1GenerateKeyPair (80a84e4, fe5cbb9c, 2, 0, fe5cbb90, fe5cbb8c) + bb
[...]
===

This is using LD_PRELOAD /usr/lib/libumem.so, but the crash also happens with regular malloc.

On a hunch, I changed ECL_MAX_FIELD_SIZE_DIGITS in ecl-priv.h from 10/20 (for 64/32 bits) to 20/40 and the crash went away. But I am in no way certain that this is the correct fix.

Comment 1

[Wan-Teh Chang]

I believe this is a duplicate of bug 319252.  Andreas,
please apply the patch in bug 319252.  If it fixes this
crash, please mark the bug VERIFIED, otherwise reopen
the bug.

For your work, please use NSS_3_11_BRANCH instead of
NSS_3_11_RTM plus the patch from bug 236245.  That
patch has been checked in on the NSS_3_11_BRANCH.


*** This bug has been marked as a duplicate of 319252 ***