1786492 - mp_invmod fails with negative modulus

Status: RESOLVED INVALID

(NSS :: Libraries, defect, P5)

Description

[Iaroslav Gridin]

User Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.5112.102 Safari/537.36

Steps to reproduce:

mp_invmod(-0x01,-0xcb7aaa)

Actual results:

MP_RANGE returned from s_mp_invmod_odd_m because modulus is < 2

Expected results:

0xcb7aa9 (as openssl computes it)

Comment 1

[John Schanck [:jschanck]]

I think it's reasonable for us to require positive m here---this isn't a general purpose arithmetic library and positive m is the only case we currently use inside of NSS. So my inclination is just to update the documentation. Is this something that came up in fuzzing? Or do you have a use case where negative values of m come up naturally?

Comment 2

[Iaroslav Gridin]

Yes, that came up in fuzzing and my main concern is that negative values are not rejected immediately with MP_RANGE but processed by algorithms that do not expect negative inputs and might behave unpredictably.

Comment 3

[Billy Brumley]

Hi Iaroslav,

John is correct---you are seeing the expected behavior here.

I know that, because I helped develop the algorithms in question in both libraries, as well as the unit tests for them. I discussed with the library developers at length, and understand very well the (expected) differences in behavior you're seeing. In this case, differential fuzzing is giving you a false positive.

So please close this out---it's not a bug.

OTOH, your other concern is indeed something worth looking into. Off the top of my head, I cannot think of any "mod" function in NSS where it would be sane to accept a non-positive integer. So if you find any of those, please go ahead and file bugs and fixes. And keep me in CC for all such reports.

Thanks,

BBB