Closed Bug 335016 Opened 15 years ago Closed 13 years ago

mpp_pprime (Miller-Rabin probabilistic primality test) may choose 0 or 1 as the random integer


(NSS :: Libraries, defect, P1)



(Not tracked)



(Reporter: wtc, Assigned: nelson)



(Whiteboard: FIPS)


(1 file)

mpp_pprime implements the Miller-Rabin probabilistic
primality test.  This algorithm is documented in
1. Handbook of Applied Cryptography (HAC), Algorithm 4.24
on page 139.
2. FIPS 186-2, Appendix 2.1 on page 13.

This algorithm has an outer loop and an inner loop.
It runs for outer loop for nt iterations, where nt
is the second parameter to mpp_pprime.

In each iteration of the outer loop, we choose a
random integer x.  The range of x is specified as
  2 <= x <=  a-2  (HAC), or
  2 <= x <=  a-1  (FIPS 186-2)
where 'a' is the integer to be tested for primality
(the first parameter to mpp_pprime).

Ignoring the a-2 vs. a-1 issue, we know that x cannot
be 0 or 1.  But mpp_pprime chooses x as follows:

    MP_CHECKOK( mp_mod(&x, a, &x) );

So x may be 0 or 1, even though the probability is
extremely low, <= (4/a)*nt.

Let's see what the consequences are if x is 0 or 1.

1. If x = 0, we initialize z = (x ** m) mod a = 0.
Then in each iteration of the inner loop, we let
z = (z ** 2) mod a = 0.  So z stays 0.  When the
inner loop ends, we report that a is NOT prime.
This could be a false composite.  The Miller-Rabin
primality test is not supposed to return false
composite.  For our purposes, getting a false
composite seems fine.

2. If x = 1, we initialize z = (x ** m) mod a = 1.
So we immediately go on to the next iteration of the
outer loop.  The consequence is that we "waste" an
iteration of the outer loop.  This seems fine.
Whiteboard: FIPS
Priority: -- → P4
Blocks: FIPS2008
Assignee: nobody → nelson
Priority: P4 → P2
Target Milestone: --- → 3.12.3
Marking P1. Request fix by Nov 17th for FIPS.
Priority: P2 → P1
My take on this is that as long as we don't ever accept 0 or 1 as a prime,
we're OK.  So, I'm not convinced this needs to be fixed. 

Some things we could do about it include:
a) set the bit whose value is 2 (1<<1) to 1 in the "random" number
b) compute a' = a - 3, then compute x = (random % a') + 2
c) test initial random value, and if less than 2, get a new random value

Of those, (a) is a single operation, non-iterative, can't fail, efficient.
(b) requires storage for one more bignum, and one extra computation. 
    (hmm, what are the bounds on the modulus a?  Could it be less than 4?)
(c) requires two extra ops (compare, conditional jump), and potentially an
extra iteration of the PRNG.  It seems the most expensive.
Would a patch like this be acceptable?

+   mpl_set_bit(&x, 1, 1);
    MP_CHECKOK( mp_mod(&x, a, &x) );

That mpl call can't fail because every mp_int always has at least one 
mp_digit, the least significant digit, even if it is zero.  But you can
be extra safe by coding it as 

+   MP_CHECKOK( mpl_set_bit(&x, 1, 1) );
    MP_CHECKOK( mp_mod(&x, a, &x) );
(I'm fine with marking this bug WONTFIX.  Might want to add
my findings in comment 0 as a comment to the function to indicate
that we considered this issue.)

This patch implements (c).  It relies on the existing outer for
loop for retrying if x <= 1.

Ignoring computational cost, I think (b) is the best solution.

(a) may have unknown consequences because the 2 (1 << 1) bit of
x will always be set.  I'd rather avoid that risk.
Attachment #348272 - Flags: review?(nelson)
Attachment #348272 - Flags: review?(nelson) → review+
Comment on attachment 348272 [details] [diff] [review]
Proposed patch for (c)

I checked in the patch on the NSS trunk (NSS 3.12.3).

Checking in mpprime.c;
/cvsroot/mozilla/security/nss/lib/freebl/mpi/mpprime.c,v  <--  mpprime.c
new revision: 1.24; previous revision: 1.23
Closed: 13 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.