436748 - mp_sqrt returns wrong result when argument is an even power of 2

Status: RESOLVED DUPLICATE of bug 1194073

(NSS :: Libraries, defect, P3)

Description

[Nelson Bolyard (seldom reads bugmail)]

When mp_sqrt is passed a number that is an even power of two, which is a 
perfect square, it returns an incorrect result.  It returns a number that 
is exactly 1 less than the actual square root.

I don't yet know if the problem occurs for all perfect squares, or only for
even powers of 2.

I don't know if we actually use mp_sqrt anywhere in freebl.  
If not, then this bug is minor or trivial.
If we do, then the severity of this bug needs to be assessed in light
of how it is used.

I have a patch in mind.  It's pretty easy to detect the perfect square
case.

Comment 1

[Wan-Teh Chang]

LXR shows that NSS doesn't use mp_sqrt.

Comment 2

[Nelson Bolyard (seldom reads bugmail)]

Thanks, Wan-Teh.  
I'm lowering the severity to minor and prioritizing accordingly.

Comment 3

[Wan-Teh Chang]

Should we just remove the mp_sqrt function?  It'd
reduce the code size a little bit.

Comment 4

[Nelson Bolyard (seldom reads bugmail)]

We could ifdef it out for NSS builds, but I don't want to lose the code
entirely because I use it for other purposes.

Comment 5

[Hanno Boeck]

Two notes:
a) I can't reproduce it, has it been fixed and this bug didn't get closed? (tried 0x100, 0x10000, which should be "perfect square with even power of 2", both produce correct result)
b) It is not true that this code is not used (maybe it was true in 2008 when this bug was opened), rsa.c uses it.

Comment 6

[Richard Barnes [:rbarnes]]

Hanno, thanks for the evaluation.  I'm going to assume that whatever was wrong here has either been fixed, or is covered by Bug 1194073.