826201 - Use sufficiently large integer when left shifting bits in security/nss/lib/util/utilpars.c

Status: RESOLVED FIXED

(NSS :: Libraries, defect)

Description

[Nickolai Zeldovich]

Attachment: mp0

User Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.17 (KHTML, like Gecko) Chrome/24.0.1312.45 Safari/537.17

Steps to reproduce:

I looked for various bugs in the Mozilla source code.


Actual results:

I found a minor issue in security/nss/lib/util/utilpars.c, where on a 64-bit system nssutil_mkSlotFlags() and nssutil_mkCipherFlags() can left-shift an ((int)1) by 31 or more bits, which is undefined behavior in C.  This can lead to some potentially surprising results: e.g., gcc-4.7.2 on a 64-bit system produces 0xffffffff80000000 as the value of (unsigned long)(1<<31).  The attached patch should fix the problem by avoiding this undefined behavior.

Comment 1

[Nickolai Zeldovich]

Attachment: An updated patch that catches a few more (1<<i) cases

Comment 2

[Kai Engert [:KaiE:]]

Nickolai, thanks a lot for your report and the patch.

(In reply to Nickolai Zeldovich from comment #0)
> gcc-4.7.2 on a 64-bit
> system produces 0xffffffff80000000 as the value of (unsigned long)(1<<31). 

Confirmed. 

r=kaie, however, I propose to replace all occurrences of
  ((unsigned long) 1)
with
  1UL
I think that's easier to read.

Comment 3

[Kai Engert [:KaiE:]]

https://hg.mozilla.org/projects/nss/rev/99fe7b8550c7