Open Bug 786557 Opened 12 years ago Updated 5 months ago

Undefined behavior caused by out-of-range shift in secmod_mkCipherFlags and secmod_mkSlotFlags

Categories

(NSS :: Libraries, defect, P3)

3.13.6
x86_64
Linux

Tracking

(Not tracked)

People

(Reporter: kinetik, Unassigned)

References

(Blocks 1 open bug)

Details

(Whiteboard: [nss-triage])

Building mozilla-central with Clang 3.2 on x86_64 with -fcatch-undefined-behavior results in a binary that crashes in secmod_mkCipherFlags and secmod_mkSlotFlags due to out-of-range left shifts:

703 secmod_mkSlotFlags(unsigned long defaultFlags)
...
706     int i,j;
...
708     for (i=0; i < sizeof(defaultFlags)*8; i++) {
709 	if (defaultFlags & (1<<i)) {

sizeof(defaultFlags) * 8 = 64 on LP64, loop terminates when i == 32.  The same pattern occurs twice in secmod_mkCipherFlags:

http://mxr.mozilla.org/mozilla-central/source/security/nss/lib/softoken/pk11pars.h#667
http://mxr.mozilla.org/mozilla-central/source/security/nss/lib/softoken/pk11pars.h#686

Also, presumably the second loop should be using sizeof(ssl1).
Severity: normal → S3
Priority: -- → P3
Whiteboard: [nss-triage]
You need to log in before you can comment on or make changes to this bug.