Closed Bug 1309159 Opened 8 years ago Closed 8 years ago

Fix multiple UBSan warnings in rijndael.c

Categories

(NSS :: Test, defect)

Product:
NSS
Component:
Test
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(firefox-esr45 wontfix)

Status:
RESOLVED FIXED
Tracking Flags:
Tracking Status
firefox-esr45 --- wontfix

People

(Reporter: ttaubert, Assigned: ttaubert)

References

Details

(Keywords: sec-low) 

We have a few unaligned pointer warnings that are easily fixed, but there's also:

rijndael.c:422:15: runtime error: left shift of 254 by 24 places cannot be represented in type 'int'
SUMMARY: AddressSanitizer: undefined-behavior rijndael.c:422:15 in [...]

The problem seems to be that SBOX() returns a PRUint8, so in SUBBYTE():

> (SBOX((w >> 24) & 0xff) << 24)

throws the warning. The left-hand operator is promoted to "int" if we don't specify PRUint32.
https://hg.mozilla.org/projects/nss/rev/ce221788f37c
Status: ASSIGNED → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
Target Milestone: --- → 3.28
Group: crypto-core-security → core-security-release
Mistakenly thought we landed 3.28.1 on the ESR-45 branch
status-firefox-esr45: fixed → affected
tracking-firefox-esr45: 51+ → ---
Group: core-security-release
You need to log in before you can comment on or make changes to this bug.