Closed Bug 1309159 Opened 4 years ago Closed 4 years ago

Fix multiple UBSan warnings in rijndael.c

Categories

(NSS :: Test, defect)

defect
Not set
normal

Tracking

(firefox-esr45 wontfix)

RESOLVED FIXED
Tracking Status
firefox-esr45 --- wontfix

People

(Reporter: ttaubert, Assigned: ttaubert)

References

Details

(Keywords: sec-low)

We have a few unaligned pointer warnings that are easily fixed, but there's also:

rijndael.c:422:15: runtime error: left shift of 254 by 24 places cannot be represented in type 'int'
SUMMARY: AddressSanitizer: undefined-behavior rijndael.c:422:15 in [...]

The problem seems to be that SBOX() returns a PRUint8, so in SUBBYTE():

> (SBOX((w >> 24) & 0xff) << 24)

throws the warning. The left-hand operator is promoted to "int" if we don't specify PRUint32.
https://hg.mozilla.org/projects/nss/rev/ce221788f37c
Status: ASSIGNED → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
Target Milestone: --- → 3.28
Group: crypto-core-security → core-security-release
Mistakenly thought we landed 3.28.1 on the ESR-45 branch
Group: core-security-release
You need to log in before you can comment on or make changes to this bug.