Closed Bug 1603257 Opened 2 years ago Closed 2 years ago

UndefinedBehaviorSanitizer: undefined-behavior ../../lib/softoken/pkcs11c.c:1241

Categories

(NSS :: Libraries, defect, P1)

defect

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: kjacobs, Assigned: kjacobs)

Details

(Keywords: sec-low)

Attachments

(1 file)

A try run for DTLS 1.3 just flagged UBSAN on https://searchfox.org/mozilla-central/source/security/nss/lib/softoken/pkcs11c.c#1241: left shift of 138 by 24 places cannot be represented in type 'int'.

We should probably update one of the tests for this to randomize the counter (as it occurs if the top bit is set, which is not true for the test).

Full log: https://firefoxci.taskcluster-artifacts.net/MReFranoQpaQ0YGG55bntA/0/public/logs/live_backing.log

This patch adds an explicit cast to fix a UBSAN issue that was flagged in https://treeherder.mozilla.org/#/jobs?repo=nss-try&selectedJob=280720441.

It also updates the test to use a random IV.

Changing this to sec-low, as the behavior in this case is what we expect. From what I can tell, this generally will only produce a wrong result with uint64 casting (where sign extension occurs).

Keywords: sec-moderatesec-low
Status: ASSIGNED → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED
Target Milestone: --- → 3.49
Group: crypto-core-security → core-security-release
Group: core-security-release
You need to log in before you can comment on or make changes to this bug.