1603257 - UndefinedBehaviorSanitizer: undefined-behavior ../../lib/softoken/pkcs11c.c:1241

Status: RESOLVED FIXED

(NSS :: Libraries, defect, P1)

Description

[Kevin Jacobs [:kjacobs]]

A try run for DTLS 1.3 just flagged UBSAN on https://searchfox.org/mozilla-central/source/security/nss/lib/softoken/pkcs11c.c#1241: left shift of 138 by 24 places cannot be represented in type 'int'.

We should probably update one of the tests for this to randomize the counter (as it occurs if the top bit is set, which is not true for the test).

Full log: https://firefoxci.taskcluster-artifacts.net/MReFranoQpaQ0YGG55bntA/0/public/logs/live_backing.log

Comment 1

[Kevin Jacobs [:kjacobs]]

Attachment: Bug 1603257 - Fix UBSAN issue in softoken CKM_NSS_CHACHA20_CTR initialization

This patch adds an explicit cast to fix a UBSAN issue that was flagged in https://treeherder.mozilla.org/#/jobs?repo=nss-try&selectedJob=280720441.

It also updates the test to use a random IV.

Comment 2

[Kevin Jacobs [:kjacobs]]

Changing this to sec-low, as the behavior in this case is what we expect. From what I can tell, this generally will only produce a wrong result with uint64 casting (where sign extension occurs).

Comment 3

[J.C. Jones [:jcj] (he/they)]

https://hg.mozilla.org/projects/nss/rev/9ca79efd6d2980fa2f3d4f0b59d0398c8fea8dfc