2039251 - Use ProcessPrng in RNG_SystemRNG in freebl on Windows

Status: RESOLVED FIXED

(NSS :: Libraries, enhancement, P2)

Description

[Bob Owen (:bobowen)]

RNG_SystemRNG currently uses RtlGenRandom.
This is bound to advapi32!SystemFunction036, which just forwards to cryptbase!SystemFunction036, which in turn just wraps bcryptprimitives!ProcessPrng.
RtlGenRandom is deprecated.
If we load and call ProcessPrng (which is available win8+) directly from bcryptprimitives.dll, we can also avoid loading cryptbase.dll.
We would still need a fallback to RtlGenRandom to maintain compatibility with Windows 7 and XP.

Rust and Chromium have already switched to using ProcessPrng instead of RtlGenRandom.

Comment 1

[Bob Owen (:bobowen)]

https://treeherder.mozilla.org/jobs?repo=nss-try&revision=808d17eb1ed6da42cf54f565369ce2efac2f80a6

Comment 2

[Bob Owen (:bobowen)]

Attachment: Bug 2039251 - Use ProcessPrng in RNG_SystemRNG in freebl on Windows. r=#nss-reviewers

SystemFunction036 (RtlGenRandom) is deprecated. In advapi32.dll it is a
forwarded export to CRYPTBASE.SystemFunction036, which then calls ProcessPrng
from bcryptprimitives.dll, so calling it loads cryptbase.dll unnecessarily.
ProcessPrng is also one of the replacement APIs suggested by the RtlGenRandom
documentation. On older Windows where ProcessPrng is unavailable, RtlGenRandom
is retained as a fallback.

Also fixed:

• PORT_SetError(SEC_ERROR_NEED_RANDOM) is now called on RNG failure as stated by
  the contract in secrng.h
• RtlGenRandom takes ULONG (32-bit) while RNG_SystemRNG takes size_t, the call
  is now guarded to avoid silent truncation on 64-bit builds.

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/projects/nss/rev/da9968085283
Use ProcessPrng in RNG_SystemRNG in freebl on Windows. r=nss-reviewers,jschanck

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/projects/nss/rev/cf204f07ee38
clang format.