Closed Bug 2056509 Opened 1 month ago Closed 20 days ago

Tab crash with IPDL protocol error: Handler returned error code!

Categories

(NSS :: Libraries, defect, P2)

Product:
NSS
Component:
Libraries
Type:
defect

Tracking

(firefox-esr153154+ fixed, firefox153+ wontfix, firefox154 unaffected, firefox155 unaffected)

Status:
RESOLVED FIXED
Tracking Flags:
Tracking Status
firefox-esr153 154+ fixed
firefox153 + wontfix
firefox154 --- unaffected
firefox155 --- unaffected

People

(Reporter: agurenko, Assigned: jschanck)

References

Details

Attachments

(3 files)

Screenshot_20260721_113504.png
217.44 KB, image/png
Details
support.json
145.13 KB, application/json
Details
Bug 2056509 - upgrade Firefox 153 to nss-rs 0.13.4.
48 bytes, text/x-phabricator-request
phab-bot
: approval-mozilla-esr153+
Details | Review

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:153.0) Gecko/20100101 Firefox/153.0

Steps to reproduce:

When opening multiple tabs (a folder from bookmark bar), some tabs are crashing. I think this happened on a single tab once, but on a group it seems relatively consistent that some random tab will crash.

Actual results:

"Gah. Your tab just crashed." message appears, following can be seen in the syslog:

Jul 21 11:35:13 firefox[5795]: IPDL protocol error: Handler returned error code!
Jul 21 11:35:13 firefox[5795]: ###!!! [Parent][DispatchAsyncMessage] Error: PClientManager::Msg_PClientSourceConstructor Processing error: message was deserialized, but the handler returned false (indicating failure)
Jul 21 11:35:13 firefox[5795]: [Parent 5795, IPC I/O Parent] WARNING: process 9180 exited on signal 15: file ipc/chromium/src/chrome/common/process_watcher_posix_sigchld.cc:161

Expected results:

Tab should've loaded successfully

This is happening on a laptop with Intel 165H CPU, if it matters. In a very limited testing on my 9950X3D Desktop yesterday, I didn't see it

Also happening on a new empty tab (+ button), not necessarily on a page load

Attached file support.json —

Here is a support file

The Bugbug bot thinks this bug should belong to the 'Firefox::Tabbed Browser' component, and is moving the bug to that component. Please correct in case you think the bot is wrong.

Component: Untriaged → Tabbed Browser

Please submit crashes from about:crashes page and post crash id here.
Thanks.

Flags: needinfo?(agurenko)

(In reply to Martin Stránský [:stransky] (ni? me) from comment #5)

Please submit crashes from about:crashes page and post crash id here.
Thanks.

I don't have anything in the about:crashes

Flags: needinfo?(agurenko)

Can you try clean profile? (run firefox -P on terminal and create/select a new one).
Thanks.

Flags: needinfo?(agurenko)

https://bugzilla.redhat.com/show_bug.cgi?id=2503108

(excerpt follows...)

  1. Create a bookmark that is going to redirect you, like "http://www.google.com" (yes, http is intentional, not https), and put it in your bookmarks toolbar.
  2. Open that bookmark by right-clicking on it and then "Open in New Tab" or "Open in New Window".
  3. Google will redirect the tab from http to https.
  4. That new tab/window will soft-crash with "Gah. Your tab just crashed." Firefox itself (with all other tabs) keeps running fine.

There's nothing in about:crashes. Firefox continues running and other tabs do not seem to be affected.

The following URL (same as above but with http instead of https which results in a redirect) should trigger the crash if opened in a new tab/window (or simply use middle mouse button instead of left):
http://bugzilla.redhat.com/show_bug.cgi?id=2503108

(In reply to Martin Stránský [:stransky] (ni? me) from comment #7)

Can you try clean profile? (run firefox -P on terminal and create/select a new one).
Thanks.

Same on a clean profile, without even logging in or installing everything. Looks like @Andreas M. Kirchwitz is right, it's the tabs that redirects

Flags: needinfo?(agurenko)

(In reply to Martin Stránský [:stransky] (ni? me) from comment #10)

Can you reproduce with plain Mozilla binaries?
https://fedoraproject.org/wiki/How_to_debug_Firefox_problems#Testing_Mozilla_binaries
Thanks.

Does not reproduce on mozilla binaries

@stransky, there is Rust in Firefox. Did Fedora build get produced with 1.97.0 or 1.97.1? Apparently, .0 can miscompile stuff...

My broken build in foot used cargo/rust 1.97.0. I am building from your srpm now and it picked up 1.97.1:

https://copr.fedorainfracloud.org/coprs/bojan/firefox/build/10758575/

Sorry, stupid autocorrect. Not foot, copr.

(In reply to Bojan Smojver from comment #12)

@stransky, there is Rust in Firefox. Did Fedora build get produced with 1.97.0 or 1.97.1? Apparently, .0 can miscompile stuff...

I have build the browser with Rust 1.97.1 and still get tabs crashing at what seems to be random.
(https://bugzilla.mozilla.org/show_bug.cgi?id=2056004)

Looks like this is a crash in IPC code, moving components. Please feel free to move back if this isn't the case.

Component: Tabbed Browser → IPC
Product: Firefox → Core

Trying a build without PGO.

AI analysis about PGO/GCC misscompilation

  Why HTTP→HTTPS Redirects Trigger It

  During a redirect, the browser creates a new ClientSource for the new origin. The child side (ClientSource::Activate) constructs a ClientSourceConstructorArgs containing a PrincipalInfo
  (an IPDL union: ContentPrincipalInfo | SystemPrincipalInfo | NullPrincipalInfo | ExpandedPrincipalInfo) and sends it to the parent. The scheme change (http → https) means a
  different-origin principal, exercising the ContentPrincipalInfo comparison and validation paths.

  Compiler Miscompilation Sensitivity

  Given this only reproduces on Fedora GCC builds (not Mozilla Clang binaries), these code paths are most vulnerable to GCC/PGO miscompilation:

  1. PrincipalInfo deserialization — It's an IPDL union type. A miscompilation could corrupt the variant tag or data layout, causing valid principals to be rejected.
  2. ValidatePrincipalCouldPotentiallyBeLoadedBy — Complex switch statement with many branches in ProcessIsolation.cpp. Incorrect branch elimination by the optimizer could reject a valid
  principal.
  3. IPCResult itself — A trivial type with a single bool mSuccess. A miscompilation of the operator bool() or initialization could flip success→failure. This mirrors Bug 1990430 where GCC
  PGO miscompiled trivially-copyable struct initialization (swapping mLength/mCapacity in AutoTArray).
  4. ClientSourceConstructorArgs serialization/deserialization — Template-heavy IPC marshalling code involving nsID (16-byte UUID), PrincipalInfo (union), TimeStamp — all sensitive to layout
  assumptions.

I'm not a Firefox or C++ internals expert, and I don't read assembly myself - I used an AI assistant to help me drive gdb and interpret what it showed. Posting this in case it's useful, but I can't vouch for the interpretation beyond what's directly shown below. Happy to be corrected.

Setup: fresh Fedora 44 (x86_64) install, official Fedora Firefox 153.0-1.fc44 package, on an AWS EC2 t3.xlarge (Intel). Reproduced the "Already registered!" tab crash 3 times using the bookmark-redirect STR from comment 8.

I attached gdb to the parent process and set a breakpoint on mozilla::ipc::IPCResult::FailImpl (the function that ultimately runs whenever
IPC_FAIL(...) is invoked anywhere). It caught the failure each time, with this backtrace:

#0 mozilla::ipc::IPCResult::FailImpl ()
#1 mozilla::ipc::IPCResult::Fail<...> ()
#2 mozilla::dom::ClientSourceParent::Init ()
at dom/clients/manager/ClientSourceParent.cpp:178
#3 mozilla::dom::ClientManagerParent::RecvPClientSourceConstructor ()
#4 mozilla::dom::PClientManagerParent::OnMessageReceived ()
#5 mozilla::ipc::PBackgroundParent::OnMessageReceived ()
#6 mozilla::ipc::MessageChannel::DispatchAsyncMessage ()
...

Line 178 of ClientSourceParent.cpp is:
cpp
if (NS_WARN_IF(!mService->AddSource(this))) {
return IPC_FAIL(Manager(), "Already registered!");
}

Stepping into ClientManagerService::AddSource() (source below, from searchfox), all 3 reproductions took the same branch:

cpp
bool ClientManagerService::AddSource(ClientSourceParent* aSource) {
auto entry = mSourceTable.Lookup(aSource->Info().Id());
if (entry) {
// Do not permit overwriting an existing ClientSource with the same
// UUID. This would allow a spoofed ClientParentSource actor to
// intercept postMessage() intended for the real actor.
if (entry.Data().is<ClientSourceParent*>()) {
return false; // <-- this is the branch my gdb session landed on each time
}
FutureClientSourceParent& placeHolder = ...
...

So, as far as I can tell: when the redirect's new tab tries to register its ClientSourceParent, the lookup table (mSourceTable, keyed by a UUID) already has a live entry under that same UUID - not a FutureClientSourceParent placeholder (which would be handled further down, and I did separately confirm that placeholder-matching path succeeds normally in unrelated, non-crashing cases during the same session). That live-entry case trips an explicit anti-spoofing guard and fails on purpose.

I don't know why the table already has a live entry - I can only speculate:

  • ClientSourceParent's entry gets removed via RemoveSource(), which is called from ActorDestroy(), which sounds like it runs as part of IPC actor teardown - i.e. asynchronously, not necessarily before the new client's constructor messagefor the redirected navigation.

  • If that's right, this might be a timing/ordering issue: whether the parent processes the old client's teardown message before or after the new client's constructor message for the same UUID. If codegen/scheduling differences (GCC+PGO vs. Clang) shift timing enough, a pre-existing race window that's rarely/never hit on Mozilla's official binaries could become much more visible on a differently-compiled binary, without either compiler doing anything incorrect.

  • There's a previous, related bug (bug https://bugzilla.mozilla.org/show_bug.cgi?id=1584007) that specifically introduced theFutureClientSourceParent placeholder mechanism to fix an earlier race in this same area. It's possible what I'm seeing is a related but distinct ordering gap that isn't covered by that fix - or it's possible this is unrelated. I genuinely don't know.

I also spent a while poking at raw memory/registers in the optimized binary trying to inspect the actual UUID values involved, but couldn't get reliable results that way, so I'm leaving that out rather than pointing anyone down a guess-based path.

No SIGSEGV, no coredump, nothing in dmesg/journalctl, nothing in about:crashes in any of the 3 reproductions - this looks like a clean, intentional IPC-level rejection rather than a memory-safety crash, for what that's worth re: the miscompilation theory in comment 19.

Firefox 153.0-1.fc44.x86_64. Happy to try more repros, or a non-PGO build if/when one is available, or to just leave this here in case someone with more context finds it useful.

BTW, the crashes are happening on non-PGO Fedora builds, so PGO is not the cause.

Used IA to investigate, got this results. May it be caused by system NSS?

Root cause: UUID PRNG not re-seeded after process fork

The log proves it:

Child 230494 generates UUID {90a0dda9-...} via CreateSourceInternal(new UUID) at this=7f2561152580
Child 230517 generates the exact same UUID {90a0dda9-...} via CreateSourceInternal(new UUID) at the same this=7f2561152580

Same UUID + same heap address = these two processes were forked from a common ancestor (fork server) and the PRNG state was duplicated without re-seeding.

Looking at the code, GenerateUUIDInPlace tries NSS first (PK11_GenerateRandomOnSlot) on the main thread. NSS uses a userspace PRNG (Fortuna). After fork, if the NSS PRNG isn't re-seeded,
both children produce identical UUID sequences.

The OS fallback (getrandom()) would be fork-safe since it's a syscall, but it's only used when NSS isn't available.

Are you using the fork server? You can check with:

Check dom.ipc.forkserver.enable in about:config.

Possible mitigations to test:

Disable fork server: set dom.ipc.forkserver.enable to false in about:config
Or set env MOZ_FORK_SERVER=0 before launching

Looks like dom.ipc.forkserver.enable does the trick.

Duplicate of this bug: 2056004

From the diff between 152.0.6 and 153.0:

--- a/ipc/glue/ForkServiceChild.cpp
+++ b/ipc/glue/ForkServiceChild.cpp
@@ -258,8 +258,6 @@ StaticRefPtr<ForkServerLauncher> ForkServerLauncher::sSingleton;
 
 ForkServerLauncher::ForkServerLauncher() = default;
 
-ForkServerLauncher::~ForkServerLauncher() = default;
-
 already_AddRefed<ForkServerLauncher> ForkServerLauncher::Create() {
   if (sSingleton == nullptr) {
     sSingleton = new ForkServerLauncher();
diff --git a/ipc/glue/ForkServiceChild.h b/ipc/glue/ForkServiceChild.h
index 923b1900b3aa..dc90fb1a5aa9 100644
--- a/ipc/glue/ForkServiceChild.h
+++ b/ipc/glue/ForkServiceChild.h
@@ -119,7 +119,7 @@ class ForkServerLauncher final : public nsIObserver {
 
  private:
   friend class ForkServiceChild;
-  ~ForkServerLauncher();
+  ~ForkServerLauncher() = default;
 
   static void RestartForkServer();

Matters?

Trying a build in copr with this patch now:

diff --git a/ipc/glue/ForkServer.cpp b/ipc/glue/ForkServer.cpp
--- a/ipc/glue/ForkServer.cpp
+++ b/ipc/glue/ForkServer.cpp
@@ -17,6 +17,8 @@
 #include "mozilla/ipc/ProtocolMessageUtils.h"
 #include "mozilla/ipc/SetProcessTitle.h"
 #include "nsTraceRefcnt.h"
+#include "nss.h"     // NSS_IsInitialized
+#include "pk11pub.h" // PK11_RandomUpdate
 
 #include <fcntl.h>
 #include <string.h>
@@ -176,6 +178,20 @@
 // passed file handles to reflect the new process.
 static void ForkedChildProcessInit(int aExecFd, int* aArgc, char*** aArgv) {
   // Remove the fork-server-specific SIGCHLD handler.
+
+  // FIX: Mix fresh, per-process entropy into NSS's global PRNG state
+  // after fork, so this child's random output (e.g. UUIDs generated
+  // via PK11_GenerateRandom) diverges from any sibling forked from
+  // the same fork-server parent. Only touch NSS if it's already
+  // initialized in this process -- the fork server intentionally
+  // spawns children before heavyweight subsystems like NSS are
+  // brought up, and we don't want to force that here.
+  if (NSS_IsInitialized()) {
+    struct { pid_t pid; PRTime time; void* addr; } entropy = {
+        getpid(), PR_Now(), &entropy};
+    PK11_RandomUpdate(&entropy, sizeof(entropy));
+  }
+
   signal(SIGCHLD, SIG_DFL);
   // This process is currently single-threaded, so the fd used by the
   // signal handler can be safely closed once the handler is removed.

No idea whether that's the right thing to do, but we'll find out, I guess.

Nope, that patch does not resolve the problem, unfortunately.

Claude suggests that not enough new randomness was injected and that could be why the above patch failed. New suggestion:

--- a/ipc/glue/ForkServer.cpp
+++ b/ipc/glue/ForkServer.cpp
@@
 #include "mozilla/ipc/ProtocolMessageUtils.h"
 #include "mozilla/ipc/SetProcessTitle.h"
 #include "nsTraceRefcnt.h"
+#include "mozilla/RandomNum.h"
+#include "nss.h"
+#include "pk11pub.h"
 
 #include <fcntl.h>
 #include <string.h>
@@
 static void ForkedChildProcessInit(int aExecFd, int* aArgc, char*** aArgv) {
   // Remove the fork-server-specific SIGCHLD handler.
+
+  // Force NSS's global RNG to diverge from any sibling forked from the
+  // same parent/template. Use entropy from a syscall (fork-safe by
+  // construction) rather than process-local values like a stack address,
+  // which can be identical across siblings when fork() isn't followed by
+  // exec() (no ASLR re-randomization between them).
+  if (NSS_IsInitialized()) {
+    uint8_t entropy[32];
+    if (mozilla::GenerateRandomBytesFromOS(entropy, sizeof(entropy))) {
+      PK11_RandomUpdate(entropy, sizeof(entropy));
+    }
+  }
+
   signal(SIGCHLD, SIG_DFL);
   // This process is currently single-threaded, so the fd used by the
   // signal handler can be safely closed once the handler is removed.

The above does not work. I will try this next:

--- a/xpcom/base/nsID.cpp
+++ b/xpcom/base/nsID.cpp
@@ -6,47 +6,13 @@
 
 #include <limits.h>
 
-#include "MainThreadUtils.h"
 #include "mozilla/Assertions.h"
 #include "mozilla/RandomNum.h"
 #include "mozilla/Sprintf.h"
-#include "nss.h"
-#include "ScopedNSSTypes.h"
-
-[[nodiscard]] static bool GenerateRandomBytesFromNSS(void* aBuffer,
-                                                     size_t aLength) {
-  MOZ_ASSERT(aBuffer);
-
-  // Bounds check that we can safely truncate size_t `aLength` to an int.
-  if (aLength == 0 || aLength > INT_MAX) {
-    MOZ_ASSERT_UNREACHABLE("Bad aLength");
-    return false;
-  }
-  int len = static_cast<int>(aLength);
-
-  // Only try to use NSS on the main thread.
-  if (!NS_IsMainThread() || !NSS_IsInitialized()) {
-    return false;
-  }
-
-  mozilla::UniquePK11SlotInfo slot(PK11_GetInternalSlot());
-  if (!slot) {
-    MOZ_ASSERT_UNREACHABLE("Null slot");
-    return false;
-  }
-
-  SECStatus srv = PK11_GenerateRandomOnSlot(
-      slot.get(), static_cast<unsigned char*>(aBuffer), len);
-  MOZ_ASSERT(srv == SECSuccess);
-  return (srv == SECSuccess);
-}
 
 nsresult nsID::GenerateUUIDInPlace(nsID& aId) {
-  // Firefox needs to generate some UUIDs before NSS has been initialized. We
-  // prefer NSS's RNG, but if NSS is not available yet or returns an error, fall
-  // back to MFBT's GenerateRandomBytes().
-  if (!GenerateRandomBytesFromNSS(&aId, sizeof(nsID)) &&
-      !mozilla::GenerateRandomBytesFromOS(&aId, sizeof(nsID))) {
+  // Always use OS for RNG
+  if (!mozilla::GenerateRandomBytesFromOS(&aId, sizeof(nsID))) {
     MOZ_ASSERT_UNREACHABLE("GenerateRandomBytesFromOS() failed");
     return NS_ERROR_NOT_AVAILABLE;
   }

That last patch is in https://copr.fedorainfracloud.org/coprs/bojan/firefox/build/10767044/. Not seeing crashes, but if you have cycles, feel free to test.

Assignee: nobody → nobody
Component: IPC → Libraries
Product: Core → NSS
Version: Firefox 153 → unspecified

Just for the record and to be not forgotten. Following is what I wrote in https://bodhi.fedoraproject.org/updates/FEDORA-2026-7d71f89d7e and I think it could be helpful here in the upstream as well.

When discussed with AI this issue and the upstream change of how destructor of the ForkServerLauncher class is now defined I got following interesting conclusion. AI guess the root of this issue is Static Initialization Order Fiasco (SIOF) and it looks like it should be fixed by the upstream. Following text is AI generated:

The Issue: Fork Server + NSS RNG State Duplication

Firefox uses a Fork Server on Linux to speed up the launching of new content processes. When a new process is needed, the main process asks the Fork Server to call fork(). The child process inherits the exact memory space of the Fork Server.

The bug occurs because the NSS (Network Security Services) library initializes its Random Number Generator (RNG) to generate UUIDs. If the NSS RNG is initialized before the fork() happens, the child process inherits the exact same RNG state. As a result, all child processes spawn with identical entropy, leading them to generate duplicate UUIDs.

Why Clang (Upstream) works, but GCC (Fedora) fails

This is a classic Static Initialization Order Fiasco (SIOF). C++ does not guarantee the order in which global/static objects are initialized across different translation units.

  • Clang (Mozilla build): Clang's compilation and linking algorithms (especially with LTO and LLD) arrange the .init_array sections in a way that happens to delay NSS initialization until after the fork occurs.
  • GCC (Fedora build): GCC and GNU ld arrange object files and initializer sections differently. In Fedora's build, NSS was initialized before the Fork Server began spawning children. (The use of out-of-line = default destructors in .cpp files affects vtable emission and linker ordering, indirectly contributing to this difference).

Relying on compiler linking order for correct runtime behavior is a latent bug.

How to Fix It Properly

The fix must be deterministic and explicitly handle the fork() boundary, rather than relying on compilation order. There are three standard approaches:

  1. Explicit Re-seeding After fork() (Best Practice):
    Immediately after fork() returns 0 (indicating we are in the child process), explicitly call an NSS API to fetch fresh entropy from the OS (e.g., via getrandom() or /dev/urandom) and re-seed the PRNG.

    pid_t pid = fork();
if (pid == 0) {
    // We are in the child process
    NSS_ReinitializeRNG(); // Fetch fresh entropy and re-seed
    // ... continue execution
}

  2. Using pthread_atfork():
    Register a handler at program startup that automatically re-seeds the RNG in the child process whenever a fork() occurs.

    pthread_atfork(nullptr, nullptr, []() {
    // Called automatically in the child process after fork()
    NSS_ReinitializeRNG(); 
});

  3. Lazy Initialization:
    Prevent NSS from initializing globally before main() or before the fork. Use lazy initialization (e.g., a function with a local static variable) so the RNG is only initialized the first time a UUID is actually requested—which should naturally happen after the child process has started its independent execution.

Also I think the solution (calling GenerateRandomBytesFromOS) found by Bojan Smojver is a better temporary workaround than just disabling dom.ipc.forkserver.enable property by default. And the real fix should be done here in the Mozilla project, most likely by the way described in my previous message.

(In reply to Martin Stránský [:stransky] (ni? me) from comment #32)
I've made a PR for Fedora package of the NSS library part of Firefox: https://src.fedoraproject.org/rpms/nss/pull-request/183

Tested by upgrading those packages

sudo dnf upgrade https://kojipkgs.fedoraproject.org//work/tasks/5109/148275109/nss-3.125.0-2.fc44.x86_64.rpm https://kojipkgs.fedoraproject.org//work/tasks/5109/148275109/nss-softokn-3.125.0-2.fc44.x86_64.rpm https://kojipkgs.fedoraproject.org//work/tasks/5109/148275109/nss-softokn-freebl-3.125.0-2.fc44.x86_64.rpm https://kojipkgs.fedoraproject.org//work/tasks/5109/148275109/nss-sysinit-3.125.0-2.fc44.x86_64.rpm https://kojipkgs.fedoraproject.org//work/tasks/5109/148275109/nss-tools-3.125.0-2.fc44.x86_64.rpm https://kojipkgs.fedoraproject.org//work/tasks/5109/148275109/nss-util-3.125.0-2.fc44.x86_64.rpm

and by using Firefox 153.0-3.fc44 with the dom.ipc.forkserver.enable property manually re-enabled in about:config.

No tab crashed so far.

(In reply to Martin Stránský [:stransky] (ni? me) from comment #32)
Hope the patch that I made in the PR for Fedora could be accepted or used as a base in the Mozilla project for the proper fix in the upstream. I'm using it in Fedora 44 with Cinnamon (i.e. with X11 because Cinnamon doesn't support Wayland) already for about two days and didn't see any tab crashed.

From Claude Opus 5:

Answer: nothing in the fork server initialises NSS. Since 153, libxul has a 
hard DT_NEEDED on libfreeblpriv3.so, and Fedora's freebl runs its FIPS power-up 
self tests from an ELF constructor that calls RNG_RNGInit(). So the DRBG is 
instantiated while dlopen(libxul) is still running, i.e. before 
ForkServer::RunForkServer() is even entered.

The chain, at FIREFOX_153_0_RELEASE (HEAD is now detached there):

1. neqo-transport has default = ["blapi"] (
third_party/rust/neqo-transport/Cargo.toml:79), and nss-rs's blapi backend calls
freebl directly, bypassing softoken (third_party/rust/nss-rs/src/freebl/mod.rs)
. gkrust therefore references FREEBL_GetVector.
2. Two commits new in 153 turned that into a link-time dependency of libxul:
   - a984743477b7 Bug 2044303 (neqo 0.29) added USE_LIBS += ["freeblpriv3"] in 
toolkit/library/moz.build:167.
   - 33b720ec8df5 Bug 2047651 added OS_LIBS += ["-lfreeblpriv3"] for 
MOZ_SYSTEM_NSS in security/moz.build:25, because Fedora's nss.pc does not list 
freebl.
  Neither is in FIREFOX_152_0_RELEASE. Before this, freebl was only ever loaded
lazily by libsoftokn3 at NSS init time.
3. browser/app/nsBrowserApp.cpp:357 calls InitXPCOMGlue() for the fork server, 
which dlopens libxul. The loader pulls in libfreeblpriv3.so eagerly and runs its
constructors.
4. security/nss/lib/freebl/fipsfreebl.c:37 declares bl_startup_tests() as 
__attribute__((constructor)) unless NSS_NO_INIT_SUPPORT is defined, and that 
function does BL_Init(); RNG_RNGInit(); (line 1999-2006). RNG_RNGInit → 
PR_CallOnce(&coRNGInit, rng_init) → theGlobalRng seeded, globalrng set, 
coRNGInit marked done (security/nss/lib/freebl/drbg.c:566).
5. The fork server then forks every content process from that address space. 
Both theGlobalRng and the coRNGInit "already done" flag are inherited verbatim, 
so RNG_RNGInit() in the child is a no-op and the child replays the parent's 
stream. nsID::GenerateUUIDInPlace prefers the NSS RNG on the main thread, so 
sibling content processes hand out the same ClientSource UUIDs, and 
ClientManagerService::AddSource hits the anti-spoofing guard you found in 
comment 20.

Why Fedora and not Mozilla builds: NSS_NO_INIT_SUPPORT controls whether that 
constructor exists. NSS's make build defines it unless NSS_FORCE_FIPS is set (
security/nss/coreconf/config.mk:220), and Fedora's nss.spec line 379 does 
export NSS_FORCE_FIPS=1. Firefox's in-tree gyp build defaults to 
disable_fips%: 1, which defines NSS_FIPS_DISABLED and NSS_NO_INIT_SUPPORT (
security/nss/coreconf/config.gypi:178-182). Same libxul dependency in both 
builds, constructor only in Fedora's freebl. So this is a build-configuration 
difference, not GCC/PGO codegen or static initialisation order.

Also, nothing recovers from it: softoken's fork detection is compiled out by 
default since PKCS#11 3.0 (security/nss/lib/softoken/softoken.h:167 requires 
DO_FORK_CHECK), and softoken is not even loaded in the fork server, so only 
freebl state exists to be duplicated.

This explains why your first two patches were no-ops: they were guarded on 
NSS_IsInitialized(), which only reflects nss_IsInitted set by NSS_Init*. In the 
fork server nothing calls NSS_Init, so the guard was false and PK11_RandomUpdate
never ran. Your third patch and Rosti's freebl-level pthread_atfork both work 
because they act below that flag. Rosti's handler survives the sandbox's raw 
clone() because mozglue interposes __register_atfork and SandboxLaunch::Fork 
replays the captured handlers manually (
mozglue/interposers/pthread_atfork_interposer.cpp, 
security/sandbox/linux/launch/SandboxLaunch.cpp:651).

Two cheap runtime confirmations on your repro box:
- ldd /usr/lib64/firefox/libxul.so | grep freebl should list libfreeblpriv3.so 
on 153 and not on 152.
- Attach gdb to the forkserver process (pgrep -f forkserver) and p globalrng / 
p theGlobalRng.isValid; non-null there is the smoking gun, and 
p NSS_IsInitialized() should return false at the same time.

For an upstream fix there are two independent angles: stop libxul from eagerly 
loading freebl (make the blapi dependency lazy, or drop the explicit link), and/
or reseed after fork. Given the fork server forks with clone() rather than 
fork(), doing it in ForkedChildProcessInit via a freebl-level call (
RNG_RandomUpdate with OS entropy, unguarded by NSS_IsInitialized) would be the 
Gecko-side equivalent of Rosti's NSS patch.

Just confirmed with ldd. Firefox 152.0.6-1 Fedora build - libxul.so DOES NOT link libfreeblpriv3.so. Build 153.0-3 does.

Apparently, already fixed upstream: https://phabricator.services.mozilla.com/D308064

With the above upstream fix applied to the Fedora build, I am also not seeing tab crashes. If you'd like to try:

https://copr.fedorainfracloud.org/coprs/bojan/firefox/build/10777064/

Or you can get scratch builds from:

https://src.fedoraproject.org/rpms/firefox/pull-request/106

Once they are done, that is.

(In reply to Bojan Smojver from comment #37)

Just confirmed with ldd. Firefox 152.0.6-1 Fedora build - libxul.so DOES NOT link libfreeblpriv3.so. Build 153.0-3 does.

This change done in Jun 17: https://github.com/mozilla-firefox/firefox/commit/30a354b6625fc79b01f9d1f36f60cea2a669248e

(In reply to Bojan Smojver from comment #38)

Apparently, already fixed upstream: https://phabricator.services.mozilla.com/D308064

This change is done in Jun 23: https://github.com/mozilla-firefox/firefox/commit/6509ea0ed474b5eb9e6a7a4337a271bdca944e03

More than a month ago. Why it wasn't included into Firefox 153.0 or into the release branch (i.e. for any future 153.0.x updates)?

Status in https://bugzilla.mozilla.org/show_bug.cgi?id=2047819

Field Value
Status RESOLVED FIXED
Milestone 154 Branch

Tracking Flags

Tracking Flag Status
firefox-esr140 --- unaffected
firefox152 --- unaffected
firefox153 + affected
firefox154 --- fixed

because the nss-rs fix has dependencies

these got merged into release branch yesterday, so there is a clear path to move forward.

See Also: → 2058230

From what I can see, based on the fact that changes from https://phabricator.services.mozilla.com/D308064 still apply to 153.0.1, that version does not carry the fix. Right?

Yes, 153.0.x is still affected.

(In reply to John Schanck [:jschanck] from comment #43)

Yes, 153.0.x is still affected.

Will it be fixed in some 153.0.x or it's better to wait for 154.0 in August 18?

153 is an ESR release and I expect we will backport patches for this issue. The next scheduled 153 dot release coincides with the 154 release on August 18th, but could potentially happen sooner.

The bug has a release status flag that shows some version of Firefox is affected, thus it will be considered confirmed.

Status: UNCONFIRMED → NEW
Ever confirmed: true
See Also: → 2047819
Attachment #9620013 - Flags: approval-mozilla-esr153?

Works fine now with FF 153.0.1 (firefox-153.0.1-1.fc44.x86_64) and NSS 3.126.0 (nss-3.126.0-1.fc44). Technically it was the nss that fixed the problem, it started to work even with 153.0.0 but same NSS version.

(In reply to Gurenko Alex from comment #48)

Works fine now with FF 153.0.1 (firefox-153.0.1-1.fc44.x86_64) and NSS 3.126.0 (nss-3.126.0-1.fc44). Technically it was the nss that fixed the problem, it started to work even with 153.0.0 but same NSS version.

The patches to fix this have not landed in NSS yet.

(In reply to John Schanck [:jschanck] from comment #49)

(In reply to Gurenko Alex from comment #48)

Works fine now with FF 153.0.1 (firefox-153.0.1-1.fc44.x86_64) and NSS 3.126.0 (nss-3.126.0-1.fc44). Technically it was the nss that fixed the problem, it started to work even with 153.0.0 but same NSS version.

The patches to fix this have not landed in NSS yet.

My bad, indeed, the nss in fedora has a cherry pick of:

* Sat Jul 25 2026 Rostislav Krasny <rostiprodev@gmail.com> - 3.125.0-2
- Reseed the freebl DRBG after fork() so forked children do not replay the
  parent's random stream (Firefox fork server duplicate UUIDs / crypto output)

[Tracking Requested - why for this release]: This is a security issue for non-Mozilla builds of Firefox that link against system NSS and use FIPS mode.

Assignee: nobody → jschanck
Severity: -- → S2
Status: NEW → ASSIGNED
tracking-firefox153: --- → ?
tracking-firefox-esr153: --- → ?
Priority: -- → P2

ESR 153.1 ships on August 18, please request an uplift request.
if we want it in the last 153.0.4 dot release next Tuesday. Uplift request deadline for this dot release is Friday EOD. Thanks

tracking-firefox153: ? → +
tracking-firefox-esr153: ? → 154+
Flags: needinfo?(jschanck)

:jschanck also this needs an uplift request form

firefox-esr153 Uplift Approval Request

  • User impact if declined/Reason for urgency: Predictable random number generator output and tab crashes on Fedora and other Linux distros that ship system NSS builds of Firefox and do not disable FIPS mode for softoken.
  • Code covered by automated testing?: yes
  • Fix verified in Nightly?: yes
  • Needs manual QE testing?: no
  • Steps to reproduce for manual QE testing:
  • Risk associated with taking this patch: low
  • Explanation of risk level: We have already upgraded to nss-rs 0.13.4 on Firefox 154+.
  • String changes made/needed?: none
  • Is Android affected?: yes

Hm, I had submitted the uplift request form in lando last week, but the comment only showed up here after clicking "edit uplift assessment" and re-submitting.

Flags: needinfo?(jschanck)
Attachment #9620013 - Flags: approval-mozilla-esr153? → approval-mozilla-esr153+
QA Whiteboard: [qa-triage-done-c155/b154]
Status: ASSIGNED → RESOLVED
Closed: 20 days ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: