Open Bug 2056509 Opened 1 day ago Updated 18 minutes ago

Tab crash with IPDL protocol error: Handler returned error code!

Categories

(Core :: IPC, defect)

Product:
Core
Component:
IPC
Version:
Firefox 153
Type:
defect

Tracking

()

Status:
UNCONFIRMED

People

(Reporter: agurenko, Unassigned)

References

Details

Attachments

(2 files)

Screenshot_20260721_113504.png
217.44 KB, image/png
Details
support.json
145.13 KB, application/json
Details

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:153.0) Gecko/20100101 Firefox/153.0

Steps to reproduce:

When opening multiple tabs (a folder from bookmark bar), some tabs are crashing. I think this happened on a single tab once, but on a group it seems relatively consistent that some random tab will crash.

Actual results:

"Gah. Your tab just crashed." message appears, following can be seen in the syslog:

Jul 21 11:35:13 firefox[5795]: IPDL protocol error: Handler returned error code!
Jul 21 11:35:13 firefox[5795]: ###!!! [Parent][DispatchAsyncMessage] Error: PClientManager::Msg_PClientSourceConstructor Processing error: message was deserialized, but the handler returned false (indicating failure)
Jul 21 11:35:13 firefox[5795]: [Parent 5795, IPC I/O Parent] WARNING: process 9180 exited on signal 15: file ipc/chromium/src/chrome/common/process_watcher_posix_sigchld.cc:161

Expected results:

Tab should've loaded successfully

This is happening on a laptop with Intel 165H CPU, if it matters. In a very limited testing on my 9950X3D Desktop yesterday, I didn't see it

Also happening on a new empty tab (+ button), not necessarily on a page load

Attached file support.json —

Here is a support file

The Bugbug bot thinks this bug should belong to the 'Firefox::Tabbed Browser' component, and is moving the bug to that component. Please correct in case you think the bot is wrong.

Component: Untriaged → Tabbed Browser

Please submit crashes from about:crashes page and post crash id here.
Thanks.

Flags: needinfo?(agurenko)

(In reply to Martin Stránský [:stransky] (ni? me) from comment #5)

Please submit crashes from about:crashes page and post crash id here.
Thanks.

I don't have anything in the about:crashes

Flags: needinfo?(agurenko)

Can you try clean profile? (run firefox -P on terminal and create/select a new one).
Thanks.

Flags: needinfo?(agurenko)

https://bugzilla.redhat.com/show_bug.cgi?id=2503108

(excerpt follows...)

  1. Create a bookmark that is going to redirect you, like "http://www.google.com" (yes, http is intentional, not https), and put it in your bookmarks toolbar.
  2. Open that bookmark by right-clicking on it and then "Open in New Tab" or "Open in New Window".
  3. Google will redirect the tab from http to https.
  4. That new tab/window will soft-crash with "Gah. Your tab just crashed." Firefox itself (with all other tabs) keeps running fine.

There's nothing in about:crashes. Firefox continues running and other tabs do not seem to be affected.

The following URL (same as above but with http instead of https which results in a redirect) should trigger the crash if opened in a new tab/window (or simply use middle mouse button instead of left):
http://bugzilla.redhat.com/show_bug.cgi?id=2503108

(In reply to Martin Stránský [:stransky] (ni? me) from comment #7)

Can you try clean profile? (run firefox -P on terminal and create/select a new one).
Thanks.

Same on a clean profile, without even logging in or installing everything. Looks like @Andreas M. Kirchwitz is right, it's the tabs that redirects

Flags: needinfo?(agurenko)

(In reply to Martin Stránský [:stransky] (ni? me) from comment #10)

Can you reproduce with plain Mozilla binaries?
https://fedoraproject.org/wiki/How_to_debug_Firefox_problems#Testing_Mozilla_binaries
Thanks.

Does not reproduce on mozilla binaries

@stransky, there is Rust in Firefox. Did Fedora build get produced with 1.97.0 or 1.97.1? Apparently, .0 can miscompile stuff...

My broken build in foot used cargo/rust 1.97.0. I am building from your srpm now and it picked up 1.97.1:

https://copr.fedorainfracloud.org/coprs/bojan/firefox/build/10758575/

Sorry, stupid autocorrect. Not foot, copr.

(In reply to Bojan Smojver from comment #12)

@stransky, there is Rust in Firefox. Did Fedora build get produced with 1.97.0 or 1.97.1? Apparently, .0 can miscompile stuff...

I have build the browser with Rust 1.97.1 and still get tabs crashing at what seems to be random.
(https://bugzilla.mozilla.org/show_bug.cgi?id=2056004)

Looks like this is a crash in IPC code, moving components. Please feel free to move back if this isn't the case.

Component: Tabbed Browser → IPC
Product: Firefox → Core

Trying a build without PGO.

AI analysis about PGO/GCC misscompilation

  Why HTTP→HTTPS Redirects Trigger It

  During a redirect, the browser creates a new ClientSource for the new origin. The child side (ClientSource::Activate) constructs a ClientSourceConstructorArgs containing a PrincipalInfo
  (an IPDL union: ContentPrincipalInfo | SystemPrincipalInfo | NullPrincipalInfo | ExpandedPrincipalInfo) and sends it to the parent. The scheme change (http → https) means a
  different-origin principal, exercising the ContentPrincipalInfo comparison and validation paths.

  Compiler Miscompilation Sensitivity

  Given this only reproduces on Fedora GCC builds (not Mozilla Clang binaries), these code paths are most vulnerable to GCC/PGO miscompilation:

  1. PrincipalInfo deserialization — It's an IPDL union type. A miscompilation could corrupt the variant tag or data layout, causing valid principals to be rejected.
  2. ValidatePrincipalCouldPotentiallyBeLoadedBy — Complex switch statement with many branches in ProcessIsolation.cpp. Incorrect branch elimination by the optimizer could reject a valid
  principal.
  3. IPCResult itself — A trivial type with a single bool mSuccess. A miscompilation of the operator bool() or initialization could flip success→failure. This mirrors Bug 1990430 where GCC
  PGO miscompiled trivially-copyable struct initialization (swapping mLength/mCapacity in AutoTArray).
  4. ClientSourceConstructorArgs serialization/deserialization — Template-heavy IPC marshalling code involving nsID (16-byte UUID), PrincipalInfo (union), TimeStamp — all sensitive to layout
  assumptions.

I'm not a Firefox or C++ internals expert, and I don't read assembly myself - I used an AI assistant to help me drive gdb and interpret what it showed. Posting this in case it's useful, but I can't vouch for the interpretation beyond what's directly shown below. Happy to be corrected.

Setup: fresh Fedora 44 (x86_64) install, official Fedora Firefox 153.0-1.fc44 package, on an AWS EC2 t3.xlarge (Intel). Reproduced the "Already registered!" tab crash 3 times using the bookmark-redirect STR from comment 8.

I attached gdb to the parent process and set a breakpoint on mozilla::ipc::IPCResult::FailImpl (the function that ultimately runs whenever
IPC_FAIL(...) is invoked anywhere). It caught the failure each time, with this backtrace:

#0 mozilla::ipc::IPCResult::FailImpl ()
#1 mozilla::ipc::IPCResult::Fail<...> ()
#2 mozilla::dom::ClientSourceParent::Init ()
at dom/clients/manager/ClientSourceParent.cpp:178
#3 mozilla::dom::ClientManagerParent::RecvPClientSourceConstructor ()
#4 mozilla::dom::PClientManagerParent::OnMessageReceived ()
#5 mozilla::ipc::PBackgroundParent::OnMessageReceived ()
#6 mozilla::ipc::MessageChannel::DispatchAsyncMessage ()
...

Line 178 of ClientSourceParent.cpp is:
cpp
if (NS_WARN_IF(!mService->AddSource(this))) {
return IPC_FAIL(Manager(), "Already registered!");
}

Stepping into ClientManagerService::AddSource() (source below, from searchfox), all 3 reproductions took the same branch:

cpp
bool ClientManagerService::AddSource(ClientSourceParent* aSource) {
auto entry = mSourceTable.Lookup(aSource->Info().Id());
if (entry) {
// Do not permit overwriting an existing ClientSource with the same
// UUID. This would allow a spoofed ClientParentSource actor to
// intercept postMessage() intended for the real actor.
if (entry.Data().is<ClientSourceParent*>()) {
return false; // <-- this is the branch my gdb session landed on each time
}
FutureClientSourceParent& placeHolder = ...
...

So, as far as I can tell: when the redirect's new tab tries to register its ClientSourceParent, the lookup table (mSourceTable, keyed by a UUID) already has a live entry under that same UUID - not a FutureClientSourceParent placeholder (which would be handled further down, and I did separately confirm that placeholder-matching path succeeds normally in unrelated, non-crashing cases during the same session). That live-entry case trips an explicit anti-spoofing guard and fails on purpose.

I don't know why the table already has a live entry - I can only speculate:

  • ClientSourceParent's entry gets removed via RemoveSource(), which is called from ActorDestroy(), which sounds like it runs as part of IPC actor teardown - i.e. asynchronously, not necessarily before the new client's constructor messagefor the redirected navigation.

  • If that's right, this might be a timing/ordering issue: whether the parent processes the old client's teardown message before or after the new client's constructor message for the same UUID. If codegen/scheduling differences (GCC+PGO vs. Clang) shift timing enough, a pre-existing race window that's rarely/never hit on Mozilla's official binaries could become much more visible on a differently-compiled binary, without either compiler doing anything incorrect.

  • There's a previous, related bug (bug https://bugzilla.mozilla.org/show_bug.cgi?id=1584007) that specifically introduced theFutureClientSourceParent placeholder mechanism to fix an earlier race in this same area. It's possible what I'm seeing is a related but distinct ordering gap that isn't covered by that fix - or it's possible this is unrelated. I genuinely don't know.

I also spent a while poking at raw memory/registers in the optimized binary trying to inspect the actual UUID values involved, but couldn't get reliable results that way, so I'm leaving that out rather than pointing anyone down a guess-based path.

No SIGSEGV, no coredump, nothing in dmesg/journalctl, nothing in about:crashes in any of the 3 reproductions - this looks like a clean, intentional IPC-level rejection rather than a memory-safety crash, for what that's worth re: the miscompilation theory in comment 19.

Firefox 153.0-1.fc44.x86_64. Happy to try more repros, or a non-PGO build if/when one is available, or to just leave this here in case someone with more context finds it useful.

BTW, the crashes are happening on non-PGO Fedora builds, so PGO is not the cause.

Used IA to investigate, got this results. May it be caused by system NSS?

Root cause: UUID PRNG not re-seeded after process fork

The log proves it:

Child 230494 generates UUID {90a0dda9-...} via CreateSourceInternal(new UUID) at this=7f2561152580
Child 230517 generates the exact same UUID {90a0dda9-...} via CreateSourceInternal(new UUID) at the same this=7f2561152580

Same UUID + same heap address = these two processes were forked from a common ancestor (fork server) and the PRNG state was duplicated without re-seeding.

Looking at the code, GenerateUUIDInPlace tries NSS first (PK11_GenerateRandomOnSlot) on the main thread. NSS uses a userspace PRNG (Fortuna). After fork, if the NSS PRNG isn't re-seeded,
both children produce identical UUID sequences.

The OS fallback (getrandom()) would be fork-safe since it's a syscall, but it's only used when NSS isn't available.

Are you using the fork server? You can check with:

Check dom.ipc.forkserver.enable in about:config.

Possible mitigations to test:

Disable fork server: set dom.ipc.forkserver.enable to false in about:config
Or set env MOZ_FORK_SERVER=0 before launching

Looks like dom.ipc.forkserver.enable does the trick.

Duplicate of this bug: 2056004

From the diff between 152.0.6 and 153.0:

--- a/ipc/glue/ForkServiceChild.cpp
+++ b/ipc/glue/ForkServiceChild.cpp
@@ -258,8 +258,6 @@ StaticRefPtr<ForkServerLauncher> ForkServerLauncher::sSingleton;
 
 ForkServerLauncher::ForkServerLauncher() = default;
 
-ForkServerLauncher::~ForkServerLauncher() = default;
-
 already_AddRefed<ForkServerLauncher> ForkServerLauncher::Create() {
   if (sSingleton == nullptr) {
     sSingleton = new ForkServerLauncher();
diff --git a/ipc/glue/ForkServiceChild.h b/ipc/glue/ForkServiceChild.h
index 923b1900b3aa..dc90fb1a5aa9 100644
--- a/ipc/glue/ForkServiceChild.h
+++ b/ipc/glue/ForkServiceChild.h
@@ -119,7 +119,7 @@ class ForkServerLauncher final : public nsIObserver {
 
  private:
   friend class ForkServiceChild;
-  ~ForkServerLauncher();
+  ~ForkServerLauncher() = default;
 
   static void RestartForkServer();

Matters?

You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: