2072864 - Crash Report [@ IPCError-browser | GPUProcessKill ] Firefox, Windows 10

Status: RESOLVED DUPLICATE of bug 1908798

(Firefox :: Untriaged, defect)

Description

[Robert Hartmann]

Steps to reproduce:

sorry no idea

Actual results:

Firefox 153.0.4 Crash Report [@ IPCError-browser | GPUProcessKill ]
https://crash-stats.mozilla.org/report/index/282be80d-ec05-4377-902c-52a8f0260827

Comment 1

[Robert Hartmann]

for me it is unclear, if this Firefox crash is related to Thunderbird crash bug 2072450 or are related to the bug 1908798 for component Graphics: WebRender.

https://crash-stats.mozilla.org/report/index/282be80d-ec05-4377-902c-52a8f0260827

Crash Reason

EXCEPTION_BREAKPOINT

Main thread:

0 	win32u.dll 	ZwUserMsgWaitForMultipleObjectsEx 		context
1 	user32.dll 	RealMsgWaitForMultipleObjectsEx(unsigned long, void* const*, unsigned long, unsigned long, unsigned long) 		cfi
2 	xul.dll 	mozilla::widget::WinUtils::WaitForMessage(unsigned long) 	widget/windows/WinUtils.cpp:406 	cfi
3 	xul.dll 	nsAppShell::ProcessNextNativeEvent(bool) 	widget/windows/nsAppShell.cpp:805 	cfi
4 	xul.dll 	nsBaseAppShell::DoProcessNextNativeEvent(bool) 	widget/nsBaseAppShell.cpp:134 	inlined
4 	xul.dll 	nsBaseAppShell::OnProcessNextEvent(nsIThreadInternal*, bool) 	widget/nsBaseAppShell.cpp:270 	inlined
4 	xul.dll 	nsThread::ProcessNextEvent(bool, bool*) 	xpcom/threads/nsThread.cpp:1118 	inlined
4 	xul.dll 	NS_ProcessNextEvent(nsIThread*, bool) 	xpcom/threads/nsThreadUtils.cpp:472 	cfi
5 	xul.dll 	mozilla::ipc::MessagePump::Run(base::MessagePump::Delegate*) 	ipc/glue/MessagePump.cpp:105 	cfi
6 	xul.dll 	MessageLoop::RunInternal() 	ipc/chromium/src/base/message_loop.cc:371 	inlined
6 	xul.dll 	MessageLoop::RunHandler() 	ipc/chromium/src/base/message_loop.cc:364 	cfi
7 	xul.dll 	MessageLoop::Run() 	ipc/chromium/src/base/message_loop.cc:346 	inlined
7 	xul.dll 	nsBaseAppShell::Run() 	widget/nsBaseAppShell.cpp:151 	cfi
8 	xul.dll 	nsAppShell::Run() 	widget/windows/nsAppShell.cpp:682 	cfi
9 	xul.dll 	XRE_RunAppShell() 	toolkit/xre/nsEmbedFunctions.cpp:656 	inlined
9 	xul.dll 	mozilla::ipc::MessagePumpForChildProcess::Run(base::MessagePump::Delegate*) 	ipc/glue/MessagePump.cpp:233 	cfi
10 	xul.dll 	MessageLoop::RunInternal() 	ipc/chromium/src/base/message_loop.cc:371 	inlined
10 	xul.dll 	MessageLoop::RunHandler() 	ipc/chromium/src/base/message_loop.cc:364 	cfi
11 	xul.dll 	MessageLoop::Run() 	ipc/chromium/src/base/message_loop.cc:346 	inlined
11 	xul.dll 	XRE_InitChildProcess(int, char**, XREChildData const*) 	toolkit/xre/nsEmbedFunctions.cpp:594 	inlined
11 	xul.dll 	mozilla::BootstrapImpl::XRE_InitChildProcess(int, char**, XREChildData const*) 	toolkit/xre/Bootstrap.cpp:59 	cfi
12 	firefox.exe 	NS_internal_main(int, char**, char**) 	browser/app/nsBrowserApp.cpp:466 	inlined
12 	firefox.exe 	wmain(int, wchar_t**) 	toolkit/xre/nsWindowsWMain.cpp:150 	cfi
13 	firefox.exe 	invoke_main() 	/builds/worker/workspace/obj-build/browser/app/D:/a/_work/1/s/src/vctools/crt/vcstartup/src/startup/exe_common.inl:90 	inlined
13 	firefox.exe 	__scrt_common_main_seh() 	/builds/worker/workspace/obj-build/browser/app/D:/a/_work/1/s/src/vctools/crt/vcstartup/src/startup/exe_common.inl:288 	cfi
14 	kernel32.dll 	BaseThreadInitThunk 		cfi
15 	ntdll.dll 	RtlUserThreadStart 		cfi

Comment 2

[Wayne Mery (:wsmwk)]

I would expect them to be the same

Comment 3

[Clouseau (:calixte)]

Crash report: https://crash-stats.mozilla.org/report/index/273606ef-3d99-408a-adaa-86c620260918

Reports with a different stack in the same spike: https://crash-stats.mozilla.org/report/index/fcc4e9b3-fc60-4949-b570-549640260919

Crash Reason:

EXCEPTION_BREAKPOINT at 0x00007ffe4788aee4

Top 10 frames of the hung main thread (nothing crashed here — a watchdog killed the process; these are what the main thread is waiting on):

0  win32u.dll  ZwUserMsgWaitForMultipleObjectsEx
1  xul.dll  mozilla::widget::WinUtils::WaitForMessage(unsigned long)  widget/windows/WinUtils.cpp:418
2  xul.dll  nsAppShell::ProcessNextNativeEvent(bool)  widget/windows/nsAppShell.cpp:755
3  xul.dll  NS_ProcessNextEvent(nsIThread*, bool)  xpcom/threads/nsThreadUtils.cpp:471
4  xul.dll  mozilla::ipc::MessagePump::Run(base::MessagePump::Delegate*)  ipc/glue/MessagePump.cpp:105
5  xul.dll  MessageLoop::RunHandler()  ipc/chromium/src/base/message_loop.cc:364
6  xul.dll  nsBaseAppShell::Run()  widget/nsBaseAppShell.cpp:151
7  xul.dll  nsAppShell::Run()  widget/windows/nsAppShell.cpp:632
8  xul.dll  mozilla::ipc::MessagePumpForChildProcess::Run(base::MessagePump::Delegate*)  ipc/glue/MessagePump.cpp:233
9  xul.dll  MessageLoop::RunHandler()  ipc/chromium/src/base/message_loop.cc:364

Filed because this signature's crash volume spiked. 6 reports from 6 distinct installations on the nightly build of 2026-09-18 (buildid 20260918092903); none of the preceding 3 build-days had any, and no build in the 21 days before had any report of it. That is an appearance from zero over the loudest of the preceding build-days and of this signature's builds over the 21 days before (Clouseau's bar is 3x, and the Poisson excess of the day against that baseline is z = 3.83, bar 3.62). This signature is not new: its first report is in build 20240514215634 (2024-05-14), 856 days before this build.

Clouseau analysis (automated -- nothing below was checked by a human; a claim it could not ground is deliberately absent):

These reports only exist when the parent succeeds in taking a paired minidump of the GPU process it is killing: GPUChild::GeneratePairedMinidump sets ipc_channel_error = "GPUProcessKill" and calls CrashReporterHost::GenerateMinidumpAndPair. That paired-minidump capture had been non-functional since the out-of-process crash-generation transition (no reports at all on 155.0a1/156.0a1/157.0a1, and none on 158.0a1 builds before 2026-09-18), and 40a64974956f (bug 1949178, 'Re-enable capturing child process shutdown hangs') restored it in exactly the spiking build 20260918092903 — the unrelated paired-minidump signature IPCError-browser | ShutDownKill reappears in the same builds after the same three-version gap. The underlying event is the long-standing one: the parent kills the GPU process after a sync IPC reply timeout (graphics_critical_error: [GFX1-]: Killing GPU process due to IPC reply timeout), and at dump time the GPU process's MainThread and Compositor thread are both idle, so the stall had already passed. No code change is warranted for the spike itself; the restored reports (and the paired browser minidump they carry) should be used to progress the existing GPU reply-timeout investigation rather than treated as a new graphics regression.

Suspected regressor (high confidence): 40a64974956f (bug 1949178) by :gsvelto.

This changeset (landed 2026-09-17, first nightly 20260918092903) replaces the breakpad in-process implementation of CrashReporter::CreateMinidumpsAndPair (gExceptionHandler->dump_path() + google_breakpad::ExceptionHandler::WriteMinidumpForChild, read at parent rev f6441079b409, toolkit/crashreporter/nsExceptionHandler.cpp:3675-3710) with a crash-helper implementation (generate_crash_report(gCrashHelperClient, aId, aTargetBlamedThread)), and moves CrashReporterHost::GenerateMinidumpAndPair out of the header into ipc/glue/CrashReporterHost.cpp. That function is the only producer of this signature (gfx/ipc/GPUChild.cpp:115-133 sets ipc_channel_error="GPUProcessKill" and calls it). It restores report generation, it does not cause GPU-process kills: the causal link is to the population change only, and it is corroborated by the identical disappearance/return of IPCError-browser | ShutDownKill, which uses the same paired-minidump machinery.

Possible path to the crash: Mechanism (Observed): nothing faulted — the minidump is generated on demand by the parent while killing the GPU process; crash_info.type is EXCEPTION_BREAKPOINT at a ret, which is the dumping artefact, and the GPU process's own threads are idle (MainThread in WinUtils::WaitForMessage, Compositor blocked in NS_ProcessNextEvent's condvar). Path of the kill (Observed): parent-side CompositorManagerChild::ShouldContinueFromReplyTimeout -> GPUProcessManager::KillProcess -> GPUProcessHost::KillHard(aGenerateMinidump=true) -> GPUChild::GeneratePairedMinidump, which annotates ipc_channel_error="GPUProcessKill" and calls CrashReporterHost::GenerateMinidumpAndPair; the representative report carries graphics_critical_error: |[0][GFX1-]: Killing GPU process due to IPC reply timeout (t=17.8056). Path of the SPIKE (Inferred, high): GenerateMinidumpAndPair -> CreateMinidumpsAndPair was failing for builds after ~2026-06-22 (crash-helper/OOP crash-generation transition), so the kills produced no reports; 40a64974956f reimplemented it on the crash helper and reports resumed in the first build containing it. Trigger of the individual crashes (Inferred, medium): a transient stall of the GPU process (or of the sync round-trip) exceeding the compositor reply timeout ~15-18 s after GPU process start; because all Gecko threads in the GPU process are idle when the dump is taken, the stalling work had already completed and is not visible in this dump. Trigger of the population change (Observed): the landing of bug 1949178 in build 20260918092903.

Checked:

• The signature is produced only by GPUChild::GeneratePairedMinidump, which sets Annotation::ipc_channel_error to "GPUProcessKill" and then calls mCrashReporter->GenerateMinidumpAndPair(mHost, "browser"_ns) (gfx/ipc/GPUChild.cpp:115-133 @ rev f3bc7391657f (raw_file, pinned build rev))
• GeneratePairedMinidump is reached from GPUProcessHost::KillHard(bool aGenerateMinidump) (gfx/ipc/GPUProcessHost.cpp:319-324), and GPUProcessManager::KillProcess is called by CompositorManagerChild::ShouldContinueFromReplyTimeout (gfx/ipc/GPUProcessHost.cpp:319-341 @ rev f3bc7391657f; searchfox calls_to mozilla::gfx::GPUProcessManager::KillProcess (mozilla-central tip): gfx/layers/ipc/CompositorManagerChild.cpp:240)
• Representative report: process_type gpu, uptime 15 s, ipc_channel_error: GPUProcessKill, graphics_critical_error: |[0][GFX1-]: Killing GPU process due to IPC reply timeout (t=17.8056), Intel 0x9a49 driver 32.0.101.7088, no bit-flip annotation, CPU family 6 model 140 (not the defective Raptor Lake model) (crash report 273606ef-3d99-408a-adaa-86c620260918)
• At dump time the GPU process was idle: MainThread in mozilla::widget::WinUtils::WaitForMessage and the Compositor thread parked in NS_ProcessNextEvent -> ConditionVariableImpl::wait (crash report 273606ef-3d99-408a-adaa-86c620260918, thread 0 (brief) and thread 5 (Compositor))
• 40a64974956f (bug 1949178, author gsvelto, date 2026-09-17, parent f6441079b409, not backed out) rewrites CreateMinidumpsAndPair to call generate_crash_report(gCrashHelperClient, ...) and adds CrashReporterHost::GenerateMinidumpAndPair to ipc/glue/CrashReporterHost.cpp (hg changeset 40a64974956f + patch diff (files: ipc/glue/CrashReporterHost.{cpp,h}, toolkit/crashreporter/nsExceptionHandler.{cpp,h}, nsDummyExceptionHandler.cpp))
• Before that changeset, CreateMinidumpsAndPair was the in-process breakpad implementation: if (!GetEnabled()) return false;, gExceptionHandler->dump_path(), google_breakpad::ExceptionHandler::WriteMinidumpForChild(aTargetHandle, ...) (toolkit/crashreporter/nsExceptionHandler.cpp:3675-3710 @ rev f6441079b409 (parent of the culprit))
• Bug 1949178 is 'Re-enable capturing minidumps of child shutdown hangs after OOP crash generation lands', Toolkit :: Crash Reporting, RESOLVED FIXED (bug 1949178)
• This signature has zero reports on 155.0a1, 156.0a1 and 157.0a1: the 180-day version facet lists 154.0a1 (151) down to 135.0a1 (2) with none of 155/156/157, and 158.0a1 has 8 (facets signature=IPCError-browser | GPUProcessKill field=version days=180 (and days=30: 158.0a1 8, remainder all <=154.0a1))

Alternatives considered:

• A real increase in GPU-process kills (a graphics regression in the 09-18 push): not supported -- an unrelated signature using the same paired-minidump machinery (IPCError-browser | ShutDownKill) returns in the very same builds after the very same three-version gap, and the restored per-1000 rate of this signature on 158.0a1 is below its own 151-153 rates.
• bug 2072747 (WebRender scene-builder/clip/interning series, Pt 1-5, same push): not supported -- no frame, annotation or facet in the spiking reports ties to it; the GPU process's Compositor, Renderer and WRRenderBackend threads are idle at dump time, and the collection explanation also accounts for the simultaneous return of a non-graphics signature.
• a29452e2f3fe / bug 1880503 (the ordinary pipeline's candidate, 'Send crash reports when GPU process doesn't respond to pause/resume on Android'): disfavored as the culprit of this spike -- the parent-side reply-timeout kill it describes is indeed the kill path (confirmed by graphics_critical_error), but bug 1880503 is GeckoView :: General and long resolved, was present in all the 158.0a1 builds from 09-10..09-17 that produced zero reports, and cannot produce a Windows-desktop population step on 2026-09-18.
• Failing hardware (bit flips / defective Raptor Lake): not supported -- 0% of this signature's yearly reports carry a bit-flip annotation, and the two inspected spike reports are family 6 model 140 (Tiger Lake) and family 6 model 42, neither the defective model; the population change is explained by a code landing in one build.
• A fault in the frames shown (WinUtils::WaitForMessage / message pump): ruled out -- the dump is generated on demand by the parent while killing the process; crash_info.instruction is ret with EXCEPTION_BREAKPOINT, the long-standing reason for 1613 of this signature's reports, and every Gecko thread in the process is parked in a wait.

Worth checking first:

• Which changeset actually disabled paired-minidump capture between the 2026-06-21 builds and the 155.0a1 cycle: check whether CrashReporter::GetEnabled() / gExceptionHandler is false in the parent once the crash helper is active, against the crash-helper landings touching toolkit/crashreporter/nsExceptionHandler.cpp on 2026-06-22 (bug 1955963, 77310ded9dfc) and 2026-06-24 (bug 2040673). This determines whether anything else is still silently not being captured.
• Why the sync IPC reply timed out in these six installations: the GPU process threads are all idle by dump time, so the stall is not in this dump -- the browser additional minidump captured alongside each of these reports (additional_minidumps = browser) should be read for the parent's blocked sync PWebRenderBridge/PCompositorManager send, and GPUProcessLaunchCount / GraphicsNumRenderers compared across the cohort.
• Why the single 158.0a1 build 20260914042611 produced 9 ShutDownKill reports before the re-land while other pre-09-18 158.0a1 builds produced none: check whether that build ran with the crash helper disabled (falling back to the in-process breakpad path) or whether those reports come from one installation.
• Whether the restored crash-helper path attaches the correct target-process annotations (the new code adds DumperError on failure): facet crash_report_keys / the DumperError annotation on this signature once more builds have shipped, to confirm the restored reports are complete and not silently degraded.

:gsvelto, can you have a look please?

Filed automatically by Clouseau, which analyses nightly crashes with an LLM. Nothing above was written or checked by a human. Please close it as INVALID if it is wrong — that is useful feedback, not a nuisance.

This is not an independent Mozilla discovery: the crash is known here only because somebody submitted the crash report linked at the top. If this duplicates an existing report, please resolve THIS bug as the duplicate and leave the credit with the earlier reporter.

Comment 4

[Gabriele Svelto [:gsvelto]]

This is an issue with the GPU process not responding to the main process.