1908798 - Crash in [@ IPCError-browser | GPUProcessKill]

Status: NEW

(Core :: Graphics: WebRender, defect, P2)

Description

[Andrew McCreight [:mccr8]]

Crash report: https://crash-stats.mozilla.org/report/index/c95a1ffd-5281-48ee-96a4-59e440240718

Reason: EXCEPTION_BREAKPOINT

Top 10 frames:

0  win32u.dll  ZwUserMsgWaitForMultipleObjectsEx
1  user32.dll  RealMsgWaitForMultipleObjectsEx(unsigned long, void* const*, unsigned long, u...
2  xul.dll  mozilla::widget::WinUtils::WaitForMessage(unsigned long)  widget/windows/WinUtils.cpp:454
3  xul.dll  nsAppShell::ProcessNextNativeEvent(bool)  widget/windows/nsAppShell.cpp:796
4  xul.dll  nsBaseAppShell::DoProcessNextNativeEvent(bool)  widget/nsBaseAppShell.cpp:131
4  xul.dll  nsBaseAppShell::OnProcessNextEvent(nsIThreadInternal*, bool)  widget/nsBaseAppShell.cpp:267
4  xul.dll  nsThread::ProcessNextEvent(bool, bool*)  xpcom/threads/nsThread.cpp:1119
4  xul.dll  NS_ProcessNextEvent(nsIThread*, bool)  xpcom/threads/nsThreadUtils.cpp:480
5  xul.dll  mozilla::ipc::MessagePump::Run(base::MessagePump::Delegate*)  ipc/glue/MessagePump.cpp:107
6  xul.dll  MessageLoop::RunInternal()  ipc/chromium/src/base/message_loop.cc:370

I don't know if this is the same as bug 1900134 or not, but the signature is slightly different, so I'll file a new bug for it. Decent volume on Nightly.

Comment 1

[BugBot [:suhaib / :marco]]

The bug is linked to a topcrash signature, which matches the following criteria:

• Top 10 desktop browser crashes on nightly (startup)
• Top 5 GPU process crashes on release (startup)

For more information, please visit BugBot documentation.

Comment 2

[Brad Werth [:bradwerth]]

This is going to be hard to investigate unless we can get a reproduction case or a more informative call stack.

The startup crashes I checked have GraphicsCriticalError of shader-cache: Shader disk cache is not supported. The non-startup crashes seem to have timeouts of different kinds, including shader-cache: Timed out before finishing loads and Killing GPU process due to IPC reply timeout.

Jamie, you're in that code that sets those errors. Are they signifying anything useful in this case?

Comment 3

[Jamie Nicol [:jnicol]]

I'm not sure why the signature for this and bug bug 1900134 differ, but the other bug is all android crashes and this is windows, so keeping them separate makes sense.

As I said in the other bug, this crash was introduced by bug 1880503, but these are pre-existing issues - previously we would just silently kill the GPU process, whereas now we get a crash report too.

I think the significance of the shader cache errors is that they indicate a really old or slow device. These crashes mostly seem to be occurring during webrender initialization (Compositor thread is in WebRenderAPI::Create, renderer thread at various stages of initialization.) Perhaps we need a longer timeout than 10 seconds? I expect the current experience is pretty terrible for affected users - several 10 second timeouts attempting to initialize webrender then eventually the GPU process gets disabled, then presumably a >10s wait to initialize webrender in the parent process.

Comment 4

[Bob Hood [:bhood]]

Dropping this into WebRender for a look-see.

Glenn, not expecting you'll be able to do a lot with this, but given the low crash volume now, do you think this remains valid as an S2?

Comment 5

[Glenn Watson [:gw]]

It does seem to have spiked and be ~100 / day, S2 is probably reasonable if we can do something about it.

Comment 6

[Bob Hood [:bhood]]

Adding a responsible party, and setting to P1 so it shows up on my internal tracking radar.

Comment 7

[Glenn Watson [:gw]]

Sotaro, in the crash report it looks like the Compositor thread is trying to shut down - any ideas what might cause this IPC error or what we should look at?

Comment 8

[Sotaro Ikeda [:sotaro]]

As in comment 3, the crash is pre-existing issues for long time. Just bug 1880503 has just been changed to generate a crash report.

The crash happened by killing GPU process from CompositorManagerChild::ShouldContinueFromReplyTimeout() by calling GPUProcessManager::KillProcess().

CompositorManagerChild::ShouldContinueFromReplyTimeout() is called when sync IPC failed with reply timeout from MessageChannel::ShouldContinueFromTimeout().

Reply timeout is set from CompositorManagerChild::SetReplyTimeout().

Comment 9

[BugBot [:suhaib / :marco]]

Set release status flags based on info from the regressing bug 1880503

Comment 10

[Sotaro Ikeda [:sotaro]]

From the crash reports, majority of the cashes seemed to happen within 60 sec from starting Firefox.

In this case, the crash reports tend to have the following functions in calling stacks.

• Compositor thread: CompositorBridgeParent::AllocPWebRenderBridgeParent()
• Compositor thread: WebRenderAPI::Create()
• Renderer thread: RenderThread::InitDeviceTask

Comment 11

[Sotaro Ikeda [:sotaro]]

There are several sync IPC message under gfx.
https://searchfox.org/mozilla-central/search?q=+sync+&path=gfx**.ipdl&case=false&regexp=false

The crash of comment 10 might be triggered by PWebRenderBridgeChild::SendEnsureConnected(). It is sync IPC.

Comment 12

[Sotaro Ikeda [:sotaro]]

It might be possible to relax CompositorManagerChild::ShouldContinueFromReplyTimeout() until RenderThread::InitDeviceTask() complete in GPU process.

Comment 13

[Sotaro Ikeda [:sotaro]]

As in comment 3, the crash seemed to tend to happen with a really old or slow device. With these devices, there might be cases that Windows system call became very slow. It might trigger sync IPC reply timeout.

Comment 14

[Sotaro Ikeda [:sotaro]]

From the following document, I obtained Firefox profiler during Firefox startup, and synchronization IPC did not take a long time during normal Firefox startup.

https://profiler.firefox.com/docs/#/./guide-startup-shutdown

Comment 15

[BugBot [:suhaib / :marco]]

Set release status flags based on info from the regressing bug 1880503

Comment 16

[Matthew Gaudet (he/him) [:mgaudet]]

A friend sent me a crash report which came to this signature. He says he "has been plagued by them for the last couple months". The user experiences for this was the following:

• Had long lived firefox session. Opened a link, and firefox appeared to freeze.
• Windows that were minimized wouldn't come back
• Only process in task manager not responding still seemed to be allocating memory (50kb/s -- not sure how he measured this)
• Ended up killing process.

If there's other debugging information desired, I can relay instructions

Comment 17

[Glenn Watson [:gw]]

Sotaro, I'm assigning this to you as you seem to have an understanding of what's going on here. Do we have a path forward to work on this?

Comment 18

[Sotaro Ikeda [:sotaro]]

(In reply to Matthew Gaudet (he/him) [:mgaudet] from comment #16)

A friend sent me a crash report which came to this signature. He says he "has been plagued by them for the last couple months". The user experiences for this was the following:

In the crash report, Compositor thread called CompositorBridgeParent::PauseComposition(). And RendererThread called
SyncObjectD3D11Host::Synchronize().

Comment 19

[Sotaro Ikeda [:sotaro]]

(In reply to Sotaro Ikeda [:sotaro] from comment #11)

There are several sync IPC message under gfx.
https://searchfox.org/mozilla-central/search?q=+sync+&path=gfx**.ipdl&case=false&regexp=false

The crash of comment 10 might be triggered by PWebRenderBridgeChild::SendEnsureConnected(). It is sync IPC.

Since there are a lot of sync IPC exist, it seems better to use a longer timeout than 10 seconds like comment 3 at first.

Comment 20

[Sotaro Ikeda [:sotaro]]

(In reply to Sotaro Ikeda [:sotaro] from comment #18)

(In reply to Matthew Gaudet (he/him) [:mgaudet] from comment #16)

A friend sent me a crash report which came to this signature. He says he "has been plagued by them for the last couple months". The user experiences for this was the following:

In the crash report, Compositor thread called CompositorBridgeParent::PauseComposition(). And RendererThread called
SyncObjectD3D11Host::Synchronize().

CompositorBridgeParent::PauseComposition() is called like the following sequence.

CanonicalBrowsingContext::RecomputeAppWindowVisibility()
->nsIWidget::PauseOrResumeCompositor()
->CompositorBridgeChild::SendPause()
// sync IPC to GPU process
->CompositorBridgeParent::RecvPause()
->CompositorBridgeParent::PauseComposition()
->WebRenderBridgeParent::Pause()
->WebRenderAPI::Pause()
->//Post task to RenderThread
->RenderThread::Pause()
->RenderCompositorANGLE::Pause() {} //Do nothing

Comment 21

[Sotaro Ikeda [:sotaro]]

WebRenderAPI::Pause() post task to RenderThread by calling WebRenderAPI::RunOnRenderThread().

It end up to call "self.low_priority_scene_sender.send(msg).unwrap()" in RenderApi::send_external_event(). It might take long time until the posted task run in RenderThread, since low_priority_scene_sender is used.

Comment 22

[Sotaro Ikeda [:sotaro]]

It is nice if RenderThread::Pause() could be called without low_priority_scene_sender. In this case, we need to care about consistency between WebRender and GL/EGL(EGLSurface).

Short term workaround for Windows is skip RenderThread::Pause() and RenderThread::Resume(), since RenderCompositorANGLE::Pause() and RenderCompositorANGLE::Resume() do nothing.

Comment 23

[Sotaro Ikeda [:sotaro]]

Bug 1922214 was created for comment 22.

Comment 24

[Sotaro Ikeda [:sotaro]]

There were several crashes with CompositorBridgeParent::RecvFlushRendering() in Compositor thread. It is triggered like the following

nsViewManager::Refresh()
->WebRenderLayerManager::FlushRendering()
->CompositorBridgeChild::SendFlushRendering() // Triggers sync IPC
->// sync IPC
->CompositorBridgeParent::RecvFlushRendering()

On Windows, WebRenderLayerManager::FlushRendering() always calls CompositorBridgeChild::SendFlushRendering(), since pref layers.force-synchronous-resize = true.
https://searchfox.org/mozilla-central/rev/ce404cd26e52d09e6a48d664c1986da25df50484/modules/libpref/init/StaticPrefList.yaml#8543

It is nice if we could remove/reduce sync FlushRendering on Windows.

Comment 25

[Sotaro Ikeda [:sotaro]]

Since Bug 1922157 fix, the amount of crashes seems to have decreased in nightly.

Comment 26

[Bob Hood [:bhood]]

Sotaro, it looks like the crash volume has increased again. Is there anything blocking us from moving forward with a fix here?

Comment 27

[Robert Hartmann]

Two Crashes with this signature.

• Firefox 141.0b3
• Firefox 142.0a1

Firefox 141.0b3 Crash Report [@ IPCError-browser | GPUProcessKill ]
Crash ID: bp-3180fcf3-bfb3-474e-ab83-13d8f0250815

Frame 	Module 	Signature 	Source 	Trust
0 	win32u.dll 	ZwUserMsgWaitForMultipleObjectsEx 		context
1 	user32.dll 	RealMsgWaitForMultipleObjectsEx(unsigned long, void* const*, unsigned long, unsigned long, unsigned long) 		cfi 
2 	xul.dll 	mozilla::widget::WinUtils::WaitForMessage(unsigned long) 	widget/windows/WinUtils.cpp:451 	cfi
3 	xul.dll 	nsAppShell::ProcessNextNativeEvent(bool) 	widget/windows/nsAppShell.cpp:796 	cfi
4 	xul.dll 	nsBaseAppShell::DoProcessNextNativeEvent(bool) 	widget/nsBaseAppShell.cpp:131 	inlined
4 	xul.dll 	nsBaseAppShell::OnProcessNextEvent(nsIThreadInternal*, bool) 	widget/nsBaseAppShell.cpp:267 	inlined
4 	xul.dll 	nsThread::ProcessNextEvent(bool, bool*) 	xpcom/threads/nsThread.cpp:1098 	inlined
4 	xul.dll 	NS_ProcessNextEvent(nsIThread*, bool) 	xpcom/threads/nsThreadUtils.cpp:480 	cfi
5 	xul.dll 	mozilla::ipc::MessagePump::Run(base::MessagePump::Delegate*) 	ipc/glue/MessagePump.cpp:107 	cfi
6 	xul.dll 	MessageLoop::RunInternal() 	ipc/chromium/src/base/message_loop.cc:369 	inlined
6 	xul.dll 	MessageLoop::RunHandler() 	ipc/chromium/src/base/message_loop.cc:362 	cfi
7 	xul.dll 	MessageLoop::Run() 	ipc/chromium/src/base/message_loop.cc:344 	inlined
7 	xul.dll 	nsBaseAppShell::Run() 	widget/nsBaseAppShell.cpp:148 	cfi
8 	xul.dll 	nsAppShell::Run() 	widget/windows/nsAppShell.cpp:673 	cfi
9 	xul.dll 	XRE_RunAppShell() 	toolkit/xre/nsEmbedFunctions.cpp:652 	inlined
9 	xul.dll 	mozilla::ipc::MessagePumpForChildProcess::Run(base::MessagePump::Delegate*) 	ipc/glue/MessagePump.cpp:235 	cfi
10 	xul.dll 	MessageLoop::RunInternal() 	ipc/chromium/src/base/message_loop.cc:369 	inlined
10 	xul.dll 	MessageLoop::RunHandler() 	ipc/chromium/src/base/message_loop.cc:362 	cfi 
11 	xul.dll 	MessageLoop::Run() 	ipc/chromium/src/base/message_loop.cc:344 	inlined
11 	xul.dll 	XRE_InitChildProcess(int, char**, XREChildData const*) 	toolkit/xre/nsEmbedFunctions.cpp:590 	inlined
11 	xul.dll 	mozilla::BootstrapImpl::XRE_InitChildProcess(int, char**, XREChildData const*) 	toolkit/xre/Bootstrap.cpp:60 	cfi
12 	firefox.exe 	NS_internal_main(int, char**, char**) 	browser/app/nsBrowserApp.cpp:397 	inlined
12 	firefox.exe 	wmain(int, wchar_t**) 	toolkit/xre/nsWindowsWMain.cpp:151 	cfi
13 	firefox.exe 	invoke_main() 	/builds/worker/workspace/obj-build/browser/app/D:/a/_work/1/s/src/vctools/crt/vcstartup/src/startup/exe_common.inl:90 	inlined
13 	firefox.exe 	__scrt_common_main_seh() 	/builds/worker/workspace/obj-build/browser/app/D:/a/_work/1/s/src/vctools/crt/vcstartup/src/startup/exe_common.inl:288 	cfi
14 	kernel32.dll 	BaseThreadInitThunk 		cfi
15 	ntdll.dll 	RtlUserThreadStart 		cfi

Firefox 142.0a1 Crash Report [@ IPCError-browser | GPUProcessKill ]
Crash ID: bp-4107237a-3317-4541-9e81-259ac0250815

Frame 	Module 	Signature 	Source 	Trust
0 	win32u.dll 	ZwUserMsgWaitForMultipleObjectsEx 		context
1 	user32.dll 	RealMsgWaitForMultipleObjectsEx(unsigned long, void* const*, unsigned long, unsigned long, unsigned long) 		cfi
2 	xul.dll 	mozilla::widget::WinUtils::WaitForMessage(unsigned long) 	widget/windows/WinUtils.cpp:451 	cfi
3 	xul.dll 	nsAppShell::ProcessNextNativeEvent(bool) 	widget/windows/nsAppShell.cpp:796 	cfi
4 	xul.dll 	nsBaseAppShell::DoProcessNextNativeEvent(bool) 	widget/nsBaseAppShell.cpp:131 	inlined
4 	xul.dll 	nsBaseAppShell::OnProcessNextEvent(nsIThreadInternal*, bool) 	widget/nsBaseAppShell.cpp:267 	inlined
4 	xul.dll 	nsThread::ProcessNextEvent(bool, bool*) 	xpcom/threads/nsThread.cpp:1098 	inlined
4 	xul.dll 	NS_ProcessNextEvent(nsIThread*, bool) 	xpcom/threads/nsThreadUtils.cpp:480 	cfi
5 	xul.dll 	mozilla::ipc::MessagePump::Run(base::MessagePump::Delegate*) 	ipc/glue/MessagePump.cpp:107 	cfi
6 	xul.dll 	MessageLoop::RunInternal() 	ipc/chromium/src/base/message_loop.cc:369 	inlined
6 	xul.dll 	MessageLoop::RunHandler() 	ipc/chromium/src/base/message_loop.cc:362 	cfi
7 	xul.dll 	MessageLoop::Run() 	ipc/chromium/src/base/message_loop.cc:344 	inlined
7 	xul.dll 	nsBaseAppShell::Run() 	widget/nsBaseAppShell.cpp:148 	cfi
8 	xul.dll 	nsAppShell::Run() 	widget/windows/nsAppShell.cpp:673 	cfi
9 	xul.dll 	XRE_RunAppShell() 	toolkit/xre/nsEmbedFunctions.cpp:652 	inlined
9 	xul.dll 	mozilla::ipc::MessagePumpForChildProcess::Run(base::MessagePump::Delegate*) 	ipc/glue/MessagePump.cpp:235 	cfi
10 	xul.dll 	MessageLoop::RunInternal() 	ipc/chromium/src/base/message_loop.cc:369 	inlined
10 	xul.dll 	MessageLoop::RunHandler() 	ipc/chromium/src/base/message_loop.cc:362 	cfi
11 	xul.dll 	MessageLoop::Run() 	ipc/chromium/src/base/message_loop.cc:344 	inlined
11 	xul.dll 	XRE_InitChildProcess(int, char**, XREChildData const*) 	toolkit/xre/nsEmbedFunctions.cpp:590 	inlined
11 	xul.dll 	mozilla::BootstrapImpl::XRE_InitChildProcess(int, char**, XREChildData const*) 	toolkit/xre/Bootstrap.cpp:60 	cfi
12 	firefox.exe 	NS_internal_main(int, char**, char**) 	browser/app/nsBrowserApp.cpp:397 	inlined
12 	firefox.exe 	wmain(int, wchar_t**) 	toolkit/xre/nsWindowsWMain.cpp:151 	cfi
13 	firefox.exe 	invoke_main() 	/builds/worker/workspace/obj-build/browser/app/D:/a/_work/1/s/src/vctools/crt/vcstartup/src/startup/exe_common.inl:90 	inlined
13 	firefox.exe 	__scrt_common_main_seh() 	/builds/worker/workspace/obj-build/browser/app/D:/a/_work/1/s/src/vctools/crt/vcstartup/src/startup/exe_common.inl:288 	cfi
14 	kernel32.dll 	BaseThreadInitThunk 		cfi
15 	ntdll.dll 	RtlUserThreadStart 		cfi

Comment 29

[Matthew Gaudet (he/him) [:mgaudet]]

This is apparently back again: https://crash-stats.mozilla.org/report/index/3d903d32-8d33-4563-81d3-9e9590251023

Comment 30

[Andy Leiserson [:aleiserson]]

I filed bug 1996653 for this signature on MacOS (where GPU process is enabled only in Nightly).

But the big uptick recently is specific to Android, which seems likely to be a separate issue. https://crash-stats.mozilla.org/signature/?signature=IPCError-browser%20%7C%20GPUProcessKill&date=%3E%3D2025-07-27T18%3A08%3A00.000Z&date=%3C2025-10-27T18%3A08%3A00.000Z#graphs (the default graph is by product, the graph by platform is also illustrative).

Comment 31

[Jamie Nicol [:jnicol]]

I had a look through the recent android crashes and couldn't see any obvious reason for a spike - they are on a range of devices, and they follow the usual pattern of the compositor thread waiting on the renderer thread, either for a pause or a readback, etc. And the renderer thread stacks are garbage. And there were no shader compilation annotations which has been a previous reason for a spike.

However, it looks like the recent spike in this signature on android exactly correlates with the decline in reports in bug 1900134. When we first started reporting these timeouts I wasn't sure why the android signatures came under IPCError-content and non-android under IPCError-browser, but for whatever reason it looks like we're still receiving roughly the same number of reports but android ones are now appearing as IPCError-browser as well.

Comment 32

[Robert Hartmann]

Windows 10, 64 bit,
Firefox 64bit 151.0b3 Crash Report [@ IPCError-browser | GPUProcessKill ]
https://crash-stats.mozilla.org/report/index/e7f6c76e-c828-40d0-8a9e-3fdbc0260529

Crashing Thread (0), Name: MainThread
Frame 	Module 	Signature 	Source 	Trust
0 	win32u.dll 	ZwUserMsgWaitForMultipleObjectsEx 		context
1 	user32.dll 	RealMsgWaitForMultipleObjectsEx(unsigned long, void* const*, unsigned long, unsigned long, unsigned long) 		cfi
2 	xul.dll 	mozilla::widget::WinUtils::WaitForMessage(unsigned long) 	widget/windows/WinUtils.cpp:406 	inlined
2 	xul.dll 	nsAppShell::ProcessNextNativeEvent(bool) 	widget/windows/nsAppShell.cpp:795 	cfi
3 	xul.dll 	nsBaseAppShell::DoProcessNextNativeEvent(bool) 	widget/nsBaseAppShell.cpp:134 	inlined
3 	xul.dll 	nsBaseAppShell::OnProcessNextEvent(nsIThreadInternal*, bool) 	widget/nsBaseAppShell.cpp:270 	inlined
3 	xul.dll 	nsThread::ProcessNextEvent(bool, bool*) 	xpcom/threads/nsThread.cpp:1118 	inlined
3 	xul.dll 	NS_ProcessNextEvent(nsIThread*, bool) 	xpcom/threads/nsThreadUtils.cpp:465 	cfi
4 	xul.dll 	mozilla::ipc::MessagePump::Run(base::MessagePump::Delegate*) 	ipc/glue/MessagePump.cpp:105 	cfi
5 	xul.dll 	MessageLoop::RunInternal() 	ipc/chromium/src/base/message_loop.cc:371 	inlined
5 	xul.dll 	MessageLoop::RunHandler() 	ipc/chromium/src/base/message_loop.cc:364 	cfi
6 	xul.dll 	MessageLoop::Run() 	ipc/chromium/src/base/message_loop.cc:346 	inlined
6 	xul.dll 	nsBaseAppShell::Run() 	widget/nsBaseAppShell.cpp:151 	cfi
7 	xul.dll 	nsAppShell::Run() 	widget/windows/nsAppShell.cpp:672 	cfi
8 	xul.dll 	XRE_RunAppShell() 	toolkit/xre/nsEmbedFunctions.cpp:652 	inlined
8 	xul.dll 	mozilla::ipc::MessagePumpForChildProcess::Run(base::MessagePump::Delegate*) 	ipc/glue/MessagePump.cpp:233 	cfi
9 	xul.dll 	MessageLoop::RunInternal() 	ipc/chromium/src/base/message_loop.cc:371 	inlined
9 	xul.dll 	MessageLoop::RunHandler() 	ipc/chromium/src/base/message_loop.cc:364 	cfi
10 	xul.dll 	MessageLoop::Run() 	ipc/chromium/src/base/message_loop.cc:346 	inlined
10 	xul.dll 	XRE_InitChildProcess(int, char**, XREChildData const*) 	toolkit/xre/nsEmbedFunctions.cpp:590 	inlined
10 	xul.dll 	mozilla::BootstrapImpl::XRE_InitChildProcess(int, char**, XREChildData const*) 	toolkit/xre/Bootstrap.cpp:59 	cfi
11 	firefox.exe 	NS_internal_main(int, char**, char**) 	browser/app/nsBrowserApp.cpp:466 	inlined
11 	firefox.exe 	wmain(int, wchar_t**) 	toolkit/xre/nsWindowsWMain.cpp:150 	cfi
12 	firefox.exe 	invoke_main() 	/builds/worker/workspace/obj-build/browser/app/D:/a/_work/1/s/src/vctools/crt/vcstartup/src/startup/exe_common.inl:90 	inlined
12 	firefox.exe 	__scrt_common_main_seh() 	/builds/worker/workspace/obj-build/browser/app/D:/a/_work/1/s/src/vctools/crt/vcstartup/src/startup/exe_common.inl:288 	cfi
13 	kernel32.dll 	BaseThreadInitThunk 		cfi
14 	ntdll.dll 	RtlUserThreadStart 		cfi

=================

And some Seconds later with same stack trace as far as I can see..:

https://crash-stats.mozilla.org/report/index/fb5f3d61-59ef-4f0f-a99a-907260260529

Comment 33

[BugBot [:suhaib / :marco]]

Based on the topcrash criteria, the crash signature linked to this bug is not a topcrash signature anymore.

For more information, please visit BugBot documentation.