1379326 - LaunchGPUProcess would crash the main process if the GPU process failed to launch

Status: NEW

(Core :: Graphics: Layers, defect, P3)

Description

[Naomi Smith [:spinda] (non-bmo mail: [e-mail])]

Attachment: gdb trace

To reproduce: simulate a GPU process launch failure by adding a "return false;" to the beginning for GPUProcessHost::Launch, recompile, and start Firefox in an environment where GPU_PROCESS is normally enabled. The main process will crash with SIGSEGV.

A gdb trace is attached. The crash occurs when LaunchGPUProcess detects that the launch failed and calls DisableGPUProcess (http://searchfox.org/mozilla-central/source/gfx/ipc/GPUProcessManager.cpp#161), which in turn attempts to set up off-main-thread compositing (http://searchfox.org/mozilla-central/source/gfx/ipc/GPUProcessManager.cpp#183-188). This ends up calling EnsureCompositorManagerChild (http://searchfox.org/mozilla-central/source/gfx/ipc/GPUProcessManager.cpp#213) which, some levels down, leads to sCompositorThreadHolder being accessed while it is still null (http://searchfox.org/mozilla-central/source/gfx/layers/ipc/CompositorThread.cpp#22). It is normally initialized when InitLayersIPC is called during gfxPlatform initialization (http://searchfox.org/mozilla-central/source/gfx/thebes/gfxPlatform.cpp#742), but this only happens *after* LaunchGPUProcess completes (http://searchfox.org/mozilla-central/source/gfx/thebes/gfxPlatform.cpp#728).

Luckily, it would appear that GPUProcessHost::Launch, despite returning a boolean value, can't ever *actually* fail, as it passes back the return value of GeckoChildProcessHost::AsyncLaunch which always returns true (http://searchfox.org/mozilla-central/source/ipc/glue/GeckoChildProcessHost.cpp#397). Perhaps, then, this recovery mechanism can be ripped out of LaunchGPUProcess entirely. But there might be other cases where DisableGPUProcess ends up being called before the compositor thread is set up.

Comment 1

[Kevin Chen[:kechen] (UTC + 8)]

Yes, this might cause some potential issues, I've cced myself and will going to investigate this.