Crash in [@ IPCError-content | GPUProcessKill]
Categories
(Core :: Graphics, defect)
Tracking
()
People
(Reporter: clouseau-bot, Assigned: RyanVM, NeedInfo)
References
(Blocks 1 open bug, Regression)
Details
(Keywords: crash, regression, topcrash)
Crash Data
Attachments
(2 external files)
Crash report: https://crash-stats.mozilla.org/report/index/01e6c1a6-5761-4053-b9f9-55cf20261015
Reports with a different stack in the same spike: https://crash-stats.mozilla.org/report/index/0045378d-940c-4888-85a3-d12c80261020, https://crash-stats.mozilla.org/report/index/02159283-332b-4436-8685-d3e6c0261017
Crash Reason:
DUMP_REQUESTED at 0x0000007eb572220c
Top 3 frames of the hung main thread (nothing crashed here — a watchdog killed the process; these are what the main thread is waiting on):
0 libc.so
1 libc.so
2 libmozglue.so mozilla::detail::ConditionVariableImpl::wait(mozilla::detail::MutexImpl&) mozglue/misc/ConditionVariable_posix.cpp:92
Filed because this signature's crash volume spiked. 618 reports from 323 distinct installations on the release build of 2026-10-07 (buildid 20261007201254); the preceding 1 build-day had 3 report, and the loudest of its 5 earlier builds in the 42 days before had 5. That is 123.6x the loudest of the preceding build-days and of this signature's builds over the 42 days before (Clouseau's bar is 3x, and the Poisson excess of the day against that baseline is z = 45.1, bar 3.62). By version, 158.0 runs at 182.8x the share of 157.0 from its first days. This signature is not new: its first report is in build 20240212120452 (2024-02-12), 968 days before this build.
Clouseau analysis (automated -- nothing below was checked by a human; a claim it could not ground is deliberately absent):
From Fenix 158.0 the parent process is killing a hung GPU process (GPUProcessKill after a sync IPC reply timeout) about 180 times more often than on 157.0, per 1000 crash reports of the same version. After the version change, PowerVR Rogue GE8322/GE8320 GPUs went from about 5% of these reports to 57%; Mali-G52 and Adreno 610 also rose. In the one 158 PowerVR report whose threads were inspected, both the WebRender Renderer thread and the WebGL CanvasRenderer thread are inside the PowerVR driver (libsrv_um.so) when the process is killed. The only Android-graphics change found among the version's changes is bug 2066020 (c425043976f8). It sets webgl.out-of-process.enable-ahardwarebuffer to true and moves the AHardwareBuffer check first in TexTypeForWebgl, so out-of-process WebGL on every Android device now shares frames through AHardwareBuffer, with no GPU/driver blocklist. Recommended: graphics should turn that pref off for the affected drivers (or off entirely) on 158/159 and check whether the release rate falls back, then add a gfxInfo blocklist entry for the drivers that stall.
Suspected regressor (medium confidence): c425043976f8 (bug 2066020) by :sotaro.
The diff changes the webgl.out-of-process.enable-ahardwarebuffer default in StaticPrefList.yaml from false to true. It also moves the UseAHardwareBufferSharedSurfaceWebglOop check ahead of the EGLImage path in TexTypeForWebgl (gfx/layers/CanvasRenderer.cpp:122-125 at eb9a6e3a7368). gfxPlatform.cpp:3268-3269 copies the pref into gfxVars on Android with no device or driver gate. So in 158, out-of-process WebGL on all Android devices moved to a new driver path (AHardwareBuffer import and sync) in the GPU process. The crash population shifted to particular GPU drivers at exactly this version, and the PowerVR report shows the CanvasRenderer and Renderer threads both inside the PowerVR driver. That fits a driver stall on the new path, but the driver-internal stacks are unsymbolized, so this is not proven. c425043976f8 shows no backout; the earlier landing 7a4f50a624e3 of the same change was backed out by 980b80614bdd, and this changeset is the reland.
Possible path to the crash:
Mechanism (observed): a watchdog kill, not a fault. The parent killed the GPU process after a sync IPC reply timeout (graphics_critical_error in 698712ce: 'Killing GPU process due to IPC reply timeout'). The GPU process main thread was idle in nsAppShell::Queue::Pop.
Path (inferred, medium):
- Bug 2066020 makes
TexTypeForWebglreturnTextureType::AndroidHardwareBufferfor out-of-process WebGL on all Android devices. - WebGL frames produced on the GPU process's
CanvasRendererthread and consumed by the WebRenderRendererthread now go through AHardwareBuffer import and sync in the vendor GLES driver. - On some drivers (PowerVR Rogue GE832x above all, also Mali-G52 and Adreno 610), both threads block inside the driver (
libsrv_um.soin 698712ce), so compositor work that waits on the render thread never replies. - The parent's sync IPC to the GPU process times out and the parent kills the GPU process, which is reported as
IPCError-content | GPUProcessKill.
Trigger (inferred, low): WebGL content being displayed on an affected driver. The driver-internal frames are unsymbolized, and no report shows WebGL usage directly.
Checked:
- 158.0 runs at 9.14 per 1000 same-version crash reports against 0.05 for 157.0, and the daily count rises with 158 adoption from 2026-10-14 (brief: crash rate by version and daily counts)
- Imagination Technologies GPUs went from 46 of 712 reports (6%) on builds before 20261007201254 to 363 of 633 (57%) from it on (facets adapter_vendor_id, split_at_build 20261007201254, 60d, Fenix release)
- From the spiking build on, PowerVR Rogue GE8322 accounts for 270, Mali-G52 141, PowerVR Rogue GE8320 85 and Adreno 610 62 of 633 reports. On older builds GE8322 had 7 and GE8320 32 of 712, and Mali-G52 was not among the top 20 (facets adapter_device_id, split_at_build 20261007201254, 60d)
- In report 698712ce (PowerVR GE8320, 158.0), the
CanvasRenderer(thread 41) andRenderer(thread 16) stacks are insidelibsrv_um.so, and the GPU Process main thread is idle waiting innsAppShell::Queue::Pop(crashstats report 698712ce-c263-4937-a9a6-642170261015, threads 12/16/41) - The kill was triggered by an IPC reply timeout (698712ce graphics_critical_error)
c425043976f8flips thewebgl.out-of-process.enable-ahardwarebufferdefault to true and moves the AHB check first inTexTypeForWebgl. The changeset shows no backout (patch diff c425043976f8; history changeset c425043976f8)- On Android, the pref is copied into
gfxVars::UseAHardwareBufferSharedSurfaceWebglOopwith no gfxInfo or driver blocklist gate (gfx/thebes/gfxPlatform.cpp@eb9a6e3a7368:3267-3270) TexTypeForWebglreturnsAndroidHardwareBufferbefore considering EGLImage or SurfaceTexture (gfx/layers/CanvasRenderer.cpp@eb9a6e3a7368:122-135)
Alternatives considered:
7a4f50a624e3(the first landing of bug 2066020): ruled out as the shipped change -- it was backed out by980b80614bdd; the identical relandc425043976f8is what shippedcbfe2c82f34f(bug 2074325, WebRender render-pass clears) and0cb7d640b527(bug 2072784, GlDevice refactor): not supported -- the earlier pipeline's second opinions found no added GPU work in either diff, and neither explains a shift toward particular GPU vendors; their diffs were not re-read in this run- Hardware fault artefact: disfavored -- 0% bit-flip annotations, and the rise is a clean version step across many devices
Worth checking first:
- Is the AHB pref the cause? Turn
webgl.out-of-process.enable-ahardwarebufferoff on release (for example through Nimbus or a pref flip in a dot release) and check whether the rate on PowerVR GE832x, Mali-G52 and Adreno 610 drops back to the 157 level - Exactly where the driver blocks (fence wait, buffer import, or eglClientWaitSync) is unknown: driver frames are unsymbolized scan frames. Annotating which
TextureTypewas in use at the hang, or making the render thread record its last GL call, would settle it - The disappearance of older GPUs (Mali-G51, Vivante GC880) from the post-158 population has not been explained (a supported-device change or simply small numbers); facet
platform_versionsplit at the build to check
Filed as a new bug rather than a comment on bug 1900134 — which is open on this signature, but it was filed before the changeset above landed, so it cannot be about this regression. Please duplicate if that is wrong.
:sotaro, can you have a look please?
Filed automatically by Clouseau, which analyses release crashes with an LLM. Nothing above was written or checked by a human. Please close it as INVALID if it is wrong — that is useful feedback, not a nuisance.
This is not an independent Mozilla discovery: the crash is known here only because somebody submitted the crash report linked at the top. If this duplicates an existing report, please resolve THIS bug as the duplicate and leave the credit with the earlier reporter.
| Assignee | ||
Updated•26 days ago
|
Comment 1•26 days ago
|
||
The bug is linked to a topcrash signature, which matches the following criterion:
- Top 10 AArch64 and ARM crashes on release
For more information, please visit BugBot documentation.
| Assignee | ||
Updated•25 days ago
|
| Assignee | ||
Comment 2•25 days ago
|
||
Updated•25 days ago
|
Comment 3•25 days ago
|
||
firefox-release Uplift Approval Request
- User impact if declined/Reason for urgency: Android topcrash
- Code covered by automated testing?: yes
- Fix verified in Nightly?: no
- Needs manual QE testing?: no
- Steps to reproduce for manual QE testing: n/a
- Risk associated with taking this patch: low
- Explanation of risk level: Just prefs us back to the pre-158 setting.
- String changes made/needed?: none
- Is Android affected?: yes
| Assignee | ||
Comment 4•25 days ago
|
||
Original Revision: https://phabricator.services.mozilla.com/D336023
Updated•25 days ago
|
Updated•25 days ago
|
Comment 7•25 days ago
|
||
| bugherder | ||
Updated•24 days ago
|
Comment 8•23 days ago
|
||
The crash seemed to happen on some specific GPUs
Crash reports by GPU model:
GPU Model Crashes Percentage
------------------------------------------------
PowerVR Rogue GE8322 367 41.7%
ARM Mali-G52 187 21.3%
PowerVR Rogue GE8320 124 14.1%
Qualcomm Adreno 610 89 10.1%
Other / Unknown 113 12.8%
------------------------------------------------
Total 880 100.0%
Comment 9•23 days ago
|
||
Analysis of the PowerVR crash stacks suggests that the GPU process hangs primarily inside the GPU driver during WebRender rendering.
- PowerVR GE8320:
GlDevice::create_texture()frequently appears in the Renderer stacks, called fromRenderer::update_texture_cache(). - PowerVR GE8322: Most Renderer stacks are inside
libsrv_um.so, although the Gecko-side call path is often unavailable. - Other GPUs: Mali-G52 and Adreno 610 also show Renderer stacks inside their respective GPU drivers.
Code inspection confirms that GlDevice::create_texture() calls glTexStorage2D() or glTexImage2D() to allocate texture storage.
These GL calls are plausible blocking points, but the exact API and root cause remain unconfirmed. AHardwareBuffer synchronization may be involved, although the stacks do not establish a direct connection.
Comment 10•23 days ago
|
||
Analysis of 880 crash reports shows that 666 (75.7%) occurred on Android 11.
The concentration is particularly strong for PowerVR GE8322 (332/367, 90.5%) and Mali-G52 (181/187, 96.8%). PowerVR GE8320 also predominantly crashes on Android 11 (97/124, 78.2%).
However, Adreno 610 crashes are distributed across multiple Android versions, with Android 13 being the most common.
This suggests that the issue may depend on specific GPU driver and Android version combinations rather than Android 11 alone. These figures represent crash counts, not exposure-adjusted crash rates.
Comment 11•11 days ago
|
||
Crashes continue after disabling webgl.out-of-process.enable-ahardwarebuffer. Fenix 159.0 still has 914 reports with nearly the same GPU distribution, and 77.5% occurred on Android 11, compared with 75.7% previously. This suggests that a similar crash population remains after the fix.
Comment 12•11 days ago
•
|
||
I wonder if the increase of crash reports might be related to Bug 1949178.
Updated•11 days ago
|
Comment 13•11 days ago
•
|
||
:RyanVM, even after Bug 2066020 back out, the crash still seems to happen. I wonder if Bug 1949178 is actual cause of the problem. How do you think about it?
| Assignee | ||
Comment 14•11 days ago
|
||
Seems possible, but gsvelto would know best.
Description
•