Crash in [@ libc.so | libart.so | boot.oat]
Categories
(Firefox for Android :: General, defect)
Tracking
()
| Tracking | Status | |
|---|---|---|
| firefox160 | --- | unaffected |
| firefox161 | --- | unaffected |
| firefox162 | --- | fixed |
People
(Reporter: diannaS, Assigned: petru)
References
(Regression, )
Details
(Keywords: crash, regression)
Crash Data
Attachments
(1 external file)
Crash report: https://crash-stats.mozilla.org/report/index/ee3e8d0d-5334-4347-b2cd-64d5c0261117
Reason:
SIGSEGV / SEGV_MAPERR
Top 10 frames:
0 libc.so libc.so@0x87878
1 libart.so libart.so@0x871590
2 libart.so libart.so@0x8717c0
3 libart.so libart.so@0x8710e0
4 libart.so libart.so@0x870c70
5 libart.so libart.so@0x866f40
6 libart.so libart.so@0x854710
7 libart.so libart.so@0x863c08
8 libart.so libart.so@0x862340
9 libart.so libart.so@0x4ee51c
| Reporter | ||
Updated•3 days ago
|
| Reporter | ||
Updated•3 days ago
|
If enable the new Customize menu in Secret Settings, it will crash immediately and then crashloop while startup, even crash report can't be sent...
(In reply to witherzuo from comment #1)
If enable the new Customize menu in Secret Settings, it will crash immediately and then crashloop while startup, even crash report can't be sent...
After updating Nightly to the latest version last night, I started experiencing the following issues:
- After startup, Nightly loads a page for approximately 1–2 seconds and then suddenly crashes without any warning or error message.
- Then I tried opening Secret Settings quickly and reset all settings to the defaults. After doing so, Nightly no longer crashed when loading a default homepage screen or built-in pages. However, attempting to load any third-party page caused Nightly to crash immediately.
- I opened Secret Settings again and re-enabled the new Customize Menu feature. Nightly crashed immediately as soon as I turned the feature on. Since then, Nightly has been completely unable to start normally, crashloop while startup, I even have no chance to interact with it...
Comment 3•2 days ago
|
||
Crash report: https://crash-stats.mozilla.org/report/index/3266abf0-5e40-4588-9d87-a4e560261117
Reports with a different stack in the same spike: https://crash-stats.mozilla.org/report/index/0fd1e5cd-7fde-4f76-bb9d-ddc1c0261117, https://crash-stats.mozilla.org/report/index/f8f2abf5-ea3f-490f-a89d-5f9ce0261117
Crash Reason:
SIGSEGV / SEGV_MAPERR at 0x0000000001fffff8
Top 10 frames:
0 libc.so
1 libart.so
2 libart.so
3 libart.so
4 libart.so
5 libart.so
6 libart.so
7 libart.so
8 libart.so
9 libart.so
Filed because this signature's crash volume spiked. 19 reports from 3 distinct installations on the nightly build of 2026-11-17 (buildid 20261117093842); the preceding 3 build-days had 0, 0, 5 reports, and the loudest of its 1 earlier build in the 21 days before had 5. That is 3.8x the loudest of the preceding build-days and of this signature's builds over the 21 days before (Clouseau's bar is 3x, and the Poisson excess of the day against that baseline is z = 4.17, bar 3.62). This signature is not new: its first report is in build 20220705093820 (2022-07-05), 1596 days before this build.
Crash-stats counts for this signature on nightly when this was posted:
| Builds of the 21 days before | Builds of 2026-11-17 | |
|---|---|---|
| Reports | 5 | 19 |
| Distinct install times | unknown | unknown |
| Reports from the busiest install time | unknown | unknown |
Clouseau analysis (automated -- nothing below was checked by a human; a claim it could not ground is deliberately absent):
Every report in this spike is a SIGSEGV on Android's own garbage-collector thread (HeapTaskDaemon). The stack runs through libc.so, libart.so and boot.oat with no symbols and no Gecko or Fenix frame, so the faulting code cannot be identified. All 24 nightly reports from the last 30 days come from two devices: 16 from a Xiaomi M2101K6G (Android 12) and 8 from a motorola edge 2024 (Android 16). Almost all crash within 60 seconds of launch (the representative one at 3 seconds), which looks like the app crashing again on every restart. The Xiaomi device was already crashing this way on the previous nightly (20261116204239), before it got the spiking build, so the spike is two devices crashing repeatedly rather than something the 2026-11-17 build introduced. The same signature has been crashing on release for years. The next step is to symbolicate the libart.so frames, or get a logcat from an affected device, to find which ART GC phase faults; no code change is supported yet.
Possible path to the crash:
Mechanism (observed): SIGSEGV / SEGV_MAPERR on thread 5 HeapTaskDaemon, the ART heap-task thread, in the parent process. The fault address is 0x1fffff8 in the representative report and 0x724a67a000 in the motorola reports. The stack is unsymbolicated system code only (frame 0 in libc.so, frames 1-13 in libart.so, frame 14 in boot.oat), so the faulting instruction and the GC phase are unknown.
Path (inferred, low): ART's concurrent GC, running on HeapTaskDaemon, reads a bad heap or object reference. That would mean the managed heap or a GC root is corrupt, but which component wrote the bad value (ART itself, a JNI or native library, or Gecko through JNI) cannot be told from these stacks. In the representative report the Gecko thread was inside js::jit::TraceCacheIRStub, which is SpiderMonkey's own GC and a separate heap. That is a single-report detail and does not link the two.
Trigger (inferred, low): the crash fires within seconds of each launch on two specific devices, consistent with a repeated crash at startup. Why those two devices are affected is unknown.
Checked:
- The representative report crashed on thread 5
HeapTaskDaemonwith SIGSEGV / SEGV_MAPERR at0x1fffff8. The stack islibc.so->libart.so(13 frames) ->boot.oat, with no symbols. (crashstats report 3266abf0-5e40-4588-9d87-a4e560261117) - The representative report has uptime 3 s,
startup_crashFalse, 108 MB of 5.8 GB physical memory free andavailable_page_file0. (crashstats report 3266abf0-5e40-4588-9d87-a4e560261117) - All 24 nightly reports from the last 7 days split between two
cpu_infovalues (family 81 model 2052 stepping 13: 16; family 65 model 3393 stepping 1: 8) and twoinstall_timevalues (1230757505: 16; 1230757797: 8). (facets cpu_info and install_time, Fenix nightly, 7d) - 23 of the 24 nightly reports have uptime under 60 s. (facets uptime interval 60, Fenix nightly, 7d)
- All 5 reports on builds before 20261117093842 come from the family 81 model 2052 CPU, the same CPU as the representative Xiaomi device. From the spiking build on, there are 11 reports from that CPU and 8 from the motorola CPU. (facets cpu_info split_at_build 20261117093842, Fenix nightly, 30d)
- Nightly reports cover three builds: 20261117093842 (15), 20261116204239 (5) and 20261117193337 (4). There are none on nightly before 2026-11-17 in the 30-day window. (facets build_id, Fenix nightly, 60d; facets cpu_info by_day, 30d)
- The same signature crashes steadily on release (roughly 15-155 reports a day), and the brief dates it to build 20220705093820. (brief: daily counts across channels and signature age)
- The two
install_timevalues are each identical across their reports and line up exactly with the two CPU models, so each device is very likely one installation crashing repeatedly. A matching timestamp does not prove it is the same installation. [inferred, medium confidence] (facets install_time and cpu_info, Fenix nightly, 7d)
Alternatives considered:
- A changeset in the 20261117093842 pushlog introducing the crash: disfavored -- the Xiaomi device was already producing this crash on the earlier build 20261116204239, and the signature has crashed on release since 2022. No pushlog change could be tied to an ART GC frame, because the stack has no Gecko or Fenix frame.
- A broad rate regression in nightly: not supported -- all 24 reports come from two devices, by
cpu_infoandinstall_time. - Hardware fault (bit flip or a known-defective CPU): not supported -- the brief shows 0% bit-flip and 0% Raptor Lake share, and the two devices use different SoCs.
Worth checking first:
- Which ART GC operation faults: symbolicate the
libart.soframes using the device builds' system symbols, or get a logcat or tombstone from one affected device. - Whether a Fenix or Gecko JNI or native component corrupts the Java heap: compare the
libart.soframe offsets between the Xiaomi and motorola reports, and check whether the release-channel population clusters on one fault offset. - Whether the motorola device first started crashing on 20261117093842: its nightly history before 2026-11-17 is not in the data. A per-install query on install_time 1230757797 across earlier builds would show it.
Posted automatically by Clouseau, which analyses nightly crashes with an LLM. Nothing above was written or checked by a human.
| Assignee | ||
Comment 4•2 days ago
|
||
I do also get Nightly 162 closing shortly after launch when the browser menu has a saved customization.
Clearing the app data fixes it, and customizing the menu again brings the crash back.
R8 changes the type of CustomizedMenuRecord.schemaVersion_ from int to boolean, although the proto declares uint32 schema_version. dexdump of the Nov 17 Nightly APK shows type Z.
As far as I can tell, protobuf-lite accesses the field as a 4-byte int through Unsafe, so it writes past the field into the next object in the Java heap.
In a local R8 build with the same narrowing, a saved record reads back with schema_version = 0x80000001, so the access does go past the field.
That fits the GC crash on HeapTaskDaemon in this report.
I haven't confirmed the overflow is what crashes the GC as my local build does not crash.
When it started: the field is boolean from the Nov 12 01:04 Nightly on, and absent from the class in the Nov 11 09:11 Nightly and earlier.
The change falls on the batch of customizable-menu commits that landed on Nov 11 (for example bug 2082394 and bug 2086545), and I did not find which one flips R8's result.
It is not the AGP update (bug 2086641 landed on Nov 12 at 21:13 UTC, after the first boolean builds).
It is also not enabling obfuscation (bug 2071819): the first builds with it have no such field, and a local build with obfuscation disabled still has the boolean field.
The libart crash signatures are already elevated in the Nov 12 and Nov 13 Nightlies.
Proposed fix: stop R8 from changing the fields of generated protobuf messages. Protobuf-javalite ships no keep rules, and -keep,allowshrinking,allowoptimization class org.mozilla.fenix.** lets R8 change them:
-keepclassmembers class * extends com.google.protobuf.GeneratedMessageLite { <fields>; }
With it, schemaVersion_ stays an int in the dex.
Records saved by affected builds have a wrong stored version, so the menu would reset to the default once after updating.
Given that only a few users have the menu customization secret setting enabled, this seems acceptable.
| Assignee | ||
Comment 5•2 days ago
|
||
R8 turned CustomizedMenuRecord.schemaVersion_ from an int into a boolean, while protobuf-lite still
reads and writes it as a 4-byte int, past the end of the field. Nightly then closed shortly after
launch when the menu had a saved customization.
Keep the fields of generated protobuf-lite messages as they are.
A customization saved by an affected build is ignored once, so the menu returns to its default
layout until it is saved again.
Updated•2 days ago
|
Comment 6•2 days ago
|
||
i have reproduced the issue using petru’s described steps in comment 4, of installing yesterdays nightly, customising the menu, and upgrading to latest nightly through the play store. now i get startup crashes
i ran this command to see how r8 had optimised the stored value
unzip -o fenix-162.0a1.multi.android-arm64-v8a.apk 'classes*.dex' -d /tmp/fenix-dex
for d in /tmp/fenix-dex/classes*.dex; do
~/Library/Android/sdk/build-tools/37.0.0/dexdump "$d" \
| rg -A40 "Class descriptor *: 'Lorg/mozilla/fenix/datastore/CustomizedMenuRecord;'" \
| rg -A2 "schemaVersion_"
done
This gives the field type used for protobuf
Broken: type: ‘Z’ (boolean)
Fixed: type: ‘I’ (int)
i got type Z as expected.
now waiting for the nightly simulation build with petru’s fix to retest
Comment 9•2 days ago
|
||
| bugherder | ||
Comment 10•2 days ago
|
||
The patch landed in nightly and beta is affected.
:petru, please make an uplift decision for beta:
- For beta, nominate the patch for beta uplift approval if the fix should be included in this release, or set
status-firefox161towontfixif the fix can wait for the next release.
Uplift requests can be submitted from Lando: D343849.
See Requesting an Uplift for documentation on how to request an uplift.
For more information, please visit BugBot documentation.
Comment 11•2 days ago
|
||
160 and 161 are marked as affected, but is the fix that landed here for a regression in 162? Possibly caused by Bug 2086545?
| Reporter | ||
Updated•2 days ago
|
| Reporter | ||
Comment 12•2 days ago
|
||
Marking 160 and 161 as unaffected since this particular fix is to address the nightly issue that spiked recently
| Reporter | ||
Updated•2 days ago
|
| Assignee | ||
Comment 13•2 days ago
|
||
The fix that I posted is specifically for a new scenario of the menu.
That feature only exists in Nightly (behind a secret setting) so for this particular menu scenario other releases aren't affected, and we also don't need an uplift.
Seeing the 77 crashes in 159 I think rather shows that the data conflates crashes from multiple sources/scenarios.
Comment 14•2 days ago
|
||
| bugherder | ||
Description
•