1637597 - Crash in [@ java.lang.Exception: at org.mozilla.gecko.mozglue.GeckoLoader.loadGeckoLibsNative(Native Method)]

Status: RESOLVED FIXED

(GeckoView :: General, defect, P2)

Description

[Emily Toop (:fluffyemily)]

This bug is for crash report bp-2a2f3a35-605a-4cda-a9d7-b8c880200513.

Java stack trace:

java.lang.Exception
	at org.mozilla.gecko.mozglue.GeckoLoader.loadGeckoLibsNative(Native Method)
	at org.mozilla.gecko.mozglue.GeckoLoader.loadGeckoLibs(GeckoLoader.java:2)
	at org.mozilla.gecko.GeckoThread.run(GeckoThread.java:50)

Comment 1

[Dylan Roeh (:droeh) (he/him)]

We're showing incorrect build ids for this crash (and all other Focus crashes) for some reason I've been unable to figure out; I confirmed that we're shipping the right GV version with Focus (by diffing libxul.so from the Focus APK on my phone against libxul.so from the appropriate GV AAR from taskcluster). Other than that the bug reports seem valid, etc.

Comment 2

[Ryan VanderMeulen [:RyanVM]]

This is the #2 overall topcrash for Focus at the moment.

Comment 3

[Emily Toop (:fluffyemily)]

GV 77. Release channel nightly. These things should not be. Focus 8.3.0 should be shipping a release version of GV 76. 99.43% of failures are happening on Android V23. Android 23 is when we start using the System Linker. We might be completely broken on V23.

Comment 4

[Emily Toop (:fluffyemily)]

May need to get glandium to take a look at this. May be worth re-examining this after upgraded Focus comes out to see if this issue is still prevalent.

Comment 5

[Rachel Tublitz [:rachel]]

S1 or S2 bugs need an assignee - could you find someone for this bug?

Comment 6

[Emily Toop (:fluffyemily)]

Dylan, I'm assigning this to you. Please can you either close, or reassign to the correct team if we still see this issue after upgrading Focus?

Comment 7

[Ryan VanderMeulen [:RyanVM]]

It's early but currently the #1 overall topcrash for Focus 8.4.0.
https://crash-stats.mozilla.org/topcrashers/?product=Focus&version=8.4.0

Comment 8

[Dylan Roeh (:droeh) (he/him)]

(In reply to Emily Toop (:fluffyemily) from comment #6)

Dylan, I'm assigning this to you. Please can you either close, or reassign to the correct team if we still see this issue after upgrading Focus?

Alright, looks like any hope of the Focus update fixing this is dead in the water. I'll see if I can figure out what's going on here.

Comment 9

[Ryan VanderMeulen [:RyanVM]]

This is the #1 overall topcrash for Focus 8.7.1. It's also showing up in the Fenix crashers. Can we take another look at this?

Comment 10

[Dylan Roeh (:droeh) (he/him)]

Clearing myself so we can discuss and re-triage.

Comment 11

[Amedyne Moya [:amoya]]

Investigate on Android 23.

Comment 12

[Dylan Roeh (:droeh) (he/him)]

(In reply to amoya from comment #11)

Investigate on Android 23.

Just confirmed that Focus works fine on 23 in an x86-64 emulator.

Comment 13

[Ryan VanderMeulen [:RyanVM]]

Still the #1 overall Focus topcrash.

Comment 14

[(No longer employed by Mozilla) Aaron Klotz]

We finally got a report with some usable information!

Comment 15

[(No longer employed by Mozilla) Aaron Klotz]

(This information is pertinent to the specific crash report from comment 14 and may not be consistent across all crash reports)

This particular dlerror message was not added to bionic until Nougat. One thing that I notice is that it does resolve the .dynamic section differently than pre-Nougat versions of the dynamic linker. I'll want to examine the libxul.so from the build in question to see how this is failing.

Comment 16

[[ex-Mozilla] Agi Sferro | :agi]

This is the output of readelf from that version:

aarch64-linux-android-readelf --dynamic libxul.so                         

Dynamic section at offset 0x77d0ba0 contains 33 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libnss3.so]
 0x0000000000000001 (NEEDED)             Shared library: [liblgpllibs.so]
 0x0000000000000001 (NEEDED)             Shared library: [libmozglue.so]
 0x0000000000000001 (NEEDED)             Shared library: [libandroid.so]
 0x0000000000000001 (NEEDED)             Shared library: [liblog.so]
 0x0000000000000001 (NEEDED)             Shared library: [libm.so]
 0x0000000000000001 (NEEDED)             Shared library: [libdl.so]
 0x0000000000000001 (NEEDED)             Shared library: [libstdc++.so]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so]
 0x000000000000000e (SONAME)             Library soname: [libxul.so]
 0x0000000000000019 (INIT_ARRAY)         0x72df138
 0x000000000000001b (INIT_ARRAYSZ)       760 (bytes)
 0x000000000000001a (FINI_ARRAY)         0x72df430
 0x000000000000001c (FINI_ARRAYSZ)       16 (bytes)
 0x0000000000000004 (HASH)               0x260
 0x0000000000000005 (STRTAB)             0x1188f0
 0x0000000000000006 (SYMTAB)             0x43a40
 0x000000000000000a (STRSZ)              3805983 (bytes)
 0x000000000000000b (SYMENT)             24 (bytes)
 0x0000000000000003 (PLTGOT)             0x77d1df0
 0x0000000000000002 (PLTRELSZ)           29352 (bytes)
 0x0000000000000014 (PLTREL)             RELA
 0x0000000000000017 (JMPREL)             0xeb31c8
 0x0000000000000007 (RELA)               0x4cb898
 0x0000000000000008 (RELASZ)             10385712 (bytes)
 0x0000000000000009 (RELAENT)            24 (bytes)
 0x000000000000001e (FLAGS)              BIND_NOW
 0x000000006ffffffb (FLAGS_1)            Flags: NOW
 0x000000006ffffffe (VERNEED)            0x4cb7f8
 0x000000006fffffff (VERNEEDNUM)         5
 0x000000006ffffff0 (VERSYM)             0x4b9c10
 0x000000006ffffff9 (RELACOUNT)          426838
 0x0000000000000000 (NULL)               0x0

Comment 17

[(No longer employed by Mozilla) Aaron Klotz]

The thing about that error message is that it happens while trying to resolve .dynamic in the ELF section table; it doesn't even think it's there.

The loader accepts an optional file offset that is used for cases where it is loading the .so straight from the APK. I am wondering whether there is some kind of problem under such a scenario where libxul.so's section table is being mapped in such a way that it is in fact being truncated in memory (or something to that effect).

Comment 18

[[ex-Mozilla] Agi Sferro | :agi]

Just noticed that something happened around December 5th 2020 that made this crash much rarer (at least in Nightly).

Comment 19

[(No longer employed by Mozilla) Aaron Klotz]

Fenix PR for forcing extractNativeLibs=true

Comment 20

[Emily Toop (:fluffyemily)]

The Fenix PR landed on Mar 29th.

Comment 21

[Emily Toop (:fluffyemily)]

Note to reopen if this recurs.