1411267 - Crash in tgkill affecting Pixel and Pixel XL

Status: RESOLVED WORKSFORME

(Firefox for Android Graveyard :: General, defect, P3)

Description

[Marcia Knous [:marcia]]

This bug was filed from the Socorro interface and is 
report bp-7678f748-9558-42e4-b4d1-28efc0171023.
=============================================================

Seen while looking at nightly android crashes: http://bit.ly/2yOnFML. Crash affects Pixel and Pixel XL. Affects 57 and other branches as well. Not sure where to bucket it - could use some help.

One comment: was playing my second or third consecutive video on YouTube, pressed the full screen button, the image stuttered and it seems that Nightly crashed 

Comment 1

[Nevin Chen(Not active on Bugzilla)]

Please help with this

Comment 2

[Blake Wu [:bwu][:blakewu]]

John,
Can you help check it?

Comment 3

[John Lin [:jhlin][:jolin]]

The log shows GL OOM error:

10-24 01:51:44.233 31707 31738 W GeckoAppShell: Unsupported wake-lock: DOM_Fullscreen
10-24 01:51:44.253 31707 31738 D GeckoScreenOrientation: locking to LANDSCAPE
10-24 01:51:44.517 31707 31707 D GeckoLayerClient: Window-size changed to (1080,1920)
10-24 01:51:44.570 31707 32548 W Adreno-GSL: <sharedmem_gpuobj_alloc:2318>: sharedmem_gpumem_alloc: mmap failed errno 12 Out of memory
...
10-24 01:51:44.600 31707 32548 E OpenGLRenderer: GL error:  Out of memory!
10-24 01:51:44.601 31707 32548 F OpenGLRenderer: GL errors! frameworks/base/libs/hwui/BakedOpRenderer.cpp:98

Comment 4

[John Lin [:jhlin][:jolin]]

All seems to happen on Pixel devices with Oreo (Android version: 26). Will try to reproduce it.

Comment 5

[Marcia Knous [:marcia]]

I have a Pixel running Oreo. I loaded some of the URLs but I haven't yet been able to reproduce this issue.

Comment 6

[Alex Thompson]

I'm getting this bug, running nightly on my Pixel running Oreo. (eg https://crash-stats.mozilla.com/report/index/b64eed6b-e418-4703-9150-cd27e0171026 ). I'm getting anywhere between 1-5 crashes a day.

I haven't noticed any pattern to the crashes, many are after unlocking/turning the screen back on but some have also been after graphically intensive tabs are pushed out of RAM and reloaded.

Happy to help debugging if I can be given any things to try/do.

Comment 7

[Marcia Knous [:marcia]]

(In reply to Alex Thompson from comment #6)
> I'm getting this bug, running nightly on my Pixel running Oreo. (eg
> https://crash-stats.mozilla.com/report/index/b64eed6b-e418-4703-9150-
> cd27e0171026 ). I'm getting anywhere between 1-5 crashes a day.
> 
> I haven't noticed any pattern to the crashes, many are after
> unlocking/turning the screen back on but some have also been after
> graphically intensive tabs are pushed out of RAM and reloaded.
> 
> Happy to help debugging if I can be given any things to try/do.

Hello Alex - Did this start happening after the 10/5 security update? My Pixel device has that update, but I am not experiencing any crashes on Nightly.

Comment 8

[Alex Thompson]

I'm not seeing a clear connection with the 10/5 security update. 

My device has an uptime of ~13.5 days so I presumably installed the update around 13/10. However, looking in about:crashes, I had the first crash on 18/10, one on 23/10 and 5 each on 24/10, 25/10, 26/10.

Comment 9

[John Lin [:jhlin][:jolin]]

Tried Beta/57, Nightly/58, and local m-c build on a Pixel XL/marlin running Oreo/8.0.0(OPR3.170623.008) and couldn't reproduce this bug. :(

Some of the logcat in those crash reports do show native crash info, but unfortunately all were cut (too large to fit into buffer, I think).

Alex and Marcia, could you please help run

 'adb shell dumpsys dropbox --print data_app_native_crash && adb shell dumpsys dropbox --print SYSTEM_TOMBSTONE'

to collect the tombstones and upload them? Thanks a lot.

Comment 10

[Alex Thompson]

Attachment: Output from adb shell dumpsys dropbox --print data_app_native_crash

Comment 11

[Alex Thompson]

Attachment: Output from adb shell dumpsys dropbox --print SYSTEM_TOMBSTONE

Comment 12

[Alex Thompson]

I've uploaded the output from those two commands (piped separately into text files). I haven't had a crash in ~17 hours, so hopefully there's something useful there.

Comment 13

[John Lin [:jhlin][:jolin]]

Thanks, Alex. It looks like the most recent crash happened when the garbage collector found some heap corruption.

Not sure if it's the same problem Marcia met. Let's wait for his tombstone and file a new bug for you otherwise.


2017-10-25 12:02:29 data_app_native_crash (compressed text, 1547 bytes)
Process: org.mozilla.fennec_aurora
PID: 8672
Flags: 0x38c83e44
Package: org.mozilla.fennec_aurora v2015519633 (58.0a1)
Foreground: No
Build: google/sailfish/sailfish:8.0.0/OPR1.170623.027/4292972:user/release-keys

*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
Build fingerprint: 'google/sailfish/sailfish:8.0.0/OPR1.170623.027/4292972:user/release-keys'
Revision: '0'
ABI: 'arm'
pid: 8672, tid: 8682, name: HeapTaskDaemon  >>> org.mozilla.fennec_aurora <<<
signal 6 (SIGABRT), code -6 (SI_TKILL), fault addr --------
Abort message: 'utils.cc:123]
...
backtrace:
    #00 pc 0004ac0c  /system/lib/libc.so (tgkill+12)
    #01 pc 0001a533  /system/lib/libc.so (abort+54)
    #02 pc 00337847  /system/lib/libart.so (_ZN3art7Runtime5AbortEPKc+370)
    #03 pc 00337e3f  /system/lib/libart.so (_ZN3art7Runtime7AborterEPKc+10)
    #04 pc 003eef59  /system/lib/libart.so (_ZN7android4base10LogMessageD1Ev+456)
    #05 pc 001c142f  /system/lib/libart.so (_ZNK3art2gc12Verification17LogHeapCorruptionENS_6ObjPtrINS_6mirror6ObjectEEENS_12MemberOffsetEPS4_b+926)
...
    #13 pc 0017c393  /system/lib/libart.so (_ZN3art2gc9collector16GarbageCollector3RunENS0_7GcCauseEb+266)

Comment 14

[Alex Thompson]

(In reply to John Lin [:jolin][:jhlin] from comment #13)
> Thanks, Alex. It looks like the most recent crash happened when the garbage
> collector found some heap corruption.
> 
> Not sure if it's the same problem Marcia met. Let's wait for his tombstone
> and file a new bug for you otherwise.
> 

Hi John,

Thank you for looking into the bug. Is the crash report generated by ADB definitely the latest report? It has the timestamp 
2017-10-25 12:02:29, however that is over 48 hours ago. I had a few more crashes after this one and managed captured the adb output. A diff between them doesn't reveal any changes, except for the number in "Drop box contents: 380 entries" changing.

Sorry if this is a silly question or bugspam.

Comment 15

[Marcia Knous [:marcia]]

(In reply to John Lin [:jolin][:jhlin] from comment #9)
> Tried Beta/57, Nightly/58, and local m-c build on a Pixel XL/marlin running
> Oreo/8.0.0(OPR3.170623.008) and couldn't reproduce this bug. :(
> 
> Some of the logcat in those crash reports do show native crash info, but
> unfortunately all were cut (too large to fit into buffer, I think).
> 
> Alex and Marcia, could you please help run
> 
>  'adb shell dumpsys dropbox --print data_app_native_crash && adb shell
> dumpsys dropbox --print SYSTEM_TOMBSTONE'
> 
> to collect the tombstones and upload them? Thanks a lot.

I can provide this, but I haven't been able to reproduce the crash on my Pixel. So far I think Alex is the only one seeing the crash.

Comment 16

[John Lin [:jhlin][:jolin]]

(In reply to Alex Thompson from comment #14)

> Thank you for looking into the bug. Is the crash report generated by ADB
> definitely the latest report? It has the timestamp 
> 2017-10-25 12:02:29, however that is over 48 hours ago. I had a few more
> crashes after this one and managed captured the adb output. A diff between
> them doesn't reveal any changes, except for the number in "Drop box
> contents: 380 entries" changing.
> 
> Sorry if this is a silly question or bugspam.

 AFAIK the dropbox should keep the latest "native" crash stack. If you encounter other crashes not showing there, they might be Java crashes. BTW all native crash dumps are kept in '/data/tombstones/tombtone*' but only accessible on rooted devices.

 Do you find messages like "Unhandled Exception ..." in your logcat captures right after crashes? Could you please help upload them? Thanks a lot.

Comment 17

[Alex Thompson]

Attachment: logcat.txt with crashes at 12:17 and 15:49 on 30/10

Hi John,

It looks like these crashes I am getting may not be "native". I have had two crashes today at 12:27 and 15:49 and when I run the adb commands from #9, they now both give "(No entries found.)"

I have uploaded my logcat (adb logcat -d > logcat.txt; then gzipped), which includes the two crashes from today. However, I can't find the text "Unhandled Exception" in the logcat.

For reference, the reported crash IDs are:
12:27 - fef5b9cc-72f4-455c-a2c3-9b2ea0171030
15:49 - 69a5f052-d6f3-44aa-8189-d20e00171030

Comment 18

[John Lin [:jhlin][:jolin]]

It looks like the log at 12:17 (fef5b9cc-72f4-455c-a2c3-9b2ea0171030) is also from LogHeapCorruption[1]:

10-30 12:17:15.719 13207 13217 F zygote  : verification.cc:82] MemMap:
10-30 12:17:15.719 13207 13217 F zygote  : verification.cc:82] [MemMap: 0x12c00000+0x40000P prot=0x3 main space (region space)]
...
10-30 12:17:15.725 13207 13217 F zygote  : verification.cc:102] GC tried to mark invalid reference 0xfffffffe
...

Again, could not tell what caused the heap corruption. Unassign myself for not knowing how to proceed.


Max, do you know who might be able to help?

[1] http://androidxref.com/8.0.0_r4/xref/art/runtime/gc/verification.cc#76

Comment 19

[John Lin [:jhlin][:jolin]]

FYI, a similar bug was filed on Android issue tracker: https://issuetracker.google.com/issues/65147194

Comment 20

[James Willcox (:snorp) ([e-mail]) (he/him)]

Jim, you've been able to debug things like this in the past, any ideas?

Comment 21

[James Willcox (:snorp) ([e-mail]) (he/him)]

(In reply to John Lin [:jolin][:jhlin] from comment #19)
> FYI, a similar bug was filed on Android issue tracker:
> https://issuetracker.google.com/issues/65147194

Hmmm, that does look similar. That link says they've fixed it in Android, but I wonder if that has gone out to anyone yet.

Comment 22

[Alex Thompson]

If this issue is from the android bug linked in #19, it may have been fixed in the latest security update to Nexus/Pixel devices. Issue 37187694 was patched in the November security update: https://source.android.com/security/bulletin/pixel/2017-11-01

I haven't had this update installed for long, so I can't comment on stability yet.

Comment 23

[Josh Matthews [:jdm]]

I observe this crash several times a day on my Pixel. Let me know if I can provide any more useful information.

Comment 24

[(not currently active) Ted Mielczarek]

I think this and bug 1392746 are probably the same issue. I've seen both on my Pixel running Oreo as well.

Comment 25

[(not currently active) Ted Mielczarek]

I think the only difference is "do we have symbols for libc.so or not", which determines whether the signature winds up as [@ tgkill ] or [@ libart.so@0x... ].

Comment 26

[Josh Matthews [:jdm]]

Sometime since my last comment these crashes stopped appearing on my Pixel.

Comment 27

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

[triage] Non top-crasher in release.

Comment 28

[Marcia Knous [:marcia]]

Only 1 crash in the 68 release. Resolving as WFM.