1374570 - Crash in Motorola libhwui.so@0x42b46

Status: RESOLVED WORKSFORME

(GeckoView :: General, defect, P2)

Description

[Gerry Chang [:gchang]]

This bug was filed from the Socorro interface and is 
report bp-bd6aa914-a26d-4c15-bce7-cf8d70170619.
=============================================================
Frame 	Module 	Signature 	Source
Ø 0 	libhwui.so 	libhwui.so@0x42b46 	
Ø 1 	libhwui.so 	libhwui.so@0x435d3 	
Ø 2 	libhwui.so 	libhwui.so@0x4365f 	
Ø 3 	libhwui.so 	libhwui.so@0x4513f 	
Ø 4 	libhwui.so 	libhwui.so@0x56662 	
Ø 5 	libc.so 	libc.so@0x1379b 	
Ø 6 	libutils.so 	libutils.so@0xef55 	
Ø 7 	libandroid_runtime.so 	libandroid_runtime.so@0xf58de 	
Ø 8 	libutils.so 	libutils.so@0xeee3 	
Ø 9 	libandroid_runtime.so 	libandroid_runtime.so@0x5fd21 	
Ø 10 	libandroid_runtime.so 	libandroid_runtime.so@0x5fcd7 	
Ø 11 	libutils.so 	libutils.so@0xeac5 	
Ø 12 	libutils.so 	libutils.so@0xea8b 	
Ø 13 	libc.so 	libc.so@0x137bb 	
Ø 14 	libc.so 	libc.so@0x1379b 	
Ø 15 	libc.so 	libc.so@0x1379b 	
Ø 16 	libc.so 	libc.so@0x1189b 	
Ø 17 	dalvik-main space (deleted) 	dalvik-main space (deleted)@0x1651d4d3

This is #19 topcrash for Firefox for android and there is a spike in the last 3 days.

Comment 1

[Marcia Knous [:marcia]]

Adding additional affect branches. This crash seems to only affect two Samsung devices, SM-N9005 and SM-N900U. Comments don't seem to be particularly useful.

Comment 2

[Marcia Knous [:marcia]]

In 57 release this signature has 1821 crashes, and there are a few other similar signatures which affects Samsung devices.

Comment 3

[Marcia Knous [:marcia]]

Adding 59 and 60 as affected. This is #6 overall browser crash for 58.0.2. It looks as if one Samsung device is generating most of the crashes: SM-N9005

Comment 4

[Marcia Knous [:marcia]]

Continues to be quite high in 58.0.2, and 98% of the crashes are GALAXY Note 3 running API 21.

Comment 5

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

[triage] #8 top crasher.

Comment 6

[:sdaswani]

Putting this on Softvision's list.

Comment 7

[Vlad Baicu]

We have tried to reproduce this issue on our Note 3 running Android 5.0 but in the absence of any STRs or actionable stacktraces we could use more info in order to go further with this

Comment 8

[:sdaswani]

Marcia, no luck trying to repro - can you suggest next steps?

Comment 9

[Marcia Knous [:marcia]]

Updating affected branches. I will take a look at the crash reports and see if there are any other clues in the comments.

Comment 10

[michael]

I also have an affected Note 3 SM-N9005 device. Firefox v59 and v60 beta crash at least daily. Is there a workaround? I've set "layers.acceleration.disabled;true" and "layers.acceleration.force-enabled;false" in about:config; but Firefox keeps crashing.

Comment 11

[michael]

Still no news for Note 3 users? Firefox keeps crashing for no reason multiple times per day. Also version 58 crashes, but not so often.

Comment 12

[Marcia Knous [:marcia]]

#6 overall top crash on release. Updating flags to reflect that this affects 62/63. SM-N9005 is the primary device crashing.

Comment 13

[Liz Henry (:lizzard)]

Let's keep this in triage to follow up on the crash rate for 63 release. 
I think the next step here is to get Vlad access to see the crash reports in the Play Store. 
It is likely too late to fix this on 63 but we could aim to investigate it for 65, and if we arrive at a fix then consider uplift to 64 beta.

Comment 14

[Marcia Knous [:marcia]]

63.0.2 has 1824 crashes, almost all from Samsung SM-N9005.

Comment 15

[Chris Peterson [:cpeterson]]

There are a lot of related or possible dupes: bug 1492183, bug 1510984, bug 1235662, bug 1430409, bug 1476077

Comment 16

[David Bolter [:davidb] (NeedInfo me for attention)]

Flagging this because it is tracking 65 and has no assignee, and it shows up on a list I check weekly.

(In reply to Marcia Knous [:marcia - needinfo? me] from comment #14)
> 63.0.2 has 1824 crashes, almost all from Samsung SM-N9005.

Crashy hwui.so -- anything we should do here? Kevin, NI'ing you in case you want to research it a bit.

Comment 17

[:sdaswani]

Clearing my NI since it's on Liz'a radar.

Comment 18

[Liz Henry (:lizzard)]

This seems very high volume on release.

Comment 19

[Marcia Knous [:marcia]]

I still see lots of crashes with Samsung devices - SM-N9005 (Samsung Galaxy Note 3) running API 21.

Other devices include:

• moto g(6)
• Samsung Galaxy S7
• Samsung Galaxy S8

ni on Ioana to see if someone can reproduce this one of the affected devices.

Comment 20

[Ioana Chiorean]

We do have the Samsung devices (but not the Moto one - i can ask in the company maybe others have it) so we will try to see if we can catch it with them.
Kevin do u have by chance the moto 6?

Comment 21

[Andrei Bodea[:andrei]]

I tested this issue on the RC build 64.0.2 but I wasn't able to reproduce it, with the following devices:
Samsung Galaxy 8(A 8.0), Samsung Galaxy S8+(A 8.0), Samsung Galaxy Note 9(A 8.1.0), Samsung Galaxy S7(A 7.0).
Note that I didn't find a Motorola g(6) and I will try to find one as Ioana said in comment 20.
I will keep an eye out for this issue while testing and if il manage to reproduce it or find any relevant information, ill get back and post them.

Comment 22

[Kevin Brosnan [Ex-Mozilla]]

I have a G5 not a G6.

Comment 23

[David Bolter [:davidb] (NeedInfo me for attention)]

Susheel do we get any interesting info out of playstore for these signatures?

Comment 24

[:sdaswani]

Sylvestre can you add David to the Play Store for Fennec and Focus so he can take a look at crash sigs? Thanks!

Comment 25

[Sylvestre Ledru [:Sylvestre]]

Sure, you should have permissions now David! (on the 3 fx products: release, beta & nightly)

Comment 26

[James Willcox (:snorp) ([e-mail]) (he/him)]

Looking at the logcats here it's OOM. Most likely just address space exhaustion.

Comment 27

[Chris Peterson [:cpeterson]]

Interesting that the crash volume steadily increased from about 400/day in October to almost 1000/day in December. Perhaps there is an external factor, like the rollout of a new OS update, contributing to this issue.

Comment 28

[Marcia Knous [:marcia]]

In looking further at some of the higher volume crashes, here are some data points:

libhwui.so@0x86da6

• 45% of the crashes are from SM-G965U (Galaxy S9)
• All of the crashes are Samsung devices running API 26

libhwui.so@0x81fde

• Largely Motorola devices, all running API 26

libhwui.so@0x86dae

• All API 26
• 83% of the crashes are from SM-N950U (Galaxy Note 8)

libhwui.so@0x9dc06

• All API 27
• 88% are from SM-N960U (Galaxy Note 9)

Comment 29

[Henrik Gemal]

https://www.power.dk/kampagne/powerweekend/#campaign-nav
Scrolling on this site seems to crash it. Using nightly on Motorola g6 https://crash-stats.mozilla.org/report/index/c70e3dc4-9ba3-4788-b123-cd39c0190323

Comment 30

[Sören Hentzschel]

Firefox 66.0.2 always crashes on www.foodist.at with Motorola G6 (four attempts, four crashes):
https://crash-stats.mozilla.org/report/index/3874a5e0-dc92-41f6-8db7-084420190402

Comment 31

[Chris Peterson [:cpeterson]]

(In reply to Henrik Gemal from comment #29)

https://www.power.dk/kampagne/powerweekend/#campaign-nav

(In reply to Sören Hentzschel from comment #30)

Firefox 66.0.2 always crashes on www.foodist.at with Motorola G6 (four attempts, four crashes):

Neither power.dk or foodist.at crash on my Moto G5 Plus, so this must be something specific to the Moto G6.

Comment 32

[Emma Humphries ☕️🎸🧞‍♀️✨ (she/they) [:emceeaich] (Pacific Time) use needinfo]

Should the stalled tag remain given this new information?

Comment 33

[Marcia Knous [:marcia]]

Updating flags. In 67b11 this was the #6 overall crash.

Comment 34

[Christian Ascheberg]

Hi, I am using Firefox 68.0b5 on a Moto G5S running Android 8.1.0. Since a few days I am experiencing situations in which Firefox stops rendering a page while it is loading. That tab and all other tabs become blank and eventually Firefox crashes. Sometimes I just close the Firefox app before it actually crashes. I think the crash started with the update to beta5. The signature is libhwui.so@0x2fe32, which crash-stats relates to this bug.

I can also always reproduce the crash on www.foodist.at mentioned in the previous comment. The symptoms are the same (all tabs become blank), but the signature is different (libESXGLESv2_adreno.so@0x3476e2).

Comment 35

[Marcia Knous [:marcia]]

The largest volume signature in the 68 release is https://mzl.la/2XNX8NB. This is currently the #11 overall top crash for Fennec 68 release.

From that, here are the correlations - high correlation to Motorola:

(100.0% in signature vs 08.69% overall) reason = SIGABRT
(95.27% in signature vs 03.17% overall) Module "u:object_r:product_prop:s0" = true
(95.27% in signature vs 06.40% overall) Module "u:object_r:display_id_prop:s0" = true
(95.27% in signature vs 06.90% overall) Module "Newfont_Light.ttf" = true
(95.27% in signature vs 07.41% overall) android_brand = motorola
(95.27% in signature vs 07.47% overall) android_manufacturer = motorola
(100.0% in signature vs 14.07% overall) android_version = 27 (REL)

Comment 36

[Chris Peterson [:cpeterson]]

Jesse, this medium-volume Android crash looks like a crash in Motorola graphics drivers. I'm not sure what we can do. I hear the Graphics team has had some luck in the past working around Android driver crashes by rearranging our GPU calls by trial and error.

Comment 37

[Jessie [:jbonisteel] pls NI]

Jamie - could you take a peak when you have a chance?

Comment 38

[Jamie Nicol [:jnicol]]

This crash is caused by OOM. I can reproduce on power.dk and foodist.at on my G6. On those sites we're layerising poorly, meaning we create a lot of textures, run out of memory, and the driver crashes.

The problem with trying to fix layerisation issues is that it's a game of whack-a-mole, changing the heuristics to make one site work better can cause issues on other sites, and even fixing specific sites can have very little effect on the overall numbers because each site has its own unique problems.

With fennec being in maintainence mode, I'm not sure this is worth putting the time in to fix. GeckoView should improve things a lot, presumably due to being multi-process (I can't reproduce a crash on those sites with geckoview). And webrender should improve things further too.

Comment 39

[Marcia Knous [:marcia]]

Some of these signatures currently show up in Fenix as well, see https://bit.ly/2YvNHSw. Will Bug 1530770 help things here?

Comment 40

[Marcia Knous [:marcia]]

I forgot to add in my last comment that this crash is #8 overall in the current Fennec 68 release.

Comment 42

[Chris Peterson [:cpeterson]]

Moving to the GeckoView Bugzilla component because this random libhwui.so crash affects both Fenix/GV and Fennec.

Comment 43

[Chris Peterson [:cpeterson]]

This Motorola crash signature's volume is still high (about 800 reports per day), but we don't have a clear solution or workaround in hand.