1207642 - Crash at hex address on Android x86 from JNI stack misalignment

Status: RESOLVED FIXED

(Firefox for Android Graveyard :: General, defect)

Description

[Kevin Brosnan [Ex-Mozilla]]

We have a crash that is taking up 27 of the top 50 spots after a day of reporting. All the crashes are at a hex address.

Comment 1

[Nick Alexander :nalexander [he/him]]

We had this reported recently in #mobile.  See also Bug 1201310 and particularly Bug 1031657, which includes crashreport links for x86.

Comment 2

[[:philipp]]

this seems to happen mainly (95%) on devices with a Imagination Technologies gpu - the most frequent crashing device (27% of all crashers) is the ASUS MeMO Pad FHD 10 (ME302C)...

https://crash-stats.mozilla.com/search/?product=FennecAndroid&version=41.0&signature=%24%40&_facets=signature&_facets=android_device&_facets=android_version&_facets=android_manufacturer&_facets=adapter_vendor_id&_facets=adapter_device_id&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#facet-adapter_vendor_id

Comment 3

[Kevin Brosnan [Ex-Mozilla]]

Snorp, you were going to order a MeMoPad. Do you have an order in?

Comment 4

[James Willcox (:snorp) ([e-mail]) (he/him)]

Yeah, I couldn't reproduce on the x86 phone I have here. I ordered a MeMoPad: http://www.amazon.com/gp/product/B00DXFDHV8?psc=1&redirect=true&ref_=od_aui_detailpages00

Comment 5

[Kevin Brosnan [Ex-Mozilla]]

Ioana would you work with the SV team to test this on x86 devices. I'll send you URLs from the crash reports, though we don't know if URLs are key to reproduce this.

Comment 6

[u421692]

Attachment: x86_crash_logs

Was able to reproduce this crash just browsing on facebook.com, or http://www.ayzdorov.ru/lechenie_ykys_blohi.php, or just about:crashes. No specific steps, just panning and zooming.
Device: ZTE Grand X In (Android 4.0.4)
Crash: https://crash-stats.mozilla.com/report/index/fc1f91ad-fa96-4309-bf86-55c522150929

Attaching logcat

Comment 7

[Xavier Laurent]

Using a ME302C, Firefox crashes very often (mostly on first or second visited page) and is not usable. It seems to me that the crashes occur when scrolling or loading a page. If I stay idle on a loaded page it does not seem to crash. Previous version was not crashing, Fx Beta is crashing the same way.

Comment 9

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

Snorp, do we better understand this issue now? This was mentioned in channel meeting as a top crash on Mobile. It would be nice if we had a potential fix in the works soon.

Comment 10

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

KaiRo, I was looking at the crash-state page and this report https://crash-stats.mozilla.com/report/list?product=FennecAndroid&range_value=7&range_unit=days&date=2015-10-07&signature=%400x82c27a8c&version=FennecAndroid%3A41.0. 

Based on product info, does it mean this is not happening on 41.0.1 or is product version 41.0 the same as 41.0.1 now?

Comment 11

[Kevin Brosnan [Ex-Mozilla]]

Firefox for Android did not build a 41.0.1.

Comment 12

[Robert Kaiser]

Right, there is no 41.0.1 for Android - yet. ;-)

Comment 13

[Sylvestre Ledru [:Sylvestre]]

Margaret, Snorp seems busy, do you know who else could help? thanks

Comment 14

[Sylvestre Ledru [:Sylvestre]]

This is a causing a huge spike in term of crashes.

Comment 15

[:Margaret Leibovic]

(In reply to Sylvestre Ledru [:sylvestre] from comment #13)
> Margaret, Snorp seems busy, do you know who else could help? thanks

Snorp is PTO, but I think he should be back Monday. But maybe jchen can help take a look or redirect in the meantime...

Comment 16

[(inactive) Jim Chen [:jchen] [:darchons]]

We suspect the random hex address is a problem with breakpad not generating valid crash reports. Bug 1069556 may fix that; it wouldn't fix the underlying cause of the crashes but would let us identify the crashes better.

Comment 17

[[e-mail]]

A very similar crash affects Samsung GT-i9195/European LTE devices (I know because v41.0 was delivered by autoupdate this evening). Firefox starts up, but crashes after idling on the home screen for perhaps a second or so. Failure is too quick to permit accessing any menus. Crashes, sends crash report, crashes.

FF is installed on the SD card, claims to have 57MB of "data" somewhere in device storage but I'm unwilling to wipe this for fear of destroying bookmarks etc. The GT-i9195 is not an x86 device and doesn't have an Imagination Technologies CPU either so this is probably a red herring. (Qualcomm Snapdragon 400/Adreno 305, ARM arch.)

Comment 18

[[e-mail]]

Sorry, GPU. The point stands; the GPU is a Qualcomm Adreno 305.

Comment 19

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

Margaret, Jim, there is a small chance that I will do another dot release for 41 in a day or two. Based on comment 16 and the fact that this is a top-crash on Fennec, do you think we will have a fix ready for uplift soon (today at best)?

Comment 20

[James Willcox (:snorp) ([e-mail]) (he/him)]

I have the ASUS device in question and can't get it to crash. I'm going to try applying the system update and see if that changes things.

Comment 21

[:Margaret Leibovic]

(In reply to Ritu Kothari (:ritu) from comment #19)
> Margaret, Jim, there is a small chance that I will do another dot release
> for 41 in a day or two. Based on comment 16 and the fact that this is a
> top-crash on Fennec, do you think we will have a fix ready for uplift soon
> (today at best)?

It doesn't look like there's been significant progress here, so I'm skeptical we'll have a patch soon. I defer to snorp.

Comment 22

[Xavier Laurent]

New signature after latest update: https://crash-stats.mozilla.com/report/list?signature=%400x83326bac

Comment 23

[Xavier Laurent]

Attachment: asus2.mov

Attached a small video, in case it can help. The crash can be triggered quite easily by hiding/displaying the address bar (when scrolling) once or twice

Comment 24

[(inactive) Jim Chen [:jchen] [:darchons]]

I looked at a Nightly crash dump and the crash was due to a SSE2 instruction accessing misaligned data, due to a misaligned stack pointer. The stack stays in libxul then goes to libdvm, so it's likely that the stack misalignment happened when libdvm calls the JNI entry point in libxul. In particular, this commit in Dalvik [1] from Honeycomb was about fixing a stack misalignment bug, so I think some x86 devices don't have this fix in their Dalvik builds (sigh). I think we can work around the Dalvik bug by telling GCC to fix the stack manually. 

[1] https://github.com/android/platform_dalvik/commit/4570ad0a7706d3338d58bd0204e102719e4d68fb

Comment 25

[(inactive) Jim Chen [:jchen] [:darchons]]

Attachment: Work around Dalvik bug by realigning stack on JNI entry (for nightly)

Assuming comment 24 is the real cause of the crashes, this patch should work.
It uses the force_align_arg_pointer attribute to force realigning stack at JNI
entry points. Because of recent JNI changes, We will need separate patches for
aurora and beta.

Comment 26

[Sylvestre Ledru [:Sylvestre]]

Jim, I guess we want that to land asap.
Could you fill the uplift request to aurora and beta? Thanks

Comment 27

[(inactive) Jim Chen [:jchen] [:darchons]]

Try for checkin: https://treeherder.mozilla.org/#/jobs?repo=try&revision=905b167794c6

Comment 28

[(inactive) Jim Chen [:jchen] [:darchons]]

Attachment: Patch for Aurora

Approval Request Comment

[Feature/regressing bug #]: N/A

[User impact if declined]: Random crashes on x86 devices

[Describe test coverage new/current, TreeHerder]: Tested locally

[Risks and why]: Very small; patch makes trivial changes to compiled code

[String/UUID change made/needed]: None

Comment 29

[(inactive) Jim Chen [:jchen] [:darchons]]

Attachment: Patch for Beta

Approval Request Comment

[Feature/regressing bug #]: N/A

[User impact if declined]: Random crashes on x86 devices

[Describe test coverage new/current, TreeHerder]: Tested locally

[Risks and why]: Very small; patch makes trivial changes to compiled code

[String/UUID change made/needed]: None

Comment 30

[Carsten Book [:Tomcat]]

this failed to apply:

renamed 1207642 -> Bug-1207642---Work-around-Dalvik-bug-by-realigning.patch
applying Bug-1207642---Work-around-Dalvik-bug-by-realigning.patch
patching file mozglue/android/nsGeckoUtils.cpp
Hunk #2 FAILED at 68
1 out of 2 hunks FAILED -- saving rejects to file mozglue/android/nsGeckoUtils.cpp.rej
patch failed, unable to continue (try -v)
patch failed, rejects left in working directory
errors during apply, please fix and refresh Bug-1207642---Work-around-Dalvik-bug-by-realigning.patch

Comment 31

[James Willcox (:snorp) ([e-mail]) (he/him)]

Jim can you fix this up for uplift

Comment 32

[(inactive) Jim Chen [:jchen] [:darchons]]

Attachment: Patch for Nightly (v2)

Use the force_align_arg_pointer attribute to force realigning stack at
JNI entry points.

Comment 33

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/0f85afd327f2

Comment 34

[Sylvestre Ledru [:Sylvestre]]

Too far in the 41 cycle to do a new dot release, tracking for 42 and we will be doing a 48 beta 9 for this.

Comment 35

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/0f85afd327f2

Comment 36

[Sylvestre Ledru [:Sylvestre]]

Comment on attachment 8676417 [details] [diff] [review]
Patch for Aurora

trying to land that again!

Comment 37

[Sylvestre Ledru [:Sylvestre]]

Comment on attachment 8676418 [details] [diff] [review]
Patch for Beta

Should be in 42 beta 9.

Comment 38

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/releases/mozilla-aurora/rev/3f6b26ed69fd
https://hg.mozilla.org/releases/mozilla-beta/rev/fbd30c48f6ec

Comment 39

[u421692]

I've verified this on Firefox 42 Beta 9 on ZTE Grand X In (Android 4.0.4):
I wasn't able to reproduce this issue following the steps from comment 6, it seems to be fixed, however we might want to check this in crash-stats to be sure. Marking this verified after checking crash-stats.

Comment 40

[Robert Kaiser]

Sigh, this seems to still happen in 42.0b9: See e.g. https://crash-stats.mozilla.com/report/list?signature=%400x4007ed69 or https://crash-stats.mozilla.com/report/list?signature=%400x4007ec79

Comment 41

[Sylvestre Ledru [:Sylvestre]]

Jim, James, any idea ? :/

Comment 42

[Ioana Chiorean]

Unfortunately our device it is a ZTE Grand X In with Android 4.0.4 ( API 15). As I can see the crashes reproduce on API 19 only ( no ZTE listed there either.) So it might be fine for our device. Is there anyone with a API 19 x86 device?

Comment 43

[(inactive) Jim Chen [:jchen] [:darchons]]

(In reply to Robert Kaiser (:[e-mail]) from comment #40)
> Sigh, this seems to still happen in 42.0b9: See e.g.
> https://crash-stats.mozilla.com/report/list?signature=%400x4007ed69 or
> https://crash-stats.mozilla.com/report/list?signature=%400x4007ec79

These two signatures are abort() calls (crash address is 0xdeadbaad). These shouldn't be as frequent as the other ones, and are not covered by the fix.

Comment 44

[Sylvestre Ledru [:Sylvestre]]

Jim, could you open a new bug to follow up with this one? Thanks