1038995 - [B2G][Flame][HERE Maps] App crashes when selecting the direction icon.

Status: RESOLVED WORKSFORME

(Firefox OS Graveyard :: Performance, defect, P3)

Description

[Yeojin Chung [:YeojinC]]

Attachment: logcat_20140715_HEREMapsCrash.txt

Description:
When the user selects the direction icon, the app crashes after the page is fully loaded.

Repro Steps:
1) Updated Flame to Build ID: 20140715000201.
2) Make sure the Flame device is set to 273mb.
3) Download HERE Maps from Marketplace, and launch the application.
4) Select "Share" on the App Permission Screen.
5) Search a place on the search field.
6) When the location is displayed, select the direction icon.
7) Observe the direction screen is loaded.

Actual:
App crashes while or after the page is loaded.

Expected:
App does NOT crash.

Flame 2.0 (273mb)

Environmental Variables:
Device: Flame 2.0
BuildID: 20140715000201
Gaia: 2c6c413ed729d465c52d6c2d5d458e2eee79e956
Gecko: d32649a24965
Version: 32.0a2 (2.0) 
Firmware Version: v122
User Agent: Mozilla/5.0 (Mobile; rv:32.0) Gecko/32.0 Firefox/32.0

Repro frequency: 100%
See attached: video, logcat
http://youtu.be/jA8SvVTF_Xk

Comment 1

[Yeojin Chung [:YeojinC]]

This issue does NOT reproduce on Flame 2.0 (512mb), Flame 2.1 (273mb), Flame 1.4 (273mb), Buri 2.1, Buri 2.0, Buri 1.4, Open C 2.1, Open C 2.0, Open C 1.4:

Flame 2.0 (512mb)

Environmental Variables:
Device: Flame 2.0
BuildID: 20140715000201
Gaia: 2c6c413ed729d465c52d6c2d5d458e2eee79e956
Gecko: d32649a24965
Version: 32.0a2 (2.0) 
Firmware Version: v122
User Agent: Mozilla/5.0 (Mobile; rv:32.0) Gecko/32.0 Firefox/32.0

Flame 2.1 (273mb)

Environmental Variables:
Device: Flame Master
Build ID: 20140715040206
Gaia: 46cd188fdda2397d2b8f3303a184dcd52952e2b2
Gecko: e032c429908b
Version: 33.0a1 (Master)
Firmware Version: v122
User Agent: Mozilla/5.0 (Mobile; rv:33.0) Gecko/33.0 Firefox/33.0
Flame 1.4 (273mb)

Buri 2.1

Environmental Variables:
Device: Buri Master
Build ID: 20140715040206
Gaia: 46cd188fdda2397d2b8f3303a184dcd52952e2b2
Gecko: e032c429908b
Version: 33.0a1 (Master)
Firmware Version: v1.2device.cfg
User Agent: Mozilla/5.0 (Mobile; rv:33.0) Gecko/33.0 Firefox/33.0

Buri 2.0

Environmental Variables:
Device: Buri 2.0
Build ID: 20140715000201
Gaia: 2c6c413ed729d465c52d6c2d5d458e2eee79e956
Gecko: d32649a24965
Version: 32.0a2 (2.0)
Firmware Version: v1.2device.cfg
User Agent: Mozilla/5.0 (Mobile; rv:32.0) Gecko/32.0 Firefox/32.0

Buri 1.4

Environmental Variables:
Device: Buri 1.4
Build ID: 20140715000202
Gaia: b7d36622c7df92c976c37520ccab25199c7ada91
Gecko: de7ecfb00955
Version: 30.0 (1.4)
Firmware Version: v1.2device.cfg
User Agent: Mozilla/5.0 (Mobile; rv:30.0) Gecko/30.0 Firefox/30.0

Open C 2.1

Environmental Variables:
Device: Open_C Master
BuildID: 20140715040206
Gaia: 46cd188fdda2397d2b8f3303a184dcd52952e2b2
Gecko: e032c429908b
Version: 33.0a1 (Master) 
Firmware Version: P821A10V1.0.0B06_LOG_DL
User Agent: Mozilla/5.0 (Mobile; rv:33.0) Gecko/33.0 Firefox/33.0

Open C 2.0

Device: Open_C 2.0
Build ID: 20140715000201
Gaia: 2c6c413ed729d465c52d6c2d5d458e2eee79e956
Gecko: d32649a24965
Version: 32.0a2 (2.0)
Firmware Version: P821A10V1.0.0B06_LOG_DL
User Agent: Mozilla/5.0 (Mobile; rv:32.0) Gecko/32.0 Firefox/32.0

Open C 1.4

Environmental Variables:
Device: Open_C 1.4
Build ID: 20140715000202
Gaia: b7d36622c7df92c976c37520ccab25199c7ada91
Gecko: de7ecfb00955
Version: 30.0 (1.4)
Firmware Version: P821A10V1.0.0B06_LOG_DL
User Agent: Mozilla/5.0 (Mobile; rv:30.0) Gecko/30.0 Firefox/30.0

App does not crash when the direction page is fully loaded.

Comment 2

[KTucker [:KTucker][Inactive 3/4/2016]]

This is a regression from 1.4. A major function of the app is unusable so nominating 2.0?

Comment 3

[David Durst [:ddurst]]

Does this depend on Marketplace for anything other than the installation of the app?

Comment 4

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

This isn't a crash, it was killed : 
07-15 15:51:39.858   310   310 E OomLogger: [Kill]: send sigkill to 5235 (HERE Maps), adj 134, size 7620

Comment 5

[Jayme Mercado [:JMercado] (Inactive 3/4/2016)]

This does not depend on Marketplace for anything other than the installation.

The window for this issue seemed to have come from B2g-Inbound but since that window was too large I checked Mozilla-inbound as well.  Both are included below.  

I found that a large amount of what was marked "Merge m-c to b2g-i" was in the Mozilla-inbound window.  I believe that I misunderstood what m-c means in this case.  I thought it stood for Mozilla Central.  Can you please clarify if that is the case, for use on future regression window checks?

B2g-inbound Regression Window

Last working
Environmental Variables:
Device: Flame Master
BuildID: 20140430053004
Gaia: adfe7c3fc8c8cf39bf14ab416ff531af1f3dd02f
Gecko: b5b912c44f82
Version: 32.0a1 (Master)
Firmware Version: v122
User Agent: Mozilla/5.0 (Mobile; rv:32.0) Gecko/32.0 Firefox/32.0


First Broken
Environmental Variables:
Device: Flame Master
BuildID: 20140430083003
Gaia: cf285caa0c65244abae9fc99cc6f88765a8d8737
Gecko: cd8d210b9a94
Version: 32.0a1 (Master)
Firmware Version: v122
User Agent: Mozilla/5.0 (Mobile; rv:32.0) Gecko/32.0 Firefox/32.0


Last working gaia / First broken gecko - Issue DOES occur
Gaia: adfe7c3fc8c8cf39bf14ab416ff531af1f3dd02f
Gecko: cd8d210b9a94

First broken gaia / Last working gecko - Issue does NOT occur
Gaia: cf285caa0c65244abae9fc99cc6f88765a8d8737
Gecko: b5b912c44f82

Gecko Pushlog:  https://hg.mozilla.org/integration/b2g-inbound/pushloghtml?fromchange=b5b912c44f82&tochange=cd8d210b9a94




Mozilla inbound regression window

Last working
Environmental Variables:
Device: Flame Master
BuildID: 20140429163002
Gaia: db3bcec51a361daddb7d3d4ba4d8a2a664b7b6aa
Gecko: de19c62cbc6b
Version: 32.0a1 (Master) 
Firmware Version: v122
User Agent: Mozilla/5.0 (Mobile; rv:32.0) Gecko/32.0 Firefox/32.0

First Broken
Environmental Variables:
Device: Flame Master
BuildID: 20140429193004
Gaia: db3bcec51a361daddb7d3d4ba4d8a2a664b7b6aa
Gecko: c0d658d3f739
Version: 32.0a1 (Master) 
Firmware Version: v122
User Agent: Mozilla/5.0 (Mobile; rv:32.0) Gecko/32.0 Firefox/32.0

Last working gaia / First broken gecko - Issue DOES occur
Gaia: db3bcec51a361daddb7d3d4ba4d8a2a664b7b6aa
Gecko: c0d658d3f739

First broken gaia / Last working gecko - Issue does NOT occur
Gaia: db3bcec51a361daddb7d3d4ba4d8a2a664b7b6aa
Gecko: de19c62cbc6b

Gecko Pushlog: https://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?fromchange=de19c62cbc6b&tochange=c0d658d3f739

Comment 6

[Joshua Mitchell (Inactive)]

Jason - these pushlogs are very tech-jargon heavy and nothing really stands out for me to nominate as the probable cause of the regression. If I HAD to nominate something it would be bug 1003320 but I'd appreciate more knowledgeable eyes on this before I go NI'ing people to take a look as it feels like I’d just be stabbing in the dark.

Comment 7

[Jason Smith [:jsmith]]

(In reply to Joshua Mitchell [:Joshua_M] from comment #6)
> Jason - these pushlogs are very tech-jargon heavy and nothing really stands
> out for me to nominate as the probable cause of the regression. If I HAD to
> nominate something it would be bug 1003320 but I'd appreciate more
> knowledgeable eyes on this before I go NI'ing people to take a look as it
> feels like I’d just be stabbing in the dark.

Right. Good news is that we've at least narrowed down the range by a lot, so we'll just need a developer from each area in the range to weigh in if they think the issue seen here is related to a bug in their relevant area.

I'm adding needinfo for the following people:

* Doug to check if the networking bug (bug 1001022) is related to this or not
* Nicholas to check if the JS bug (bug 952650) is related to this or not
* Ben to check if the Canvas bug (bug 999841) is related to this or not

Comment 8

[Ben Kelly [:bkelly, not reviewing]]

You can experiment with bug 999841 by setting gfx.canvas.max-size-for-skia-gl.  Values are:

   0   means unlimited size
   > 0 means use value as an absolute threshold
   < 0 means use the number of screen pixels as a threshold

You probably want to set it to some artificially low absolute threshold, like 1, to force software canvas.

Still, even if that avoids the crash, its not clear what actually is using more memory vs 1.4 from this test.

Comment 9

[Ben Kelly [:bkelly, not reviewing]]

What would be useful would be to get memory reports on 1.4 and 2.0 with the 512MB configuration.  The higher memory config will let the code survive long enough for us to take the memory report.  The memory reports should be taken immediately after the direction screen is loaded (when it would have crashed before).

Comment 10

[Jayme Mercado [:JMercado] (Inactive 3/4/2016)]

Attachment: Flame_1.4_about_memory.zip

Comment 11

[Jayme Mercado [:JMercado] (Inactive 3/4/2016)]

Attachment: Flame_2.0_about_memory.zip

Comment 12

[Jayme Mercado [:JMercado] (Inactive 3/4/2016)]

> What would be useful would be to get memory reports on 1.4 and 2.0 with the
> 512MB configuration.  The higher memory config will let the code survive
> long enough for us to take the memory report.  The memory reports should be
> taken immediately after the direction screen is loaded (when it would have
> crashed before).

I have uploaded about memory logs for 1.4 and 2.0 while the device is set to 512 mb.(In reply to Ben Kelly [:bkelly] from comment #9)

Comment 13

[Nicholas Nethercote [inactive]]

Bug 952650 was a very mechanical refactoring of some utility functions that involve JS values. These functions are both simple and very heavily used, so it seems unlikely that any problem with them would only manifest in this fashion.

Also, considering that the 512 MiB Flame 2.0 config works but the 273 MiB Flame 2.0 config does not, and
it's an OOM crash, I'm confident that bug 952650 is not at fault. It smells to me like some kind of graphics memory issue...

Comment 14

[Ben Kelly [:bkelly, not reviewing]]

(In reply to Jayme Mercado [:JMercado] from comment #12)
> I have uploaded about memory logs for 1.4 and 2.0 while the device is set to
> 512 mb.(In reply to Ben Kelly [:bkelly] from comment #9)

Were these taken with the -m command line argument to minimize memory?  Sorry I forgot to mention that, but its probably necessary for this test.

As of right now these reports don't make much sense at first glance.  For example, diffing from 1.4 to 2.0 for HERE maps:

 -1.55 MB ── resident
-15.03 MB ── resident-unique

For overall system memory we see this, though:

0.00 MB (100.0%) -- mem
├──-4.38 MB (100.0%) ++ processes
├──17.79 MB (100.0%) ── other
└──-13.41 MB (100.0%) ── free

Not sure whats in the "other" category, but it seems graphics memory is likely to fall in that bucket.

Comment 15

[Ben Kelly [:bkelly, not reviewing]]

Forgot to flag qawanted again in the last comment.  Can you please collect new memory reports using -m flag this time to force minimization.  Sorry for not catching this the first time!

Comment 16

[Jayme Mercado [:JMercado] (Inactive 3/4/2016)]

Attachment: Flame 1.4 about memory.zip

Comment 17

[Jayme Mercado [:JMercado] (Inactive 3/4/2016)]

Attachment: Flame 2.0 about memory.zip

Comment 18

[Jayme Mercado [:JMercado] (Inactive 3/4/2016)]

I have replaced the about memory logs with ones using the following command: 
MOZ_IGNORE_NUWA_PROCESS=1 ./get_about_memory.py -m

Please let me know if that wasn't what you needed.

Comment 19

[Kai-Chih Hu [:khu]]

[Blocking Requested - why for this release]:
Should we have reasonable testing environment first, especially for LMK / OOMK / ZRAM size,... in kernel configuration? Thanks.

Comment 20

[Jason Smith [:jsmith]]

(In reply to Kevin Hu [:khu] from comment #19)
> [Blocking Requested - why for this release]:
> Should we have reasonable testing environment first, especially for LMK /
> OOMK / ZRAM size,... in kernel configuration? Thanks.

QA Wanted to retest on Flame 319 MB.

Comment 21

[Jayme Mercado [:JMercado] (Inactive 3/4/2016)]

This issue does not occur on Flame 2.0 319 MB.  The app does not OOM when getting directions.

Environmental Variables:
Device: Flame 2.0
BuildID: 20140724081209
Gaia: 68226b3fd4eba752307daa5e917238bde253f5ab
Gecko: 8b920340ee28
Version: 32.0 (2.0) 
Firmware Version: v122
User Agent: Mozilla/5.0 (Mobile; rv:32.0) Gecko/32.0 Firefox/32.0

Comment 22

[Joshua Mitchell (Inactive)]

Issue does not repro in 319 mb - closing as Works for Me

Comment 23

[Joshua Mitchell (Inactive)]

leaving open for perf investigation

Comment 24

[Ben Kelly [:bkelly, not reviewing]]

I don't see anything obvious in the memory reports.  HERE maps is using 6.94MB less in v2.1 vs v1.4.  The main process is using 7.05MB more.  Seems like a wash.

If I had to guess I'd say the original problem is due to increased transient memory usage in skia canvas acceleration.  Canvas memory usage is being discussed over in bug 1042291.

Comment 25

[:Eli Perelman]

FxOS Perf Triage: Nothing to investigate, closing.