1018286 - [Gaia] 1200 ms cold launch time regression across all apps

Status: RESOLVED INVALID

(Firefox OS Graveyard :: Performance, defect, P1)

Description

[Mason Chang [Inactive] [:mchang]]

Large regression, 1200ms across all the apps. 

Good Gecko and Gaia:
Gaia Revision: 26d8fcab9b61f464
Gecko Revision: e6f113c83095

Bad Gecko and Gaia:
Gaia Revision: 26d8fcab9b61f464
Gecko Revision: 76432b693fc4

Looks like a Gecko regression.

Comment 1

[Jason Smith [:jsmith]]

I'm wondering if this is related to the problem I'm seeing in bug 1018185.

Comment 2

[Jason Smith [:jsmith]]

(In reply to Jason Smith [:jsmith] from comment #1)
> I'm wondering if this is related to the problem I'm seeing in bug 1018185.

Actually this is unrelated.

Comment 3

[Mason Chang [Inactive] [:mchang]]

Gaia: 26d8fca on hamachi.

Tip
Gecko - 185840:2208a2ed9745
Results for Settings, cold_load_time: median:788, mean:789, std: 11, max:825, min:767, all:804,795,785,770,779,767,789,788,798,776,789,781,796,825,796,793,793,793,785,784,785,795,782,809,788,789,778,782,790,786

Results for Settings, cold_load_time: median:788, mean:789, std: 12, max:819, min:768, all:819,785,802,790,812,792,791,790,799,795,785,778,805,776,787,787,798,768,786,782,770,809,792,801,777,780,781,772,784,794

Results for Phone, cold_load_time: median:942, mean:948, std: 23, max:1050, min:926, all:954,926,1050,934,942,947,944,942,941,967,933,943,936,995,939,932,932,937,960,933,952,942,937,961,951,953,936,961,930,944

Bad Gecko - 76432b693fc4 
Results for Settings, cold_load_time: median:789, mean:791, std: 14, max:843, min:771, all:818,803,782,787,781,771,791,796,788,843,803,789,789,789,775,785,780,790,805,783,804,791,797,787,798,794,775,778,779,801

Results for Settings, cold_load_time: median:789, mean:791, std: 12, max:825, min:772, all:825,795,776,794,794,786,790,783,772,809,773,806,802,796,776,799,814,787,810,776,787,797,798,776,792,787,786,789,781,788

Results for Phone, cold_load_time: median:944, mean:949, std: 23, max:1060, min:919, all:1060,955,954,941,940,919,958,944,978,951,949,936,968,951,937,953,934,945,942,951,948,934,935,947,941,966,926,932,941,937

Good Gecko - e6f113c83095
Results for Phone, cold_load_time: median:971, mean:979, std: 24, max:1037, min:953, all:994,965,959,975,962,957,957,970,980,971,1024,954,1028,962,967,973,955,1018,1022,960,985,971,971,953,978,1013,961,969,1037,984

Results for Phone, cold_load_time: median:970, mean:976, std: 24, max:1041, min:948, all:979,974,961,980,959,959,1041,952,976,973,969,968,953,1037,956,966,948,976,964,953,1033,967,995,968,974,984,961,972,981,1015

BackoutBot results running on a flame. Doesn't cover the exact gecko range, but updated mozilla-central from yesterday around noon to today around 10 AM pst. 
Gaia: c729c2fa9159c2211744b531a9ee8403e7174848 Gecko: 185726:38c5e21a80fa1d8a8fd061bb0294913d92280cb2
Test: Contacts	Median: 734	Mean: 733	Std Dev: 18
Test: Phone	Median: 530	Mean: 536	Std Dev: 17
Test: Camera	Median: 507	Mean: 511	Std Dev: 16
Test: Email	Median: 283	Mean: 282	Std Dev: 19
Test: Settings	Median: 447	Mean: 455	Std Dev: 32

Gaia: e98fe1e94d3d80ad36903500d8ca3333904b162c Gecko: 185839:2208a2ed9745fba675f2b6fc552c238b4a93db0c
Test: Contacts	Median: 715	Mean: 719	Std Dev: 23
Test: Phone	Median: 535	Mean: 534	Std Dev: 12
Test: Camera	Median: 508	Mean: 512	Std Dev: 19
Test: Email	Median: 280	Mean: 280	Std Dev: 13
Test: Settings	Median: 456	Mean: 452	Std Dev: 22

I'm fairly sure this is a problem with datazilla and not a real regression.

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

the bug is filed as 1200ms cold launch time, the values are not even 1200ms, does the data in datazilla look right?  is the graph/ui misleading?  Could it be that just the alert piece is incorrect?

Comment 5

[Mason Chang [Inactive] [:mchang]]

Attachment: http://selenium.qa.mtv2.mozilla.com:8080/view/B2G%20Perf/job/b2g.hamachi.mozilla-central.perf/1944/

Comment 6

[Mason Chang [Inactive] [:mchang]]

Attachment: http://selenium.qa.mtv2.mozilla.com:8080/view/B2G%20Perf/job/b2g.hamachi.mozilla-central.perf/1945/

I'm fairly sure this is an issue with datazilla. Attached are the good / bad logs. What we see is that the hamachi build has a big red icon meaning it failed. There are lots of time outs, and JavaScript errors in the log file of the bad version. Test data that is generated is very far off from the good log, but I'm not sure what happens if a previous b2g perf run fails or time outs and we run one right after that without rebooting the device. Davehunt, do you know?

Jmaher, or stephend, can you please take a look into why datazilla builds are currently failing? http://selenium.qa.mtv2.mozilla.com:8080/view/B2G%20Perf/job/b2g.hamachi.mozilla-central.perf/

Comment 7

[Mason Chang [Inactive] [:mchang]]

@Joel from comment 4, I took the before / after and subtracted the means. The actual values are all over the place on datazilla.

Comment 8

[Joel Maher ( :jmaher ) (UTC -8)]

some thoughts:
* random data between runs are a good sign of a device that is struggling
* if the individual data points are random- that could be an issue of memory on the device

Comment 9

[:Eli Perelman]

After confirming with Mason that this is indeed an issue within Datazilla, and that the numbers have since recovered, I am closing this as invalid.

Comment 10

[Mason Chang [Inactive] [:mchang]]

I've been getting this error locally when running b2gperf:

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/b2gperf/b2gperf.py", line 362, in run
    self.test()
  File "/usr/local/lib/python2.7/dist-packages/b2gperf/b2gperf.py", line 401, in test
    'launch_app("%s")' % self.app_name)
  File "/usr/local/lib/python2.7/dist-packages/marionette/marionette.py", line 1177, in execute_async_script
    filename=os.path.basename(frame[0]))
  File "/usr/local/lib/python2.7/dist-packages/marionette/decorators.py", line 35, in _
    return func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/marionette/marionette.py", line 624, in _send_message
    self._handle_error(response)
  File "/usr/local/lib/python2.7/dist-packages/marionette/marionette.py", line 669, in _handle_error
    raise JavascriptException(message=message, status=status, stacktrace=stacktrace)
JavascriptException: JavascriptException: TypeError: GaiaApps.getActiveApp(...) is undefined
stacktrace:
        execute_async_script @b2gperf.py, line 401
        inline javascript, line 1081
        src: "      if (GaiaApps.getActiveApp().origin == origin) {"

This is around gaia commit: dfead902971a6ecbfb60aac57a3812530d534246

Comment 11

[Dave Hunt [:davehunt] [he/him] ⌚BST]

I think it's inaccurate to say this is an issue with Datazilla, as that's just the dashboard that displays the results. I can see that the b2gperf job in Jenkins has been failing, which may indicate an issue with b2gperf.

Looking at the first failed build it would appears that the Contacts app failed to report results past the 23rd iteration. This could indicate a crash or similar issue. The same happened with the Music app after the 14th iteration, the Clock app after the 7th iteration, the the FM Radio app after the 2nd iteration. The Camera app failed due to a JavaScript exception after the 2nd iteration. The Browser app failed with the "GaiaApps.getActiveApp(...) is undefined" exception that's noted in comment 10.

A sudden onset of such issues would indicate a regression of some sort. It's interesting that since these issues started we actually had two non-consecutive successful builds, which would either imply these issues are intermittent or that there was a backout (and perhaps relanding).

Comment 12

[Dave Hunt [:davehunt] [he/him] ⌚BST]

It's also worth noting that our mozilla-b2g30_v1_4 job is unaffected.

Comment 13

[Dave Hunt [:davehunt] [he/him] ⌚BST]

(In reply to Mason Chang [:mchang] from comment #6)
> I'm not sure what happens if a previous b2g perf run fails or
> time outs and we run one right after that without rebooting the device.
> Davehunt, do you know?

b2gperf performs a reset by stopping the b2g process, removing persistent storage and profile, and then starting the b2g process again. Each application tested should therefore start in a known good state. The removal of files is optional, and enabled by passing --reset on the command line.

Comment 14

[Dave Hunt [:davehunt] [he/him] ⌚BST]

I've replicated the issue with build 20140530050928. It appears that the homescreen is continually crashing. I'm going to file a separate bug for this issue.

Comment 15

[Dave Hunt [:davehunt] [he/him] ⌚BST]

I've raised bug 1018909, which I believe to be related.