1180164 - [raptor] Add mark of tapping application icon for cold launch test

Status: RESOLVED WONTFIX

(Firefox OS Graveyard :: Gaia::PerformanceTest, defect, P2)

Description

[Ting-Yu Chou [:ting] (away)]

Now the first mark of cold launch test is appLaunch, which is before Homescreen calling app.launch(). I'd like to add a new one (tapAppIcon) when tapping application icon and make it the first, this should reflect what user really see.

Comment 1

[GitHub Autolander Bot]

Attachment: [gaia] janus926:bug-1180164 > mozilla-b2g:master

Comment 2

[:Eli Perelman]

:ting, can you run this and tell me what the average time difference is between the current appLaunch marker and tapAppIcon is? We need to determine what this number is, because this will be the amount of regression that all apps will start to exhibit, and we need to determine whether this is something that we want to accept. Honestly we should probably just go with this change, but I just want to be more informed first. I also need to think through how we can still achieve this once we switch app launching back to Marionette.

Comment 3

[Ting-Yu Chou [:ting] (away)]

This is what I got from a Flame 319MB with command |$ RUNS=30 APP=sms node tests/raptor/launch_test|:

Metric                            Mean      Median    Min       Max       StdDev   p95     
--------------------------------  --------  --------  --------  --------  -------  --------
coldlaunch.navigationLoaded       879.133   868.500   820.000   1002.000  41.832   947.000 
coldlaunch.willRenderThreads      913.100   906.500   856.000   1033.000  41.264   982.000 
coldlaunch.navigationInteractive  918.533   910.500   862.000   1037.000  41.147   990.000 
coldlaunch.visuallyLoaded         1242.933  1240.000  1194.000  1373.000  33.468   1289.000
coldlaunch.contentInteractive     1801.567  1791.000  1728.000  2020.000  54.504   1867.000
coldlaunch.objectsInitEnd         1831.267  1818.000  1762.000  2057.000  55.780   1910.000
coldlaunch.fullyLoaded            3056.733  3029.500  2927.000  3333.000  100.618  3289.000

Metric                            Mean      Median    Min       Max       StdDev   p95     
--------------------------------  --------  --------  --------  --------  -------  --------
coldlaunch.appLaunch              109.723   109.110   104.753   118.182   3.550    116.603 
coldlaunch.navigationLoaded       984.823   960.967   925.221   1254.798  62.538   1069.019
coldlaunch.willRenderThreads      1018.790  995.064   959.221   1299.798  64.637   1101.697
coldlaunch.navigationInteractive  1024.023  1001.064  963.221   1303.798  64.556   1105.697
coldlaunch.visuallyLoaded         1361.923  1336.333  1306.221  1668.798  67.909   1453.697
coldlaunch.contentInteractive     1920.456  1904.064  1847.829  2253.798  77.938   2034.697
coldlaunch.objectsInitEnd         1952.156  1937.564  1875.829  2280.798  76.992   2064.697
coldlaunch.fullyLoaded            3200.290  3180.056  2987.871  3500.798  129.082  3457.681

Comment 4

[:Eli Perelman]

Comment on attachment 8629784 [details] [review]
[gaia] janus926:bug-1180164 > mozilla-b2g:master

Unfortunately I don't think we can take this in its current form; it causes metrics to report a 100ms regression. Given that we can't have any apps regressing performance for v2.5 (bug 1180696), we will be hard-pressed to justify allowing a major change in the way we capture the numbers.

This also wouldn't able to be aligned with the numbers captured by telemetry. Also, the performance delta between tapAppIcon and appLaunch would be very hardware-specific. I also can't think of a way where we can *reliably* port this same logic to Marionette once we use it for launching apps.

:(

Comment 5

[Ting-Yu Chou [:ting] (away)]

(In reply to :Eli Perelman from comment #4)
> telemetry. Also, the performance delta between tapAppIcon and appLaunch
> would be very hardware-specific.

:(

For now there're two bugs in this period: bug 1150349 and bug 1130993. If we don't have this mark it will be difficult to detect regression between tapAppIcon and appLaunch.

Why are we switching to Marionette for launching apps though? Another idea is to create a test case only to mointor tapAppIcon and appLaunch...

Comment 6

[Ting-Yu Chou [:ting] (away)]

Bobby and Gregor, just want to let you know that the launch numbers Raptor now measures start from Homescreen calling app.launch(), not from tapping application icon. And it is >100ms between these two points now.

Eli has his concerns in comment 4 and we will not go with this change, please let us know if you have any other ideas.

Comment 7

[Bobby Chien]

[Tracking Requested - why for this release]:

my opinion is making another raptor test for the time between tapAppIcon and appLaunch (comment 5). This test could be new indicator which shows the time margin before homescreen launch any app. So the total app launch time will be homescreen-launchs-app-time + app-launch-time. Based on this measurements, we could keep current app-launch-time design with no change. 

What do you think? Eli.

Comment 8

[:Eli Perelman]

It will be seen whether we can use this approach with the Marionette refactor or not. As soon as I get my refactor landed we can discuss it further.