1189469 - [Raptor] Cache application coordinates to improve application launch variance

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Gaia::PerformanceTest, defect)

Description

[:Eli Perelman]

When Raptor uses Marionette, we search for and launch the application at every run. During this process, we start a Marionette session, launch the app, and while launch kill the Marionette session. I believe this activity during app launch may be contributing to some variance in times.

Cache the coordinates using Marionette in advance of launch and use those coordinates each time.

Comment 1

[:Eli Perelman]

Attachment: Link to Github pull-request: https://github.com/mozilla-b2g/raptor/pull/15

Comment 2

[Robert Wood [:rwood]]

(In reply to :Eli Perelman from comment #1)
> Created attachment 8641239 [details] [review]
> Link to Github pull-request: https://github.com/mozilla-b2g/raptor/pull/15

I tested launching each of the 12 apps on aries, flame, and emulator and it worked. However retries doesn't work as I expected. If I manually scroll the lcd before an app launch, so that the wrong app is launched, I had expected a retry to scroll the homescreen back into place so the next tap could work, however there's no retry and there's a marionette error instead:

https://gist.github.com/rwood-moz/bb68156d8400b44f75d2

Comment 3

[:Eli Perelman]

I'm changing up the way the Marionette connection works, so hopefully I can get this resolved soon.

Comment 4

[:Eli Perelman]

OK, PR updated. Please let me know if the retry works better now.

Comment 5

[Robert Wood [:rwood]]

Comment on attachment 8641239 [details] [review]
Link to Github pull-request: https://github.com/mozilla-b2g/raptor/pull/15

LGTM. Retries are working again with latest update. The unrelated marionette polling socket timeout issue (Bug 1188990) prevented me from trying the latest update on the emulator, but I'm satisfied with the patch as it works great on device.

Comment 6

[:Eli Perelman]

In master: https://github.com/mozilla-b2g/raptor/commit/0d2151892b2f8254636b7e02d5e75c3bb8b0e8a0
In npm: @mozilla/raptor v1.4.5

Comment 7

[:Eli Perelman]

As an update, here is a comparison against the Clock app, 30 runs:

[No coordinate cache]

| Metric                           | Mean     | Median   | Min     | Max      | StdDev  | p95      |
| -------------------------------- | -------- | -------- | ------- | -------- | ------- | -------- |
| coldlaunch.navigationLoaded      | 967.200  | 992.500  | 702.000 | 1106.000 | 94.081  | 1078.000 |
| coldlaunch.navigationInteractive | 1057.933 | 1091.000 | 802.000 | 1210.000 | 99.256  | 1165.000 |
| coldlaunch.visuallyLoaded        | 1356.733 | 1404.500 | 975.000 | 1527.000 | 137.964 | 1466.000 |
| coldlaunch.contentInteractive    | 1357.000 | 1404.500 | 975.000 | 1529.000 | 138.021 | 1467.000 |
| coldlaunch.fullyLoaded           | 1357.400 | 1405.500 | 975.000 | 1530.000 | 138.091 | 1467.000 |
| coldlaunch.pss                   | 19.134   | 19.127   | 18.835  | 19.423   | 0.109   | 19.339   |
| coldlaunch.uss                   | 14.501   | 14.475   | 14.406  | 14.773   | 0.079   | 14.676   |
| coldlaunch.rss                   | 31.042   | 31.023   | 30.953  | 31.277   | 0.071   | 31.195   |

---

[Coordinate Cache]

| Metric                           | Mean     | Median   | Min      | Max      | StdDev | p95      |
| -------------------------------- | -------- | -------- | -------- | -------- | ------ | -------- |
| coldlaunch.navigationLoaded      | 1076.467 | 1080.000 | 1010.000 | 1175.000 | 47.993 | 1166.000 |
| coldlaunch.navigationInteractive | 1170.800 | 1174.500 | 1107.000 | 1278.000 | 45.740 | 1261.000 |
| coldlaunch.visuallyLoaded        | 1412.367 | 1400.500 | 1299.000 | 1565.000 | 59.067 | 1540.000 |
| coldlaunch.contentInteractive    | 1412.467 | 1401.000 | 1299.000 | 1565.000 | 59.094 | 1540.000 |
| coldlaunch.fullyLoaded           | 1412.733 | 1401.000 | 1300.000 | 1565.000 | 59.051 | 1540.000 |
| coldlaunch.rss                   | 31.078   | 31.117   | 30.785   | 31.234   | 0.121  | 31.184   |
| coldlaunch.pss                   | 19.217   | 19.271   | 18.936   | 19.378   | 0.134  | 19.336   |
| coldlaunch.uss                   | 14.549   | 14.588   | 14.254   | 14.695   | 0.119  | 14.652   |

---

Big thing to note is the improvement in standard deviation from around 138ms down to 59ms, which is pretty significant. The means and p95s look smaller because greater ranges were produced, not because there were any actual improvements.

Comment 8

[:Eli Perelman]

Also, here is a set of runs from when Raptor was in the Gaia tree, Clock, 30 runs:

Metric                            Mean      Median    Min       Max       StdDev  p95
--------------------------------  --------  --------  --------  --------  ------  --------
coldlaunch.navigationLoaded       981.067   993.000   762.000   1108.000  84.570  1076.000
coldlaunch.navigationInteractive  1074.000  1095.000  829.000   1194.000  94.260  1167.000
coldlaunch.visuallyLoaded         1297.867  1318.000  1049.000  1418.000  96.694  1399.000
coldlaunch.contentInteractive     1297.933  1318.000  1049.000  1418.000  96.729  1399.000
coldlaunch.fullyLoaded            1298.367  1318.500  1049.000  1418.000  96.825  1399.000
coldlaunch.pss                    21.139    21.482    18.776    21.578    0.823   21.573
coldlaunch.uss                    16.489    16.830    14.148    16.926    0.818   16.918
coldlaunch.rss                    32.967    33.310    30.551    33.414    0.832   33.414

---

Standard deviation sits around 96ms, so we are even improved from that!