Closed Bug 1189469 Opened 9 years ago Closed 9 years ago

[Raptor] Cache application coordinates to improve application launch variance

Categories

(Firefox OS Graveyard :: Gaia::PerformanceTest, defect)

Product:
Firefox OS Graveyard
Component:
Gaia::PerformanceTest
Platform:
ARM
Gonk (Firefox OS)
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED
Milestone:
FxOS-S4 (07Aug)

People

(Reporter: Eli, Assigned: Eli)

Details

(Whiteboard: [systemsfe])

Attachments

(1 file)

Link to Github pull-request: https://github.com/mozilla-b2g/raptor/pull/15
45 bytes, text/x-github-pull-request
rwood
: review+
Details | Review
When Raptor uses Marionette, we search for and launch the application at every run. During this process, we start a Marionette session, launch the app, and while launch kill the Marionette session. I believe this activity during app launch may be contributing to some variance in times. Cache the coordinates using Marionette in advance of launch and use those coordinates each time.
(In reply to :Eli Perelman from comment #1) > Created attachment 8641239 [details] [review] > Link to Github pull-request: https://github.com/mozilla-b2g/raptor/pull/15 I tested launching each of the 12 apps on aries, flame, and emulator and it worked. However retries doesn't work as I expected. If I manually scroll the lcd before an app launch, so that the wrong app is launched, I had expected a retry to scroll the homescreen back into place so the next tap could work, however there's no retry and there's a marionette error instead: https://gist.github.com/rwood-moz/bb68156d8400b44f75d2
I'm changing up the way the Marionette connection works, so hopefully I can get this resolved soon.
OK, PR updated. Please let me know if the retry works better now.
Comment on attachment 8641239 [details] [review] Link to Github pull-request: https://github.com/mozilla-b2g/raptor/pull/15 LGTM. Retries are working again with latest update. The unrelated marionette polling socket timeout issue (Bug 1188990) prevented me from trying the latest update on the emulator, but I'm satisfied with the patch as it works great on device.
Attachment #8641239 - Flags: review?(rwood) → review+
In master: https://github.com/mozilla-b2g/raptor/commit/0d2151892b2f8254636b7e02d5e75c3bb8b0e8a0 In npm: @mozilla/raptor v1.4.5
Status: ASSIGNED → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
Whiteboard: [systemsfe]
As an update, here is a comparison against the Clock app, 30 runs: [No coordinate cache] | Metric | Mean | Median | Min | Max | StdDev | p95 | | -------------------------------- | -------- | -------- | ------- | -------- | ------- | -------- | | coldlaunch.navigationLoaded | 967.200 | 992.500 | 702.000 | 1106.000 | 94.081 | 1078.000 | | coldlaunch.navigationInteractive | 1057.933 | 1091.000 | 802.000 | 1210.000 | 99.256 | 1165.000 | | coldlaunch.visuallyLoaded | 1356.733 | 1404.500 | 975.000 | 1527.000 | 137.964 | 1466.000 | | coldlaunch.contentInteractive | 1357.000 | 1404.500 | 975.000 | 1529.000 | 138.021 | 1467.000 | | coldlaunch.fullyLoaded | 1357.400 | 1405.500 | 975.000 | 1530.000 | 138.091 | 1467.000 | | coldlaunch.pss | 19.134 | 19.127 | 18.835 | 19.423 | 0.109 | 19.339 | | coldlaunch.uss | 14.501 | 14.475 | 14.406 | 14.773 | 0.079 | 14.676 | | coldlaunch.rss | 31.042 | 31.023 | 30.953 | 31.277 | 0.071 | 31.195 | --- [Coordinate Cache] | Metric | Mean | Median | Min | Max | StdDev | p95 | | -------------------------------- | -------- | -------- | -------- | -------- | ------ | -------- | | coldlaunch.navigationLoaded | 1076.467 | 1080.000 | 1010.000 | 1175.000 | 47.993 | 1166.000 | | coldlaunch.navigationInteractive | 1170.800 | 1174.500 | 1107.000 | 1278.000 | 45.740 | 1261.000 | | coldlaunch.visuallyLoaded | 1412.367 | 1400.500 | 1299.000 | 1565.000 | 59.067 | 1540.000 | | coldlaunch.contentInteractive | 1412.467 | 1401.000 | 1299.000 | 1565.000 | 59.094 | 1540.000 | | coldlaunch.fullyLoaded | 1412.733 | 1401.000 | 1300.000 | 1565.000 | 59.051 | 1540.000 | | coldlaunch.rss | 31.078 | 31.117 | 30.785 | 31.234 | 0.121 | 31.184 | | coldlaunch.pss | 19.217 | 19.271 | 18.936 | 19.378 | 0.134 | 19.336 | | coldlaunch.uss | 14.549 | 14.588 | 14.254 | 14.695 | 0.119 | 14.652 | --- Big thing to note is the improvement in standard deviation from around 138ms down to 59ms, which is pretty significant. The means and p95s look smaller because greater ranges were produced, not because there were any actual improvements.
Also, here is a set of runs from when Raptor was in the Gaia tree, Clock, 30 runs: Metric Mean Median Min Max StdDev p95 -------------------------------- -------- -------- -------- -------- ------ -------- coldlaunch.navigationLoaded 981.067 993.000 762.000 1108.000 84.570 1076.000 coldlaunch.navigationInteractive 1074.000 1095.000 829.000 1194.000 94.260 1167.000 coldlaunch.visuallyLoaded 1297.867 1318.000 1049.000 1418.000 96.694 1399.000 coldlaunch.contentInteractive 1297.933 1318.000 1049.000 1418.000 96.729 1399.000 coldlaunch.fullyLoaded 1298.367 1318.500 1049.000 1418.000 96.825 1399.000 coldlaunch.pss 21.139 21.482 18.776 21.578 0.823 21.573 coldlaunch.uss 16.489 16.830 14.148 16.926 0.818 16.918 coldlaunch.rss 32.967 33.310 30.551 33.414 0.832 33.414 --- Standard deviation sits around 96ms, so we are even improved from that!
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: