Closed Bug 1120828 Opened 5 years ago Closed 5 years ago

[Raptor] Output basic statistical information in console at each suite completion

Categories

(Firefox OS Graveyard :: Gaia::PerformanceTest, defect)

x86_64
Linux
defect
Not set

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: julienw, Assigned: Eli)

Details

(Keywords: perf)

Attachments

(3 files)

"median" results are actually more useful than "average" results.

Note that we could use MathJS (http://mathjs.org/) to provide this.

This would also be better if this result would be used for datazilla. I don't know if this would make comparing with previous results more difficult though.
Datazilla for test-perf data will be deprecated, but the default metric for Raptor data will be 95th percentile (p95). What's great is that while this will be the default view, all the data is in place for us to change to numerous metrics right in the UI, including mean and median, so those calculations are still accessible and visualizable in the UI.
Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → WONTFIX
Eli, datazilla can actually provide "median" calculation already today.

This bug is to output the calculation from the command line tool.
Status: RESOLVED → REOPENED
Resolution: WONTFIX → ---
OK, I understand now. The test-perf tool is also being deprecated, and so there wont be any modifications to it other than critical bug fixes. Would you mind if I commandeered this bug to provide those metrics in the command line of the replacement tool?
Any command line tool works for me :)
Thanks Julien!

So the purpose of this bug is to now output some simple statistics about the metrics that Raptor generates, but do this at the command line. Currently Raptor outputs a pseudo-JSON file which contains the base numbers of all runs, but no calculations are generated from this data because we have saved that work for the Raptor UI.

In order to make using Raptor more efficient, developer-friendly, quick to set up, and have better CLI feature parity with test-perf, we will output some metrics at the command line. At the completion of each *suite* run in Raptor, we will aggregate the results into some pretty console output containing mean, median, mode, min, max, standard deviation, and 95th percentile. This should be pretty similar to how we generated the emulator data using the test-stats repo script [1], except the console output will be different, and we wont be reading the data from a file. In fact, I'm pretty sure we have all the data already aggregated in Phase#runs [2][3], we just need to format it once the suite is complete.

[1] https://github.com/eliperelman/test-stats/blob/master/index.js
[2] https://github.com/mozilla-b2g/raptor/blob/master/lib/suite/phase.js#L25
[3] https://github.com/mozilla-b2g/raptor/blob/master/lib/suite/phase.js#L168
Assignee: nobody → eperelman
Status: REOPENED → ASSIGNED
Keywords: perf
Summary: Provide "median" results instead or in addition of average results. → [Raptor] Output basic statistical information in console at each suite completion
When raptor is running on production, I am assuming that all of the raw data will still be stored in the database though correct (and not the statistical output)?
Flags: needinfo?(eperelman)
Yes. I plan on outputting the stats *in addition* to our current reporting methods. So while the console will show this output, we will still write to raptor.log, and send the data to the DB if configured.
Flags: needinfo?(eperelman)
Maybe as an added bonus, we can have some command that will read in the raptor.log and output the stats for it, just like we do for test-stats.
IMO this could even be enough. 2 commands instead of 1 is just as easy to automate.
Experimenting with outputting stats and formatting in the console from Raptor:

$ RUNS=30 APPS="clock" node tests/raptor/launch_test

Metric                            Mean      Median    Min       Max       Std Dev  p95
--------------------------------  --------  --------  --------  --------  -------  --------
coldlaunch.navigationLoaded       858.467   850.000   752.000   987.000   61.592   980.000
coldlaunch.navigationInteractive  979.933   963.500   873.000   1107.000  62.390   1099.000
coldlaunch.visuallyLoaded         1194.933  1187.500  1096.000  1326.000  60.338   1317.000
coldlaunch.contentInteractive     1195.800  1188.000  1096.000  1326.000  60.174   1318.000
coldlaunch.fullyLoaded            1196.800  1188.500  1097.000  1327.000  60.017   1319.000
coldlaunch.uss                    10.170    10.700    8.100     10.900    0.981    10.900
coldlaunch.rss                    26.857    27.200    24.500    28.700    1.122    28.500
coldlaunch.pss                    14.873    15.400    12.700    15.700    0.988    15.600
Oh, I also removed mode. It was pretty pointless now I think about it, since it's pretty rare that 2 numbers will be exact.
Comment on attachment 8576095 [details] [review]
Link to Github pull-request: https://github.com/mozilla-b2g/raptor/pull/20

R+ with the latest updates you made in the PR.

Tested by running the launch test and verifying the stats console output with the values in the raptor.log, on: Flame-kk device, single app, single and multiple runs; flame-kk device multiple apps, single and multiple runs; once on emulator single app single run.

This is an awesome feature and will save me time too! Thanks.
Attachment #8576095 - Flags: review?(rwood) → review+
Comment on attachment 8576229 [details] [review]
[gaia-node-modules] eliperelman:bug-1120828 > mozilla-b2g:master

Carrying over r+ for package bump.
Attachment #8576229 - Flags: review+
Comment on attachment 8576231 [details] [review]
[gaia] eliperelman:bug-1120828 > mozilla-b2g:master

Carrying over r+ for package bump.
Attachment #8576231 - Flags: review+
In master: https://github.com/mozilla-b2g/gaia/commit/3f0f535a1af0a08709d421f18e26f40f50ae8c9b
Status: ASSIGNED → RESOLVED
Closed: 5 years ago5 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.