1120828 - [Raptor] Output basic statistical information in console at each suite completion

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Gaia::PerformanceTest, defect)

Description

[Julien Wajsberg [:julienw]]

"median" results are actually more useful than "average" results.

Note that we could use MathJS (http://mathjs.org/) to provide this.

This would also be better if this result would be used for datazilla. I don't know if this would make comparing with previous results more difficult though.

Comment 1

[:Eli Perelman]

Datazilla for test-perf data will be deprecated, but the default metric for Raptor data will be 95th percentile (p95). What's great is that while this will be the default view, all the data is in place for us to change to numerous metrics right in the UI, including mean and median, so those calculations are still accessible and visualizable in the UI.

Comment 2

[Julien Wajsberg [:julienw]]

Eli, datazilla can actually provide "median" calculation already today.

This bug is to output the calculation from the command line tool.

Comment 3

[:Eli Perelman]

OK, I understand now. The test-perf tool is also being deprecated, and so there wont be any modifications to it other than critical bug fixes. Would you mind if I commandeered this bug to provide those metrics in the command line of the replacement tool?

Comment 4

[Julien Wajsberg [:julienw]]

Any command line tool works for me :)

Comment 5

[:Eli Perelman]

Thanks Julien!

So the purpose of this bug is to now output some simple statistics about the metrics that Raptor generates, but do this at the command line. Currently Raptor outputs a pseudo-JSON file which contains the base numbers of all runs, but no calculations are generated from this data because we have saved that work for the Raptor UI.

In order to make using Raptor more efficient, developer-friendly, quick to set up, and have better CLI feature parity with test-perf, we will output some metrics at the command line. At the completion of each *suite* run in Raptor, we will aggregate the results into some pretty console output containing mean, median, mode, min, max, standard deviation, and 95th percentile. This should be pretty similar to how we generated the emulator data using the test-stats repo script [1], except the console output will be different, and we wont be reading the data from a file. In fact, I'm pretty sure we have all the data already aggregated in Phase#runs [2][3], we just need to format it once the suite is complete.

[1] https://github.com/eliperelman/test-stats/blob/master/index.js
[2] https://github.com/mozilla-b2g/raptor/blob/master/lib/suite/phase.js#L25
[3] https://github.com/mozilla-b2g/raptor/blob/master/lib/suite/phase.js#L168

Comment 6

[Robert Wood [:rwood]]

When raptor is running on production, I am assuming that all of the raw data will still be stored in the database though correct (and not the statistical output)?

Comment 7

[:Eli Perelman]

Yes. I plan on outputting the stats *in addition* to our current reporting methods. So while the console will show this output, we will still write to raptor.log, and send the data to the DB if configured.

Comment 8

[:Eli Perelman]

Maybe as an added bonus, we can have some command that will read in the raptor.log and output the stats for it, just like we do for test-stats.

Comment 9

[Julien Wajsberg [:julienw]]

IMO this could even be enough. 2 commands instead of 1 is just as easy to automate.

Comment 10

[:Eli Perelman]

Experimenting with outputting stats and formatting in the console from Raptor:

$ RUNS=30 APPS="clock" node tests/raptor/launch_test

Metric                            Mean      Median    Min       Max       Std Dev  p95
--------------------------------  --------  --------  --------  --------  -------  --------
coldlaunch.navigationLoaded       858.467   850.000   752.000   987.000   61.592   980.000
coldlaunch.navigationInteractive  979.933   963.500   873.000   1107.000  62.390   1099.000
coldlaunch.visuallyLoaded         1194.933  1187.500  1096.000  1326.000  60.338   1317.000
coldlaunch.contentInteractive     1195.800  1188.000  1096.000  1326.000  60.174   1318.000
coldlaunch.fullyLoaded            1196.800  1188.500  1097.000  1327.000  60.017   1319.000
coldlaunch.uss                    10.170    10.700    8.100     10.900    0.981    10.900
coldlaunch.rss                    26.857    27.200    24.500    28.700    1.122    28.500
coldlaunch.pss                    14.873    15.400    12.700    15.700    0.988    15.600

Comment 11

[:Eli Perelman]

Oh, I also removed mode. It was pretty pointless now I think about it, since it's pretty rare that 2 numbers will be exact.

Comment 12

[:Eli Perelman]

Attachment: Link to Github pull-request: https://github.com/mozilla-b2g/raptor/pull/20

Comment 13

[Robert Wood [:rwood]]

Comment on attachment 8576095 [details] [review]
Link to Github pull-request: https://github.com/mozilla-b2g/raptor/pull/20

R+ with the latest updates you made in the PR.

Tested by running the launch test and verifying the stats console output with the values in the raptor.log, on: Flame-kk device, single app, single and multiple runs; flame-kk device multiple apps, single and multiple runs; once on emulator single app single run.

This is an awesome feature and will save me time too! Thanks.

Comment 14

[GitHub Autolander Bot]

Attachment: [gaia-node-modules] eliperelman:bug-1120828 > mozilla-b2g:master

Comment 15

[:Eli Perelman]

Comment on attachment 8576229 [details] [review]
[gaia-node-modules] eliperelman:bug-1120828 > mozilla-b2g:master

Carrying over r+ for package bump.

Comment 16

[GitHub Autolander Bot]

Attachment: [gaia] eliperelman:bug-1120828 > mozilla-b2g:master

Comment 17

[:Eli Perelman]

Comment on attachment 8576231 [details] [review]
[gaia] eliperelman:bug-1120828 > mozilla-b2g:master

Carrying over r+ for package bump.

Comment 18

[:Eli Perelman]

In master: https://github.com/mozilla-b2g/gaia/commit/3f0f535a1af0a08709d421f18e26f40f50ae8c9b