Closed Bug 1120828 Opened 6 years ago Closed 6 years ago
[Raptor] Output basic statistical information in console at each suite completion
45 bytes, text/x-github-pull-request
|Details | Review|
57 bytes, text/x-github-pull-request
|Details | Review|
46 bytes, text/x-github-pull-request
|Details | Review|
"median" results are actually more useful than "average" results. Note that we could use MathJS (http://mathjs.org/) to provide this. This would also be better if this result would be used for datazilla. I don't know if this would make comparing with previous results more difficult though.
Datazilla for test-perf data will be deprecated, but the default metric for Raptor data will be 95th percentile (p95). What's great is that while this will be the default view, all the data is in place for us to change to numerous metrics right in the UI, including mean and median, so those calculations are still accessible and visualizable in the UI.
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → WONTFIX
Eli, datazilla can actually provide "median" calculation already today. This bug is to output the calculation from the command line tool.
Status: RESOLVED → REOPENED
Resolution: WONTFIX → ---
OK, I understand now. The test-perf tool is also being deprecated, and so there wont be any modifications to it other than critical bug fixes. Would you mind if I commandeered this bug to provide those metrics in the command line of the replacement tool?
Any command line tool works for me :)
Thanks Julien! So the purpose of this bug is to now output some simple statistics about the metrics that Raptor generates, but do this at the command line. Currently Raptor outputs a pseudo-JSON file which contains the base numbers of all runs, but no calculations are generated from this data because we have saved that work for the Raptor UI. In order to make using Raptor more efficient, developer-friendly, quick to set up, and have better CLI feature parity with test-perf, we will output some metrics at the command line. At the completion of each *suite* run in Raptor, we will aggregate the results into some pretty console output containing mean, median, mode, min, max, standard deviation, and 95th percentile. This should be pretty similar to how we generated the emulator data using the test-stats repo script , except the console output will be different, and we wont be reading the data from a file. In fact, I'm pretty sure we have all the data already aggregated in Phase#runs , we just need to format it once the suite is complete.  https://github.com/eliperelman/test-stats/blob/master/index.js  https://github.com/mozilla-b2g/raptor/blob/master/lib/suite/phase.js#L25  https://github.com/mozilla-b2g/raptor/blob/master/lib/suite/phase.js#L168
Assignee: nobody → eperelman
Status: REOPENED → ASSIGNED
Summary: Provide "median" results instead or in addition of average results. → [Raptor] Output basic statistical information in console at each suite completion
When raptor is running on production, I am assuming that all of the raw data will still be stored in the database though correct (and not the statistical output)?
Yes. I plan on outputting the stats *in addition* to our current reporting methods. So while the console will show this output, we will still write to raptor.log, and send the data to the DB if configured.
Maybe as an added bonus, we can have some command that will read in the raptor.log and output the stats for it, just like we do for test-stats.
IMO this could even be enough. 2 commands instead of 1 is just as easy to automate.
Experimenting with outputting stats and formatting in the console from Raptor: $ RUNS=30 APPS="clock" node tests/raptor/launch_test Metric Mean Median Min Max Std Dev p95 -------------------------------- -------- -------- -------- -------- ------- -------- coldlaunch.navigationLoaded 858.467 850.000 752.000 987.000 61.592 980.000 coldlaunch.navigationInteractive 979.933 963.500 873.000 1107.000 62.390 1099.000 coldlaunch.visuallyLoaded 1194.933 1187.500 1096.000 1326.000 60.338 1317.000 coldlaunch.contentInteractive 1195.800 1188.000 1096.000 1326.000 60.174 1318.000 coldlaunch.fullyLoaded 1196.800 1188.500 1097.000 1327.000 60.017 1319.000 coldlaunch.uss 10.170 10.700 8.100 10.900 0.981 10.900 coldlaunch.rss 26.857 27.200 24.500 28.700 1.122 28.500 coldlaunch.pss 14.873 15.400 12.700 15.700 0.988 15.600
Oh, I also removed mode. It was pretty pointless now I think about it, since it's pretty rare that 2 numbers will be exact.
Comment on attachment 8576095 [details] [review] Link to Github pull-request: https://github.com/mozilla-b2g/raptor/pull/20 R+ with the latest updates you made in the PR. Tested by running the launch test and verifying the stats console output with the values in the raptor.log, on: Flame-kk device, single app, single and multiple runs; flame-kk device multiple apps, single and multiple runs; once on emulator single app single run. This is an awesome feature and will save me time too! Thanks.
Attachment #8576095 - Flags: review?(rwood) → review+
Comment on attachment 8576229 [details] [review] [gaia-node-modules] eliperelman:bug-1120828 > mozilla-b2g:master Carrying over r+ for package bump.
Attachment #8576229 - Flags: review+
Comment on attachment 8576231 [details] [review] [gaia] eliperelman:bug-1120828 > mozilla-b2g:master Carrying over r+ for package bump.
Attachment #8576231 - Flags: review+
Status: ASSIGNED → RESOLVED
Closed: 6 years ago → 6 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.