Closed
Bug 1120828
Opened 6 years ago
Closed 6 years ago
[Raptor] Output basic statistical information in console at each suite completion
Categories
(Firefox OS Graveyard :: Gaia::PerformanceTest, defect)
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: julienw, Assigned: Eli)
Details
(Keywords: perf)
Attachments
(3 files)
"median" results are actually more useful than "average" results. Note that we could use MathJS (http://mathjs.org/) to provide this. This would also be better if this result would be used for datazilla. I don't know if this would make comparing with previous results more difficult though.
Assignee | ||
Comment 1•6 years ago
|
||
Datazilla for test-perf data will be deprecated, but the default metric for Raptor data will be 95th percentile (p95). What's great is that while this will be the default view, all the data is in place for us to change to numerous metrics right in the UI, including mean and median, so those calculations are still accessible and visualizable in the UI.
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → WONTFIX
Reporter | ||
Comment 2•6 years ago
|
||
Eli, datazilla can actually provide "median" calculation already today. This bug is to output the calculation from the command line tool.
Status: RESOLVED → REOPENED
Resolution: WONTFIX → ---
Assignee | ||
Comment 3•6 years ago
|
||
OK, I understand now. The test-perf tool is also being deprecated, and so there wont be any modifications to it other than critical bug fixes. Would you mind if I commandeered this bug to provide those metrics in the command line of the replacement tool?
Reporter | ||
Comment 4•6 years ago
|
||
Any command line tool works for me :)
Assignee | ||
Comment 5•6 years ago
|
||
Thanks Julien! So the purpose of this bug is to now output some simple statistics about the metrics that Raptor generates, but do this at the command line. Currently Raptor outputs a pseudo-JSON file which contains the base numbers of all runs, but no calculations are generated from this data because we have saved that work for the Raptor UI. In order to make using Raptor more efficient, developer-friendly, quick to set up, and have better CLI feature parity with test-perf, we will output some metrics at the command line. At the completion of each *suite* run in Raptor, we will aggregate the results into some pretty console output containing mean, median, mode, min, max, standard deviation, and 95th percentile. This should be pretty similar to how we generated the emulator data using the test-stats repo script [1], except the console output will be different, and we wont be reading the data from a file. In fact, I'm pretty sure we have all the data already aggregated in Phase#runs [2][3], we just need to format it once the suite is complete. [1] https://github.com/eliperelman/test-stats/blob/master/index.js [2] https://github.com/mozilla-b2g/raptor/blob/master/lib/suite/phase.js#L25 [3] https://github.com/mozilla-b2g/raptor/blob/master/lib/suite/phase.js#L168
Assignee | ||
Updated•6 years ago
|
Summary: Provide "median" results instead or in addition of average results. → [Raptor] Output basic statistical information in console at each suite completion
Comment 6•6 years ago
|
||
When raptor is running on production, I am assuming that all of the raw data will still be stored in the database though correct (and not the statistical output)?
Flags: needinfo?(eperelman)
Assignee | ||
Comment 7•6 years ago
|
||
Yes. I plan on outputting the stats *in addition* to our current reporting methods. So while the console will show this output, we will still write to raptor.log, and send the data to the DB if configured.
Flags: needinfo?(eperelman)
Assignee | ||
Comment 8•6 years ago
|
||
Maybe as an added bonus, we can have some command that will read in the raptor.log and output the stats for it, just like we do for test-stats.
Reporter | ||
Comment 9•6 years ago
|
||
IMO this could even be enough. 2 commands instead of 1 is just as easy to automate.
Assignee | ||
Comment 10•6 years ago
|
||
Experimenting with outputting stats and formatting in the console from Raptor: $ RUNS=30 APPS="clock" node tests/raptor/launch_test Metric Mean Median Min Max Std Dev p95 -------------------------------- -------- -------- -------- -------- ------- -------- coldlaunch.navigationLoaded 858.467 850.000 752.000 987.000 61.592 980.000 coldlaunch.navigationInteractive 979.933 963.500 873.000 1107.000 62.390 1099.000 coldlaunch.visuallyLoaded 1194.933 1187.500 1096.000 1326.000 60.338 1317.000 coldlaunch.contentInteractive 1195.800 1188.000 1096.000 1326.000 60.174 1318.000 coldlaunch.fullyLoaded 1196.800 1188.500 1097.000 1327.000 60.017 1319.000 coldlaunch.uss 10.170 10.700 8.100 10.900 0.981 10.900 coldlaunch.rss 26.857 27.200 24.500 28.700 1.122 28.500 coldlaunch.pss 14.873 15.400 12.700 15.700 0.988 15.600
Assignee | ||
Comment 11•6 years ago
|
||
Oh, I also removed mode. It was pretty pointless now I think about it, since it's pretty rare that 2 numbers will be exact.
Assignee | ||
Comment 12•6 years ago
|
||
Attachment #8576095 -
Flags: review?(rwood)
Comment 13•6 years ago
|
||
Comment on attachment 8576095 [details] [review] Link to Github pull-request: https://github.com/mozilla-b2g/raptor/pull/20 R+ with the latest updates you made in the PR. Tested by running the launch test and verifying the stats console output with the values in the raptor.log, on: Flame-kk device, single app, single and multiple runs; flame-kk device multiple apps, single and multiple runs; once on emulator single app single run. This is an awesome feature and will save me time too! Thanks.
Attachment #8576095 -
Flags: review?(rwood) → review+
Comment 14•6 years ago
|
||
Assignee | ||
Comment 15•6 years ago
|
||
Comment on attachment 8576229 [details] [review] [gaia-node-modules] eliperelman:bug-1120828 > mozilla-b2g:master Carrying over r+ for package bump.
Attachment #8576229 -
Flags: review+
Comment 16•6 years ago
|
||
Assignee | ||
Comment 17•6 years ago
|
||
Comment on attachment 8576231 [details] [review] [gaia] eliperelman:bug-1120828 > mozilla-b2g:master Carrying over r+ for package bump.
Attachment #8576231 -
Flags: review+
Assignee | ||
Comment 18•6 years ago
|
||
In master: https://github.com/mozilla-b2g/gaia/commit/3f0f535a1af0a08709d421f18e26f40f50ae8c9b
Status: ASSIGNED → RESOLVED
Closed: 6 years ago → 6 years ago
Resolution: --- → FIXED
You need to log in
before you can comment on or make changes to this bug.
Description
•