849558 - Allow easier debug of talos tests outside of the talos environment

Status: RESOLVED FIXED

(Datazilla Graveyard :: Metrics, defect)

Description

[Avi Halachmi (:avih)]

Attachment: talos-debug.js

As suggested at bug 845943 comment #7, attached is talos-debug.js. Just include it from any talos test, run the test in a browser, and tpRecordTime will display the summary.

Here's an excerpt from the file itself:

/******************************************************************
 * window.talosDebug provides some statistical functions
 * (sum, average, median, stddev) and a tpRecordTime method which
 * reports some statistics about the data set, including detected
 * stability point of the data (detecting first few noisy elements).
 * If tpRecordTime doesn't exist globally (i.e. running outside of
 * talos, e.g. in a browser), then it points it to window.talosDebug.tpRecordTime
 * Can be controlled by few properties (disable display, hardcoded
 * stability point, etc)
 *****************************************************************/

The warmup-preiod detection was fine tuned with the noise patterns of the tscroll tests (in various browsers), and while completely unscientific, provides surprisingly useful and consistent results (e.g. usually shows a value of 1-10 for the tscroll tests, which I manually confirmed as valid).

We might need to improve it if this proves useful for tests other than tscroll, but for now, it can serve as a starting point.

Comment 1

[Avi Halachmi (:avih)]

Attachment: v2 - minor fix: also handle single, non-string reported value

Also, here's a sample output of the SVG scroll test on my system. Notice how after (automatically) skipping 2 items (of 44), stddev goes from 10.8% down to 2.7%:

-------------------
Count: 44
Average: 16.38
Median: 16.88
StdDev: 1.77 (10.83% of average)

Warmup auto-detected: 2
After ignoring first 2 items:
Count: 42
Average: 16.70
Median: 16.89
StdDev: 0.47 (2.79% of average)

Recorded:
5.68, 13.59, [warmed-up:], 17.26, 16.93, 15.95, 17.05, 16.98, 16.08, 16.94, 16.91, 16.39, 16.72, 17.03, 16.03, 16.96, 17.72, 16.03, 17.04, 17.07, 15.92, 16.99, 17.09, 15.94, 16.99, 16.99, 16.01, 16.99, 17.08, 15.96, 16.84, 17.21, 16.11, 16.86, 16.92, 16.21, 16.86, 16.88, 16.19, 16.88, 16.97, 16.21, 16.74, 17.11, 16.21

Stddev from item NN to last:
1.77, 0.66, [warmed-up:], 0.47, 0.46, 0.47, 0.46, 0.46, 0.46, 0.46, 0.46, 0.47, 0.47, 0.48, 0.49, 0.48, 0.49, 0.45, 0.44, 0.44, 0.45, 0.43, 0.43, 0.43, 0.41, 0.42, 0.42, 0.40, 0.41, 0.41, 0.38, 0.39, 0.37, 0.35, 0.37, 0.37, 0.37, 0.38, 0.40, 0.39, 0.42, 0.44, 0.45, 0.64, 0.00

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 723203 [details] [diff] [review]
v2 - minor fix: also handle single, non-string reported value

Review of attachment 723203 [details] [diff] [review]:
-----------------------------------------------------------------

I would add a license header to this.

Comment 3

[Avi Halachmi (:avih)]

Landed talos-debug.js:
https://hg.mozilla.org/build/talos/rev/053f69f5d92d

Added an MPL clause:
https://hg.mozilla.org/build/talos/rev/f443093a8ce3