From Edwin Smith: Some suggestions to make #s more relevant: 1. calculate %stdev instead of stdev, that is, %stdev = stdev/avg. this will normalize all the stdev results, currently they are in the measurement units and its easier to understand if they are unitless %. 2. calculates "significance" as %diff / (%stdev1 + %stdev2). Test with large standard devitions (lots of noise) thus require a larger %diff before the change can be considered significant. Not to touch on religious issues here... some small changes are still desireable (or not desireable) even if they are "insignificant" calculated this way. but more importantly this will highlight the truly significant changes. (with signifiance outside the noise band, say > 1 or < -1).
Created attachment 407915 [details] [diff] [review] Improve performance reporting Add %diff and %stddev to report as well as significance indicators.
Assignee: nobody → cpeyer
Status: NEW → ASSIGNED
Attachment #407915 - Flags: review?(brbaker)
Created attachment 408155 [details] [diff] [review] Updated patch - also includes improved cygwin support Patch also adds the ability to specify files in cygwin path-format even when using windows python.
Comment on attachment 408155 [details] [diff] [review] Updated patch - also includes improved cygwin support - the /bold/ does not work on windows and is actually messes up the formatting a little because there are additional characters being inserted. I am guessing you could see the same issue if you piped the output to a file on mac - could you comment a little around the "--", "-", "+" "++" code, exactly what is this trying to tell the user?
Attachment #408155 - Flags: review?(brbaker) → review+
Comment on attachment 408155 [details] [diff] [review] Updated patch - also includes improved cygwin support Have you checked to see what the performance difference is, if any, with compiling with -AS3? Should new baseline data be generated so that we can compare the performance of milestone builds properly?
Created attachment 408954 [details] [diff] [review] Add -p --prettyprint option -p option will bold tests with significant performance change +/- also indicates significant performance change: - : -2 < sig < -1 -- : sig < -2 + : 1 < sig < 2 ++ : sig > 2
Attachment #408155 - Attachment is obsolete: true
pushed redux 2923:5f7b60c15f76
Status: ASSIGNED → RESOLVED
Last Resolved: 9 years ago
Resolution: --- → FIXED
Created attachment 409086 [details] [diff] [review] Update to not fix cygwin paths on winmobile. Was previously trying to fix winmobile paths which was causing failures.
Comment on attachment 409086 [details] [diff] [review] Update to not fix cygwin paths on winmobile. looks ok. on mac I did no see the results appear in bold on my shell but the ++,+,--,- is useful to point out signficant changes in pretty-print mode.
pushed (fixed patch) changeset 2940 e962fef8d694
Status: RESOLVED → VERIFIED
You need to log in before you can comment on or make changes to this bug.