Closed Bug 523863 Opened 10 years ago Closed 10 years ago

Misc improvements to performance runtests to improve reporting.

Categories

(Tamarin Graveyard :: Tools, enhancement)

enhancement
Not set

Tracking

(Not tracked)

VERIFIED FIXED

People

(Reporter: cpeyer, Assigned: cpeyer)

Details

Attachments

(1 file, 3 obsolete files)

From Edwin Smith:

Some suggestions to make #s more relevant:

1. calculate %stdev instead of stdev, that is, %stdev = stdev/avg.  this will
normalize all the stdev results, currently they are in the measurement units
and its easier to understand if they are unitless %.

2. calculates "significance" as %diff / (%stdev1 + %stdev2).  Test with large
standard devitions (lots of noise) thus require a larger %diff before the
change can be considered significant.

Not to touch on religious issues here... some small changes are still
desireable (or not desireable) even if they are "insignificant" calculated this
way.  but more importantly this will highlight the truly significant changes. 
(with signifiance outside the noise band, say > 1 or < -1).
Flags: in-testsuite-
Flags: flashplayer-triage+
Flags: flashplayer-qrb?
Attached patch Improve performance reporting (obsolete) — Splinter Review
Add %diff and %stddev to report as well as significance indicators.
Assignee: nobody → cpeyer
Status: NEW → ASSIGNED
Attachment #407915 - Flags: review?(brbaker)
Patch also adds the ability to specify files in cygwin path-format even when using windows python.
Attachment #407915 - Attachment is obsolete: true
Attachment #408155 - Flags: review?(brbaker)
Attachment #407915 - Flags: review?(brbaker)
Comment on attachment 408155 [details] [diff] [review]
Updated patch - also includes improved cygwin support

- the /bold/ does not work on windows and is actually messes up the formatting a little because there are additional characters being inserted. I am guessing you could see the same issue if you piped the output to a file on mac

- could you comment a little around the "--", "-", "+" "++" code, exactly what is this trying to tell the user?
Attachment #408155 - Flags: review?(brbaker) → review+
Comment on attachment 408155 [details] [diff] [review]
Updated patch - also includes improved cygwin support

Have you checked to see what the performance difference is, if any, with compiling with -AS3? Should new baseline data be generated so that we can compare the performance of milestone builds properly?
Attached patch Add -p --prettyprint option (obsolete) — Splinter Review
-p option will bold tests with significant performance change

+/- also indicates significant performance change:
- : -2 < sig < -1
-- : sig < -2
+ : 1 < sig < 2
++ : sig > 2
Attachment #408155 - Attachment is obsolete: true
pushed redux 2923:5f7b60c15f76
Status: ASSIGNED → RESOLVED
Closed: 10 years ago
Flags: flashplayer-qrb?
Resolution: --- → FIXED
Was previously trying to fix winmobile paths which was causing failures.
Attachment #408954 - Attachment is obsolete: true
Attachment #409086 - Flags: review?(dschaffe)
Attachment #409086 - Flags: review?(dschaffe) → review+
Comment on attachment 409086 [details] [diff] [review]
Update to not fix cygwin paths on winmobile.

looks ok.  on mac I did no see the results appear in bold on my shell but the ++,+,--,- is useful to point out signficant changes in pretty-print mode.
pushed (fixed patch) changeset 2940	e962fef8d694
Status: RESOLVED → VERIFIED
You need to log in before you can comment on or make changes to this bug.