Misc improvements to performance runtests to improve reporting.

VERIFIED FIXED

Status

--
enhancement
VERIFIED FIXED
9 years ago
9 years ago

People

(Reporter: cpeyer, Assigned: cpeyer)

Tracking

unspecified
Bug Flags:
in-testsuite -
flashplayer-triage +

Details

Attachments

(1 attachment, 3 obsolete attachments)

(Assignee)

Description

9 years ago
From Edwin Smith:

Some suggestions to make #s more relevant:

1. calculate %stdev instead of stdev, that is, %stdev = stdev/avg.  this will
normalize all the stdev results, currently they are in the measurement units
and its easier to understand if they are unitless %.

2. calculates "significance" as %diff / (%stdev1 + %stdev2).  Test with large
standard devitions (lots of noise) thus require a larger %diff before the
change can be considered significant.

Not to touch on religious issues here... some small changes are still
desireable (or not desireable) even if they are "insignificant" calculated this
way.  but more importantly this will highlight the truly significant changes. 
(with signifiance outside the noise band, say > 1 or < -1).
Flags: in-testsuite-
Flags: flashplayer-triage+
Flags: flashplayer-qrb?
(Assignee)

Comment 1

9 years ago
Created attachment 407915 [details] [diff] [review]
Improve performance reporting

Add %diff and %stddev to report as well as significance indicators.
Assignee: nobody → cpeyer
Status: NEW → ASSIGNED
Attachment #407915 - Flags: review?(brbaker)
(Assignee)

Comment 2

9 years ago
Created attachment 408155 [details] [diff] [review]
Updated patch - also includes improved cygwin support

Patch also adds the ability to specify files in cygwin path-format even when using windows python.
Attachment #407915 - Attachment is obsolete: true
Attachment #408155 - Flags: review?(brbaker)
Attachment #407915 - Flags: review?(brbaker)

Comment 3

9 years ago
Comment on attachment 408155 [details] [diff] [review]
Updated patch - also includes improved cygwin support

- the /bold/ does not work on windows and is actually messes up the formatting a little because there are additional characters being inserted. I am guessing you could see the same issue if you piped the output to a file on mac

- could you comment a little around the "--", "-", "+" "++" code, exactly what is this trying to tell the user?
Attachment #408155 - Flags: review?(brbaker) → review+

Comment 4

9 years ago
Comment on attachment 408155 [details] [diff] [review]
Updated patch - also includes improved cygwin support

Have you checked to see what the performance difference is, if any, with compiling with -AS3? Should new baseline data be generated so that we can compare the performance of milestone builds properly?
(Assignee)

Comment 5

9 years ago
Created attachment 408954 [details] [diff] [review]
Add -p --prettyprint option

-p option will bold tests with significant performance change

+/- also indicates significant performance change:
- : -2 < sig < -1
-- : sig < -2
+ : 1 < sig < 2
++ : sig > 2
Attachment #408155 - Attachment is obsolete: true
(Assignee)

Comment 6

9 years ago
pushed redux 2923:5f7b60c15f76
Status: ASSIGNED → RESOLVED
Last Resolved: 9 years ago
Flags: flashplayer-qrb?
Resolution: --- → FIXED
(Assignee)

Comment 7

9 years ago
Created attachment 409086 [details] [diff] [review]
Update to not fix cygwin paths on winmobile.

Was previously trying to fix winmobile paths which was causing failures.
Attachment #408954 - Attachment is obsolete: true
Attachment #409086 - Flags: review?(dschaffe)

Updated

9 years ago
Attachment #409086 - Flags: review?(dschaffe) → review+

Comment 8

9 years ago
Comment on attachment 409086 [details] [diff] [review]
Update to not fix cygwin paths on winmobile.

looks ok.  on mac I did no see the results appear in bold on my shell but the ++,+,--,- is useful to point out signficant changes in pretty-print mode.
(Assignee)

Comment 9

9 years ago
pushed (fixed patch) changeset 2940	e962fef8d694
Status: RESOLVED → VERIFIED
You need to log in before you can comment on or make changes to this bug.