589007 - performance runtests.py: default output should be max 78 chars

Status: VERIFIED FIXED

(Tamarin Graveyard :: Tools, defect)

Description

[Chris Peyer]

Bugzilla wraps comments at 78 chars, so it would be nice to have output that is cut/paste friendly.

Also - to shorten the output, test dirs should be printed on a separate line from the tests themselves

Comment 1

[Chris Peyer]

Felix also noticed that 0.0 results are printed as -0.0

Comment 2

[Felix S. Klock II [:pnkfelix, :fklock]]

(In reply to comment #0) 
> Also - to shorten the output, test dirs should be printed on a separate line
> from the tests themselves

Note that Ed and I both thought it would be good if some portion of the path-prefix were still printed on the line with the filename (essentially the same way it is now, except you'll now be free to make it much shorter since the full context will be indicated from the separate line that has the full path).

Also, if you're already mucking with this, I liked you idea of dropping the ".as" suffix from the test names.  (Where can this lead us astray -- on abs files being used as tests?  Still think it'd be a net win: just set things up to only drop the ".as" suffix when its actually present, rather than unconditionally dropping the last three characters of the path.)

Comment 3

[Felix S. Klock II [:pnkfelix, :fklock]]

As long as you're fixing things, you might change the header(s) for the "stdev" column to say "covar" or "cv" instead -- because, I believe, you are not presenting the standard deviation, you are presenting a quantity that statistics professionals call "coefficient of variation" (and that the runtests.py comments assigns the term "relative standard deviation").  Unless I misunderstand.

The above may just be a nit; someone trying to use this output "for real" is probably going to have to look at the code the same way I did to really interpret these columns anyway.  (Although there is an argument that having good headers *does* matter because some of this output is being logged or put on bugzilla tickets, and that point one loses the connection to the original source code for runtests.py...

Comment 4

[Chris Peyer]

I believe that relative standard deviation is the correct name for the measurement.  It is defined here:

http://en.wikipedia.org/wiki/Relative_standard_deviation

as "relative standard deviation is the absolute value of the coefficient of variation."

Comment 5

[Chris Peyer]

Attachment: Shorten output to fit to bugzilla comment width

Output is now 73 col wide when comparing two vms with multiple iterations.  Requires python 2.6+ for all advantages of new formatting.

In order to still display full path for each test, use the --fullpath option.

Comment 6

[Felix S. Klock II [:pnkfelix, :fklock]]

(In reply to comment #4)
> I believe that relative standard deviation is the correct name for the
> measurement.  It is defined here:
> 
> http://en.wikipedia.org/wiki/Relative_standard_deviation
> 
> as "relative standard deviation is the absolute value of the coefficient of
> variation."

I stand corrected!

Comment 7

[Felix S. Klock II [:pnkfelix, :fklock]]

(In reply to comment #5)
> Created attachment 469213 [details] [diff] [review]
> Shorten output to fit to bugzilla comment width
> 
> Output is now 73 col wide when comparing two vms with multiple iterations. 
> Requires python 2.6+ for all advantages of new formatting.
> 
> In order to still display full path for each test, use the --fullpath option.

Hm, ... now that we've freed up this room that was taken up by the longish names, is it out of scope of this ticket to fix the column alignment for --detail mode?

I'm looking in particular at output like this:

Dir: jsbench/
  Crypt                     [   6608 :   6821]  6721.3 ± 1.3%   [   6525 :   6530]  6527.7 ± 0.0%    1.3%    2.9     
  Euler                     [  12399 :  12598]   12480 ± 0.7%   [  11974 :  12136]   12028 ± 0.6%    3.4%    3.6 +   
  FFT                       [  12511 :  12535]  12522.7 ± 0.1%   [  12247 :  12306]  12268.7 ± 0.2%    2.1%    2.0 +   
  HeapSort                  [   5112 :   5144]  5126.7 ± 0.3%   [   5019 :   5048]  5029.7 ± 0.3%    1.8%    1.9 +   
  LUFact                    [  12675 :  12736]   12702 ± 0.2%   [  12425 :  12586]   12489 ± 0.6%    2.0%    1.7 +   
  Moldyn                    [  17737 :  17938]  17807.7 ± 0.5%   [  17428 :  17494]   17464 ± 0.2%    1.7%    1.9 +   
  RayTracer                 [  12665 :  12729]  12690.3 ± 0.2%   [  12437 :  12521]  12474.7 ± 0.3%    1.8%    1.7 +   

If its too much trouble, then don't bother; I'm switching to using CSV output now so that I can get at the raw numbers, so its not so important to have columns that line up in --detail mode.

Comment 8

[Felix S. Klock II [:pnkfelix, :fklock]]

Comment on attachment 469213 [details] [diff] [review]
Shorten output to fit to bugzilla comment width

r+.  It is more important to me that we land this patch soon rather than worry about the --detail thing that I mentioned in my previous coment.

Comment 9

[Edwin Smith]

Comment on attachment 469213 [details] [diff] [review]
Shorten output to fit to bugzilla comment width

I couldn't get the patch to apply cleanly but it looks okay to the untrained eye.

Comment 10

[Chris Peyer]

Probably cause I already pushed the changes:d

http://hg.mozilla.org/tamarin-redux/rev/26817661d9ff