581340 - jsmicro/asmicro need "floor-less" mode for stats

Status: RESOLVED WONTFIX

(Tamarin Graveyard :: Tools, defect)

Description

[Felix S. Klock II [:pnkfelix, :fklock]]

Below snagged from discussion in asteam chat room yesterday evening:

fklockii: the short story is: jsmicro/driver.js [and asmicro/driver.as] each calculate a count of #iterations/sec
fklockii: and *then* they both take the floor of that value, and present that as the benchmark measurement
fklockii: this accentuates error.  It also sometimes increases std dev (but sometimes decreases it, when everything ends up in one bucket)
fklockii: In any case, I am now thinking that when we are hooked into runtests.py, the drivers should not take the floor.  Instead they should present results at full precision, and let runtests.py be in charge of prettifying the presentation of the final calculated values.
Chris Peyer: agreed
fklockii: the only draw back I can think of is that one would need for runtests.py to communicate with the driver code to tell it to print at full precision.  Presumably it would be a trivial command line parameter to add; I don't see it as much of a draw back, but could processing that sort of input introduce new noise into the measurements?  I might be over-thinking things at this point...
Chris Peyer: i haven't looked at the driver code, but why can't we always print at full precision?
fklockii: I got the impression from lars that he wants the presentation to be quick and dirty, easy for human eyes to digest
fklockii: (i can't say I blame him; it feels easier to read.  Though reading it also feels wrong since one knows that what one is reading is highly suspect.)
fklockii: don't read too much into what I'm attributing to Lars
fklockii: (its based on an off-the-cuff discussion we had in #asteam back on the morning of June 10th [1])


[1] http://asteam.macromedia.com/irc/log/asteam/asteam/20100611

Comment 1

[Lars T Hansen]

The main problem with "full precision" is that it's variable-length and has a lot of irrelevant data.  I could live with a couple of decimal places, as long as it was always the same number of decimals.  In case that would take some pressure off.

Comment 2

[Felix S. Klock II [:pnkfelix, :fklock]]

Sure, I wouldn't want to read more than three or four significant figures either.

Note that my discussion with Chris suggested that full-precision mode be something that we toggle on/off; presumably the default would be off, but the runtests.py driver would turn it on, and then floor the results for the final presentation.  (But then again, some experiments I did with this last week led me to think that I was getting worked up over very little and that this is not introducing an absurd amount of error into our results.  So this is not a high priority item.)

Comment 3

[Sylvestre Ledru [:Sylvestre]]

Tamarin is a dead project now. Mass WONTFIX.

Comment 4

[Sylvestre Ledru [:Sylvestre]]

Tamarin isn't maintained anymore. WONTFIX remaining bugs.