Below snagged from discussion in asteam chat room yesterday evening: fklockii: the short story is: jsmicro/driver.js [and asmicro/driver.as] each calculate a count of #iterations/sec fklockii: and *then* they both take the floor of that value, and present that as the benchmark measurement fklockii: this accentuates error. It also sometimes increases std dev (but sometimes decreases it, when everything ends up in one bucket) fklockii: In any case, I am now thinking that when we are hooked into runtests.py, the drivers should not take the floor. Instead they should present results at full precision, and let runtests.py be in charge of prettifying the presentation of the final calculated values. Chris Peyer: agreed fklockii: the only draw back I can think of is that one would need for runtests.py to communicate with the driver code to tell it to print at full precision. Presumably it would be a trivial command line parameter to add; I don't see it as much of a draw back, but could processing that sort of input introduce new noise into the measurements? I might be over-thinking things at this point... Chris Peyer: i haven't looked at the driver code, but why can't we always print at full precision? fklockii: I got the impression from lars that he wants the presentation to be quick and dirty, easy for human eyes to digest fklockii: (i can't say I blame him; it feels easier to read. Though reading it also feels wrong since one knows that what one is reading is highly suspect.) fklockii: don't read too much into what I'm attributing to Lars fklockii: (its based on an off-the-cuff discussion we had in #asteam back on the morning of June 10th )  http://asteam.macromedia.com/irc/log/asteam/asteam/20100611
The main problem with "full precision" is that it's variable-length and has a lot of irrelevant data. I could live with a couple of decimal places, as long as it was always the same number of decimals. In case that would take some pressure off.
Sure, I wouldn't want to read more than three or four significant figures either. Note that my discussion with Chris suggested that full-precision mode be something that we toggle on/off; presumably the default would be off, but the runtests.py driver would turn it on, and then floor the results for the final presentation. (But then again, some experiments I did with this last week led me to think that I was getting worked up over very little and that this is not introducing an absurd amount of error into our results. So this is not a high priority item.)
Priority: -- → P4
Target Milestone: --- → Future
You need to log in before you can comment on or make changes to this bug.