Closed Bug 755633 Opened 12 years ago Closed 12 years ago

Talos V8 tests are rounded to the nearest millisecond, introducing rounding errors and aliasing

Categories

(Testing :: Talos, defect)

defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: mbrubeck, Unassigned)

References

Details

(Whiteboard: [SfN])

Talos V8 is extremely bimodal.  Because of the way its results are averaged, it is currently always returning exactly 9.133 or 9.167.  (In the past it has also returned other specific values like 9.500 or 9.000 -- always an even multiple of 1/6.)

A typical Talos V8 run looks like this:

NOISE: |0;richards.html;3;3;3;3;3;3;3;3;3;3;3;3;3;3;3;3;3;3;3;3
NOISE: |1;deltablue.html;1;1;1;1;1;1;1;1;1;1;1;1;1;1;1;1;1;1;1;1
NOISE: |2;crypto.html;3;3;3;3;3;3;3;3;3;3;3;3;3;3;3;3;3;3;3;3
NOISE: |3;raytrace.html;54;35;34;35;34;35;35;35;33;34;34;37;35;34;35;35;35;33;35;35
NOISE: |4;earley-boyer.html;14;13;13;13;13;13;13;13;13;13;13;13;13;13;13;13;13;13;13;13
NOISE: |5;regexp.html;118;116;115;114;115;115;115;116;115;116;116;116;115;116;116;116;116;116;116;116
NOISE: |6;splay.html;1;1;1;1;1;1;1;1;1;1;1;1;1;1;1;1;1;1;1;1

The numbers are average times in milliseconds for each test.  Since they are rounded to whole milliseconds, they are not capable of detecting even fairly large changes in the tests that take ~1 ms or ~3ms to run.  And the resulting aliasing of the final result leads to the false alarms mentioned above.

I think the problem is the Math.round call here:
http://hg.mozilla.org/build/talos/file/d00eba6055f5/talos/page_load_test/v8/run-splay.html#l53

If Talos cannot handle fractional numbers in the output, perhaps we should multiply the result by 1000 before rounding it.  Or we could calculate the score in the same way the standard V8 test suite does, by starting with a reference score and dividing by the average time in ms to get a "bigger is better" number.  That would make the number much easier for the JS team to understand.  Currently we have this calculation commented out:
http://hg.mozilla.org/build/talos/file/d00eba6055f5/talos/page_load_test/v8/base.js#l179
There are a few places where Talos is hard-coded to use ints :( This should probably be fixed
Whiteboard: [SfN]
Fixed by bug 767226.
Status: NEW → RESOLVED
Closed: 12 years ago
Depends on: 767226
Resolution: --- → FIXED
Blocks: 859495
You need to log in before you can comment on or make changes to this bug.