Closed
Bug 755633
Opened 13 years ago
Closed 12 years ago
Talos V8 tests are rounded to the nearest millisecond, introducing rounding errors and aliasing
Categories
(Testing :: Talos, defect)
Testing
Talos
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: mbrubeck, Unassigned)
References
Details
(Whiteboard: [SfN])
Talos V8 is extremely bimodal. Because of the way its results are averaged, it is currently always returning exactly 9.133 or 9.167. (In the past it has also returned other specific values like 9.500 or 9.000 -- always an even multiple of 1/6.)
A typical Talos V8 run looks like this:
NOISE: |0;richards.html;3;3;3;3;3;3;3;3;3;3;3;3;3;3;3;3;3;3;3;3
NOISE: |1;deltablue.html;1;1;1;1;1;1;1;1;1;1;1;1;1;1;1;1;1;1;1;1
NOISE: |2;crypto.html;3;3;3;3;3;3;3;3;3;3;3;3;3;3;3;3;3;3;3;3
NOISE: |3;raytrace.html;54;35;34;35;34;35;35;35;33;34;34;37;35;34;35;35;35;33;35;35
NOISE: |4;earley-boyer.html;14;13;13;13;13;13;13;13;13;13;13;13;13;13;13;13;13;13;13;13
NOISE: |5;regexp.html;118;116;115;114;115;115;115;116;115;116;116;116;115;116;116;116;116;116;116;116
NOISE: |6;splay.html;1;1;1;1;1;1;1;1;1;1;1;1;1;1;1;1;1;1;1;1
The numbers are average times in milliseconds for each test. Since they are rounded to whole milliseconds, they are not capable of detecting even fairly large changes in the tests that take ~1 ms or ~3ms to run. And the resulting aliasing of the final result leads to the false alarms mentioned above.
I think the problem is the Math.round call here:
http://hg.mozilla.org/build/talos/file/d00eba6055f5/talos/page_load_test/v8/run-splay.html#l53
If Talos cannot handle fractional numbers in the output, perhaps we should multiply the result by 1000 before rounding it. Or we could calculate the score in the same way the standard V8 test suite does, by starting with a reference score and dividing by the average time in ms to get a "bigger is better" number. That would make the number much easier for the JS team to understand. Currently we have this calculation commented out:
http://hg.mozilla.org/build/talos/file/d00eba6055f5/talos/page_load_test/v8/base.js#l179
Comment 1•13 years ago
|
||
There are a few places where Talos is hard-coded to use ints :( This should probably be fixed
Updated•13 years ago
|
Whiteboard: [SfN]
Reporter | ||
Comment 2•12 years ago
|
||
Fixed by bug 767226.
You need to log in
before you can comment on or make changes to this bug.
Description
•