Talos V8 tests are rounded to the nearest millisecond, introducing rounding errors and aliasing

RESOLVED FIXED

Status

RESOLVED FIXED
7 years ago
6 years ago

People

(Reporter: mbrubeck, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [SfN])

(Reporter)

Description

7 years ago
Talos V8 is extremely bimodal.  Because of the way its results are averaged, it is currently always returning exactly 9.133 or 9.167.  (In the past it has also returned other specific values like 9.500 or 9.000 -- always an even multiple of 1/6.)

A typical Talos V8 run looks like this:

NOISE: |0;richards.html;3;3;3;3;3;3;3;3;3;3;3;3;3;3;3;3;3;3;3;3
NOISE: |1;deltablue.html;1;1;1;1;1;1;1;1;1;1;1;1;1;1;1;1;1;1;1;1
NOISE: |2;crypto.html;3;3;3;3;3;3;3;3;3;3;3;3;3;3;3;3;3;3;3;3
NOISE: |3;raytrace.html;54;35;34;35;34;35;35;35;33;34;34;37;35;34;35;35;35;33;35;35
NOISE: |4;earley-boyer.html;14;13;13;13;13;13;13;13;13;13;13;13;13;13;13;13;13;13;13;13
NOISE: |5;regexp.html;118;116;115;114;115;115;115;116;115;116;116;116;115;116;116;116;116;116;116;116
NOISE: |6;splay.html;1;1;1;1;1;1;1;1;1;1;1;1;1;1;1;1;1;1;1;1

The numbers are average times in milliseconds for each test.  Since they are rounded to whole milliseconds, they are not capable of detecting even fairly large changes in the tests that take ~1 ms or ~3ms to run.  And the resulting aliasing of the final result leads to the false alarms mentioned above.

I think the problem is the Math.round call here:
http://hg.mozilla.org/build/talos/file/d00eba6055f5/talos/page_load_test/v8/run-splay.html#l53

If Talos cannot handle fractional numbers in the output, perhaps we should multiply the result by 1000 before rounding it.  Or we could calculate the score in the same way the standard V8 test suite does, by starting with a reference score and dividing by the average time in ms to get a "bigger is better" number.  That would make the number much easier for the JS team to understand.  Currently we have this calculation commented out:
http://hg.mozilla.org/build/talos/file/d00eba6055f5/talos/page_load_test/v8/base.js#l179

Comment 1

7 years ago
There are a few places where Talos is hard-coded to use ints :( This should probably be fixed

Updated

7 years ago
Whiteboard: [SfN]
(Reporter)

Comment 2

7 years ago
Fixed by bug 767226.
Status: NEW → RESOLVED
Last Resolved: 7 years ago
Depends on: 767226
Resolution: --- → FIXED
(Reporter)

Updated

6 years ago
Blocks: 859495
You need to log in before you can comment on or make changes to this bug.