1128908 - Mozbench: Add geometric mean to Unity WebGLBenchmark

Status: RESOLVED FIXED

(Testing :: General, defect)

Description

[Dan Minor [:dminor]]

We've seen problematic results for this benchmark on Windows where the test will not render correctly causing it to run extremely quickly. See Bug 1128639.

Since this potentially impacts both Chrome and Firefox, we should remove this test from the summary score until we're sure it is running properly. This appears to have caused a dramatic shift in relative performance of Chrome and Firefox on the test hardware in the past.

Comment 1

[Dan Minor [:dminor]]

It looks like this calculation is buried somewhere in a 24MB minified javascript file. Grepping for obvious text strings like "benchmark", "overall", "mandelbrot" is not getting me anywhere.

The blog post describing the benchmarks has this to say:

If you run all of the benchmarks, it will also show an Overall Score value. This score is calculated as a weighted sum of all the individual test results. Since the different benchmarks have very different result scales, they each carry different weights in the overall score.  The weights are calibrated to produce similar result scales on my development machine running in WebGL. But that is somewhat arbitrary, so, when comparing results, the individual scores are much more meaningful than the Overall Score!

Since it seems the current overall score is somewhat arbitrary, I'd propose we start calculating the geometric mean of the individual results (excluding the flaky Mandelbrot test) and graphing that for now. Does this sound ok?

Comment 2

[Martin Best (:mbest)]

The Mandelbrot test works if you use dx11.  I'm thinking we should make sure we are testing under those conditions.  There is an Angle bug preventing it from working under Dx9 that the graphics team is aware of.  Storing each result sounds good.  DX11 is available in nightly and Aurora currently.

Comment 3

[Dan Minor [:dminor]]

I've switched the mozbench dashboard over to display the geometric mean by default which should make us less vulnerable to a single flaky test in the future. I've verified the test machine is using DX11.