1384245 - Microbenchmarks should run multiple times and take the median

Status: RESOLVED FIXED

(Testing :: General, enhancement)

Description

[William Lachance (:wlach)]

For Talos we typically run each tests a set number of times and take the median result, to reduce the effect of outliers. IMO we should do the same for microbenchmarks.

I would propose for a start running each test 5 times and then taking the median of the result. This should be enough to get more stable results hopefully without making things take horribly long. Ideally we would store the replicate data inside the PERFHERDER_DATA json for reference just as we do with talos. E.g.:

11:01:50     INFO -  PERFHERDER_DATA: {"framework": {"name": "talos"}, "suites": [{"subtests": [{"replicates": [10.90374999999997, 10.988266666666675, 11.424299999999954, 11.065683333333313, 10.898749999999987, 11.340833333333315, 11.24523333333333, 11.090799999999975, 11.550283333333356, 10.926816666666669, 11.392216666666656, 11.55555, 11.004833333333341, 11.816033333333271, 11.081083333333291, 11.376366666666703, 11.03756666666667, 11.22735, 10.983716666666652, 11.273833333333314], "name": "tresize", "value": 11.24523333333333}], "extraOptions": ["e10s"], "name": "tresize"}, {"extraOptions": ["e10s"], "name": "tcanvasmark", "lowerIsBetter": false, "alertThreshold": 2.0, "value": 10097.5, "subtests": [{"name": "Asteroids - Bitmaps", "lowerIsBetter": false, "alertThreshold": 2.0, "replicates": [1063.0, 1068.0, 1092.0, 1074.0, 973.0], "value": 1071.0, "unit": "score"}, {"name": "Asteroids - Vectors", "lowerIsBetter": false, "alertThreshold": 2.0, "replicates": [1910.0, 1943.0, 1913.0, 1892.0, 1921.0], "value": 1917.0, "unit": "score"}, {"name": "Asteroids - Bitmaps- shapes- text", "lowerIsBetter": false, "alertThreshold": 2.0, "replicates": [1858.0, 1869.0, 1869.0, 1845.0, 1851.0], "value": 1860.0, "unit": "score"}, {"name": "Asteroids - Shapes- shadows- blending", "lowerIsBetter": false, "alertThreshold": 2.0, "replicates": [617.0, 620.0, 622.0, 614.0, 619.0], "value": 619.5, "unit": "score"}, {"name": "Arena5 - Vectors- shadows- bitmaps- text", "lowerIsBetter": false, "alertThreshold": 2.0, "replicates": [424.0, 428.0, 433.0, 430.0, 430.0], "value": 430.0, "unit": "score"}, {"name": "Plasma - Maths- canvas shapes", "lowerIsBetter": false, "alertThreshold": 2.0, "replicates": [1145.0, 1145.0, 1128.0, 1122.0, 1146.0], "value": 1136.5, "unit": "score"}, {"name": "3D Rendering - Maths- polygons- image transforms", "lowerIsBetter": false, "alertThreshold": 2.0, "replicates": [1371.0, 1361.0, 1368.0, 1364.0, 1368.0], "value": 1366.0, "unit": "score"}, {"name": "Pixel blur - Math- getImageData- putImageData", "lowerIsBetter": false, "alertThreshold": 2.0, "replicates": [1695.0, 1712.0, 1691.0, 1704.0, 1686.0], "value": 1697.5, "unit": "score"}]}]}

I believe the only file that would need to be modified is this one:

http://searchfox.org/mozilla-central/rev/3a3af33f513071ea829debdfbc628caebcdf6996/testing/gtest/mozilla/MozGTestBench.cpp

Comment 1

[William Lachance (:wlach)]

I'm going to take a stab at this -- I want to use this as a case study in a blog post I'm writing. ;)

Comment 2

[William Lachance (:wlach)]

Attachment: Bug 1384245 - Run gtest benchmarks multiple times and take median

Review commit: https://reviewboard.mozilla.org/r/163098/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/163098/

Comment 3

[Milan Sreckovic [:milan] (needinfo for best results)]

Comment on attachment 8892095 [details]
Bug 1384245 - Run gtest benchmarks multiple times and take median

https://reviewboard.mozilla.org/r/163098/#review168378

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/7ac289e62574
Run gtest benchmarks multiple times and take median r=milan

Comment 5

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/7ac289e62574