Microbenchmarks should run multiple times and take the median

RESOLVED FIXED in Firefox 56

Status

enhancement
RESOLVED FIXED
2 years ago
2 years ago

People

(Reporter: wlach, Assigned: wlach)

Tracking

unspecified
mozilla56
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(firefox56 fixed)

Details

Attachments

(1 attachment)

For Talos we typically run each tests a set number of times and take the median result, to reduce the effect of outliers. IMO we should do the same for microbenchmarks.

I would propose for a start running each test 5 times and then taking the median of the result. This should be enough to get more stable results hopefully without making things take horribly long. Ideally we would store the replicate data inside the PERFHERDER_DATA json for reference just as we do with talos. E.g.:

11:01:50     INFO -  PERFHERDER_DATA: {"framework": {"name": "talos"}, "suites": [{"subtests": [{"replicates": [10.90374999999997, 10.988266666666675, 11.424299999999954, 11.065683333333313, 10.898749999999987, 11.340833333333315, 11.24523333333333, 11.090799999999975, 11.550283333333356, 10.926816666666669, 11.392216666666656, 11.55555, 11.004833333333341, 11.816033333333271, 11.081083333333291, 11.376366666666703, 11.03756666666667, 11.22735, 10.983716666666652, 11.273833333333314], "name": "tresize", "value": 11.24523333333333}], "extraOptions": ["e10s"], "name": "tresize"}, {"extraOptions": ["e10s"], "name": "tcanvasmark", "lowerIsBetter": false, "alertThreshold": 2.0, "value": 10097.5, "subtests": [{"name": "Asteroids - Bitmaps", "lowerIsBetter": false, "alertThreshold": 2.0, "replicates": [1063.0, 1068.0, 1092.0, 1074.0, 973.0], "value": 1071.0, "unit": "score"}, {"name": "Asteroids - Vectors", "lowerIsBetter": false, "alertThreshold": 2.0, "replicates": [1910.0, 1943.0, 1913.0, 1892.0, 1921.0], "value": 1917.0, "unit": "score"}, {"name": "Asteroids - Bitmaps- shapes- text", "lowerIsBetter": false, "alertThreshold": 2.0, "replicates": [1858.0, 1869.0, 1869.0, 1845.0, 1851.0], "value": 1860.0, "unit": "score"}, {"name": "Asteroids - Shapes- shadows- blending", "lowerIsBetter": false, "alertThreshold": 2.0, "replicates": [617.0, 620.0, 622.0, 614.0, 619.0], "value": 619.5, "unit": "score"}, {"name": "Arena5 - Vectors- shadows- bitmaps- text", "lowerIsBetter": false, "alertThreshold": 2.0, "replicates": [424.0, 428.0, 433.0, 430.0, 430.0], "value": 430.0, "unit": "score"}, {"name": "Plasma - Maths- canvas shapes", "lowerIsBetter": false, "alertThreshold": 2.0, "replicates": [1145.0, 1145.0, 1128.0, 1122.0, 1146.0], "value": 1136.5, "unit": "score"}, {"name": "3D Rendering - Maths- polygons- image transforms", "lowerIsBetter": false, "alertThreshold": 2.0, "replicates": [1371.0, 1361.0, 1368.0, 1364.0, 1368.0], "value": 1366.0, "unit": "score"}, {"name": "Pixel blur - Math- getImageData- putImageData", "lowerIsBetter": false, "alertThreshold": 2.0, "replicates": [1695.0, 1712.0, 1691.0, 1704.0, 1686.0], "value": 1697.5, "unit": "score"}]}]}

I believe the only file that would need to be modified is this one:

http://searchfox.org/mozilla-central/rev/3a3af33f513071ea829debdfbc628caebcdf6996/testing/gtest/mozilla/MozGTestBench.cpp
Blocks: 1369807
See Also: → 1369807
I'm going to take a stab at this -- I want to use this as a case study in a blog post I'm writing. ;)
Assignee: nobody → wlachance
Comment on attachment 8892095 [details]
Bug 1384245 - Run gtest benchmarks multiple times and take median

https://reviewboard.mozilla.org/r/163098/#review168378
Attachment #8892095 - Flags: review?(milan) → review+

Comment 4

2 years ago
Pushed by wlachance@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/7ac289e62574
Run gtest benchmarks multiple times and take median r=milan

Comment 5

2 years ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/7ac289e62574
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla56
You need to log in before you can comment on or make changes to this bug.