Microbenchmarks should run multiple times and take the median

RESOLVED FIXED in Firefox 56

Status

Testing
General
RESOLVED FIXED
3 months ago
3 months ago

People

(Reporter: wlach, Assigned: wlach)

Tracking

unspecified
mozilla56
Points:
---

Firefox Tracking Flags

(firefox56 fixed)

Details

MozReview Requests

Submitter Diff Changes Open Issues Last Updated
Loading...
Error loading review requests:

Attachments

(1 attachment)

For Talos we typically run each tests a set number of times and take the median result, to reduce the effect of outliers. IMO we should do the same for microbenchmarks.

I would propose for a start running each test 5 times and then taking the median of the result. This should be enough to get more stable results hopefully without making things take horribly long. Ideally we would store the replicate data inside the PERFHERDER_DATA json for reference just as we do with talos. E.g.:

11:01:50     INFO -  PERFHERDER_DATA: {"framework": {"name": "talos"}, "suites": [{"subtests": [{"replicates": [10.90374999999997, 10.988266666666675, 11.424299999999954, 11.065683333333313, 10.898749999999987, 11.340833333333315, 11.24523333333333, 11.090799999999975, 11.550283333333356, 10.926816666666669, 11.392216666666656, 11.55555, 11.004833333333341, 11.816033333333271, 11.081083333333291, 11.376366666666703, 11.03756666666667, 11.22735, 10.983716666666652, 11.273833333333314], "name": "tresize", "value": 11.24523333333333}], "extraOptions": ["e10s"], "name": "tresize"}, {"extraOptions": ["e10s"], "name": "tcanvasmark", "lowerIsBetter": false, "alertThreshold": 2.0, "value": 10097.5, "subtests": [{"name": "Asteroids - Bitmaps", "lowerIsBetter": false, "alertThreshold": 2.0, "replicates": [1063.0, 1068.0, 1092.0, 1074.0, 973.0], "value": 1071.0, "unit": "score"}, {"name": "Asteroids - Vectors", "lowerIsBetter": false, "alertThreshold": 2.0, "replicates": [1910.0, 1943.0, 1913.0, 1892.0, 1921.0], "value": 1917.0, "unit": "score"}, {"name": "Asteroids - Bitmaps- shapes- text", "lowerIsBetter": false, "alertThreshold": 2.0, "replicates": [1858.0, 1869.0, 1869.0, 1845.0, 1851.0], "value": 1860.0, "unit": "score"}, {"name": "Asteroids - Shapes- shadows- blending", "lowerIsBetter": false, "alertThreshold": 2.0, "replicates": [617.0, 620.0, 622.0, 614.0, 619.0], "value": 619.5, "unit": "score"}, {"name": "Arena5 - Vectors- shadows- bitmaps- text", "lowerIsBetter": false, "alertThreshold": 2.0, "replicates": [424.0, 428.0, 433.0, 430.0, 430.0], "value": 430.0, "unit": "score"}, {"name": "Plasma - Maths- canvas shapes", "lowerIsBetter": false, "alertThreshold": 2.0, "replicates": [1145.0, 1145.0, 1128.0, 1122.0, 1146.0], "value": 1136.5, "unit": "score"}, {"name": "3D Rendering - Maths- polygons- image transforms", "lowerIsBetter": false, "alertThreshold": 2.0, "replicates": [1371.0, 1361.0, 1368.0, 1364.0, 1368.0], "value": 1366.0, "unit": "score"}, {"name": "Pixel blur - Math- getImageData- putImageData", "lowerIsBetter": false, "alertThreshold": 2.0, "replicates": [1695.0, 1712.0, 1691.0, 1704.0, 1686.0], "value": 1697.5, "unit": "score"}]}]}

I believe the only file that would need to be modified is this one:

http://searchfox.org/mozilla-central/rev/3a3af33f513071ea829debdfbc628caebcdf6996/testing/gtest/mozilla/MozGTestBench.cpp
Blocks: 1369807
See Also: → bug 1369807
I'm going to take a stab at this -- I want to use this as a case study in a blog post I'm writing. ;)
Assignee: nobody → wlachance
Comment hidden (mozreview-request)

Comment 3

3 months ago
mozreview-review
Comment on attachment 8892095 [details]
Bug 1384245 - Run gtest benchmarks multiple times and take median

https://reviewboard.mozilla.org/r/163098/#review168378
Attachment #8892095 - Flags: review?(milan) → review+

Comment 4

3 months ago
Pushed by wlachance@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/7ac289e62574
Run gtest benchmarks multiple times and take median r=milan

Comment 5

3 months ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/7ac289e62574
Status: NEW → RESOLVED
Last Resolved: 3 months ago
status-firefox56: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla56
You need to log in before you can comment on or make changes to this bug.