Closed
Bug 1384245
Opened 7 years ago
Closed 7 years ago
Microbenchmarks should run multiple times and take the median
Categories
(Testing :: General, enhancement)
Testing
General
Tracking
(firefox56 fixed)
RESOLVED
FIXED
mozilla56
Tracking | Status | |
---|---|---|
firefox56 | --- | fixed |
People
(Reporter: wlach, Assigned: wlach)
References
Details
Attachments
(1 file)
For Talos we typically run each tests a set number of times and take the median result, to reduce the effect of outliers. IMO we should do the same for microbenchmarks. I would propose for a start running each test 5 times and then taking the median of the result. This should be enough to get more stable results hopefully without making things take horribly long. Ideally we would store the replicate data inside the PERFHERDER_DATA json for reference just as we do with talos. E.g.: 11:01:50 INFO - PERFHERDER_DATA: {"framework": {"name": "talos"}, "suites": [{"subtests": [{"replicates": [10.90374999999997, 10.988266666666675, 11.424299999999954, 11.065683333333313, 10.898749999999987, 11.340833333333315, 11.24523333333333, 11.090799999999975, 11.550283333333356, 10.926816666666669, 11.392216666666656, 11.55555, 11.004833333333341, 11.816033333333271, 11.081083333333291, 11.376366666666703, 11.03756666666667, 11.22735, 10.983716666666652, 11.273833333333314], "name": "tresize", "value": 11.24523333333333}], "extraOptions": ["e10s"], "name": "tresize"}, {"extraOptions": ["e10s"], "name": "tcanvasmark", "lowerIsBetter": false, "alertThreshold": 2.0, "value": 10097.5, "subtests": [{"name": "Asteroids - Bitmaps", "lowerIsBetter": false, "alertThreshold": 2.0, "replicates": [1063.0, 1068.0, 1092.0, 1074.0, 973.0], "value": 1071.0, "unit": "score"}, {"name": "Asteroids - Vectors", "lowerIsBetter": false, "alertThreshold": 2.0, "replicates": [1910.0, 1943.0, 1913.0, 1892.0, 1921.0], "value": 1917.0, "unit": "score"}, {"name": "Asteroids - Bitmaps- shapes- text", "lowerIsBetter": false, "alertThreshold": 2.0, "replicates": [1858.0, 1869.0, 1869.0, 1845.0, 1851.0], "value": 1860.0, "unit": "score"}, {"name": "Asteroids - Shapes- shadows- blending", "lowerIsBetter": false, "alertThreshold": 2.0, "replicates": [617.0, 620.0, 622.0, 614.0, 619.0], "value": 619.5, "unit": "score"}, {"name": "Arena5 - Vectors- shadows- bitmaps- text", "lowerIsBetter": false, "alertThreshold": 2.0, "replicates": [424.0, 428.0, 433.0, 430.0, 430.0], "value": 430.0, "unit": "score"}, {"name": "Plasma - Maths- canvas shapes", "lowerIsBetter": false, "alertThreshold": 2.0, "replicates": [1145.0, 1145.0, 1128.0, 1122.0, 1146.0], "value": 1136.5, "unit": "score"}, {"name": "3D Rendering - Maths- polygons- image transforms", "lowerIsBetter": false, "alertThreshold": 2.0, "replicates": [1371.0, 1361.0, 1368.0, 1364.0, 1368.0], "value": 1366.0, "unit": "score"}, {"name": "Pixel blur - Math- getImageData- putImageData", "lowerIsBetter": false, "alertThreshold": 2.0, "replicates": [1695.0, 1712.0, 1691.0, 1704.0, 1686.0], "value": 1697.5, "unit": "score"}]}]} I believe the only file that would need to be modified is this one: http://searchfox.org/mozilla-central/rev/3a3af33f513071ea829debdfbc628caebcdf6996/testing/gtest/mozilla/MozGTestBench.cpp
Assignee | ||
Updated•7 years ago
|
Assignee | ||
Comment 1•7 years ago
|
||
I'm going to take a stab at this -- I want to use this as a case study in a blog post I'm writing. ;)
Assignee: nobody → wlachance
Comment hidden (mozreview-request) |
Comment 3•7 years ago
|
||
mozreview-review |
Comment on attachment 8892095 [details] Bug 1384245 - Run gtest benchmarks multiple times and take median https://reviewboard.mozilla.org/r/163098/#review168378
Attachment #8892095 -
Flags: review?(milan) → review+
Pushed by wlachance@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/7ac289e62574 Run gtest benchmarks multiple times and take median r=milan
Comment 5•7 years ago
|
||
bugherder |
https://hg.mozilla.org/mozilla-central/rev/7ac289e62574
Status: NEW → RESOLVED
Closed: 7 years ago
status-firefox56:
--- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla56
You need to log in
before you can comment on or make changes to this bug.
Description
•