Closed Bug 1443788 Opened 2 years ago Closed 2 years ago

motionmark tests appear to be reporting the complexity, not the score

Categories

(Testing :: Talos, enhancement)

enhancement
Not set

Tracking

(firefox60 fixed)

RESOLVED FIXED
mozilla60
Tracking Status
firefox60 --- fixed

People

(Reporter: jmaher, Assigned: jmaher)

Details

Attachments

(1 file)

it was brought to my attention that the data is 100% uniform over time for motionmark.  This is odd, and looking into it what we report is 100% uniform.

here is an example of a page we load:
loaded http://localhost:49218/tests/webkit/PerformanceTests/MotionMark/developer.html?test-interval=15&display=minimal&tiles=big&controller=fixed&frame-rate=30&kalman-process-error=1&kalman-measurement-error=4&time-measurement=performance&suite-name=HTMLsuite&test-name=CompositedTransforms&complexity=75

and here is what we see reported:
15:19:57     INFO -  PID 1712 | [#10] Composited_Transforms  Cycles:5  Average:75.00  Median:75.00  stddev:0.00 (0.0%)  stddev-sans-first:0.00
15:19:57     INFO -  PID 1712 | Values: 75.0  75.0  75.0  75.0  75.0

all iterations show 75 which is the complexity.

the code we added into motionmark to record the results is here:
https://searchfox.org/mozilla-central/source/third_party/webkit/PerformanceTests/MotionMark/resources/debug-runner/animometer.js#631

which is this code:
        var item = dashboard._results['iterationsResults'][0];
        var fullNames = new Array;
        var values = new Array;
        for (var suite in item['testsResults']) {
            for (var subtest in item['testsResults'][suite.toString()]) {
                fullNames.push(suite.toString() + "-" + subtest.toString().replace(/ /g, '_'));
                values.push(item['testsResults'][suite.toString()][subtest.toString()]['controller']['average']);
           }
        }
        if (typeof tpRecordTime !== "undefined") {
            tpRecordTime(values.join(','), 0, fullNames.join(','));
        }


I suspect the values we get from controller['average'] is not what we need- it might be the score reported when running the test, but it is invalid.

We can run this live online with this url:
http://browserbench.org/MotionMark/developer.html?test-interval=15&display=minimal&tiles=big&controller=fixed&frame-rate=30&kalman-process-error=1&kalman-measurement-error=4&time-measurement=performance&suite-name=HTMLsuite&test-name=CompositedTransforms&complexity=75

I get 75 as the score- so I need to know what we should be measuring.
:gw- if you run the above url, what data should we be recording from that to track over time?
Flags: needinfo?(gwatson)
The value to record is the FPS value that is listed for the test. Thanks.
Flags: needinfo?(gwatson)
here you can see some variation in the try results:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=351562edb0d089394fb9290c1a54e3214709fe1f
Assignee: nobody → jmaher
Status: NEW → ASSIGNED
Attachment #8957216 - Flags: review?(rwood)
Comment on attachment 8957216 [details] [diff] [review]
report the FPS instead of average

Review of attachment 8957216 [details] [diff] [review]:
-----------------------------------------------------------------

LGTM
Attachment #8957216 - Flags: review?(rwood) → review+
Pushed by jmaher@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/461bb0cfa848
motionmark tests need to report the FPS not the average. r=rwood
https://hg.mozilla.org/mozilla-central/rev/461bb0cfa848
Status: ASSIGNED → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla60
You need to log in before you can comment on or make changes to this bug.