1689537 - Update MotionMark to r233463 or later

Status: RESOLVED FIXED

(Testing :: Performance, task, P1)

Description

[Tetsuharu OHZEKI [:tetsuharu] (UTC+9)]

WebKit folks apply some fixes to MotionMark recently. I'm not sure about that they have a plan to release as newer version to https://browserbench.org/, but these patches looks like contains some fix related to calc the score.

Comment 1

[Dzmitry Malyshau [:kvark]]

MM originally landed in tree with bug 1428435.
Joel, do we need any manual action to pick this up?

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

redirect to :davehunt who is in charge of perf tests now. we would need to import the latest code from motion mark, ensure it still works, and rebaseline numbers; could be quick like a day or two, or if there are tricky things or odd behaviors a week of work.

Comment 3

[Dave Hunt [:davehunt] [he/him] ⌚BST]

(In reply to Joel Maher ( :jmaher ) (UTC -0800) from comment #2)

redirect to :davehunt who is in charge of perf tests now. we would need to import the latest code from motion mark, ensure it still works, and rebaseline numbers; could be quick like a day or two, or if there are tricky things or odd behaviors a week of work.

We're planning to migrate all of our benchmarks to browsertime over the coming months. We can look into updating MotionMark at the same time. :bebe could you open a bug for migrating MotionMark and set this bug as a dependency?

Comment 4

[Florin Strugariu [:Bebe]]

on it

Comment 5

[Florin Strugariu [:Bebe]]

:tetsuharu to update the we need to figure out what changes and sub-tests we need to run.

Currently we have 2 tests based on the motion mark benchmark suite: animometer and htmlsuite
Also there are some custom changes made to the original code of motionmark to run in Raptor.

Here is the original commit:
https://hg.mozilla.org/mozilla-central/rev/b51ba2011412289a3471cb88c91e82221aa45be8

I will try to migrate it but as I have limited knowledge about the benchmark code I would nee some help

Comment 6

[Tetsuharu OHZEKI [:tetsuharu] (UTC+9)]

(In reply to Florin Strugariu [:Bebe] (needinfo me) from comment #5)

:tetsuharu to update the we need to figure out what changes and sub-tests we need to run.

Currently we have 2 tests based on the motion mark benchmark suite: animometer and htmlsuite
Also there are some custom changes made to the original code of motionmark to run in Raptor.

Here is the original commit:
https://hg.mozilla.org/mozilla-central/rev/b51ba2011412289a3471cb88c91e82221aa45be8

I will try to migrate it but as I have limited knowledge about the benchmark code I would nee some help

Sorry to late this reply.
First of all, I did not touch it when we had introduced MotionMark to Reptor, and I'm not sure about it well...
To be honest, I just found WebKit folks update MotionMark but I don't follow details of their activity. Thus I doubt that I can help your migration effort....

Comment 7

[Florin Strugariu [:Bebe]]

:Dave is this still something we want to do?
This will take some time to update/modify the benchmark to output the data we need.

Comment 8

[Dave Hunt [:davehunt] [he/him] ⌚BST]

(In reply to Florin Strugariu [:Bebe] (needinfo me) from comment #7)

:Dave is this still something we want to do?
This will take some time to update/modify the benchmark to output the data we need.

We should come up with a policy for reviewing and updating our benchmarks as many of them haven't been updated for years. Let's keep this open and create a meta bug for reviewing all benchmarks. The performance strategy meeting would be a good place to discuss how we proceed.

Comment 9

[Dave Hunt [:davehunt] [he/him] ⌚BST]

The latest version of MotionMark is 1.2. Based on comment 5 this upgrade may not be straightforward, and I would suggest treating this as a new benchmark. This would allow us to disregard how we calculated and reported MotionMark results in the past (we have separate tests for animometer and htmlsuite) and allow us to approach this with a fresh perspective. :bas :denispal :jgilbert (as the point of contact for the existing tests) do you have thoughts?

Comment 10

[Bas Schouten (:bas.schouten)]

That seems fine to me. The existing scores in CI seem to have no correlation with the actual test results when running locally on any device I have, so I don't particularly care for the results in CI.

Comment 11

[Kelsey Gilbert [:jgilbert]]

I don't mind if we make a new test key for the new version, since that lets us clearly de-correlate the old-vs-new results.

Comment 12

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Moving this to Testing::Performance so we can prioritise this work.

Comment 13

[Kash Shampur [:kshampur] ⌚EST]

Attachment: WIP: Bug 1689537 - Update MotionMark to 1.2 (34a15d3).

Comment 14

[Kash Shampur [:kshampur] ⌚EST]

Attachment: Bug 1689537 - Update config files for MotionMark 1.3 benchmark. r?#perftest

This patch updates the toml/yml files necessary to run motionmark 1.3.
An external repo model is used here rather than vendoring in-tree, as
this should be more advantageous going forward.
Brand new files (motionmark-1-3-<desktop/mobile>.toml) are added rather
than adding it to the existing motionmark animometer/htmlsuite configfiles.
This should be easier to maintain going forward (if for example we
decide to remove the previous motionmark). And also there has has been
so many changes since motionmark 1.0, this is effectively a new
benchmark.

It is also worth noting that what was formerly known as
motionmark-animometer is not just motionmark. However htmlsuite is
still the same.

Comment 15

[Kash Shampur [:kshampur] ⌚EST]

Attachment: Bug 1689537 - Add MotionMark 1.3 browsertime script. r?#perftest

This patch adds a custom test script to run motionmark with browsertime.
As we are not vendoring in tree and applying custom patches, it is
convenient instead here to make use javascript directly to start the
benchmark controller.

Comment 16

[Kash Shampur [:kshampur] ⌚EST]

Attachment: Bug 1689537 - Add Python support class for MotionMark 1.3. r?#perftest

This patch uses our raptor support class for custom tests. The data
handling shares a lot of similarities to existing benchmark tests
e.g. speedometer3, and that is leveraged here.

Comment 17

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/a1ba13c7c518
Update config files for MotionMark 1.3 benchmark. r=perftest-reviewers,aglavic
https://hg.mozilla.org/integration/autoland/rev/42a9a375f4b8
Add MotionMark 1.3 browsertime script. r=perftest-reviewers,aglavic
https://hg.mozilla.org/integration/autoland/rev/867bf0a90600
Add Python support class for MotionMark 1.3. r=perftest-reviewers,sparky

Comment 18

[Cristian Tuns]

https://hg.mozilla.org/mozilla-central/rev/a1ba13c7c518
https://hg.mozilla.org/mozilla-central/rev/42a9a375f4b8
https://hg.mozilla.org/mozilla-central/rev/867bf0a90600