1473365 - migrate AWFY assortedDOM tests to a raptor benchmark

Status: RESOLVED FIXED

(Testing :: Raptor, enhancement)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

as part of the AWFY -> Talos migration we have many tests that need to be run in talos/raptor/jsshell-bench.  One test is assortedDOM:
https://arewefastyet.com/#machine=36&view=breakdown&suite=assorteddom

this runs in a browser (not shell).  here is the source for the test:
https://github.com/mozilla/arewefastyet/tree/2696cbea7fbbe50c3a8cbf70a0ac3c70f135852e/benchmarks/misc-desktop/tests/assorted

These are not large (<200KB), and we can add them in-tree.  :bz recommended making these into new perf-reftest style tests, I assume that will be straightforward and we can figure that out.

The one quirk here by doing that is perf-reftests run in talos and we don't have a good method for running talos against google chrome.  raptor does and maybe we could port perf-reftest tests from talos->raptor if all APIs are compatible on the backend (can we do this in compliant web extension code only)

Comment 1

[Boris Zbarsky [:bzbarsky]]

I'm a lot more interested in the Firefox numbers for these than the Chrome numbers, fwiw.

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

:rwood, you had done a lot of the work on getting perf-reftest tests to run, is this something you can pick up in the next week or two?

Comment 3

[Robert Wood [:rwood]]

(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #2)
> :rwood, you had done a lot of the work on getting perf-reftest tests to run,
> is this something you can pick up in the next week or two?

That won't be easy - perf-reftests use their own test code and measure / calculate their own values (in each test), it's not a like a pageload test that Raptor currently supports i.e. capturing specific event(s). Also they run sets of two pages in one test and then compare the values and that is the actual result. It would be a fair amount of work in Raptor, definitely not something quick. If perf-reftests aren't really required on google chrome then I would recommend leaving them in Talos. If there is a requirement to having them on google chrome then maybe we could schedule some time to look into this further on i.e. for raptor M2?

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

:rwood, I think keeping it on talos is a good and reasonable plan.  We can look in detail at these tests and see if there is a reasonable path forward.  lets look next week.

Comment 5

[Robert Wood [:rwood]]

I had a closer look at the tests - from what I can tell they don't have to be compared against a baseline or anything (like a perf-reftest) they are standalone (more like perf-reftest-singletons). In which case yes I will see if I can port them to Raptor. I should be able to get them to work like other benchmarks in Raptor - that report their own results via the webext benchmark-relay.js etc.

Comment 6

[Robert Wood [:rwood]]

Attachment: assorted-dom.png

Had a closer look at the assorted dom tests and ran them locally via the arewefastyet repo. I see that all of the small tests are already combined into one benchmark which is run via /hosted/index.html. I think the best way to port this to raptor is to take a snapshot of this benchmark and modify it a bit to get it running like the other benchmarks (autorun, and post results to the raptor webext).

Attaching a screenshot of the tests as run currently in arewefastyet for reference.

Comment 7

[Robert Wood [:rwood]]

I added the assorted-dom benchmark source to the perf-automation repo [1] so now with this patch the new raptor benchmark (raptor-assorted-dom) can be run locally (but not yet in production until the fetch task is done in Bug 1481930).

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=1481930#c1

Comment 8

[Robert Wood [:rwood]]

Attachment: Bug 1473365 - migrate AWFY assortedDOM tests to a raptor benchmark

Comment 9

[Robert Wood [:rwood]]

Comment on attachment 8998962 [details]
Bug 1473365 - migrate AWFY assortedDOM tests to a raptor benchmark

Thanks for the review, I made the changes and updated the phabricator diff (but didn't get an email for some reason).

https://phabricator.services.mozilla.com/D3046?id=7600

Comment 10

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8998962 [details]
Bug 1473365 - migrate AWFY assortedDOM tests to a raptor benchmark

Joel Maher ( :jmaher ) (UTC+2) has approved the revision.

Comment 11

[Robert Wood [:rwood]]

Comment on attachment 8998962 [details]
Bug 1473365 - migrate AWFY assortedDOM tests to a raptor benchmark

Sorry :jmaher, one more update - added taskcluster configs and switched to geometric mean. Thanks!

Comment 12

[Robert Wood [:rwood]]

Comment on attachment 8998962 [details]
Bug 1473365 - migrate AWFY assortedDOM tests to a raptor benchmark

https://phabricator.services.mozilla.com/D3046?id=7903

Comment 13

[Robert Wood [:rwood]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=c61b43dc04d275b272e122452fefc3bc08e9bc5a

Comment 14

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8998962 [details]
Bug 1473365 - migrate AWFY assortedDOM tests to a raptor benchmark

Joel Maher ( :jmaher ) (UTC+2) has approved the revision.

Comment 15

[Robert Wood [:rwood]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=e67dd6a5c22b6762095890b93f4be3882390b8b9

Comment 16

[Robert Wood [:rwood]]

Note: It's red on try due to the same issue as the 'ugl' job (fetch tasks not working at the moment) so I set it to tier 3 so it can land.

Comment 17

[Robert Wood [:rwood]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=b6dc18b196e0e371d1a0d3d794601891b7dc40d5

Comment 18

[Robert Wood [:rwood]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=8c2ca548f61ec8951f2986d50c35d5bfd5b3eba6

Comment 19

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/a2876bd19572
migrate AWFY assortedDOM tests to a raptor benchmark r=jmaher

Comment 20

[Arthur Iakab [arthur_iakab]]

https://hg.mozilla.org/mozilla-central/rev/a2876bd19572