Closed Bug 1473365 Opened Last year Closed Last year

migrate AWFY assortedDOM tests to a raptor benchmark

Categories

(Testing :: Raptor, enhancement)

enhancement
Not set

Tracking

(firefox63 fixed)

RESOLVED FIXED
mozilla63
Tracking Status
firefox63 --- fixed

People

(Reporter: jmaher, Assigned: rwood)

References

Details

Attachments

(2 files)

as part of the AWFY -> Talos migration we have many tests that need to be run in talos/raptor/jsshell-bench.  One test is assortedDOM:
https://arewefastyet.com/#machine=36&view=breakdown&suite=assorteddom

this runs in a browser (not shell).  here is the source for the test:
https://github.com/mozilla/arewefastyet/tree/2696cbea7fbbe50c3a8cbf70a0ac3c70f135852e/benchmarks/misc-desktop/tests/assorted

These are not large (<200KB), and we can add them in-tree.  :bz recommended making these into new perf-reftest style tests, I assume that will be straightforward and we can figure that out.

The one quirk here by doing that is perf-reftests run in talos and we don't have a good method for running talos against google chrome.  raptor does and maybe we could port perf-reftest tests from talos->raptor if all APIs are compatible on the backend (can we do this in compliant web extension code only)
I'm a lot more interested in the Firefox numbers for these than the Chrome numbers, fwiw.
:rwood, you had done a lot of the work on getting perf-reftest tests to run, is this something you can pick up in the next week or two?
Flags: needinfo?(rwood)
(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #2)
> :rwood, you had done a lot of the work on getting perf-reftest tests to run,
> is this something you can pick up in the next week or two?

That won't be easy - perf-reftests use their own test code and measure / calculate their own values (in each test), it's not a like a pageload test that Raptor currently supports i.e. capturing specific event(s). Also they run sets of two pages in one test and then compare the values and that is the actual result. It would be a fair amount of work in Raptor, definitely not something quick. If perf-reftests aren't really required on google chrome then I would recommend leaving them in Talos. If there is a requirement to having them on google chrome then maybe we could schedule some time to look into this further on i.e. for raptor M2?
Flags: needinfo?(rwood)
:rwood, I think keeping it on talos is a good and reasonable plan.  We can look in detail at these tests and see if there is a reasonable path forward.  lets look next week.
I had a closer look at the tests - from what I can tell they don't have to be compared against a baseline or anything (like a perf-reftest) they are standalone (more like perf-reftest-singletons). In which case yes I will see if I can port them to Raptor. I should be able to get them to work like other benchmarks in Raptor - that report their own results via the webext benchmark-relay.js etc.
Assignee: nobody → rwood
Status: NEW → ASSIGNED
Depends on: 1478057
Depends on: 1478773
No longer depends on: 1478773, 1478057
Attached image assorted-dom.png
Had a closer look at the assorted dom tests and ran them locally via the arewefastyet repo. I see that all of the small tests are already combined into one benchmark which is run via /hosted/index.html. I think the best way to port this to raptor is to take a snapshot of this benchmark and modify it a bit to get it running like the other benchmarks (autorun, and post results to the raptor webext).

Attaching a screenshot of the tests as run currently in arewefastyet for reference.
Component: Talos → Raptor
Summary: migrate AWFY assortedDOM tests to perf-reftests → migrate AWFY assortedDOM tests to a raptor benchmark
Depends on: 1481930
I added the assorted-dom benchmark source to the perf-automation repo [1] so now with this patch the new raptor benchmark (raptor-assorted-dom) can be run locally (but not yet in production until the fetch task is done in Bug 1481930).

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=1481930#c1
Comment on attachment 8998962 [details]
Bug 1473365 - migrate AWFY assortedDOM tests to a raptor benchmark

Thanks for the review, I made the changes and updated the phabricator diff (but didn't get an email for some reason).

https://phabricator.services.mozilla.com/D3046?id=7600
Attachment #8998962 - Flags: review?(jmaher)
Comment on attachment 8998962 [details]
Bug 1473365 - migrate AWFY assortedDOM tests to a raptor benchmark

Joel Maher ( :jmaher ) (UTC+2) has approved the revision.
Attachment #8998962 - Flags: review+
Comment on attachment 8998962 [details]
Bug 1473365 - migrate AWFY assortedDOM tests to a raptor benchmark

Sorry :jmaher, one more update - added taskcluster configs and switched to geometric mean. Thanks!
Attachment #8998962 - Flags: review+
Comment on attachment 8998962 [details]
Bug 1473365 - migrate AWFY assortedDOM tests to a raptor benchmark

https://phabricator.services.mozilla.com/D3046?id=7903
Comment on attachment 8998962 [details]
Bug 1473365 - migrate AWFY assortedDOM tests to a raptor benchmark

Joel Maher ( :jmaher ) (UTC+2) has approved the revision.
Attachment #8998962 - Flags: review+
Note: It's red on try due to the same issue as the 'ugl' job (fetch tasks not working at the moment) so I set it to tier 3 so it can land.
Depends on: 1482344
Pushed by rwood@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/a2876bd19572
migrate AWFY assortedDOM tests to a raptor benchmark r=jmaher
Attachment #8998962 - Flags: review?(jmaher) → review+
https://hg.mozilla.org/mozilla-central/rev/a2876bd19572
Status: ASSIGNED → RESOLVED
Closed: Last year
Resolution: --- → FIXED
Target Milestone: --- → mozilla63
You need to log in before you can comment on or make changes to this bug.