Use a single test INI and taskcluster configuration file for all raptor-browsertime desktop page-load tests
Categories
(Testing :: Raptor, enhancement, P1)
Tracking
(firefox73 fixed)
| Tracking | Status | |
|---|---|---|
| firefox73 | --- | fixed |
People
(Reporter: rwood, Assigned: rwood)
References
Details
Attachments
(1 file)
As part of the raptor-browsertime test INI and taskcluster configs refactor, use a single test INI specification file for all raptor-browsertime desktop page-load tests.
Also have a single corresponding taskcluster browsertime-desktop.yml configuration file which will contain the settings for all raptor-browsertime desktop page-load tests, with a single entry for 'tp6-desktop'. Use taskcluster transforms to auto-generate one taskcluster raptor-browsertime test job per each test-url to be included in the tp6-desktop test.
|
Assignee | |
Updated•5 years ago
|
|
|
Assignee | |
Comment 1•5 years ago
|
||
Put together a potential solution and pushed to try:
https://hg.mozilla.org/try/rev/c2ae4b1c5b3d6d303eda28dab4803f0059c4b742
:ahal, :jlorenzo - any thoughts and advice on the above approach? Thanks!
|
||
Comment 2•5 years ago
|
||
I'm no expert in the way tests are defined in mozilla-central, but this solution looks sensible to me.
I thought we had a nice transform to easily chunk jobs, but it turns out we have many implementations across taskgraph. Therefore, I'm fine not factorizing this piece just yet. Taskgraph has grown organically in order to finish the Taskcluster migration last year, and we can clean this up later.
|
||
Comment 3•5 years ago
|
||
Yeah, that looks reasonable to me too.
This is unrelated to this issue and there's probably a good answer to this question.. but why do we have to run these each in a separate task? Why not have a single tp6 task that runs all of the pageloads in serial (restarting the browser in-between of course).
|
|
Assignee | |
Comment 4•5 years ago
|
||
Thanks for the feedback guys!
(In reply to Andrew Halberstadt [:ahal] from comment #3)
This is unrelated to this issue and there's probably a good answer to this question.. but why do we have to run these each in a separate task? Why not have a single
tp6task that runs all of the pageloads in serial (restarting the browser in-between of course).
That would be super nice but we split them out mainly because even just one URL cold page-load job takes around 25 min on Windows. We do 25 browser-cycles (or 25 page-cycles on warm page load). Also just easier for automated perf alerts and for sheriffs bisecting/retriggering etc.
|
|
Assignee | |
Comment 5•5 years ago
|
||
|
|
Assignee | |
Comment 6•5 years ago
|
||
|
|
Assignee | |
Comment 7•5 years ago
|
||
|
|
||
Comment 8•5 years ago
|
||
(In reply to Robert Wood [:rwood] from comment #4)
That would be super nice but we split them out mainly because even just one URL cold page-load job takes around 25 min on Windows. We do 25 browser-cycles (or 25 page-cycles on warm page load). Also just easier for automated perf alerts and for sheriffs bisecting/retriggering etc.
Ah, yes that is a good reason to keep them separate :)
|
|
Assignee | |
Comment 9•5 years ago
|
||
|
|
Assignee | |
Comment 10•5 years ago
|
||
This patch will allow us to have a single raptor test INI and corresponding taskcluster config entry for all raptor-browsertime tp6 desktop tests. In the tp6-desktop.ini file all the URLs to be measured are listed as subtests below the default test settings, with a simple name (i.e. ‘amazon’). In the raptor.yml taskcluster config new 'browsertime-tp6-desktop' entry, there is a ‘raptor-subtests’ key with a list of matching URLs/subtests. Using a taskcluster transform, taskcluster will automatically build out one raptor-browsertime tp6 desktop test job for every URL/subtest. When hovering over the 'tp6-N' jobs you will see the URL/subtest name is appended to the task description.
When pushing to try locally using ‘mach try fuzzy’ (or chooser) each test URL will be listed separately so you can choose individual pages to push to try, including warm page-load or cold page-load, firefox or chrome, etc.
When running './mach raptor' locally you still use the same format but with the shorter subtest names, i.e. ‘./mach raptor --browsertime --test amazon’. In order to differentiate between warm and cold page-load tests, since using the same subtest names for both, this patch also adds a new --cold’ command line argument. When running locally you would simply use ‘./mach raptor --browsertime --test amazon --cold’.
|
|
Assignee | |
Comment 11•5 years ago
|
||
|
|
Assignee | |
Comment 12•5 years ago
|
||
|
|
Assignee | |
Comment 13•5 years ago
|
||
Hi :ahal, :jlorenzo, if you could add this patch to your review queue please that'd be great! I think in phabricator it fell off the 'must review list' because :sparky reviewed (thanks :sparky) and I made minor updates since then. No rush but when you get chance that'd be great thanks!
|
|
||
Comment 14•5 years ago
|
||
Review done at https://phabricator.services.mozilla.com/D52185#1663222 🙂
|
|
Assignee | |
Comment 15•5 years ago
|
||
Thanks for your reviews Andrew and Johan, appreciated!
|
|
Assignee | |
Comment 16•5 years ago
|
||
|
|
Assignee | |
Comment 17•5 years ago
|
||
|
|
Assignee | |
Comment 18•4 years ago
|
||
|
||
Comment 19•4 years ago
|
||
Pushed by rwood@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/ebc95c785899 Use a single test INI and taskcluster configuration file for all raptor-browsertime desktop page-load tests r=perftest-reviewers,ahal,sparky,jlorenzo
|
||
Comment 20•4 years ago
|
||
| bugherder | ||
Description
•