Closed Bug 1590561 Opened 3 months ago Closed 2 months ago

Use a single test INI and taskcluster configuration file for all raptor-browsertime desktop page-load tests

Categories

(Testing :: Raptor, enhancement, P1)

Version 3
enhancement

Tracking

(firefox73 fixed)

RESOLVED FIXED
mozilla73
Tracking Status
firefox73 --- fixed

People

(Reporter: rwood, Assigned: rwood)

References

Details

Attachments

(1 file)

As part of the raptor-browsertime test INI and taskcluster configs refactor, use a single test INI specification file for all raptor-browsertime desktop page-load tests.

Also have a single corresponding taskcluster browsertime-desktop.yml configuration file which will contain the settings for all raptor-browsertime desktop page-load tests, with a single entry for 'tp6-desktop'. Use taskcluster transforms to auto-generate one taskcluster raptor-browsertime test job per each test-url to be included in the tp6-desktop test.

Summary: Use a single test INI specification file for all raptor-browsertime tp6 desktop page-load tests → Use a single test INI and taskcluster configuration file for all raptor-browsertime desktop page-load tests

Put together a potential solution and pushed to try:

https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=d27104684415ef9b2d8d359b10e62f01b4603ef3

https://hg.mozilla.org/try/rev/c2ae4b1c5b3d6d303eda28dab4803f0059c4b742

:ahal, :jlorenzo - any thoughts and advice on the above approach? Thanks!

Flags: needinfo?(jlorenzo)
Flags: needinfo?(ahal)

I'm no expert in the way tests are defined in mozilla-central, but this solution looks sensible to me.

I thought we had a nice transform to easily chunk jobs, but it turns out we have many implementations across taskgraph. Therefore, I'm fine not factorizing this piece just yet. Taskgraph has grown organically in order to finish the Taskcluster migration last year, and we can clean this up later.

Flags: needinfo?(jlorenzo)

Yeah, that looks reasonable to me too.

This is unrelated to this issue and there's probably a good answer to this question.. but why do we have to run these each in a separate task? Why not have a single tp6 task that runs all of the pageloads in serial (restarting the browser in-between of course).

Flags: needinfo?(ahal)

Thanks for the feedback guys!

(In reply to Andrew Halberstadt [:ahal] from comment #3)

This is unrelated to this issue and there's probably a good answer to this question.. but why do we have to run these each in a separate task? Why not have a single tp6 task that runs all of the pageloads in serial (restarting the browser in-between of course).

That would be super nice but we split them out mainly because even just one URL cold page-load job takes around 25 min on Windows. We do 25 browser-cycles (or 25 page-cycles on warm page load). Also just easier for automated perf alerts and for sheriffs bisecting/retriggering etc.

(In reply to Robert Wood [:rwood] from comment #4)

That would be super nice but we split them out mainly because even just one URL cold page-load job takes around 25 min on Windows. We do 25 browser-cycles (or 25 page-cycles on warm page load). Also just easier for automated perf alerts and for sheriffs bisecting/retriggering etc.

Ah, yes that is a good reason to keep them separate :)

This patch will allow us to have a single raptor test INI and corresponding taskcluster config entry for all raptor-browsertime tp6 desktop tests. In the tp6-desktop.ini file all the URLs to be measured are listed as subtests below the default test settings, with a simple name (i.e. ‘amazon’). In the raptor.yml taskcluster config new 'browsertime-tp6-desktop' entry, there is a ‘raptor-subtests’ key with a list of matching URLs/subtests. Using a taskcluster transform, taskcluster will automatically build out one raptor-browsertime tp6 desktop test job for every URL/subtest. When hovering over the 'tp6-N' jobs you will see the URL/subtest name is appended to the task description.

When pushing to try locally using ‘mach try fuzzy’ (or chooser) each test URL will be listed separately so you can choose individual pages to push to try, including warm page-load or cold page-load, firefox or chrome, etc.

When running './mach raptor' locally you still use the same format but with the shorter subtest names, i.e. ‘./mach raptor --browsertime --test amazon’. In order to differentiate between warm and cold page-load tests, since using the same subtest names for both, this patch also adds a new --cold’ command line argument. When running locally you would simply use ‘./mach raptor --browsertime --test amazon --cold’.

Hi :ahal, :jlorenzo, if you could add this patch to your review queue please that'd be great! I think in phabricator it fell off the 'must review list' because :sparky reviewed (thanks :sparky) and I made minor updates since then. No rush but when you get chance that'd be great thanks!

Flags: needinfo?(jlorenzo)
Flags: needinfo?(ahal)
Flags: needinfo?(jlorenzo)

Thanks for your reviews Andrew and Johan, appreciated!

Flags: needinfo?(ahal)
Blocks: 1600392
Pushed by rwood@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/ebc95c785899
Use a single test INI and taskcluster configuration file for all raptor-browsertime desktop page-load tests r=perftest-reviewers,ahal,sparky,jlorenzo
Status: ASSIGNED → RESOLVED
Closed: 2 months ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla73
Regressions: 1601970
See Also: → 1605121
You need to log in before you can comment on or make changes to this bug.