1638132 - Extend WebRender visual metrics pageload tests [Desktop]

Status: RESOLVED FIXED

(Testing :: Performance, enhancement, P3)

Description

[Andrew Creskey [:acreskey]]

Opening this bug to discuss extending our visual metrics pageload tests to explicitly include WebRender on Desktop.

WebRender is performance-impacting feature and it would be useful to quantify this in various pageload scenarios.

There are a few existing pageload tests, -qr, which I believe enable WebRender. However they are not running and also broader coverage of sites would be desired.

test-vismet-windows10-64-shippable-qr/opt-browsertime-tp6-firefox-cold-amazon
test-vismet-windows10-64-shippable-qr/opt-browsertime-tp6-firefox-amazon
test-vismet-linux64-shippable-qr/opt-browsertime-tp6-firefox-amazon

Comment 1

[Jeff Muizelaar [:jrmuizel]]

Yes please.

Comment 2

[Greg Mierzwinski [:sparky]]

:davehunt, do you know who could work on this? We would need them to start by creating desktop browsertime tasks for these and then determining where they should run (if at all).

Comment 3

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Can we define what it is we'd like before we get this assigned? Do we want this running in CI or just locally? If CI, what platforms/sites would we like to have running? Do we need to add these results to health.graphics or arewefastyet.com?

It makes sense for the perftest team to take this, and we can schedule it for our July sprint. If it's needed earlier we can either drop something from June or provide support for someone outside of the perftest team to work on it.

Comment 4

[Greg Mierzwinski [:sparky]]

:acreskey, can you address :davehunt's questions?

Comment 5

[Andrew Creskey [:acreskey]]

(In reply to Dave Hunt [:davehunt] [he/him] ⌚BST from comment #3)

Can we define what it is we'd like before we get this assigned? Do we want this running in CI or just locally? If CI, what platforms/sites would we like to have running? Do we need to add these results to health.graphics or arewefastyet.com?

It makes sense for the perftest team to take this, and we can schedule it for our July sprint. If it's needed earlier we can either drop something from June or provide support for someone outside of the perftest team to work on it.

I don't have a complete plan for what's needed.

For local testing we can already remove the preferences that disable WebRender and do one-off testing.
https://github.com/sitespeedio/browsertime/blob/60cfa96f81c30a9f7d68440bdaa024e5384ae6a7/lib/firefox/webdriver/firefoxPreferences.js#L52

So this bug is about adding some to-be-determined set of Desktop visual metrics pageload tests with WebRender enabled in CI.

As I understand it, visual metric tests for desktop are quite limited at the moment.

In terms of the scope of pages and where the results would be visible, I think we would need to have a meeting and get some people together to discuss.
I'm personally focused on Fenix performance at the moment so while I'm happy to share my thoughts, I won't be able to drive this effort.

It might be useful to learn from how this work is progress on Android, Bug 1623421.

Comment 6

[Greg Mierzwinski [:sparky]]

Sorry, I forgot to add a priority to this bug - we've discussed it at out meetings already.

Comment 7

[Dave Hunt [:davehunt] [he/him] ⌚BST]

@acreskey my understanding is that we've been running visual page load tests with WebRender on desktop for some time. Is there anything left to do here or can we mark it as resolved?

Comment 8

[Andrew Creskey [:acreskey]]

Dave -- yes, and we're running them on Android as well.
I can't think of anything else that needs to be done here.