1624331 - Make sure that profile conditioning exercises WebRender shaders, so that cold page load rendering will use cached shaders

Status: NEW

(Testing :: Performance, enhancement, P3)

Description

[Markus Stange [:mstange]]

WebRender has some first-run overhead: Different visual elements (text, borders, background colors, gradients, filter effects, transforms) require various shaders for rendering, and shaders have to be compiled on the device before they can be used. For complicated web pages, this compilation can take 50-250ms. After compilation, the compiled form is cached in the profile directory, keyed by Gecko build ID, OpenGL renderer name, and GPU device ID. Subsequent uses of a cached shader, even after browser restarts, are fast.

WebRender only compiles (and caches) shaders once it encounters them on a web page. There is no shader warm-up during first run.
At the moment, this means that some shaders are encountered for the first time when the cold page load test is run, so the page load numbers are skewed by the first-time compilation of these shaders.

In reality, when a user is doing a cold page load of some website, odds are that the user has encountered similar visual elements before on other websites, and that the necessary shaders have already been compiled and cached to the user's profile directory. That means that in practice, cold page loads usually don't hit the first-time shader compilation overhead.
We should make our cold page load performance tests closer to reality by triggering compilation and caching of WebRender shaders during profile conditioning. We can do this by loading an HTML file that is constructed in such a way that it exercises all WebRender shaders.
Because of the device-specific format of the cached shaders, this requires profile conditioning to be run on the same type of device that also runs the actual tests.

Comment 1

[Markus Stange [:mstange]]

(In reply to Markus Stange [:mstange] from comment #0)

We can do this by loading an HTML file that is constructed in such a way that it exercises all WebRender shaders.

I'm going to try to create such an HTML file in bug 1624335.

Comment 2

[Greg Mierzwinski [:sparky]]

So that all makes sense and sounds good. I agree with how you say the cold page-load numbers are skewed, but isn't this something we want to know? If we do this profile conditioning, we'd lose all measurements that contain this extra processing, and we would no longer be checking its performance (at least for regression detection).

Since users will definitely hit this code, I think it's important for us to test it in some way rather than completely ignoring it so I'm against changing our current cold page loads to use this profile, but I think it would be fine if we add some new tests that use this new profile. Or maybe we could do it the other way around: change all current cold page load tests to use this, and add new tests that exercise this caching/setup stage. As long as we have at least a few sheriffed tests that don't use this profile, we should be fine.

Comment 3

[Markus Stange [:mstange]]

I agree that it would be nice to know if first-time compilation time regresses. However, isn't the same true for other things that are done during profile conditioning? Do we have tests that, for example, test how long it takes to create and initialize a new profile? Or a test that checks how long startup takes if no JSM files are cached in the JSM startup cache?

Comment 4

[Greg Mierzwinski [:sparky]]

(In reply to Markus Stange [:mstange] from comment #3)

I agree that it would be nice to know if first-time compilation time regresses. However, isn't the same true for other things that are done during profile conditioning?

At the moment we use this profile for cold/warm page-loads: https://dxr.mozilla.org/mozilla-central/source/testing/condprofile/condprof/scenarii/settled.py
So there is no conditioning that is in use in CI that opens web pages.

(In reply to Markus Stange [:mstange] from comment #3)

Do we have tests that, for example, test how long it takes to create and initialize a new profile? Or a test that checks how long startup takes if no JSM files are cached in the JSM startup cache?

Talos xperf might be the closest test we have for this, but it barely addresses it - it's more for resource usage during startup. We should add the tests you mention to ./mach perftest script when it's available.

I think that as long as we have a 3-4 pageload tests that continue using the settled profile rather than this compiled-n-cached profile, then we'll have it partially covered until mach perftest is available where we could easily add all those special tests.

Comment 5

[Jamie Nicol [:jnicol]]

webrender has an option to compile all shaders at startup. it might be easier to expose this as a pref, rather than making (and importantly, maintaining) a page which we hope will cause every shader to be compiled.

Comment 6

[Markus Stange [:mstange]]

(In reply to Greg Mierzwinski [:sparky] from comment #4)

(In reply to Markus Stange [:mstange] from comment #3)

I agree that it would be nice to know if first-time compilation time regresses. However, isn't the same true for other things that are done during profile conditioning?

At the moment we use this profile for cold/warm page-loads: https://dxr.mozilla.org/mozilla-central/source/testing/condprofile/condprof/scenarii/settled.py

I see, it just waits 30 seconds and doesn't do anything else.

In other words, if WebRender started compiling and caching its shaders within those 30 seconds, nothing would need to be done in this bug, and the "settled" profile would be equivalent to the "compiled-and-cached" profile. I've proposed doing this in bug 1624051 and can look into that a bit.

(In reply to Jamie Nicol [:jnicol] from comment #5)

webrender has an option to compile all shaders at startup. it might be easier to expose this as a pref, rather than making (and importantly, maintaining) a page which we hope will cause every shader to be compiled.

Good point about the maintenance.

Comment 7

[Greg Mierzwinski [:sparky]]

:mstange, sounds good. By the way, :tarek can help you out with adding this new conditioned profile when the time comes.

Comment 8

[Andrew Creskey [:acreskey]]

We've also used gfx.webrender.precache-shaders (Bug 1643832) in some locals performance tests to compile them at startup.