1683647 - 3.84 - 13.74% raptor-tp6-instagram-firefox-cold / raptor-tp6-instagram-firefox-cold loadtime (windows10-64-shippable-qr) regression on push 6307b67c66ffa34fdbdb8481cc95f58a5cc8df29 (Fri December 18 2020)

Status: RESOLVED WONTFIX

(Core :: Layout, defect)

Description

[Florin Strugariu [:Bebe]]

Perfherder has detected a raptor performance regression from push 6307b67c66ffa34fdbdb8481cc95f58a5cc8df29. As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

Ratio	Suite	Test	Platform	Options	Absolute values (old vs new)
14%	raptor-tp6-instagram-firefox-cold	loadtime	windows10-64-shippable-qr	webrender	1,305.71 -> 1,485.17
4%	raptor-tp6-instagram-firefox-cold		windows10-64-shippable-qr	webrender	400.40 -> 415.78

Improvements:

Ratio	Suite	Test	Platform	Options	Absolute values (old vs new)
39%	raptor-tp6-cnn-ampstories-firefox-cold	loadtime	linux64-shippable-qr	live nocondprof webrender	1,876.17 -> 1,135.50

Details of the alert can be found in the alert summary, including links to graphs and comparisons for each of the affected tests. Please follow our guide to handling regression bugs and let us know your plans within 3 business days, or the offending patch(es) will be backed out in accordance with our regression policy.

For more information on performance sheriffing please see our FAQ.

Comment 1

[Florin Strugariu [:Bebe]]

:gw one of the 3 flagged bugs has generated this regression. Can you take a look pleasse

Comment 2

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

This should be a regression from bug 1682365 part 2 - it landed each time directly before the regression. Bug 1682473 cannot be the regressor as it didn't modify the application but only update test expectations (haven't checked the other bugs).

Comment 3

[Glenn Watson [:gw]]

Yep, it looks like it's the patch that Aryx identified above. I'll investigate today / tomorrow to see what's causing it, I suspect it might be caused by a different set of shaders being used on initial pageload than previously. It's interesting in that perf alert that there is one pageload test that's a dramatic improvement too.

Comment 4

[Glenn Watson [:gw]]

I was able to confirm that the set of shaders compiled is different with the new patch applied.

However, with the new patch, WR compiles 18 shaders, while it compiles 20 shaders previously for this scenario. So if anything, I'd expect the page load to be faster.

I think there might be some weird interactions going on related to which shaders are getting cached to disk, which would also explain why it only occurs on Windows.

I'm currently running some try experiments that change how / when WR decides which shaders to cache to see if they make a difference.

Comment 5

[Florin Strugariu [:Bebe]]

Thanks Aryx for correctly reassigning/identifying this alert

Comment 6

[Florin Strugariu [:Bebe]]

Also some alerts from browsertime:
== Change summary for alert #28212 (as of Wed, 23 Dec 2020 09:20:55 GMT) ==

Regressions:

Ratio	Suite	Test	Platform	Options	Absolute values (old vs new)
2%	instagram	SpeedIndex	android-hw-p2-8-0-android-aarch64-shippable	nocondprof warm webrender	917.58 -> 940.08
2%	instagram	SpeedIndex	android-hw-p2-8-0-android-aarch64-shippable	nocondprof warm webrender	920.12 -> 942.00

For up to date results, see: https://treeherder.mozilla.org/perfherder/alerts?id=28212

Comment 7

[BugBot [:suhaib / :marco]]

Set release status flags based on info from the regressing bug 1682365

Comment 8

[Glenn Watson [:gw]]

I'm back from PTO today so will resume looking into this. I've kicked off a number of raptor try runs to try and identify what the differences are here.

Comment 9

[Glenn Watson [:gw]]

No luck finding anything specific on try runs. The regression on android points towards my original thought though - since Windows and Android are the only platforms that make use of the shader compilation cache.

I'm now in the process of setting up a local Windows environment that I will try to repro directly on.

Comment 10

[Glenn Watson [:gw]]

Looked into this some more. The shader cache doesn't appear to be involved here (at least, on my local machine the shader cache isn't active, not sure why yet). Also verified that with the new patch we compile fewer shaders, so we're doing less overall work. My current theory is that although less work is done, it might be starting later than previously, which might account for the regression. Doing some profile try runs to check that idea out.

Comment 11

[Glenn Watson [:gw]]

I've spent more time looking into this today, and consulted with jrmuizel and mstange on a number of profile runs.

We don't have a perfect explanation for this, but what we do know is:

• The CPU time spent in the WR renderer thread is significantly better with this patch (fewer shaders are compiled, resulting in ~100-200ms less time spent in the render thread).

• In all the profiles we looked at, it appears that the regression in the metrics comes from the network requests completing later than they do before the patch.

These seem unrelated.

A plausible theory is that doing less work in the WR renderer thread results in OS thread scheduling differences causing the network thread(s) to begin or complete later than they previously were, but I'm not sure how to prove this.

Since the patch in question is a definite improvement to WR startup times, and also to rendering times in general, I think we should probably just accept this metric regression.

Of note, there was also a 39% improvement in the load time of another raptor test, which might also support the idea that this is due to random thread scheduling.

Does that seem reasonable?

Comment 12

[Jeff Muizelaar [:jrmuizel]]

It does.

Comment 13

[Markus Stange [:mstange]]

I agree, let's not lose more sleep over this. Instead we should focus on bug 1624331, so that we don't compile shaders during page load tests in the first place.