1600256 - 10.05 - 42.43% raptor-tp6-yahoo-mail-firefox-cold / raptor-tp6-yahoo-mail-firefox-cold loadtime (windows10-64-shippable-qr) regression on push 1bcbb2e808a7ff611a33bc336bc3f2a8a514f2f8 (Thu November 28 2019)

Status: RESOLVED WONTFIX

(Core :: Graphics: WebRender, defect)

Description

[Florin Strugariu [:Bebe]]

Raptor has detected a Firefox performance regression from push:

https://hg.mozilla.org/integration/autoland/pushloghtml?changeset=1bcbb2e808a7ff611a33bc336bc3f2a8a514f2f8

As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

42% raptor-tp6-yahoo-mail-firefox-cold loadtime windows10-64-shippable-qr opt 967.08 -> 1,377.42
35% raptor-tp6-yahoo-mail-firefox-cold loadtime windows10-64-shippable-qr opt 1,021.58 -> 1,376.17
10% raptor-tp6-yahoo-mail-firefox-cold windows10-64-shippable-qr opt 531.20 -> 584.60

You can find links to graphs and comparison views for each of the above tests at: https://treeherder.mozilla.org/perf.html#/alerts?id=24176

On the page above you can see an alert for each affected platform as well as a link to a graph showing the history of scores for this test. There is also a link to a Treeherder page showing the Raptor jobs in a pushlog format.

To learn more about the regressing test(s) or reproducing them, please see: https://wiki.mozilla.org/TestEngineering/Performance/Raptor

*** Please let us know your plans within 3 business days, or the offending patch(es) will be backed out! ***

Our wiki page outlines the common responses and expectations: https://wiki.mozilla.org/TestEngineering/Performance/Talos/RegressionBugsHandling

Comment 1

[Jim Mathies [:jimm]]

Hey Jessie, can we get some help moving this along. Seems like a bad 72 regression.

Comment 2

[Jessie [:jbonisteel] pls NI]

Will follow up about this when Nical is back from PTO tomorrow so we can update bug with next steps

Comment 3

[Nicolas Silva [:nical]]

I am looking into this. I just haven't figured out yet what makes the shader compilation times regress in that configuration.

Comment 4

[Nicolas Silva [:nical]]

If this gets backed out, https://hg.mozilla.org/integration/autoland/rev/35d6558a2145 should get backed out in the process, in order to apply the backout cleanly since it's on top of the same code (it won't add a regression).

I got far enough in my investigation to know that the shader compile time regression does not affect my windows laptop. Shader compilation times for some of some of the big shaders like alpha variant of brush_image and ps_text_run are in the hundreds of ms (both before and after the regressing patches) which is pretty bad so it would be nice to improve it overall.

From some online searches, register allocation is where the D3D shader compiler spends most of its time, and the more nesting there is (functions, loops, branches) the worst it gets. Also textureLods instead of texture that avoid depending on partial derivatives (And the regression almost certainly in the vertex shader anyway) help a lot but I think that we already try hard to do textureLods wherever we can.

Comment 5

[Julien Cristau [:jcristau]]

Sebastian, let's get bug 1594128 backed out of beta for 72.0b4 please.

Comment 6

[Nicolas Silva [:nical]]

Is there something special to know in order to get raptor-tp6-yahoo-mail-firefox-cold running on try? I've tried a few try queries like raptor cold windows shippable-qr but it only runs amazon-related tests (which don't seem to be affected?) and no yahoo.

Comment 7

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Backout of bug 1594128 from beta pushed: https://hg.mozilla.org/releases/mozilla-beta/rev/621bd2532945

Robert, can you guide Nicolas how to run the tp6 yahoo test from comment 0 on central, please?

Comment 8

[Robert Wood [:rwood]]

(In reply to Sebastian Hengst [:aryx] [limited availability until start of 2020] (needinfo on intermittent or backout) from comment #7)

Backout of bug 1594128 from beta pushed: https://hg.mozilla.org/releases/mozilla-beta/rev/621bd2532945

Robert, can you guide Nicolas how to run the tp6 yahoo test from comment 0 on central, please?

Please see the wiki, that should get you going, thanks! Any further questions feel free to ask in slack #perftest.

https://wiki.mozilla.org/TestEngineering/Performance/Raptor#Running_Locally_2

Comment 9

[Liz Henry (:lizzard) (relman/hg->git project)]

Nical, are you still looking at this? We may need to backout from 73 beta as well if not.

Comment 10

[Nicolas Silva [:nical]]

I am looking at this, although i'm not making much progress. I don't think that it is worth backing out because it affects a very specific scenario: cold startup on a certain page (yahoo-related tests only on the CI) on a certain hardware/driver configuration. We landed stuff on top of it so in my opinion avoiding the regression is not worth the risk of backing out or reverting some changes manually on beta.
I've been comparing cold startups on a number of pages with and without the patches and it doesn't make a difference on the hardware I have.

Comment 11

[Ryan VanderMeulen [:RyanVM]]

It looks to me like things have gone back to normal now (in fact, it appears to now be favoring the lower bound of the bimodal distribution when it was favoring the upper bound when this regression was reported). Also, Beta doesn't show any obvious regression when Gecko 73 was merged to it a week ago:
https://treeherder.mozilla.org/perf.html#/graphs?highlightAlerts=1&series=autoland,2134905,1,10&series=mozilla-beta,2141074,1,10&timerange=5184000
https://treeherder.mozilla.org/perf.html#/graphs?highlightAlerts=1&series=autoland,2134909,1,10&series=mozilla-beta,2141085,1,10&timerange=5184000

Florin, do you agree with my analysis above? Is this bug worth keeping open at this point?

Comment 12

[Florin Strugariu [:Bebe]]

:RyanVM I agree with you the graph looks like it got stable and its at the lower value of the bimodal.

If there is nothing else here we can mack it as WORKSFORME

Comment 13

[Ryan VanderMeulen [:RyanVM]]

Thanks, glad to see this went away.

Comment 14

[Dave Hunt [:davehunt] [he/him] ⌚BST]

I don't think regression bugs raised via alerts should be resolved as WORKSFORME. I'm going to mark this as a regression that was accepted (WONTFIX), which was later reversed by an unidentified series of commits.