1564835 - 5.29 - 25.11% raptor-tp6-google-firefox-cold / raptor-tp6-google-firefox-cold fcp (windows10-64-shippable-qr) regression on push 1a4dad7c74164cbd979b261f1e3f6a02b499d425 (Mon July 8 2019)

Status: RESOLVED FIXED

(Testing :: Raptor, defect, P3)

Description

[Florin Strugariu [:Bebe]]

Raptor has detected a Firefox performance regression from push:

https://hg.mozilla.org/integration/autoland/pushloghtml?changeset=1a4dad7c74164cbd979b261f1e3f6a02b499d425

As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

25% raptor-tp6-google-firefox-cold fcp windows10-64-shippable-qr opt 295.88 -> 370.17
23% raptor-tp6-google-firefox-cold fcp windows10-64-shippable-qr opt 295.58 -> 364.33
5% raptor-tp6-google-firefox-cold windows10-64-shippable-qr opt 317.30 -> 334.08

You can find links to graphs and comparison views for each of the above tests at: https://treeherder.mozilla.org/perf.html#/alerts?id=21794

On the page above you can see an alert for each affected platform as well as a link to a graph showing the history of scores for this test. There is also a link to a Treeherder page showing the Raptor jobs in a pushlog format.

To learn more about the regressing test(s) or reproducing them, please see: https://wiki.mozilla.org/Performance_sheriffing/Raptor

*** Please let us know your plans within 3 business days, or the offending patch(es) will be backed out! ***

Our wiki page outlines the common responses and expectations: https://wiki.mozilla.org/Performance_sheriffing/Talos/RegressionBugsHandling

Comment 1

[Florin Strugariu [:Bebe]]

Are we expecting this regression?

Comment 2

[Robert Wood [:rwood]]

I don't see how that could be related at all, the patch just shuts down the browser a bit faster than before (removed a 15s wait).

Comment 3

[Dave Hunt [:davehunt] [he/him] ⌚BST]

This is a significant regression. I see that the graphs have since dropped back to previous levels. Could it be this was actually caused by bug 1557282, which was identified as a performance regression and fixed in bug 1567236?

Comment 4

[Dave Hunt [:davehunt] [he/him] ⌚BST]

I performed some additional retriggers, and this definitely seems to be the cause of the regression. Can we push to try with this change backed out, to see if it affects the results? As mentioned in comment 3 the results appear to have come back down since this regression, so it could be that this commit interacted poorly with something else that has since been addressed.

Comment 5

[Florin Strugariu [:Bebe]]

Running a try build with the manual backout of the bug. (hg oops -er <rev> returned errors)

Base (Central):
https://treeherder.mozilla.org/#/jobs?repo=try&revision=1b94a271f53f106688a13e931da23e22e45dba62
Backout:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=ef67fb3cf75141ccea86009e8cf9ba8ac40d7e6b

Compare:
https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=1b94a271f53f106688a13e931da23e22e45dba62&newProject=try&newRevision=ef67fb3cf75141ccea86009e8cf9ba8ac40d7e6b&framework=10

Comment 6

[Florin Strugariu [:Bebe]]

From the compare view it looks like Dave's suggestion was true.
The commit interacted poorly with something else that has since been addressed.

Reverting the patch has not shown any improvement in performance.