1592921 - 3.73 - 5.4% raptor-tp6-imgur-firefox / raptor-tp6-imgur-firefox fcp / raptor-tp6-imgur-firefox loadtime (linux64-shippable-qr) regression on push 1bbcb17120eb305e78bd3be7807440b8f436ee6c (Sat October 26 2019)

Status: RESOLVED WORKSFORME

(Toolkit :: Video/Audio Controls, defect, P2)

Description

[Florin Strugariu [:Bebe]]

Raptor has detected a Firefox performance regression from push:

https://hg.mozilla.org/integration/autoland/pushloghtml?changeset=1bbcb17120eb305e78bd3be7807440b8f436ee6c

As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

5% raptor-tp6-imgur-firefox linux64-shippable-qr opt 291.86 -> 307.62
5% raptor-tp6-imgur-firefox fcp linux64-shippable-qr opt 281.75 -> 296.83
5% raptor-tp6-imgur-firefox linux64-shippable-qr opt 291.80 -> 306.61
4% raptor-tp6-imgur-firefox loadtime linux64-shippable-qr opt 408.56 -> 423.79

Improvements:

38% raptor-tp6m-cnn-geckoview-cold loadtime android-hw-g5-7-0-arm7-api-16 pgo 14,550.25 -> 8,994.58
10% raptor-tp6m-cnn-geckoview-cold android-hw-g5-7-0-arm7-api-16 pgo 4,624.51 -> 4,148.08

You can find links to graphs and comparison views for each of the above tests at: https://treeherder.mozilla.org/perf.html#/alerts?id=23637

On the page above you can see an alert for each affected platform as well as a link to a graph showing the history of scores for this test. There is also a link to a Treeherder page showing the Raptor jobs in a pushlog format.

To learn more about the regressing test(s) or reproducing them, please see: https://wiki.mozilla.org/TestEngineering/Performance/Raptor

*** Please let us know your plans within 3 business days, or the offending patch(es) will be backed out! ***

Our wiki page outlines the common responses and expectations: https://wiki.mozilla.org/TestEngineering/Performance/Talos/RegressionBugsHandling

Comment 1

[Mike Conley (:mconley) (:⚙️)]

*** Please let us know your plans within 3 business days, or the offending patch(es) will be backed out! ***

The feature that was enabled is Nightly-only for now. Please do not back me out.

Comment 2

[Mike Conley (:mconley) (:⚙️)]

I pushed to try to compare current mozilla-central (with PiP enabled for Linux) against a patch with PiP disabled.

https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=d34e5254283463a8bc53c29784535b4aceb6d024&newProject=try&newRevision=da27f4823476ba3c2d178ccafade3afc6c33ceed&framework=10

No change to imgur raptor performance there. In fact, if you take a look at each of the graphs for the regressed subtests:

• https://treeherder.mozilla.org/perf.html#/graphs?timerange=2592000&series=autoland,1927374,1,10&series=mozilla-inbound,673353d51df434a2bac3e729567b3c2da2162f77,1,10
• https://treeherder.mozilla.org/perf.html#/graphs?timerange=2592000&series=autoland,1927372,1,10&series=mozilla-inbound,5eeb0de097840a11f57bc6b04f28a5ee537d6ff1,1,10
• https://treeherder.mozilla.org/perf.html#/graphs?timerange=2592000&series=autoland,1978132,1,10&series=mozilla-inbound,66eaaf68267964ea2f36ba1fa83d039157230cc4,1,10
• https://treeherder.mozilla.org/perf.html#/graphs?timerange=2592000&series=autoland,1927372,1,10&series=mozilla-inbound,5eeb0de097840a11f57bc6b04f28a5ee537d6ff1,1,10

You can see them all dip back down again around November 1st.

So, conclusion:

1. Disabling PiP doesn't cause an improvement
2. Original score on the regressing graphs has been re-achieved since approximately Nov 1st

So I think the regression is gone. Closing as WFM.