1673891 - 3.8 - 4.34% booking ContentfulSpeedIndex / booking SpeedIndex (android-hw-g5-7-0-arm7-api-16-shippable) regression on push aba7c7caae105cc5ace3c8a60293783b2257fdc8 (Tue October 20 2020)

Status: RESOLVED WORKSFORME

(Testing :: Performance, defect, P3)

Description

[Florin Strugariu [:Bebe]]

Perfherder has detected a browsertime performance regression from push aba7c7caae105cc5ace3c8a60293783b2257fdc8. As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

Ratio	Suite	Test	Platform	Options	Absolute values (old vs new)
4%	booking	SpeedIndex	android-hw-g5-7-0-arm7-api-16-shippable	warm	992.42 -> 1,035.50
4%	booking	ContentfulSpeedIndex	android-hw-g5-7-0-arm7-api-16-shippable	warm	771.25 -> 800.83
4%	booking	SpeedIndex	android-hw-g5-7-0-arm7-api-16-shippable	warm	993.46 -> 1,031.25

Details of the alert can be found in the alert summary, including links to graphs and comparisons for each of the affected tests. Please follow our guide to handling regression bugs and let us know your plans within 3 business days, or the offending patch(es) will be backed out in accordance with our regression policy.

For more information on performance sheriffing please see our FAQ.

Comment 1

[Julian Descottes [:jdescottes]]

Hi! Thanks for the heads up, butI can't see any reason why my patch would regress android performance.
I removed dead code, which is most likely not even evaluated in those android tests (unless USB debugging is turned on for them).

Looking at the graph for the tests mentioned here, they seem to be very flaky: eg https://treeherder.mozilla.org/perf.html#/graphs?highlightAlerts=1&series=autoland,2701483,1,13&timerange=1209600&zoom=1603160057388,1603283306121,836.7769607843138,1203.3962418300655

Looking at the alert,booking SpeedIndex android-hw-g5-7-0-arm7-api-16-shippable opt warm moved from 993 to 1035.
Right after my patches, we can see other pushes which go above 1035 (eg 1063 for https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=c264e34897e39b5d748084387041ddd9160bb52d&tochange=cb586468ae3643c6d130f5168f7c26816b946b72 ) as well as below 993 (988 for https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=ddedbce9ad9f49965c061252ee77f5ea981b716d&tochange=ac382cc4a6e069a6121297a6e8952650e8cf0807)

Also my patch was backed out with https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=30b04bd63bb855746a472877754903c51dd026b4&tochange=31a89e69110e09b89bafbc90c9a2f2702a153566 which scores even higher at 1043ms. I relanded it a couple of hours later but still. If my patch was responsible for the regression, then the push with the backout should have recovered the issue and it did not.

In my opinion this means that the variance for this test is too high to decide that Bug 1671676 regressed anything here.
Now there seems to be some kind of regression for this test around the time my patch landed, but I would look at earlier pushes.

Maybe https://bugzilla.mozilla.org/show_bug.cgi?id=1646810 & https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=421ef74443011799bfb10c7f58138ad3709e34e5&tochange=f087af9e75e69519555b6b2e7a4601e31558f895 ?
Push date is Tue, 20 Oct 2020 16:32:00 +0000 , which is 3 minutes before my patch, and this one is actually about mobile.

Forwarding the ni? to :agi

Comment 2

[[ex-Mozilla] Agi Sferro | :agi]

I agree this looks like just noise to me.

Comment 3

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1671676

Comment 4

[Julian Descottes [:jdescottes]]

Based on what I explained in comment 1, I am updating the regression field and moving the bug back to Testing. I still think Bug 1646810 could be linked to the increased noise level in the bug, but someone would need to look into that in more details.

Comment 5

[[ex-Mozilla] Agi Sferro | :agi]

Ah sorry I missed the mention about Bug 1646810. Note that we landed a perf fix for that: https://bugzilla.mozilla.org/show_bug.cgi?id=1673316

Comment 6

[Alexandru Ionescu (needinfo me) [:alexandrui]]

The culprit is already set up correctly. No action needed here.

Comment 7

[Ryan VanderMeulen [:RyanVM]]

AFAICT these tests aren't running anymore. Is there a different one we should be looking at now? Also, I'm a bit confused about this regression to begin with. There was clearly a change around 14-Oct before bug 1646810 landed, but other than a momentary bit of noise around the time of this bug being filed, it appears that all the remain results from 21-Oct on were in the same range as before this was filed?

Is there anything actionable to do here still?

Comment 8

[Dave Hunt [:davehunt] [he/him] ⌚BST]

The tests are still running, but were changed to run with WebRender enabled, and then with conditioned profiles disabled. You can see the transitions in this graph. Looking at the results, I agree with comment 7 in that there's not really any sustained or actionable regression here. It may have been fixed, or perhaps a temporary infrastructure issue.