1904643 - 14 - 9.78% amazon ContentfulSpeedIndex / amazon SpeedIndex + 3 more (Android) regression on Thu June 13 2024

Status: RESOLVED DUPLICATE of bug 1909842

(Core :: Networking, defect, P2)

Description

[Florin Bilt]

Perfherder has detected a browsertime performance regression from push 236df657bbf367aee05c08d7ecbcc0a348e46470. As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

Ratio	Test	Platform	Options	Absolute values (old vs new)	Performance Profiles
14%	amazon ContentfulSpeedIndex	android-hw-a51-11-0-aarch64-shippable-qr	warm webrender	517.63 -> 590.12	Before/After
13%	amazon PerceptualSpeedIndex	android-hw-a51-11-0-aarch64-shippable-qr	warm webrender	466.85 -> 525.72	Before/After
12%	amazon ContentfulSpeedIndex	android-hw-a51-11-0-aarch64-shippable-qr	warm webrender	536.30 -> 598.91	Before/After
10%	amazon LastVisualChange	android-hw-a51-11-0-aarch64-shippable-qr	warm webrender	1,571.75 -> 1,736.22	Before/After
10%	amazon SpeedIndex	android-hw-a51-11-0-aarch64-shippable-qr	warm webrender	483.70 -> 531.01	Before/After

Improvements:

Ratio	Test	Platform	Options	Absolute values (old vs new)	Performance Profiles
50%	sina loadtime	android-hw-a51-11-0-aarch64-shippable-qr	warm webrender	2,798.42 -> 1,391.12	Before/After
43%	amazon loadtime	android-hw-a51-11-0-aarch64-shippable-qr	warm webrender	405.24 -> 230.60	Before/After
32%	amazon loadtime	android-hw-a51-11-0-aarch64-shippable-qr	warm webrender	443.35 -> 302.47	Before/After
23%	ebay-kleinanzeigen-search loadtime	android-hw-a51-11-0-aarch64-shippable-qr	warm webrender	1,572.52 -> 1,206.81	Before/After
18%	ebay-kleinanzeigen loadtime	android-hw-a51-11-0-aarch64-shippable-qr	warm webrender	1,410.87 -> 1,152.70	Before/After
...	...	...	...	...	...
9%	sina SpeedIndex	android-hw-a51-11-0-aarch64-shippable-qr	warm webrender	4,568.72 -> 4,178.79	Before/After

Details of the alert can be found in the alert summary, including links to graphs and comparisons for each of the affected tests. Please follow our guide to handling regression bugs and let us know your plans within 3 business days, or the patch(es) may be backed out in accordance with our regression policy.

If you need the profiling jobs you can trigger them yourself from treeherder job view or ask a sheriff to do that for you.

You can run these tests on try with ./mach try perf --alert 891

For more information on performance sheriffing please see our FAQ.

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1889771

Comment 2

[Andrew Creskey [:acreskey]]

Hi Florin, I'm glad that the CI tests picked up this change.

We're measuring the improvements via telemetry since the CI pageload tests are not an ideal environment for this kind of tuning.
(As you know, all of the socket connections are going through https proxy to the mitm server and there is not network latency nor bandwidth restrictions).

We have actually temporarily backed out this change to validate the improvements that we saw in telemetry here:
https://bugzilla.mozilla.org/show_bug.cgi?id=1889771#c7

But let me take a look at the videos and see what changed in the CI tests.
(Also note the improvements in the ci tests)

Comment 3

[Donal Meehan [:dmeehan]]

Setting Fx129 as Fixed since the regressor Bug 1889771 was backed out of central

Comment 4

[Andrew Creskey [:acreskey]]

(In reply to Florin Bilt from comment #0)

Perfherder has detected a browsertime performance regression from push 236df657bbf367aee05c08d7ecbcc0a348e46470. As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

Ratio	Test	Platform	Options	Absolute values (old vs new)	Performance Profiles
14%	amazon ContentfulSpeedIndex	android-hw-a51-11-0-aarch64-shippable-qr	warm webrender	517.63 -> 590.12	Before/After
13%	amazon PerceptualSpeedIndex	android-hw-a51-11-0-aarch64-shippable-qr	warm webrender	466.85 -> 525.72	Before/After
12%	amazon ContentfulSpeedIndex	android-hw-a51-11-0-aarch64-shippable-qr	warm webrender	536.30 -> 598.91	Before/After
10%	amazon LastVisualChange	android-hw-a51-11-0-aarch64-shippable-qr	warm webrender	1,571.75 -> 1,736.22	Before/After
10%	amazon SpeedIndex	android-hw-a51-11-0-aarch64-shippable-qr	warm webrender	483.70 -> 531.01	Before/After

Improvements:

Ratio	Test	Platform	Options	Absolute values (old vs new)	Performance Profiles
50%	sina loadtime	android-hw-a51-11-0-aarch64-shippable-qr	warm webrender	2,798.42 -> 1,391.12	Before/After
43%	amazon loadtime	android-hw-a51-11-0-aarch64-shippable-qr	warm webrender	405.24 -> 230.60	Before/After
32%	amazon loadtime	android-hw-a51-11-0-aarch64-shippable-qr	warm webrender	443.35 -> 302.47	Before/After
23%	ebay-kleinanzeigen-search loadtime	android-hw-a51-11-0-aarch64-shippable-qr	warm webrender	1,572.52 -> 1,206.81	Before/After
18%	ebay-kleinanzeigen loadtime	android-hw-a51-11-0-aarch64-shippable-qr	warm webrender	1,410.87 -> 1,152.70	Before/After
...	...	...	...	...	...
9%	sina SpeedIndex	android-hw-a51-11-0-aarch64-shippable-qr	warm webrender	4,568.72 -> 4,178.79	Before/After

Details of the alert can be found in the alert summary, including links to graphs and comparisons for each of the affected tests. Please follow our guide to handling regression bugs and let us know your plans within 3 business days, or the patch(es) may be backed out in accordance with our regression policy.

If you need the profiling jobs you can trigger them yourself from treeherder job view or ask a sheriff to do that for you.

You can run these tests on try with ./mach try perf --alert 891

For more information on performance sheriffing please see our FAQ.

Florin, I'm not able to access the before and after profiles (the links are to "https://profiler.firefox.com/from-url/N%2FA")
Is there any other way I can access them?

Comment 5

[Florin Bilt]

Hello, we have a bug when generating the links for comment 0.
Here you can find the run with profiles before and after

Comment 6

[Andrew Creskey [:acreskey]]

Thanks Florin.

So far on the amazon warm visual metric regression I'm seeing that the CSS animation gwm-LoadingIndicator-animation tends to run longer with the patch (higher socket count).

We're also seeing a vismet improvement of similar scale on sina.com as well as loadtime improvements from 18% to 50% (ebay, sina, amazon).

And we saw improvements in telemetry when this patch was landed: https://bugzilla.mozilla.org/show_bug.cgi?id=1889771#c7

I'm still intending to run local browsertime tests on live sites to see if I can pick up any impact that way.

Comment 7

[Sunil Mayya]

Andrew, i am putting this into our queue.
Looks like you are preparing patches for fixing this.
Can I assign this to you to close the loop?

Comment 8

[Andrew Creskey [:acreskey]]

(In reply to Sunil Mayya from comment #7)

Andrew, i am putting this into our queue.
Looks like you are preparing patches for fixing this.
Can I assign this to you to close the loop?

Yes, I'll take this one Sunil.
We will likely choose to "not fix" this one as the real world network environment is showing improvements to pageload metrics.

Comment 9

[Andrew Creskey [:acreskey]]

Old triggered results from initial landing of increased socket pool.
Please see comments here: https://bugzilla.mozilla.org/show_bug.cgi?id=1909842#c3