1812303 - 90.86 - 17.66% sina LastVisualChange / sina loadtime + 7 more (Android) regression on Fri January 20 2023

Status: RESOLVED INVALID

(Core :: DOM: Networking, defect, P2)

Description

[Alexandru Ionescu (needinfo me) [:alexandrui]]

Perfherder has detected a browsertime performance regression from push 0c260a33541a67b1c4f0ab83a1d1380f466fcf9b. As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

Ratio	Test	Platform	Options	Absolute values (old vs new)
91%	sina LastVisualChange	android-hw-a51-11-0-aarch64-shippable-qr	cold webrender	3,254.25 -> 6,210.92
90%	sina PerceptualSpeedIndex	android-hw-a51-11-0-aarch64-shippable-qr	cold webrender	2,735.58 -> 5,207.17
89%	sina FirstVisualChange	android-hw-a51-11-0-aarch64-shippable-qr	cold webrender	2,256.56 -> 4,256.92
83%	sina fcp	android-hw-a51-11-0-aarch64-shippable-qr	cold webrender	2,295.94 -> 4,211.29
75%	sina SpeedIndex	android-hw-a51-11-0-aarch64-shippable-qr	cold webrender	3,045.00 -> 5,330.42
71%	sina ContentfulSpeedIndex	android-hw-a51-11-0-aarch64-shippable-qr	cold webrender	2,816.50 -> 4,829.25
54%	sina loadtime	android-hw-a51-11-0-aarch64-shippable-qr	warm webrender	1,140.41 -> 1,754.54
20%	sina ContentfulSpeedIndex	android-hw-a51-11-0-aarch64-shippable-qr	warm webrender	287.92 -> 346.17
18%	sina loadtime	android-hw-a51-11-0-aarch64-shippable-qr	cold webrender	7,982.31 -> 9,391.75

Improvements:

Ratio	Test	Platform	Options	Absolute values (old vs new)
44%	sina FirstVisualChange	android-hw-a51-11-0-aarch64-shippable-qr	cold webrender	4,346.08 -> 2,413.92
34%	sina loadtime	android-hw-a51-11-0-aarch64-shippable-qr	warm webrender	1,738.40 -> 1,153.69

Details of the alert can be found in the alert summary, including links to graphs and comparisons for each of the affected tests. Please follow our guide to handling regression bugs and let us know your plans within 3 business days, or the offending patch(es) may be backed out in accordance with our regression policy.

If you need the profiling jobs you can trigger them yourself from treeherder job view or ask a sheriff to do that for you.

For more information on performance sheriffing please see our FAQ.

Comment 1

[Release mgmt bot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1351231

Comment 2

[Release mgmt bot [:suhaib / :marco/ :calixte]]

The bug is marked as tracked for firefox111 (nightly). We have limited time to fix this, the soft freeze is in 14 days. However, the bug still isn't assigned.

:ghess, could you please find an assignee for this tracked bug? Given that it is a regression and we know the cause, we could also simply backout the regressor. If you disagree with the tracking decision, please talk with the release managers.

For more information, please visit auto_nag documentation.

Comment 3

[Eden Chuang[:edenchuang]]

PFetch (dom.workers.pFetch.enabled) is currently enabled in nightly build only.

According to the bug description, it seems that PFetch causes some performance issues on the android platform.
I will take a look these days to figure the root cause out.

Since PFetch would not be turned on in Beta and release, we still want to keep it enabled in nightly.

Comment 4

[Eden Chuang[:edenchuang]]

I am wondering if this performance regression is really related to bug 1351231.

The new time values with the patches in the bug description are almost the same with results before Jan 19.
https://treeherder.mozilla.org/perfherder/graphs?timerange=1209600&series=autoland,4375260,1,13

The interesting part is most of regression test cases only got better results in Jan 19 and Jan 20.

To make sure if this really related to bug 1351231, I pushed a try run which force disable PFetch to see if we get the same result or not.

https://treeherder.mozilla.org/jobs?repo=try&revision=3c9f471d086495f40900a086cef38c4add580b6e

Comment 5

[Andrew Sutherland [:asuth] (he/him)]

(In reply to Eden Chuang[:edenchuang] from comment #4)

To make sure if this really related to bug 1351231, I pushed a try run which force disable PFetch to see if we get the same result or not.

https://treeherder.mozilla.org/jobs?repo=try&revision=3c9f471d086495f40900a086cef38c4add580b6e

One piece of guidance I've received is that comparing metrics to m-c from a try branch push can be deceptive and it's better to use mach try perf which can launch try performance runs for both with and without your try stack, which should be more representative.

Comment 6

[Donal Meehan [:dmeehan]]

:edenchuang this is the final week of 111 nightly, 111 goes to beta next week.
bug 1351231 only impact nightly builds, but there was some doubt in Comment 4 that this was the regressor.
Any updates on the investigation?

Comment 7

[Eden Chuang[:edenchuang]]

Hi Donal

After running some perf try runs w/o enabling PFetch(bug 1351231), the results have no big difference(almost less than 5%).
So I think this regression might not be caused by bug 1351231.
I am trying to run perf try runs again by not only disabling but removing the patches of bug 1351231 to ensure this is not related to bug 1351231.
I will update the results on the bug later.

Comment 8

[Release mgmt bot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1351231

Comment 9

[Donal Meehan [:dmeehan]]

:edenchuang 111 is now in beta, any updates from comment 7?
Since Bug 1351231 only impacts nightly, I would set firefox111 to disabled and remove tracking. Though would need to confirm your investigation first.

Comment 10

[Eden Chuang[:edenchuang]]

After investigating many try perf runs, I think this is not a performance regression caused by PFetch.

Following is one of try perf runs that compares enabling PFetch(old) and disabling PFetch(new).

https://treeherder.mozilla.org/perfherder/compare?originalProject=try&originalRevision=b8362a7bfa46f44f8235cb1674f080f7f04c8791&newProject=try&newRevision=5d65cd2ca792ec6c13f0e39e504aedeef027b204&page=1&framework=13&showOnlyComparable=1

Focus on the regression case sina, and there is no significant difference between enable and disable.
In some significant cases, I always found that enabling (old) has better results than disabling (new).

So I think we can set firefox 111 to disabled and remove tracking

Comment 11

[Alexandru Ionescu (needinfo me) [:alexandrui]]

I'm closing this as invalid, thanks :edenchuang.