1902302 - 7.72 - 3.33% cnn-nav.landing SpeedIndex / cnn-nav.world PerceptualSpeedIndex (Windows) regression on Tue June 11 2024

Status: RESOLVED WONTFIX

(Core :: Privacy: Anti-Tracking, defect, P2)

Description

[Florin Bilt]

Perfherder has detected a browsertime performance change from push 859498ffdfa51365ba1d695f5eab2a9550805b43.

Regressions:

Ratio	Test	Platform	Options	Absolute values (old vs new)	Performance Profiles
211%	allrecipes LastVisualChange	android-hw-a51-11-0-aarch64-shippable-qr	warm webrender	1,914.13 -> 5,953.96	Before/After
187%	allrecipes LastVisualChange	android-hw-a51-11-0-aarch64-shippable-qr	cold webrender	2,582.42 -> 7,422.08	Before/After
161%	reddit loadtime	android-hw-a51-11-0-aarch64-shippable-qr	warm webrender	605.85 -> 1,579.17	Before/After
156%	allrecipes ContentfulSpeedIndex	android-hw-a51-11-0-aarch64-shippable-qr	warm webrender	853.69 -> 2,185.73	Before/After
132%	reddit loadtime	android-hw-a51-11-0-aarch64-shippable-qr	cold webrender	948.17 -> 2,197.48	Before/After
129%	ebay-kleinanzeigen-search loadtime	android-hw-a51-11-0-aarch64-shippable-qr	cold webrender	1,104.47 -> 2,531.99	Before/After
119%	ebay-kleinanzeigen loadtime	android-hw-a51-11-0-aarch64-shippable-qr	cold webrender	1,138.13 -> 2,497.50	Before/After
108%	allrecipes PerceptualSpeedIndex	android-hw-a51-11-0-aarch64-shippable-qr	warm webrender	762.84 -> 1,583.48	Before/After
92%	booking PerceptualSpeedIndex	android-hw-a51-11-0-aarch64-shippable-qr	warm webrender	412.02 -> 789.23	Before/After
84%	allrecipes loadtime	android-hw-a51-11-0-aarch64-shippable-qr	warm webrender	677.81 -> 1,246.49	Before/After
...	...	...	...	...	...
4%	espn SpeedIndex	android-hw-a51-11-0-aarch64-shippable-qr	warm webrender	1,762.89 -> 1,837.25	Before/After
4%	espn LastVisualChange	android-hw-a51-11-0-aarch64-shippable-qr	warm webrender	2,442.73 -> 2,529.08	Before/After
3%	cnn-nav.world PerceptualSpeedIndex	windows10-64-shippable-qr	cold fission webrender	721.74 -> 745.77	Before/After
3%	cnn-nav.world loadtime	windows10-64-shippable-qr	cold fission webrender	830.18 -> 855.91	Before/After
3%	stackoverflow loadtime	android-hw-a51-11-0-aarch64-shippable-qr	warm webrender	393.05 -> 404.50	Before/After

Improvements:

Ratio	Test	Platform	Options	Absolute values (old vs new)	Performance Profiles
28%	imgur loadtime	linux1804-64-shippable-qr	cold fission webrender	2,756.04 -> 1,990.51	Before/After
28%	cnn loadtime	android-hw-a51-11-0-aarch64-shippable-qr	cold webrender	3,452.32 -> 2,501.33	Before/After
25%	buzzfeed loadtime	macosx1015-64-shippable-qr	cold fission webrender	1,336.96 -> 1,001.37	Before/After
10%	cnn fcp	android-hw-a51-11-0-aarch64-shippable-qr	warm webrender	1,602.42 -> 1,436.12	Before/After
10%	cnn largestContentfulPaint	android-hw-a51-11-0-aarch64-shippable-qr	warm webrender	1,632.04 -> 1,465.56	Before/After
...	...	...	...	...	...
3%	fandom SpeedIndex	linux1804-64-shippable-qr	fission warm webrender	252.04 -> 245.66	Before/After

As author of one of the patches included in that push, we need your help to address this regression.
Details of the alert can be found in the alert summary, including links to graphs and comparisons for each of the affected tests. Please follow our guide to handling regression bugs and let us know your plans within 3 business days, or the patch(es) may be backed out in accordance with our regression policy.

If you need the profiling jobs you can trigger them yourself from treeherder job view or ask a sheriff to do that for you.

You can run these tests on try with ./mach try perf --alert 711

For more information on performance sheriffing please see our FAQ.

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1899359

Comment 2

[Donal Meehan [:dmeehan]]

:timhuang could this be triaged for severity?
Fx129 soft code freeze starts next week on 2024-07-04 ahead of Fx128 going to beta.

Comment 3

[Tim Huang[:timhuang]]

This only affects the Nightly build for now.

Comment 4

[Harshit Sohaney [:hsohaney]]

We will look deeper into this when we consider moving forward from Nightly

Comment 5

[Harshit Sohaney [:hsohaney]]

Attachment: Screenshot 2024-07-30 at 4.32.23 PM.png

After doing some debugging using the profiler and adding a start marker at downloadUpdates() and an end marker at StreamFinished(), it looks like there is no clear performance change between remote settings and shavar. I have attached screenshots of the time measured between the two.

shavar time measured at 1.7s

Comment 6

[Harshit Sohaney [:hsohaney]]

Attachment: Screenshot 2024-07-30 at 4.33.01 PM.png

remote settings time measured at 1.2s

Comment 7

[Harshit Sohaney [:hsohaney]]

Does this testing seem satisfactory? Is there more to test here to decide if the performance regression is caused by this change?

Comment 8

[Florin Bilt]

In the meantime, several alerts have appeared indicating that this patch is the culprit.
I updated comment 0.
I am quite sure that this patch caused it.

Comment 9

[Harshit Sohaney [:hsohaney]]

Alright, I think I might have placed the marker in the wrong place then, will look into next steps for this

Comment 10

[Tim Huang[:timhuang]]

We found the root cause of the performance regression is that we don't get the tracking-protection lists from remote setting. The remote settings server is set to a dummy server where doesn't have any list available. So, the traffic pattern changes because Firefox no longer classifies trackers.

I tried to change the server location to the prod server, but it doesn't work. I still get no tracking protection lists on Remote Settings. This might be that the external network connect to remote setting server is blocked by the BrowseTime test suite. But, the network loading to the Shavar server is still available.

Florin, do you know how can we access the remote settings server for the pref test on Android?

Comment 11

[Tim Huang[:timhuang]]

After talking with :sparky and running several live site tests, we realized that we might need to record the testing again with a real remote settings server. And we need to set the pref to use the real remote settings server for playback testing.

Comment 12

[Tim Huang[:timhuang]]

Attachment: Bug 1902302 - Enable the real RS server for browsertime playback tests. r?sparky!

Comment 13

[Greg Mierzwinski [:sparky]]

I think it's just some re-recordings that need to be done here, not enabling the live remote settings during the playback tests. :fbilt, can you re-record the allrecipes test to see if that resolves the regression?

Comment 14

[Florin Bilt]

(In reply to Greg Mierzwinski [:sparky] from comment #13)

I think it's just some re-recordings that need to be done here, not enabling the live remote settings during the playback tests. :fbilt, can you re-record the allrecipes test to see if that resolves the regression?

Sorry for my late replay, I had some issues with recording job.
Here are allrecipes jobs with a new record
And here is a comparasion between my push and the first revision after this revision (859498ffdfa51365ba1d695f5eab2a9550805b43) where the android tests are passing.
It seems that tre regression has been fixed. What do you think sparky?

Comment 15

[Greg Mierzwinski [:sparky]]

I agree looks like the rerecording resolved the issue, thanks for testing this! Can you make a patch to land the allrecipes rerecording?

Comment 16

[Florin Bilt]

Hi sparky and hsohaney.
The patch for the Bug 1917839 was landed on Monday.
I expect new alerts to appear for allrecipes .

Comment 17

[Greg Mierzwinski [:sparky]]

Thanks for the fix :fbilt! I think we can close this bug as WONTFIX now since the rerecording brought us back to the original values prior to this regression.

Comment 18

[Harshit Sohaney [:hsohaney]]

Is this good to close as per :sparky 's comment?