1919295 - 96.89 - 1.28% google-docs PerceptualSpeedIndex / nytimes SpeedIndex + 16 more (Linux, OSX, Windows) regression on Thu September 5 2024

Status: RESOLVED FIXED

(Core :: JavaScript Engine, defect, P1)

Description

[Treeherder Bug Filer]

Perfherder has detected a browsertime performance regression from push 21015e92d7f99239985eb8257cc7f7107fcad6cc. As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

Ratio	Test	Platform	Options	Absolute values (old vs new)	Performance Profiles
97%	google-docs PerceptualSpeedIndex	windows11-64-shippable-qr	fission warm webrender	1,218.31 -> 2,398.68
89%	google-docs LastVisualChange	windows11-64-shippable-qr	fission warm webrender	2,646.32 -> 4,998.44
81%	google-docs LastVisualChange	linux1804-64-shippable-qr	fission warm webrender	5,289.66 -> 9,584.51	Before/After
72%	google-docs PerceptualSpeedIndex	linux1804-64-shippable-qr	fission warm webrender	2,445.99 -> 4,213.97	Before/After
62%	google-docs ContentfulSpeedIndex	windows11-64-shippable-qr	fission warm webrender	797.53 -> 1,289.37
60%	google-docs SpeedIndex	windows11-64-shippable-qr	fission warm webrender	532.01 -> 848.80
52%	google-docs PerceptualSpeedIndex	macosx1015-64-shippable-qr	fission warm webrender	2,580.17 -> 3,926.82
47%	google-docs LastVisualChange	macosx1015-64-shippable-qr	fission warm webrender	5,255.98 -> 7,750.23
38%	google-docs ContentfulSpeedIndex	macosx1015-64-shippable-qr	fission warm webrender	1,430.92 -> 1,970.11
36%	google-docs SpeedIndex	macosx1015-64-shippable-qr	fission warm webrender	976.13 -> 1,323.02
...	...	...	...	...	...
8%	google-docs largestContentfulPaint	windows11-64-shippable-qr	fission warm webrender	710.06 -> 766.30
5%	google-docs largestContentfulPaint	macosx1015-64-shippable-qr	fission warm webrender	968.96 -> 1,018.38
2%	nytimes SpeedIndex	linux1804-64-shippable-qr	fission warm webrender	1,042.06 -> 1,066.21	Before/After
2%	nytimes LastVisualChange	linux1804-64-shippable-qr	fission warm webrender	1,323.41 -> 1,352.70	Before/After
1%	nytimes SpeedIndex	linux1804-64-shippable-qr	fission warm webrender	1,043.72 -> 1,057.10	Before/After

Details of the alert can be found in the alert summary, including links to graphs and comparisons for each of the affected tests. Please follow our guide to handling regression bugs and let us know your plans within 3 business days, or the patch(es) may be backed out in accordance with our regression policy.

If you need the profiling jobs you can trigger them yourself from treeherder job view or ask a sheriff to do that for you.

You can run these tests on try with ./mach try perf --alert 2114

For more information on performance sheriffing please see our FAQ.

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1905987

Comment 2

[Matthew Gaudet (he/him) [:mgaudet]]

Ok: Something is going on here. Let's use Google Docs Perceptual Speed Index before/after Warm profiles as our discussion points:

1. The profiles are actually quite misleading: If you don't look at the screenshots view, and instead look at the Perceptual Visual Completeness, you'd be mislead into thinking the "After" profile is faster. After all, Perceptual Visual Completeness hits 100% at 10.37s on the before profile; whereas it hits 100% at 6.13s on the after profile. However If you compare the screenshot tracks, it becomes clear that the profile for the After track hasn't actually captured the whole page load; the sidebar hasn't yet rendered by the time the After profile stops
2. Now, given the regressing patch as identified uses invalidation, I would normally ask how many we have ( window.filteredMarkers.filter((x) => x.name == "Invalidate").length) ; On the Before profile, we record 367 Invalidations. On the After profile we record only 220. But given the apparent incompleteness of the after profile, this doesn't actually help.

So next investigative step is to try and see if backing out will fix this; after that a more detailed investigation with a total profile will be needed.

Try push with backout is here -- we'll see results start rolling in over the next couple hours.

Comment 3

[Matthew Gaudet (he/him) [:mgaudet]]

Ok. Big improvement.

I have one idea I want to try before I get Bug 1905987 backed out, but that will be the likely outcome here.

Comment 4

[Matthew Gaudet (he/him) [:mgaudet]]

Ok, here's my 1 idea: Limit invalidation to expected generation zero -- I'll check the results on this tomorrow morning. If it doesn't fix the issues for google docs, we'll back out Bug 1905987.

Comment 5

[Matthew Gaudet (he/him) [:mgaudet]]

Ok -- So limiting to gen 0 hurts Jetstream and just hurts GDocs too.

So, I'll request backout of Bug 1905987.

Comment 6

[Ryan VanderMeulen [:RyanVM]]

Fixed by backout.