1758449 - 16.16 - 2.47% matrix-react-bench / cnn fcp + 9 more (Linux, OSX, Windows) regression on Thu March 3 2022

Status: RESOLVED WONTFIX

(Core :: DOM: Navigation, defect)

Description

[Alexandru Ionescu (needinfo me) [:alexandrui]]

Perfherder has detected a browsertime performance regression from push 979db4220201d00aa3b5fd2437f9b4dd46c02ce8. As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

Ratio	Test	Platform	Options	Absolute values (old vs new)
16%	matrix-react-bench	macosx1015-64-shippable-qr	fission webrender	30.15 -> 35.02
13%	cnn dcf	linux1804-64-shippable-qr	fission warm webrender	478.71 -> 541.62
13%	cnn dcf	linux1804-64-shippable-qr	fission warm webrender	480.17 -> 541.17
13%	matrix-react-bench	macosx1015-64-shippable-qr	fission webrender	30.47 -> 34.31
10%	matrix-react-bench	linux1804-64-shippable-qr	fission webrender	32.64 -> 35.97
8%	matrix-react-bench	windows10-64-shippable-qr	fission webrender	32.93 -> 35.64
4%	cnn ContentfulSpeedIndex	macosx1015-64-shippable-qr	fission warm webrender	513.08 -> 535.33
4%	cnn PerceptualSpeedIndex	macosx1015-64-shippable-qr	fission warm webrender	515.38 -> 536.17
4%	cnn PerceptualSpeedIndex	macosx1015-64-shippable-qr	fission warm webrender	515.46 -> 533.67
3%	cnn fnbpaint	macosx1015-64-shippable-qr	fission warm webrender	397.71 -> 408.62
2%	cnn fcp	macosx1015-64-shippable-qr	fission warm webrender	396.46 -> 406.25

Details of the alert can be found in the alert summary, including links to graphs and comparisons for each of the affected tests. Please follow our guide to handling regression bugs and let us know your plans within 3 business days, or the offending patch(es) will be backed out in accordance with our regression policy.

For more information on performance sheriffing please see our FAQ.

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1741671

Comment 2

[Sean Feng [:sefeng]]

I am looking into this, the regressions are not expected.

This feature is behind a pref and nightly only, so status-firefox99 should be unaffected.

Comment 3

[Bas Schouten (:bas.schouten)]

Ted, since you know the test best, any chance you have an idea of how the test could interact with the work in bug 1741671?

Comment 4

[Sean Feng [:sefeng]]

For matrix-react-bench, In my local testing, I noticed this benchmark did dozens of registering and unregistering beforeunload handlers via the React's componentWillUnmount and componentDidMount methods. With the patch, we start to bfcache pages with beforeunload handlers, so presumably the bfcache work caused the regressions, and we hit this code a lot in this benchmark.

I think the ideal fix for it is to not doing the registering and unregistering beforeunload handlers because this is probably not the intention of the benchmark.

Comment 5

[Sean Feng [:sefeng]]

I was about to proposing a fix to disable bfcache for matrix-react-bench, because

1. I was able to fix the regression locally with bfcache disabled
2. This test has nothing to do with bfcache/pageload
3. I couldn't find the exact reason of why my patches caused the regression.

I guess a plausible explanation was the matrix-react-bench unintentionally measured the pageload...though I don't have any evidence.

The good news is that bug 1758992 fixed the regression by updating the test version to v1.1, so it's more likely that something was wrong in the test. Anyway, I think we are good with matrix-react-bench.

Comment 6

[Sean Feng [:sefeng]]

I am also not going to worry about cnn related test, as it's known that CNN had many processes and BFCached had caused some regressions earlier when it was enabled in Fission.

So I am closing this bug as WONTFIX.