1877481 - 13.22 - 6.07% espn PerceptualSpeedIndex / espn loadtime (Linux) regression on Tue January 23 2024

Status: RESOLVED FIXED

(Core :: DOM: Navigation, defect)

Description

[Alex Finder|PTO, back on March 31st]

Perfherder has detected a browsertime performance regression from push 8af7503ecb577e1840763fe1596800deb30d03f7. As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

Ratio	Test	Platform	Options	Absolute values (old vs new)	Performance Profiles
13%	espn PerceptualSpeedIndex	linux1804-64-shippable-qr	cold fission webrender	839.58 -> 950.54	Before/After
6%	espn loadtime	linux1804-64-shippable-qr	fission warm webrender	794.63 -> 842.90	Before/After

Details of the alert can be found in the alert summary, including links to graphs and comparisons for each of the affected tests. Please follow our guide to handling regression bugs and let us know your plans within 3 business days, or the patch(es) may be backed out in accordance with our regression policy.

If you need the profiling jobs you can trigger them yourself from treeherder job view or ask a sheriff to do that for you.

You can run these tests on try with ./mach try perf --alert 41229

For more information on performance sheriffing please see our FAQ.

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1875040

Comment 2

[Andreas Farre [:farre]]

So looking at the profiler results from with the patch and the patch backed out we have cold with patch FirstVisualChange=835, PerceptualSpeedIndex=931, fnbpaint=804 and cold with patch backed out FirstVisualChange=530, PerceptualSpeedIndex=756, fnbpaint=517. The perfherder compare is here.

My problem here is that I don't know how to interpret neither FirstVisualChange nor PerceptualSpeedIndex and fnbpaint. So I generated a side by side, and it made me a bit worried. Because the differing visual change is that the backed out version only manages some kind of blank paint before the patch in bug 1875040, but is actually slower in starting to draw actual content. And I can't visually determine a speed for the pages either. Also, for reference, in the side by side comparison tests, try / b59eb13532f9 is with the patch in bug 1875040 and try / bed10b41093b is the patch in 1875040 backed out.

Also, it should be noted that the patch in bug 1875040 is generally ~10% faster on more than half of the warm load tests.

I don't really know how to proceed on this, since perf says there is a regression but I don't understand it. Is there someone I can ask for advice on how to find out more?

Comment 3

[Greg Mierzwinski [:sparky]]

Attachment: last-visual-change.png

The PerceptualSpeedIndex (PSI) regression is likely coming from the difference in the FirstVisualChange/LastVisualChange. I triggered another side-by-side on the autoland push since I noticed it was missing there: https://treeherder.mozilla.org/jobs?repo=autoland&group_state=expanded&tier=1%2C2%2C3&tochange=8af7503ecb577e1840763fe1596800deb30d03f7&fromchange=2437c2ca5bec35fe4ab02eb938e0e02457cd079b&searchStr=espn&selectedTaskRun=B4-SSrqlRYSz3ZCOomx84w.0

The PSI metric is similar to SpeedIndex (SI), but it uses Structural Similarity instead of Histograms to calculate the visual progress. You can find some more information about these here: https://www.sitespeed.io/documentation/sitespeed.io/metrics/#visual-metrics

For FirstVisualChange/fnbpaint, they're similar to each other but obtained in different ways. They indicate the first point in time at which there's a non-blank paint.

Looking at the side-by-side that I generated on autoland (linked above) I'm seeing that the first non-blank paint happens later when your patch is applied. Note that the grey paint happens much earlier in the before vs. the after.

For the last visual change, it's the last point in time that a change was found. I've attached a screenshot showing that the bar at the top of the page seems to be going faster there when your patch is applied. I think the FirstVisualChange is impacting the metric more here though. (I'll attach the screenshot in the next comment).

Comment 4

[Greg Mierzwinski [:sparky]]

Attachment: first-visual-change.png

Comment 5

[Greg Mierzwinski [:sparky]]

I'm currently thinking that the majority of the regression is coming from the grey non-blank paint taking 300ms longer to appear when your patch is applied, and that's also impacting the PSI metric.

I'm looking at this video for the screenshots: https://firefoxci.taskcluster-artifacts.net/B4-SSrqlRYSz3ZCOomx84w/0/public/build/side-by-side/browsertime-tp6-firefox-espn/cold-side-by-side.mp4

Comment 6

[Andreas Farre [:farre]]

Yeah, I don't know what to do to resolve this. Is this a backout Alex?

Comment 7

[Alex Finder|PTO, back on March 31st]

(In reply to Andreas Farre [:farre] from comment #6)

Yeah, I don't know what to do to resolve this. Is this a backout Alex?

I can try re-running the job with the culprit revision 8af7503ecb577e1840763fe1596800deb30d03f7 backed out to see if we observe the same results.

Comment 8

[Andreas Farre [:farre]]

Yeah, I can't really argue for keeping this. Alex, can you orchestrate the backout?

Comment 9

[Alex Finder|PTO, back on March 31st]

(In reply to Andreas Farre [:farre] from comment #8)

Yeah, I can't really argue for keeping this. Alex, can you orchestrate the backout?

Re-ran the tests for autoland range ba5a29797a73f - 5c38b3e5008b2 on the try repo, with 8af7503ecb577e1840763fe1596800deb30d03f7 backed out. At first glance, the revision 5ba224ddf96a93a2bcc743278351cc0bf6d5b0f2, highlighted in the previous graph, which is the backout of 8af7503ecb577e1840763fe1596800deb30d03f7, doesn't seem to fix the detected regression. I also retriggered the jobs a few times, so we can exclude any noise.

Comment 10

[Alex Finder|PTO, back on March 31st]

Leaving a :ni for myself to recheck the results next Monday, on my return from PTO.

Comment 11

[Iulian Moraru]

This should be fixed by backout of Bug 1875040. Backout link: https://hg.mozilla.org/integration/autoland/rev/54bf47d03582dce757f4d87be8f557734e9900ca

Comment 12

[Alex Finder|PTO, back on March 31st]

(In reply to Iulian Moraru from comment #11)

This should be fixed by backout of Bug 1875040. Backout link: https://hg.mozilla.org/integration/autoland/rev/54bf47d03582dce757f4d87be8f557734e9900ca

Retriggered the jobs for this range, since the datapoints were not showing up on the graph. Will confirm the fix once the jobs are finished running.

Comment 13

[Alex Finder|PTO, back on March 31st]

The graph for PerceptualSpeedIndex confirms the fix introduced by the backout.