1770116 - 4.78 - 3.92% pinterest PerceptualSpeedIndex / pinterest PerceptualSpeedIndex + 1 more (OSX, Windows) regression on Fri May 13 2022

Status: RESOLVED WONTFIX

(Core :: IPC, defect)

Description

[Alexandru Ionescu (needinfo me) [:alexandrui]]

Perfherder has detected a browsertime performance regression from push 3f4ebef6a4b1b4fb858d9d8eba17a4ccae268d1a. As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

Ratio	Test	Platform	Options	Absolute values (old vs new)
5%	pinterest PerceptualSpeedIndex	macosx1015-64-shippable-qr	fission warm webrender	1,543.25 -> 1,617.00
4%	outlook SpeedIndex	macosx1015-64-shippable-qr	fission warm webrender	974.96 -> 1,016.92
4%	pinterest PerceptualSpeedIndex	windows10-64-shippable-qr	fission warm webrender	1,782.92 -> 1,852.75

Improvements:

Ratio	Test	Platform	Options	Absolute values (old vs new)
34%	pinterest ContentfulSpeedIndex	linux1804-64-shippable-qr	fission warm webrender	1,109.71 -> 730.58
33%	pinterest ContentfulSpeedIndex	linux1804-64-shippable-qr	fission warm webrender	1,099.42 -> 735.75
33%	pinterest ContentfulSpeedIndex	windows10-64-shippable-qr	fission warm webrender	1,207.08 -> 808.67
27%	pinterest ContentfulSpeedIndex	macosx1015-64-shippable-qr	fission warm webrender	993.75 -> 722.83
24%	pinterest SpeedIndex	windows10-64-shippable-qr	fission warm webrender	952.42 -> 719.58
21%	pinterest SpeedIndex	macosx1015-64-shippable-qr	fission warm webrender	852.75 -> 671.00

Details of the alert can be found in the alert summary, including links to graphs and comparisons for each of the affected tests. Please follow our guide to handling regression bugs and let us know your plans within 3 business days, or the offending patch(es) will be backed out in accordance with our regression policy.

If you need the profiling jobs you can trigger them yourself from treeherder job view or ask a sheriff to do that for you.

For more information on performance sheriffing please see our FAQ.

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1754004

Comment 2

[Nika Layzell [:nika] (ni? for response)]

I'm vaguely guessing that the small regressions are caused by slight changes in timing, where something is happening at a slightly different time than before (perhaps slightly faster), thus delaying whatever the perceptual speed index is measuring.

:jesup, do you think it's worth looking into the regressions? As I mentioned above they're probably caused by slight timing changes, but we might be able to see what exactly is going on if we record some before/after profiles with IPC markers?

Comment 3

[Randell Jesup [:jesup] (needinfo me)]

Given the speedups for pinterest for SpeedIndex (which we monitor/care about more than Perceptual SpeedIndex), I think this is fine. Likely it's some image that ends up painted marginally later or moves slightly. PSI uses SSIM video-style comparison; SI uses a color and luma histogram (and so is impervious to movement); CSI uses a comparison based on edges (more weight to things with lots of high-contrast edges like text).

Comment 4

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1754004

Comment 5

[BugBot [:suhaib / :marco/ :calixte]]

The severity field is not set for this bug.
:jld, could you have a look please?

For more information, please visit auto_nag documentation.

Comment 6

[Jed Davis [:jld] ⟨⏰|UTC-8⟩ ⟦he/him⟧]

WONTFIX based on comment #3.