1856527 - 37.46 - 34.11% reddit-billgates-post-2.billg LastVisualChange / reddit-billgates-post-2.billg LastVisualChange (Windows) regression on Wed September 27 2023

Status: RESOLVED INVALID

(Firefox :: Theme, defect)

Description

[Alexandru Ionescu (needinfo me) [:alexandrui]]

Perfherder has detected a browsertime performance regression from push dc441449616ec76547adebb86e8a67419b00ddf3. As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

Ratio	Test	Platform	Options	Absolute values (old vs new)
37%	reddit-billgates-post-2.billg LastVisualChange	windows10-64-shippable-qr	cold fission webrender	1,803.43 -> 2,479.00
34%	reddit-billgates-post-2.billg LastVisualChange	windows10-64-shippable-qr	cold fission webrender	1,870.35 -> 2,508.28

Details of the alert can be found in the alert summary, including links to graphs and comparisons for each of the affected tests. Please follow our guide to handling regression bugs and let us know your plans within 3 business days, or the offending patch(es) may be backed out in accordance with our regression policy.

If you need the profiling jobs you can trigger them yourself from treeherder job view or ask a sheriff to do that for you.

For more information on performance sheriffing please see our FAQ.

Comment 1

[Emilio Cobos Álvarez (:emilio)]

That makes zero sense, that patch only changed some colours in the front end UI. Can you double check the push that caused that?

Comment 2

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1855353

Comment 3

[Alexandru Ionescu (needinfo me) [:alexandrui]]

Hi Emilio. I agree, it looks odd, but the graph is pretty clear and contains retriggers. Unfortunately the side-by-side is not available yet for interactive tests. The least I can do is to backout your patch on try and see if the regression goes away.

Comment 4

[Emilio Cobos Álvarez (:emilio)]

That'd be useful. The patch also affected macOS so I'm curious why macOS wouldn't get hit by this.

Comment 5

[Emilio Cobos Álvarez (:emilio)]

Does LastVisualChange look somehow at the pixels in the browser UI also, in a way such that a color change could've messed with the heuristics or something?

Comment 6

[Alexandru Ionescu (needinfo me) [:alexandrui]]

https://treeherder.mozilla.org/jobs?repo=try&revision=ffb65782836541e967dc5b6cf6ab0398b93af979

Comment 7

[Greg Mierzwinski [:sparky]]

Attachment: billg-side-by-side.gif

It looks at everything within the orange screen. This includes the scrollbar as well if that was changed, and locally, I'm finding that the scroll bar is changing right around the time as the last visual change.

Comment 8

[Alexandru Ionescu (needinfo me) [:alexandrui]]

https://treeherder.mozilla.org/perfherder/compare?originalProject=autoland&originalRevision=dc441449616ec76547adebb86e8a67419b00ddf3&newProject=try&newRevision=ffb65782836541e967dc5b6cf6ab0398b93af979&framework=13&page=1&showOnlyComparable=1&filter=LastVisualChange

Comment 9

[Emilio Cobos Álvarez (:emilio)]

But the scrollbar also changes in comment 7 way after the LastVisualChanges is recorded right? I have to be missing something. I don't see anything obviously worse on the side-by-side diff. I'm pretty sure this is not a real regression.

Comment 10

[Greg Mierzwinski [:sparky]]

I agree with this being invalid, I don't see any problematic differences. What I'm thinking happened is that changing the colours changed something with the fuzzing for finding the last frame/image: https://github.com/sitespeedio/browsertime/blob/main/browsertime/visualmetrics-portable.py#L874-L897

It's been an issue in the past as well: https://bugzilla.mozilla.org/show_bug.cgi?id=1815970
I'll make a new bug for this one though since we don't have anything on file for it.

Comment 11

[Emilio Cobos Álvarez (:emilio)]

Okay I'll close this based on the discussion above.

Comment 12

[Greg Mierzwinski [:sparky]]

Setting as INVALID since it's not clear that there was a genuine regression here.