1861992 - 18.65 - 15.76% google LastVisualChange / google SpeedIndex (Android) regression on Tue October 24 2023

Status: RESOLVED WONTFIX

(GeckoView :: General, defect)

Description

[Alexandru Ionescu (needinfo me) [:alexandrui]]

Perfherder has detected a browsertime performance regression from push aaa658ba6ff75d75e2b80d803280d13f9a92e4dc. As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

Ratio	Test	Platform	Options	Absolute values (old vs new)
19%	google LastVisualChange	android-hw-a51-11-0-aarch64-shippable-qr	cold webrender	702.08 -> 833.03
16%	google SpeedIndex	android-hw-a51-11-0-aarch64-shippable-qr	cold webrender	323.56 -> 374.57

Details of the alert can be found in the alert summary, including links to graphs and comparisons for each of the affected tests. Please follow our guide to handling regression bugs and let us know your plans within 3 business days, or the patch(es) may be backed out in accordance with our regression policy.

If you need the profiling jobs you can trigger them yourself from treeherder job view or ask a sheriff to do that for you.

For more information on performance sheriffing please see our FAQ.

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1860161

Comment 2

[Gregory Pappas [:gregp]]

Hello!

• Are before/after profiles available ?
• Are before/after videos available ?

Comment 3

[Timothy Nikkel (:tnikkel)]

You can get videos by finding a job in treeherder and then selecting it and then clicking "artifacts and debugging tools" then you can download one of the files called "browsertime*". I found a job by clicking on the percentage regression link in the first comment, then zooming in to the regression range in that graph, then clicking on one dot before the regression and then open job. Once you've found the job you can then click on the performance tab and click generate performance profile. I've done such already, it should it ready soon.

Before job:
https://treeherder.mozilla.org/jobs?repo=autoland&revision=7f25d62e49a2325004433386f09955d3c2f64c3c&group_state=expanded&selectedTaskRun=fdg9A9vmQ4S4G4OSyOjDag.0

After job:
https://treeherder.mozilla.org/jobs?repo=autoland&revision=aaa658ba6ff75d75e2b80d803280d13f9a92e4dc&group_state=expanded&selectedTaskRun=FsgRnGd5RaK9Zs6yJP0v4Q.0

Comment 4

[Gregory Pappas [:gregp]]

Wow, thanks so much!!!!

Comment 5

[Gregory Pappas [:gregp]]

Attachment: warm-side-by-side-slow-motion.gif

Comment 6

[Gregory Pappas [:gregp]]

Attachment: cold-side-by-side-slow-motion.gif

Comment 7

[Gregory Pappas [:gregp]]

I don't think this is a real regression. I think these numbers became worse because the scrollbar fade is increasing the amount of time it technically takes for the page to reach its final 'stable' state, but the page is not actually loading slower.

So, we could:

• Close this bug as INVALID (because it kinda is ?) and accept the new baseline
• Set ui.scrollbarFadeDuration to 0 in the test config to make the numbers go back down
  • Pros: People unlike computers will not be confused by the presence of fading scrollbars and think the page is loading slower than it actually is. So maybe doing this will make the test more useful?
  • Cons: The environment will no longer be representative of what users experience.
• do something else ?

Comment 8

[Alexandru Ionescu (needinfo me) [:alexandrui]]

Thank you, :grepg, for pointing this out. We'll close it as won't fix, and I'll ping :sparky about this to see whether we can do something about this with our tooling.

Comment 9

[Timothy Nikkel (:tnikkel)]

Presumably we already have to deal with the same issue on our desktop tests? So we if don't disable scrollbar fadeout there, we should probably do the same on android.

Comment 10

[Greg Mierzwinski [:sparky]]

We don't disable scrollbar fade on desktop either. I think the fade is something we should disable though because it has the potential to add variability to our results. You can already see this in the cold/warm side-by-side's that :gregp provided.

Comment 11

[Timothy Nikkel (:tnikkel)]

What do you mean by it adds variability? Without the fade out we would still show the scrollbar, but just hide it immediately without a fadeout.

Comment 12

[Greg Mierzwinski [:sparky]]

That's correct, but with the fade, when we look at the warm and cold videos above, you can see that they don't fade in the same way. I'm not sure if that's the browser causing it, or the recording but it would introduce more variability to the last visual change.

Comment 13

[Timothy Nikkel (:tnikkel)]

(In reply to Greg Mierzwinski [:sparky] from comment #12)

That's correct, but with the fade, when we look at the warm and cold videos above, you can see that they don't fade in the same way.

What do you mean "fade in the same way"? I've just spent several minutes comparing the "after" in the warm to the "after" in the cold of the two gifs on this bug on my screen at the same time and I do not see a difference. Can you be more specific as to what difference I am looking for?

Comment 14

[Greg Mierzwinski [:sparky]]

I find the scrollbar fade in the after for the warm pageload to be janky versus the cold pageload one. Another example I've found is gmaps: https://treeherder.mozilla.org/jobs?repo=try&revision=da4d431eef4407fb3b4276996982099350e25cad&group_state=expanded&selectedTaskRun=Cq869vLLRT671xtwLYg-YQ.0

In the warm after video, I'm seeing it disappear very quickly, but in the cold version it takes a bit more time for the fade to complete.

Comment 15

[Greg Mierzwinski [:sparky]]

:tnikkel, is it possible that there's a bug that's causing these differences? I don't think the fade itself is an issue for the tests, but it seems like the rate of change of the fade is variable.

Comment 16

[Timothy Nikkel (:tnikkel)]

Thanks for that additional link.

I would guess that the number of frames per second captured is different (changing in a random fashion) which is what makes it look faster or slower. Do our captures actually capture every single frame that Firefox produces? I suspect that it doesn't. For example, in the warm after video there the scrollbar looks noticeably lighter when it first appears then the scrollbar on the left side of that video. So my guess is that we didn't capture a frame while the scrollbar was at full opacity, but instead our first frame of it is after the fade out animation is in progress. So the animation seems shorter. But the timestamp in the video when the scrollbar disappears should still be accurate (ie thats when the scrollbar disappeared on screen if we had an externally taken video of the screen). So visual pageload analysis should still be able to be done and produce accurate results, no?

Comment 17

[Greg Mierzwinski [:sparky]]

Adding a ni? for myself to look into the recording on android.

Comment 18

[Greg Mierzwinski [:sparky]]

(In reply to Timothy Nikkel (:tnikkel) from comment #16)

Thanks for that additional link.

I would guess that the number of frames per second captured is different (changing in a random fashion) which is what makes it look faster or slower. Do our captures actually capture every single frame that Firefox produces? I suspect that it doesn't.

Good point, we use the adb screenrecord to record videos on Android, and it isn't synchronized to the changes that happen in the browser window so we probably had an unrecorded change happen between frames. That's likely where this issue is coming from.

(In reply to Timothy Nikkel (:tnikkel) from comment #16)

But the timestamp in the video when the scrollbar disappears should still be accurate (ie thats when the scrollbar disappeared on screen if we had an externally taken video of the screen). So visual pageload analysis should still be able to be done and produce accurate results, no?

It'll still impact the *SpeedIndex calculations since they depend on visual completeness, and this gets calculated from the LastVisualChange, and the histograms of the frames leading up to that point which are impacted by the fade out.

That said, I agree that the variability issue I'm seeing is unrelated to the fade out, and it's related to the screen recording instead. We don't need to disable this fade out feature in our pageload tests.