1862156 - 27.52% reddit-billgates-post-1.comments ContentfulSpeedIndex (Windows) regression on Mon October 23 2023

Status: RESOLVED WORKSFORME

(Firefox :: Address Bar, defect, P2)

Description

[Alex Finder-PTO|back on Sept 16th]

Perfherder has detected a browsertime performance regression from push 785ce00db669bd935155ddad043e488d5a0dc22f. As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

Ratio	Test	Platform	Options	Absolute values (old vs new)
28%	reddit-billgates-post-1.comments ContentfulSpeedIndex	windows10-64-shippable-qr	cold fission webrender	174.05 -> 221.95

Improvements:

Ratio	Test	Platform	Options	Absolute values (old vs new)
2%	reddit-billgates-post-1.billg loadtime	windows10-64-shippable-qr	cold fission webrender	1,313.13 -> 1,286.55
2%	reddit-billgates-post-1.comments loadtime	windows10-64-shippable-qr	cold fission webrender	1,313.13 -> 1,286.55
2%	reddit-billgates-post-1.posts loadtime	windows10-64-shippable-qr	cold fission webrender	1,313.13 -> 1,286.55

Details of the alert can be found in the alert summary, including links to graphs and comparisons for each of the affected tests. Please follow our guide to handling regression bugs and let us know your plans within 3 business days, or the patch(es) may be backed out in accordance with our regression policy.

If you need the profiling jobs you can trigger them yourself from treeherder job view or ask a sheriff to do that for you.

For more information on performance sheriffing please see our FAQ.

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1853418

Comment 2

[Dão Gottwald [:dao]]

From what I understand the insecure connection label should be used in this test. Trimming https:// would be part of it, but that should be the case for many tests, so that wouldn't explain this particular regression.

It also looks like numbers may have come down to previous levels today: https://treeherder.mozilla.org/perfherder/graphs?highlightAlerts=1&highlightChangelogData=1&highlightCommonAlerts=0&selected=4060519,1784144926&series=autoland,4060519,1,13&timerange=1209600
Nevermind, this seems to have been an illusion.

Does this test load real reddit.com rather than a copy? If so I suppose the fluctuation could be entirely unrelated to any change on our end...

Comment 3

[Marco Bonardo [:mak]]

We should investigate starting from https://treeherder.mozilla.org/jobs?repo=autoland&fromchange=7a7af6939fe90e2308af1f0d49d87ac46c9d3925&tochange=785ce00db669bd935155ddad043e488d5a0dc22f&selectedTaskRun=QlZcL5l6RCu2O_O1Jg8Ssg.0 and compare both the videos and the Perf profiles between the enabling of the feature and a previous revision.

We noticed an update of the browsertime harness not too long before the regression, we don't know if there's any relation to that.

Comment 4

[Marco Bonardo [:mak]]

We noticed a slight increase also on Linux.

Comment 5

[Marco Bonardo [:mak]]

The regression returned to baseline around Dec 6, I couldn't find an alert for the improvement, and examining improvment changesets didn't bring up anything particularly useful (some GC fixes, some external string handling in js that maybe helped?).
Comparing the profiles the only relevant difference I see is a ValueChange event dispatched for accessibility reasons taking a bit more time, and some GC timing shift.
In the "bad" version the content also appears earlier... I don't think we'll extract any useful action from this report.

Comment 6

[Dão Gottwald [:dao]]

Taking this over from mseibert. Looks like we'll likely close this as WORKSFORME, but before doing so I'll investigate a bit further.

Comment 7

[Dão Gottwald [:dao]]

Confirmed that there's no regression anymore. This is current mozilla-central with the security.insecure_connection_text.enabled, security.insecure_connection_text.pbmode.enabled , and browser.urlbar.trimHttps prefs disabled (that's the pre-bug 1853418 state; result: 151.85 ± 19.07) vs. them enabled (result: 146.47 ± 24.36):
https://treeherder.mozilla.org/perfherder/compare?originalProject=try&originalRevision=c0574c65f9fadd99fe21cca7f165ba982e79faf0&newProject=try&newRevision=a07434d6353dd69c7673083b2abcbcbde74542bb&page=1&framework=13&filter=64-shippable+ContentfulSpeedIndex+fission+comments

So for this try run it's actually a -3.55% improvement.