1919487 - 8.34% google-search FirstVisualChange (Linux) regression on Thu September 5 2024

Status: RESOLVED WONTFIX

(WebExtensions :: General, defect)

Description

[Treeherder Bug Filer]

Perfherder has detected a browsertime performance regression from push df22aff84e9a290fa59741e43bed14154df6b9ab. As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

Ratio	Test	Platform	Options	Absolute values (old vs new)	Performance Profiles
8%	google-search FirstVisualChange	linux1804-64-shippable-qr	fission warm webrender	109.51 -> 118.64	Before/After

Improvements:

Ratio	Test	Platform	Options	Absolute values (old vs new)	Performance Profiles
8%	google-search fcp	windows11-64-shippable-qr	fission warm webrender	44.15 -> 40.54	Before/After
8%	google-search largestContentfulPaint	linux1804-64-shippable-qr	fission warm webrender	67.62 -> 62.47	Before/After
6%	bing-search fcp	windows11-64-shippable-qr	fission warm webrender	51.68 -> 48.38	Before/After
6%	google-search largestContentfulPaint	windows11-64-shippable-qr	fission warm webrender	54.40 -> 51.15	Before/After
6%	bing-search loadtime	macosx1015-64-shippable-qr	fission warm webrender	79.22 -> 74.83	Before/After
...	...	...	...	...	...
3%	google-search loadtime	linux1804-64-shippable-qr	fission warm webrender	178.36 -> 173.23	Before/After

Details of the alert can be found in the alert summary, including links to graphs and comparisons for each of the affected tests. Please follow our guide to handling regression bugs and let us know your plans within 3 business days, or the patch(es) may be backed out in accordance with our regression policy.

If you need the profiling jobs you can trigger them yourself from treeherder job view or ask a sheriff to do that for you.

You can run these tests on try with ./mach try perf --alert 2091

For more information on performance sheriffing please see our FAQ.

Comment 1

[Florin Bilt]

I know that overall it is an improvement, but it is in our procedure that if contains a regression, I have to file a bug

Comment 2

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1916240

Comment 3

[Rob Wu [:robwu]]

When I open the "Marker Table", and search for "web", I can see that the "WebDriver:Navigate" marker, starting at 3.047 in the "Before" profile and 3.024 in the "After" profile. Since the reported absolute value in the report here is 109.51 vs 118.64, I set the marker range to 120. I did so by editing the URL of the profiler and setting range=3046m120 and range=3023m120. This guarantees that the two profiles have the same time scale.

• Before: https://share.firefox.dev/3XAKJIC
• After: https://share.firefox.dev/3B8Xh2H

Now when I compare the Screenshots between the two profiles, I see:

• Before: 3.163 (blank) ~ 3.164 (something visible), delta vs 3.047 start time is 116ms.
• After: 3.116 (blank) ~ 3.117 (something visible), delta vs 3.024 start time is 92ms.

Visually, this clearly looks like an improvement. The time from the start of the navigation until something was visually visible went down, by 24ms.

The main intended change that resulted in this reduction is the faster network response.
The Network tab shows the following relevant info for the time before the response was received:

• Before: 22ms (waiting for socket thread) + 12ms (request + awaiting response) = 34ms. In the Marker Chart, at the "Extension Request", I see that the request was suspended for 13ms.
• After: 1.3ms + 3.7ms = 5ms. The request was no longer suspended.

We thus know that the patch sped up the network request handling by at least 13ms. Since 13ms is larger than the supposed 9ms regression in FirstVisualChange, the overall difference is a net improvement.

I'm inclined to close this bug since the patch resulted in the expected improvements, and it is unlikely for the original work to directly have caused the regression. If there is anything, it would likely be a test-specific issue or an unrelated issue that got exposed after the cause of a slower network went away.

Comment 4

[Rob Wu [:robwu]]

:sparky, the reported regression is surprising considering that a manual inspection shows that the time between navigation and anything visual improved. I'm using the "start of navigation" as initial reference, because I don't know the exact starting point for "first visual change".

Could you take a look to see if this warrants further action? See comment 3 for my analysis.

Comment 5

[Greg Mierzwinski [:sparky]]

I had a look at the graph for this test, and it looks like the google-search FirstVisualChange improved in the patch that caused bug 1916240 (it doesn't look like an alert was generated for this metric there): https://treeherder.mozilla.org/perfherder/graphs?highlightAlerts=1&highlightChangelogData=1&highlightCommonAlerts=0&replicates=0&series=autoland,3776971,1,13&timerange=5184000&zoom=1721995857310,1727232387573,72.21111111111111,143.98888888888888

The patch you landed in bug 1916240 then reverted that change, so we're back to original values now. Here's a bing-search graph for comparison to see that the range is similar between regressor+regressor-fix for it and google-search: https://treeherder.mozilla.org/perfherder/graphs?highlightAlerts=1&highlightChangelogData=1&highlightCommonAlerts=0&replicates=0&series=autoland,3869614,1,13&timerange=5184000

I agree regarding setting this as a wontfix.

Comment 6

[Rob Wu [:robwu]]

Thanks for confirming!