1695879 - 13.83 - 15.07% microsoft FirstVisualChange / microsoft PerceptualSpeedIndex / microsoft SpeedIndex (linux1804-64-shippable) regression on push ff46eda9cc314eb255bb17fccb9c052c1d2107a0 (Fri February 26 2021)

Status: RESOLVED FIXED

(Core :: Networking: HTTP, defect)

Description

[Ionuț Goldan [:igoldan]]

Perfherder has detected a browsertime performance regression from push ff46eda9cc314eb255bb17fccb9c052c1d2107a0. As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

Ratio	Suite	Test	Platform	Options	Absolute values (old vs new)
15%	microsoft	FirstVisualChange	linux1804-64-shippable	cold nocondprof	486.67 -> 560.00
14%	microsoft	SpeedIndex	linux1804-64-shippable	cold nocondprof	494.92 -> 566.08
14%	microsoft	PerceptualSpeedIndex	linux1804-64-shippable	cold nocondprof	505.50 -> 575.42

Details of the alert can be found in the alert summary, including links to graphs and comparisons for each of the affected tests. Please follow our guide to handling regression bugs and let us know your plans within 3 business days, or the offending patch(es) will be backed out in accordance with our regression policy.

For more information on performance sheriffing please see our FAQ.

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1689554

Comment 2

[Dragana Damjanovic [:dragana]]

The link to the push( ff46eda9cc314eb255bb17fccb9c052c1d2107a0) does not show bug 1689554

Can you check which part is incorrect the bug number or the change set.

Comment 3

[Dave Hunt [:davehunt] [he/him] ⌚BST]

(In reply to Dragana Damjanovic [:dragana] from comment #2)

The link to the push( ff46eda9cc314eb255bb17fccb9c052c1d2107a0) does not show bug 1689554

Can you check which part is incorrect the bug number or the change set.

The push refers to when Perfherder detected the regression, however as we don't run performance tests on every push it's rarely the cause of the regression. The performance sheriffs retrigger and backfill to determine the patch that appears to cause the regression. You can see the result of these retriggers in this chart. The solid vertical line is where the alert was detected, and the earlier datapoint with a similar result was determined to be the cause. This is ba635fc1e41aca5570bf23b712094d6fa6438a6b and bug 1689554.

Comment 4

[Dragana Damjanovic [:dragana]]

Thanks,
can someone point me to the test code that regress?

These patches almost only touch HTTP3 and will be only used if HTTP3 is used and HTTP3 is disabled by default.

There is one that influence doe when HTTP3 is disabled.

Comment 5

[Ionuț Goldan [:igoldan]]

Alex, can you answer Dragana' s question above?

Comment 6

[Alexandru Ionescu (needinfo me) [:alexandrui]]

Attachment: warm-side-by-side.mp4

Dragana, I think this is what you need

Comment 7

[Ionuț Goldan [:igoldan]]

Found other regressions as well, from browsertime this time:

== Change summary for alert #28962 (as of Mon, 01 Mar 2021 13:08:29 GMT) ==

Regressions:

Ratio	Suite	Test	Platform	Options	Absolute values (old vs new)
114%	raptor-tp6-cnn-ampstories-firefox-cold	loadtime	linux1804-64-shippable-qr	live nocondprof webrender	1,026.17 -> 2,193.50
107%	raptor-tp6-cnn-ampstories-firefox-cold	loadtime	linux1804-64-shippable	live nocondprof	1,053.75 -> 2,176.50
89%	raptor-tp6-cnn-ampstories-firefox-cold	loadtime	windows10-64-shippable	live nocondprof	1,032.29 -> 1,951.50
24%	raptor-tp6-cnn-ampstories-firefox-cold		linux1804-64-shippable-qr	live nocondprof webrender	426.65 -> 528.23
23%	raptor-tp6-cnn-ampstories-firefox-cold		linux1804-64-shippable	live nocondprof	439.83 -> 540.30
14%	raptor-tp6-cnn-ampstories-firefox-cold	fnbpaint	linux1804-64-shippable-qr	live nocondprof webrender	557.96 -> 634.00
11%	raptor-tp6-cnn-ampstories-firefox-cold	fnbpaint	linux1804-64-shippable	live nocondprof	578.42 -> 640.17
11%	raptor-tp6-cnn-ampstories-firefox-cold	fnbpaint	linux1804-64-shippable-qr	live nocondprof webrender	578.00 -> 639.08
5%	raptor-tp6-cnn-ampstories-firefox-cold	fcp	linux1804-64-shippable-qr	live nocondprof webrender	1,048.58 -> 1,103.33

For up to date results, see: https://treeherder.mozilla.org/perfherder/alerts?id=28962

Comment 8

[Dragana Damjanovic [:dragana]]

Attachment: Removing QUIC from nsSocketTransport was incorrect

This cause some of the socket options not to be set.

Comment 9

[Dragana Damjanovic [:dragana]]

https://treeherder.mozilla.org/jobs?repo=try&revision=4d588e644f19cafcc0255b6c1ffe9c9bd940801a

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/e48072625e17
Removing QUIC from nsSocketTransport was incorrect r=necko-reviewers,valentin

Comment 11

[Dorel Luca [:dluca]]

https://hg.mozilla.org/mozilla-central/rev/e48072625e17

Comment 12

[Ionuț Goldan [:igoldan]]

I confirm the regressions were fixed:

== Change summary for alert #29162 (as of Wed, 10 Mar 2021 06:55:20 GMT) ==

Improvements:

Ratio	Suite	Test	Platform	Options	Absolute values (old vs new)
58%	raptor-tp6-cnn-ampstories-firefox-cold	loadtime	macosx1015-64-shippable-qr	live nocondprof webrender	2,068.17 -> 877.15
32%	raptor-tp6-cnn-ampstories-firefox-cold		macosx1015-64-shippable-qr	live nocondprof webrender	501.83 -> 342.15
28%	raptor-tp6-cnn-ampstories-firefox-cold	fcp	macosx1015-64-shippable-qr	live nocondprof webrender	1,046.08 -> 755.92
26%	raptor-tp6-cnn-ampstories-firefox-cold	fnbpaint	macosx1015-64-shippable-qr	live nocondprof webrender	585.67 -> 432.31
20%	raptor-tp6-cnn-ampstories-firefox-cold	loadtime	linux64-shippable	live nocondprof	2,137.42 -> 1,701.42
20%	raptor-tp6-cnn-ampstories-firefox-cold	dcf	linux64-shippable	live nocondprof	51.67 -> 41.42

For up to date results, see: https://treeherder.mozilla.org/perfherder/alerts?id=29162

Comment 13

[Florin Strugariu [:Bebe]]

== Change summary for alert #29205 (as of Thu, 11 Mar 2021 06:23:13 GMT) ==

Improvements:

Ratio	Suite	Test	Platform	Options	Absolute values (old vs new)
24%	youtube-watch	fcp	android-hw-g5-7-0-arm7-api-16-shippable	cold nocondprof	1,845.06 -> 1,401.58
18%	microsoft	LastVisualChange	windows10-64-shippable	nocondprof warm	438.18 -> 358.17
12%	bing		android-hw-p2-8-0-android-aarch64-shippable	cold nocondprof webrender	300.97 -> 263.91
12%	yahoo-news	loadtime	windows10-64-shippable	cold nocondprof	1,876.04 -> 1,658.62
11%	google		android-hw-p2-8-0-android-aarch64-shippable	cold nocondprof	347.01 -> 307.68
10%	google		android-hw-p2-8-0-android-aarch64-shippable	cold nocondprof	340.42 -> 307.63
8%	apple	loadtime	linux1804-64-shippable	nocondprof warm	300.53 -> 277.46
8%	apple		linux1804-64-shippable	nocondprof warm	293.97 -> 271.83
7%	amazon-search	loadtime	android-hw-g5-7-0-arm7-api-16-shippable	cold nocondprof	3,997.50 -> 3,701.00
7%	apple	fcp	linux1804-64-shippable	nocondprof warm	293.39 -> 272.67
5%	ebay-kleinanzeigen	ContentfulSpeedIndex	android-hw-g5-7-0-arm7-api-16-shippable	cold nocondprof webrender	2,090.29 -> 1,988.75
5%	apple	fcp	linux64-shippable	cold nocondprof	585.94 -> 559.58
4%	apple	fcp	linux64-shippable-qr	nocondprof warm webrender	301.29 -> 288.90
4%	google-mail	loadtime	linux64-shippable-qr	nocondprof warm webrender	618.62 -> 594.08
4%	apple	ContentfulSpeedIndex	linux1804-64-shippable-qr	cold nocondprof webrender	766.26 -> 735.92
4%	apple	SpeedIndex	linux1804-64-shippable-qr	cold nocondprof webrender	630.21 -> 605.50
4%	apple		linux64-shippable-qr	nocondprof warm webrender	299.50 -> 288.33
3%	apple	loadtime	linux64-shippable-qr	cold nocondprof webrender	1,045.52 -> 1,009.71
3%	youtube-watch	loadtime	android-hw-g5-7-0-arm7-api-16-shippable	cold nocondprof	2,446.42 -> 2,362.89
3%	cnn	fcp	android-hw-g5-7-0-arm7-api-16-shippable	cold nocondprof webrender	3,178.44 -> 3,070.71
3%	apple	loadtime	linux64-shippable-qr	nocondprof warm webrender	301.83 -> 291.79
3%	apple		linux64-shippable-qr	cold nocondprof webrender	676.77 -> 654.63
3%	cnn	FirstVisualChange	android-hw-g5-7-0-arm7-api-16-shippable	cold nocondprof webrender	3,656.42 -> 3,549.17
3%	twitter	ContentfulSpeedIndex	linux64-shippable	cold nocondprof	1,691.75 -> 1,648.58
2%	twitter	PerceptualSpeedIndex	linux64-shippable	cold nocondprof	1,201.38 -> 1,173.75
2%	microsoft	ContentfulSpeedIndex	windows10-64-shippable	nocondprof warm	206.30 -> 202.17

For up to date results, see: https://treeherder.mozilla.org/perfherder/alerts?id=29205

Comment 14

[Florin Strugariu [:Bebe]]

== Change summary for alert #29676 (as of Tue, 13 Apr 2021 06:05:26 GMT) ==

Improvements:

Ratio	Suite	Test	Platform	Options	Absolute values (old vs new)
8%	raptor-tp6-cnn-ampstories-firefox-cold	dcf	macosx1015-64-shippable-qr	live nocondprof webrender	49.29 -> 45.25
7%	raptor-tp6-cnn-ampstories-firefox-cold	dcf	macosx1015-64-shippable-qr	live nocondprof webrender	48.21 -> 44.75

For up to date results, see: https://treeherder.mozilla.org/perfherder/alerts?id=29676