1816545 - Speculative connections are disabled in performance test infrastructure

Status: RESOLVED FIXED

(Testing :: Performance, defect, P1)

Description

[Andrew Creskey [:acreskey]]

In the perf user profile the number of speculative connections is set to 0.

For many performance tests this isn't ideal because it effectively disables important performance codepaths.

Note: we used to have this problem in marionette, but it was fixed there.

As per the comment), we may have to look out for and solve issues with open sockets on shutdown.

This change may also affect baselines (I'm running a partial test).

Comment 1

[Andrew Creskey [:acreskey]]

Overall I think this change is worth making.
But note that some speculative connects will still be aborted by the runtime because we are connected to an https proxy in the perf test environment. See https://bugzilla.mozilla.org/show_bug.cgi?id=1816678#c2

Comment 2

[Alexandru Ionescu (needinfo me) [:alexandrui]]

Attachment: Bug 1816545 - Speculative connections are disabled in performance test infrastructure r=#perftest

Comment 3

[Andrew Creskey [:acreskey]]

In looking at Alex's try push we see some small improvements and small regressions and also a large loadtime regression on Office.

Ultimately this change configures our performance tests as the shipped product, so the baseline shift is warranted, I think.

I did look at the office loadtime regression.
Baseline
With the change, re-enabling the socket pool
In this case the loadTime was shifted because the loadgroup ended up including significantly more resources.
But note that the visual metrics are not affected.

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/b90e80efa3a4
Speculative connections are disabled in performance test infrastructure r=perftest-reviewers,acreskey,sparky

Comment 5

[Cristina Horotan [:chorotan]]

https://hg.mozilla.org/mozilla-central/rev/b90e80efa3a4

Comment 6

[Alexandru Ionescu (needinfo me) [:alexandrui]]

== Change summary for alert #37779 (as of Tue, 28 Mar 2023 05:20:26 GMT) ==

Regressions:

Ratio	Test	Platform	Options	Absolute values (old vs new)	Performance Profiles
3%	buzzfeed ContentfulSpeedIndex	linux1804-64-shippable-qr	fission warm webrender	472.00 -> 485.17	Before/After

Improvements:

Ratio	Test	Platform	Options	Absolute values (old vs new)	Performance Profiles
5%	buzzfeed fcp	linux1804-64-shippable-qr	cold fission webrender	750.96 -> 712.00	Before/After
3%	paypal fcp	windows10-64-shippable-qr	cold fission webrender	308.10 -> 297.83	Before/After

For up to date results, see: https://treeherder.mozilla.org/perfherder/alerts?id=37779

Comment 7

[Andrej (:aglavic)]

== Change summary for alert #37911 (as of Fri, 31 Mar 2023 00:47:44 GMT) ==

Improvements:

Ratio	Test	Platform	Options	Absolute values (old vs new)
8%	tsvg_static	linux1804-64-shippable-qr	e10s fission stylo webrender	79.52 -> 73.07
3%	tp5o	windows10-64-shippable-qr	e10s fission stylo webrender	223.39 -> 217.56
2%	tp5o	windows10-64-shippable-qr	e10s fission stylo webrender	223.60 -> 219.54

For up to date results, see: https://treeherder.mozilla.org/perfherder/alerts?id=37911

Comment 8

[Nicolas Chevobbe [:nchevobbe][PTO back September 7]]

== Change summary for alert #37806 (as of Tue, 28 Mar 2023 19:08:29 GMT) ==

Regressions:

Ratio	Test	Platform	Options	Absolute values (old vs new)
7%	damp custom.inspector.expandall.balanced	windows10-64-shippable-qr	e10s fission stylo webrender-sw	550.61 -> 588.32

Improvements:

Ratio	Test	Platform	Options	Absolute values (old vs new)
4%	damp complicated.styleeditor.reload.DAMP	windows10-64-shippable-qr	e10s fission stylo webrender	1,316.11 -> 1,259.05
4%	damp complicated.styleeditor.reload.DAMP	windows10-64-shippable-qr	e10s fission stylo webrender-sw	1,310.36 -> 1,256.96
4%	damp complicated.jsdebugger.reload.DAMP	windows10-64-shippable-qr	e10s fission stylo webrender-sw	1,567.12 -> 1,509.52
3%	damp complicated.jsdebugger.reload.DAMP	windows10-64-shippable-qr	e10s fission stylo webrender	1,572.99 -> 1,518.01
3%	damp custom.jsdebugger.reload.DAMP	windows10-64-shippable-qr	e10s fission stylo webrender-sw	741.21 -> 715.88
...	...	...	...	...
2%	damp complicated.webconsole.reload.DAMP	windows10-64-shippable-qr	e10s fission stylo webrender-sw	1,578.58 -> 1,544.33

For up to date results, see: https://treeherder.mozilla.org/perfherder/alerts?id=37806