1733166 - Update microsoft warm page load test to navigate to sub-page rather than about:blank

Status: RESOLVED FIXED

(Testing :: Performance, task, P1)

Description

[Kimberly Sereduck :kimberlythegeek]

+++ This bug was initially created as a clone of Bug #1724261 +++

https://bugzilla.mozilla.org/show_bug.cgi?id=1716825 landed a patch to navigate to a sub-page instead of about:blank for warm page load tests.

in this bug we need to:
1 - determine what secondary URL should be used for each test
2 - implement a secondary URL and re-record page load

Comment 1

[Kimberly Sereduck :kimberlythegeek]

Attachment: Bug 1733166: Update microsoft warm page load test to navigate to sub-page rather than about:blank

Comment 2

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/281143cff91f
Update microsoft warm page load test to navigate to sub-page rather than about:blank r=perftest-reviewers,AlexandruIonescu

Comment 3

[Natalia Csoregi [:nataliaCs]]

https://hg.mozilla.org/mozilla-central/rev/281143cff91f

Comment 4

[Alexandru Ionescu (needinfo me) [:alexandrui]]

== Change summary for alert #31689 (as of Tue, 05 Oct 2021 08:00:35 GMT) ==

Regressions:

Ratio	Test	Platform	Options	Absolute values (old vs new)
124%	microsoft ContentfulSpeedIndex	linux1804-64-shippable-qr	warm webrender	240.33 -> 538.08
115%	microsoft ContentfulSpeedIndex	macosx1015-64-shippable-qr	warm webrender	251.75 -> 541.67
114%	microsoft ContentfulSpeedIndex	macosx1014-64-shippable-qr	warm webrender	306.25 -> 656.50
114%	microsoft FirstVisualChange	macosx1014-64-shippable-qr	cold webrender	753.33 -> 1,610.00
111%	microsoft SpeedIndex	macosx1014-64-shippable-qr	cold webrender	765.50 -> 1,613.58
110%	microsoft PerceptualSpeedIndex	macosx1014-64-shippable-qr	cold webrender	784.17 -> 1,647.42
108%	microsoft LastVisualChange	linux1804-64-shippable-qr	warm webrender	520.00 -> 1,083.33
104%	microsoft ContentfulSpeedIndex	macosx1014-64-shippable-qr	cold webrender	801.67 -> 1,638.50
104%	microsoft LastVisualChange	macosx1015-64-shippable-qr	warm webrender	563.33 -> 1,146.67
103%	microsoft PerceptualSpeedIndex	linux1804-64-shippable-qr	cold webrender	540.92 -> 1,096.17
...	...	...	...	...
18%	reddit LastVisualChange	macosx1015-64-shippable-qr	cold webrender	2,633.33 -> 3,103.33
15%	nytimes loadtime	macosx1014-64-shippable-qr	warm webrender	1,724.46 -> 1,982.46
13%	microsoft dcf	macosx1014-64-shippable-qr	warm webrender	338.96 -> 383.33
6%	microsoft dcf	windows10-64-shippable-qr	cold webrender	425.92 -> 450.96
4%	microsoft loadtime	windows10-64-shippable-qr	warm webrender	294.29 -> 307.50

Improvements:

Ratio	Test	Platform	Options	Absolute values (old vs new)
43%	microsoft loadtime	linux1804-64-shippable-qr	cold webrender	709.08 -> 403.83
42%	microsoft loadtime	macosx1014-64-shippable-qr	cold webrender	1,062.33 -> 615.00
39%	microsoft fcp	macosx1014-64-shippable-qr	cold webrender	692.17 -> 421.92
37%	microsoft fnbpaint	macosx1014-64-shippable-qr	cold webrender	704.71 -> 443.67
37%	microsoft fcp	windows10-64-shippable-qr	cold webrender	406.33 -> 257.33
...	...	...	...	...
30%	microsoft dcf	linux1804-64-shippable-qr	cold webrender	514.33 -> 359.42

For up to date results, see: https://treeherder.mozilla.org/perfherder/alerts?id=31689