1730597 - Update cnn page load test to navigate to sub-page rather than about:blank

Status: RESOLVED FIXED

(Testing :: Performance, task, P3)

Description

[Kimberly Sereduck :kimberlythegeek]

+++ This bug was initially created as a clone of Bug #1724261 +++

https://bugzilla.mozilla.org/show_bug.cgi?id=1716825 landed a patch to navigate to a sub-page instead of about:blank for warm page load tests.

in this bug we need to:
1 - determine what secondary URL should be used for each test
2 - implement a secondary URL and re-record page load

Comment 1

[Kimberly Sereduck :kimberlythegeek]

Attachment: Bug 1730597: Update cnn page load test to navigate to sub-page rather than about:blank

Comment 2

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/74f2e0ba916e
Update cnn page load test to navigate to sub-page rather than about:blank r=perftest-reviewers,AlexandruIonescu

Comment 3

[Pulsebot]

Backout by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/c8d8dfab31ef
Backed out changeset 74f2e0ba916e for causing Btime performance test failures.

Comment 4

[Marian-Vasile Laza]

Backed out changeset 74f2e0ba916e (Bug 1730597) for causing Btime performance test failures.
Backout link
Push with failures - cnn-viesmet
Failure Log

Comment 5

[Kimberly Sereduck :kimberlythegeek]

triggered a try run to see if this is an issue with my changes in this patch or caused by re-recording in mitm6

Comment 6

[Kimberly Sereduck :kimberlythegeek]

The mitm6 re-recording did not cause the same failures on cnn-vismet. Will investigate further

Comment 7

[Alexandru Ionescu (needinfo me) [:alexandrui]]

== Change summary for alert #31561 (as of Wed, 29 Sep 2021 19:13:45 GMT) ==

Regressions:

Ratio	Test	Platform	Options	Absolute values (old vs new)
15%	cnn ContentfulSpeedIndex	macosx1015-64-shippable-qr	warm webrender	1,272.77 -> 1,458.79

For up to date results, see: https://treeherder.mozilla.org/perfherder/alerts?id=31561

Comment 8

[Kimberly Sereduck :kimberlythegeek]

Attachment: WIP: Bug 1730597: mitm7 cnn secondary url

Depends on D127740

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/7a13a0605ca5
Update cnn page load test to navigate to sub-page rather than about:blank r=perftest-reviewers,AlexandruIonescu,sparky

Comment 10

[Cristian Tuns]

https://hg.mozilla.org/mozilla-central/rev/7a13a0605ca5

Comment 11

[Alexandru Ionescu (needinfo me) [:alexandrui]]

(In reply to Pulsebot from comment #9)

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/7a13a0605ca5
Update cnn page load test to navigate to sub-page rather than about:blank
r=perftest-reviewers,AlexandruIonescu,sparky

== Change summary for alert #31816 (as of Tue, 12 Oct 2021 05:20:14 GMT) ==

Regressions:

Ratio	Test	Platform	Options	Absolute values (old vs new)
168%	cnn LastVisualChange	macosx1014-64-shippable-qr	cold webrender	4,873.33 -> 13,060.00
161%	cnn loadtime	macosx1014-64-shippable-qr	cold webrender	3,450.67 -> 9,023.25
158%	cnn loadtime	macosx1014-64-shippable-qr	cold webrender	3,449.94 -> 8,909.75
76%	cnn loadtime	macosx1014-64-shippable-qr	warm webrender	2,106.67 -> 3,713.00
50%	cnn SpeedIndex	macosx1014-64-shippable-qr	cold webrender	3,568.58 -> 5,343.50
48%	cnn LastVisualChange	macosx1014-64-shippable-qr	warm webrender	3,966.67 -> 5,866.67
26%	cnn ContentfulSpeedIndex	macosx1014-64-shippable-qr	cold webrender	2,416.67 -> 3,048.42

Improvements:

Ratio	Test	Platform	Options	Absolute values (old vs new)
36%	cnn dcf	macosx1014-64-shippable-qr	warm webrender	1,320.54 -> 842.92
34%	cnn PerceptualSpeedIndex	macosx1014-64-shippable-qr	warm webrender	2,360.00 -> 1,563.92
22%	cnn fnbpaint	macosx1014-64-shippable-qr	warm webrender	1,102.88 -> 860.96
22%	cnn fcp	macosx1014-64-shippable-qr	warm webrender	1,098.04 -> 859.46
20%	cnn FirstVisualChange	macosx1014-64-shippable-qr	warm webrender	1,123.33 -> 900.00
...	...	...	...	...
11%	cnn FirstVisualChange	macosx1014-64-shippable-qr	cold webrender	1,603.33 -> 1,430.00

For up to date results, see: https://treeherder.mozilla.org/perfherder/alerts?id=31816