1735196 - Update twitch page load test to mitm7 and add secondary url

Status: RESOLVED FIXED

(Testing :: Raptor, task, P1)

Description

[Kimberly Sereduck :kimberlythegeek]

+++ This bug was initially created as a clone of Bug #1724261 +++

https://bugzilla.mozilla.org/show_bug.cgi?id=1716825 landed a patch to navigate to a sub-page instead of about:blank for warm page load tests.

in this bug we need to:
1 - determine what secondary URL should be used for each test
2 - implement a secondary URL and re-record page load

Comment 1

[Andrej (:aglavic)]

Attachment: Bug 1735196 Update twitch to mitm7 and add secondary url r?#perftest

Comment 2

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/abb3901bfaca
Update twitch to mitm7 and add secondary url r=perftest-reviewers,AlexandruIonescu

Comment 3

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/abb3901bfaca

Comment 4

[Alexandru Ionescu (needinfo me) [:alexandrui]]

== Change summary for alert #33199 (as of Wed, 09 Feb 2022 04:10:22 GMT) ==

Regressions:

Ratio	Test	Platform	Options	Absolute values (old vs new)
46%	twitch fcp	macosx1015-64-shippable-qr	cold fission webrender	52.71 -> 77.17
45%	twitch fnbpaint	macosx1015-64-shippable-qr	cold fission webrender	53.42 -> 77.33
38%	twitch fcp	linux1804-64-shippable-qr	cold fission webrender	95.58 -> 132.38
38%	twitch fnbpaint	linux1804-64-shippable-qr	cold fission webrender	95.38 -> 132.00
34%	twitch LastVisualChange	windows10-64-shippable-qr	cold fission webrender	4,639.08 -> 6,236.25
34%	twitch FirstVisualChange	windows10-64-shippable-qr	cold fission webrender	100.00 -> 133.83
28%	twitch ContentfulSpeedIndex	linux1804-64-shippable-qr	cold fission webrender	1,107.67 -> 1,417.58
26%	twitch fnbpaint	windows10-64-shippable-qr	cold fission webrender	80.96 -> 102.12
26%	twitch fcp	windows10-64-shippable-qr	cold fission webrender	81.00 -> 101.67
23%	twitch dcf	linux1804-64-shippable-qr	fission warm webrender	534.40 -> 659.58
...	...	...	...	...
15%	twitch loadtime	windows10-64-shippable-qr	fission warm webrender	588.67 -> 675.83
14%	twitch loadtime	linux1804-64-shippable-qr	cold fission webrender	911.92 -> 1,043.33
12%	twitch dcf	windows10-64-shippable-qr	cold fission webrender	644.75 -> 725.04
10%	twitch dcf	linux1804-64-shippable-qr	cold fission webrender	679.02 -> 745.21
9%	twitch SpeedIndex	macosx1015-64-shippable-qr	fission warm webrender	351.50 -> 381.92

Improvements:

Ratio	Test	Platform	Options	Absolute values (old vs new)
69%	twitch ContentfulSpeedIndex	macosx1015-64-shippable-qr	fission warm webrender	881.83 -> 276.08
50%	twitch ContentfulSpeedIndex	windows10-64-shippable-qr	fission warm webrender	1,155.42 -> 577.50
49%	twitch ContentfulSpeedIndex	linux1804-64-shippable-qr	fission warm webrender	895.83 -> 456.58
28%	twitch PerceptualSpeedIndex	macosx1015-64-shippable-qr	fission warm webrender	257.75 -> 184.67
21%	twitch PerceptualSpeedIndex	windows10-64-shippable-qr	cold fission webrender	764.33 -> 600.42
...	...	...	...	...
7%	twitch LastVisualChange	macosx1015-64-shippable-qr	fission warm webrender	6,143.33 -> 5,743.33

For up to date results, see: https://treeherder.mozilla.org/perfherder/alerts?id=33199