1831856 - Replace median with geomean for performance tests

Status: RESOLVED FIXED

(Testing :: Raptor, task, P1)

Description

[Alexandru Ionescu (needinfo me) [:alexandrui]]

Replace median with geomean for performance tests

Comment 1

[Alexandru Ionescu (needinfo me) [:alexandrui]]

Attachment: Bug 1831856 - Replace median with geomean for performance tests r=#perftest

Comment 2

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/7974ac7e053a
Replace median with geomean for performance tests r=perftest-reviewers,sparky

Comment 3

[Norisz Fay [:noriszfay]]

https://hg.mozilla.org/mozilla-central/rev/7974ac7e053a

Comment 4

[Alex Finder| PTO returning Dec. 8th]

(In reply to Pulsebot from comment #2)

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/7974ac7e053a
Replace median with geomean for performance tests r=perftest-reviewers,sparky

== Change summary for alert #38585 (as of Wed, 07 Jun 2023 12:19:51 GMT) ==

Regressions:

Ratio	Test	Platform	Options	Absolute values (old vs new)	Performance Profiles
96%	speedometer3	windows10-64-nightlyasrelease-qr	fission webrender	253.69 -> 10.24	Before/After
96%	speedometer3	macosx1015-64-nightlyasrelease-qr	fission webrender	251.61 -> 10.82	Before/After
96%	speedometer3	linux1804-64-nightlyasrelease-qr	fission webrender	176.27 -> 7.57	Before/After
96%	speedometer3	android-hw-a51-11-0-aarch64-shippable-qr	webrender	44.01 -> 1.93
27%	buzzfeed loadtime	macosx1015-64-shippable-qr	cold fission webrender	928.94 -> 1,182.51	Before/After
25%	buzzfeed PerceptualSpeedIndex	linux1804-64-shippable-qr	cold fission webrender	1,007.32 -> 1,261.51	Before/After
23%	buzzfeed LastVisualChange	linux1804-64-shippable-qr	cold fission webrender	1,781.67 -> 2,191.59	Before/After
18%	fandom fcp	windows10-64-shippable-qr	bytecode-cached cold fission webrender	292.04 -> 344.07
17%	fandom fcp	windows10-64-shippable-qr	cold fission webrender	289.04 -> 339.44
17%	imgur LastVisualChange	macosx1015-64-shippable-qr	fission warm webrender	3,326.67 -> 3,890.88	Before/After
...	...	...	...	...	...
4%	paypal SpeedIndex	windows10-64-shippable-qr	cold fission webrender	526.58 -> 547.68	Before/After
3%	buzzfeed FirstVisualChange	linux1804-64-shippable-qr	fission warm webrender	160.00 -> 164.97	Before/After
3%	yahoo-mail FirstVisualChange	linux1804-64-shippable-qr	fission warm webrender	160.00 -> 164.92	Before/After
2%	paypal LastVisualChange	windows10-64-shippable-qr	cold fission webrender	1,342.62 -> 1,371.97	Before/After
2%	facebook fcp	windows10-64-shippable-qr	cold fission webrender	836.69 -> 854.92

Improvements:

Ratio	Test	Platform	Options	Absolute values (old vs new)	Performance Profiles
31%	google-docs fcp	windows10-64-shippable-qr	fission warm webrender	552.46 -> 381.38	Before/After
30%	google-docs FirstVisualChange	windows10-64-shippable-qr	fission warm webrender	590.38 -> 414.81	Before/After
21%	instagram FirstVisualChange	linux1804-64-shippable-qr	fission warm webrender	80.00 -> 63.23	Before/After
10%	google-mail fcp	macosx1015-64-shippable-qr	fission warm webrender	470.96 -> 424.38	Before/After
8%	imdb SpeedIndex	macosx1015-64-shippable-qr	fission warm webrender	480.67 -> 441.05	Before/After
...	...	...	...	...	...
4%	google-slides FirstVisualChange	windows10-64-shippable-qr	fission warm webrender	266.92 -> 256.28	Before/After

For up to date results, see: https://treeherder.mozilla.org/perfherder/alerts?id=38585