1864921 - Run Speedometer 3 once instead of 5 times per job. (Additionally, this bug was supposed to switch median to mean as the summary statistic, but that part didn't work, see bug 1867959)

Status: RESOLVED FIXED

(Testing :: Raptor, defect, P1)

Description

[Jeff Muizelaar [:jrmuizel]]

Using the mean matches how upstream speedometer combines the scores.

Only running the suite once makes sure that we weight the first run the
same as when running it in the default configuration.

Comment 1

[Jeff Muizelaar [:jrmuizel]]

Attachment: Bug 1864921 - Switch SP3 to use the mean as the summary statistic and do cold iterations.

Using the mean matches how upstream speedometer combines the scores.

Only running the suite once makes sure that we weight the first run the
same as when running it in the default configuration.

Comment 2

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/f1790974c5ec
Switch SP3 to use the mean as the summary statistic and do cold iterations. r=sparky,perftest-reviewers

Comment 3

[Cristian Tuns]

Backed out for causing sp3 failures

• Backout link
• Push with failures
• Failure Log
• Failure line: Error copying results /opt/worker/tasks/task_170062173159417/build/raptor.json to upload dir /opt/worker/tasks/task_170062173159417/build/blobber_upload_dir/perfherder-data.json

Comment 4

[Jeff Muizelaar [:jrmuizel]]

Failure was:

ask 2023-11-22T03:43:22.242Z] 03:43:22    ERROR -  Traceback (most recent call last):
[task 2023-11-22T03:43:22.242Z] 03:43:22     INFO -    File "/opt/worker/tasks/task_170062173159417/build/tests/raptor/raptor/raptor.py", line 186, in <module>
[task 2023-11-22T03:43:22.242Z] 03:43:22     INFO -      main()
[task 2023-11-22T03:43:22.242Z] 03:43:22     INFO -    File "/opt/worker/tasks/task_170062173159417/build/tests/raptor/raptor/raptor.py", line 60, in main
[task 2023-11-22T03:43:22.242Z] 03:43:22     INFO -      raptor_test_list = get_raptor_test_list(args, mozinfo.os)
[task 2023-11-22T03:43:22.242Z] 03:43:22     INFO -    File "/opt/worker/tasks/task_170062173159417/build/tests/raptor/raptor/manifest.py", line 518, in get_raptor_test_list
[task 2023-11-22T03:43:22.242Z] 03:43:22     INFO -      next_test["expected_browser_cycles"] = int(next_test["browser_cycles"])
[task 2023-11-22T03:43:22.242Z] 03:43:22     INFO -  KeyError: 'browser_cycles'

Greg, any idea the cause of this?

Comment 5

[Greg Mierzwinski [:sparky]]

browser_cycles needs to be defined in the manifests that you've added cold = true to. In your case, you'd want to add this in the speedometer3 tests for desktop, and mobile:

browser_cycles = 1

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/8366d9b1b0ca
Switch SP3 to use the mean as the summary statistic and do cold iterations. r=sparky,perftest-reviewers

Comment 7

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/8366d9b1b0ca

Comment 8

[Alex Finder]

(In reply to Pulsebot from comment #6)

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/8366d9b1b0ca
Switch SP3 to use the mean as the summary statistic and do cold iterations.
r=sparky,perftest-reviewers

== Change summary for alert #40387 (as of Fri, 24 Nov 2023 09:58:13 GMT) ==

Improvements:

Ratio	Test	Platform	Options	Absolute values (old vs new)	Performance Profiles
2%	speedometer3	macosx1300-64-shippable-qr	fission webrender	15.82 -> 16.14	Before/After

For up to date results, see: https://treeherder.mozilla.org/perfherder/alerts?id=40387

Comment 9

[Markus Stange [:mstange]]

Changing summary to describe the effects the patch ended up having.