1812589 - Ensure the CPU frequency, and other optimizations are enabled on the A51

Status: RESOLVED FIXED

(Testing :: Raptor, task, P1)

Description

[Greg Mierzwinski [:sparky]]

This bug is for making sure the CPU frequency, and other performance settings (or optimizations), are setup on the A51 devices.

Comment 1

[Andrej (:aglavic)]

Looks like Peter H. setup work on browsertime to use pinned cpu frequencies:
https://github.com/sitespeedio/browsertime/pulls?q=is%3Apr+a51
Will investigate along this path and follow up with results

Comment 2

[Andrej (:aglavic)]

Double check that any optimizations done for p2 and g5 are also being done for the a51 see also: https://searchfox.org/mozilla-central/source/testing/raptor/raptor/performance_tuning.py

Comment 3

[Andrej (:aglavic)]

Attachment: Bug 1812589 - Ensure the CPU frequency, and other optimizations are enabled on the A51. r?#perftest

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/24de8014b48f
Ensure the CPU frequency, and other optimizations are enabled on the A51. r=perftest-reviewers,sparky

Comment 5

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/24de8014b48f

Comment 6

[Chris Peterson [:cpeterson]]

== Change summary for alert #38826 (as of Thu, 22 Jun 2023 14:53:17 GMT) ==

Improvements:

Ratio	Test	Platform	Options	Absolute values (old vs new)
38%	sina PerceptualSpeedIndex	android-hw-a51-11-0-aarch64-shippable-qr	warm webrender	351.45 -> 219.29
37%	sina SpeedIndex	android-hw-a51-11-0-aarch64-shippable-qr	warm webrender	354.27 -> 222.18
37%	sina FirstVisualChange	android-hw-a51-11-0-aarch64-shippable-qr	warm webrender	345.80 -> 217.83
37%	sina ContentfulSpeedIndex	android-hw-a51-11-0-aarch64-shippable-qr	warm webrender	347.21 -> 219.20
35%	sina fcp	android-hw-a51-11-0-aarch64-shippable-qr	warm webrender	221.45 -> 144.19
...	...	...	...	...
4%	imdb LastVisualChange	android-hw-a51-11-0-aarch64-shippable-qr	warm webrender	7,390.59 -> 7,058.85

For up to date results, see: https://treeherder.mozilla.org/perfherder/alerts?id=38826

Comment 7

[Olli Pettay [:smaug][[e-mail]]]

What was the reason for this change? Don't we want to run Android devices closer to the way users run them. The change seems to have artificially improved also sp3 score. This makes it hard to test thread scheduling changes on CI, since the results not match what one can see locally.

Comment 8

[Greg Mierzwinski [:sparky]]

:smaug, we made the change to help stabilize test results from android. We can disable this tuning on sp3 if needed: https://searchfox.org/mozilla-central/source/testing/raptor/raptor/cmdline.py#345

Comment 9

[Olli Pettay [:smaug][[e-mail]]]

I think at least on a51 we should disable it, since that is used for sp3. But it is unclear to me why we need this tuning anywhere?
Bas might have an opinion too.

Comment 10

[Greg Mierzwinski [:sparky]]

Ok, we'll file a bug to take care of disabling it on sp3. These CPU settings provide us with more stable values that improve our regression detection capabilities (catching smaller changes, and less invalid alerts).

Comment 11

[Bas Schouten (:bas.schouten)]

(In reply to Olli Pettay [:smaug][[e-mail]] from comment #9)

I think at least on a51 we should disable it, since that is used for sp3. But it is unclear to me why we need this tuning anywhere?
Bas might have an opinion too.

I think the reality in benchmarking is that the stability of the system requires us to minimize circumstantial frequency boosts and such (they introduce issues such as noisy neighbor effects, differences due to temperature of the environment, etc.) so sacrificing some score for stability makes sense. Especially since this also makes Speedometer scores more reliably comparable between platforms.

Comment 12

[Olli Pettay [:smaug][[e-mail]]]

We got higher scores because of the change. Higher scores and better stability, but less realistic numbers.

Comment 13

[Olli Pettay [:smaug][[e-mail]]]

er, so do we still run sp3 with this? That means unrealistic numbers

Comment 14

[Greg Mierzwinski [:sparky]]

Based on what Bas mentioned, I didn't make any patches to disable it. I think it would be good to discuss this in the next staff meeting if it's still a concern.

Comment 15

[Chris Peterson [:cpeterson]]

Do we test Chrome with these same A51 performance settings?

To get stable but more realistic numbers, perhaps we could use the "powersave" scaling_governor policy instead of "performance"? "powersave" forces the CPU to use the lowest possible clock frequency and not change on demand.

Comment 16

[Greg Mierzwinski [:sparky]]

Yes, all android devices/tests have this tuning. We could look into changing to powersave. IIRC the values we set here for the minimum frequency are the maximum values that the CPU can run at: https://searchfox.org/mozilla-central/source/testing/raptor/raptor/performance_tuning.py#169-170

Comment 17

[Olli Pettay [:smaug][[e-mail]]]

We should use whatever is the default on the devices. Android devices have rather different types of CPUs comparing to desktop and it is very important to get such testing where big cores may run fast, but little cores then a lot slower, since that seems to happen normally.
Browsers have so many threads and processes these days that the scheduling affects performance a lot.