Linux speedometer3 has a high amount of bi-modality
Categories
(Testing :: Raptor, defect, P2)
Tracking
(Not tracked)
People
(Reporter: sparky, Assigned: kshampur)
References
Details
(Whiteboard: [fxp][operational])
Attachments
(2 files)
The linux sp3 test is showing a lot of bimodality at the moment. See here for a sample graph of it: https://treeherder.mozilla.org/perfherder/graphs?highlightAlerts=1&highlightChangelogData=0&highlightCommonAlerts=0&highlightInitialDataPoints=0&replicates=0&series=autoland,5691152,1,13&timerange=1209600&zoom=1774982911573,1776203969593,11.033555555555555,15.059444444444445
It started to become much more prominent once we enabled the 12 task duplicates on autoland.
Updated•7 months ago
|
Comment 1•7 months ago
|
||
We should check if some machines are slower than others
Comment 2•7 months ago
|
||
Pretty sure it was bimodal before, but then we stopped reporting all the subscores briefly, then the bimodality came back.
Comment 3•7 months ago
|
||
(Doesn't -seem- to be machine though)
| Reporter | ||
Comment 4•7 months ago
|
||
I made a graph here, and I don't see an obvious pattern related to specific machines being different from others: https://sql.telemetry.mozilla.org/queries/117597#287629
Comment 6•7 months ago
|
||
https://quiet-haupia-492082.netlify.app/speedometer?os=linux&replicates=true&repository=autoland&colorByMachine=true&range=week also suggests that this is not correlated to the machine the test runs on. i.e. the same machine will have fast runs and slow runs.
The chrome runs are noisy too which further suggests that this might not be Firefox specific.
It might be worth trying to collect CPU clocks from fast runs and slow runs to see if there's a noticeable difference there.
Here are some example profiles that are from fast runs and slow runs. I haven't verified that the profile run also shows the same speed difference.
Linux profiles from CI: https://share.firefox.dev/3OmgoND and https://share.firefox.dev/4cVbsHV
corresponding to https://treeherder.mozilla.org/jobs?repo=autoland&revision=3b5c70d393508355636debff19de554c715aa4d6&selectedTaskRun=NbatlmoiSmiTpatOSWIjZA.0 and https://treeherder.mozilla.org/jobs?repo=autoland&revision=235c31a3441a5804b35a2d2bef0dc2662620cc4c&selectedTaskRun=WdfwCZ41TLayB2B8DUxkKw.0
| Assignee | ||
Comment 7•7 months ago
|
||
not everyone is looking at the really long slack thread (understandably) so just want to summarize a few linux related details to consolidate info in this one bug. Maybe something will be useful in the future.
- Hardware: All Linux perf machines run on HPE Moonshot Specs at https://wiki.mozilla.org/Performance/Platforms#HPE_Moonshot (confirmed current by aerickson).
- Former Windows machines: Linux 24.04 workers were reclaimed from old Windows hardware. 18.04 and 24.04 share the same hardware.
- PSU concern: (currently unknown) Since they were previously old Windows machines, there's an open question of whether they might also have under-specced PSUs
- High-density chassis + age is a plausible explanation for the high Linux variance seen historically.
also just to be clear also this seems to affect all linux perftests not just sp3
Comment 8•6 months ago
•
|
||
It seems to depend only on push timing, no machine correlation, apparently. Note that ./mach try perf --rebuild X will apparently queue the base runs first, making it more likely to run the new revision on warm (slower) machines, IIUC. A separate batch of only new runs later did end up in the higher bucket for all runs across all touched machines.
I vaguely remember less variance when the runners were still on 18.04, so maybe some governor has become more sensible to temperature with the new kernel?
| Assignee | ||
Comment 9•6 months ago
•
|
||
fwiw my own queries also support your findings that it is not machine specific (that is, multiple machines will appear in both the high/low score groups, checked over a 60 day time period with autoland data)
I vaguely remember less variance when the runners were still on 18.04, so maybe some governor has become more sensible to temperature with the new kernel?
going to loop in :aerickson who might be able to better answer this
Comment 10•6 months ago
|
||
Jens, can you make a graph that shows the time correlation?
Comment 11•6 months ago
•
|
||
Claude can. It is actually not the first processing but the retriggers via perfcompare that were skewed vs. base higher perf, but the result is probably the same. I'd read out of this that up to ten runs are likely to be faster than more running at the same time? Obviously I do not have datapoints of what else was going on on those machines here.
Comment 12•6 months ago
|
||
Is that just for one push? I was hoping for a graph of correlation over all of the jobs
Comment 13•6 months ago
•
|
||
That is just one push, I did not do more than looking at that. And honestly I do not know how to best do that, because any load of any test running on those machines would need to be tracked?
But already this small amount of data seems to show pretty clearly that the only possible correlation is time, not specific machines.
Comment 14•6 months ago
|
||
(In reply to Greg Mierzwinski [:sparky] from comment #4)
I made a graph here, and I don't see an obvious pattern related to specific machines being different from others: https://sql.telemetry.mozilla.org/queries/117597#287629
Maybe Greg can update/extend this query a bit (starting from the date range) ? IIUC this seems to look only at sp3 runs on those machines, and bi-modality is clearly visible there, too. What misses is correlation to general load and/or even better temperature, not sure if we have data for that.
| Reporter | ||
Comment 15•6 months ago
|
||
We don't have temperature's being tracked in our DB, but I tried to do something for overall load on the machines here (could be a bad query, so it would be good if someone could double-check): https://sql.telemetry.mozilla.org/queries/119338
I see that when we hit 110+ concurrent, the data seems to often get into the lower band, and below 110 we get into the upper band more. So it looks like there's a possibility that something related to overall load is happening here. It's difficult to say for certain since this pattern isn't always true, but that may be due to machine rack placement.
I'll see if I can start gathering temps on these machines.
| Reporter | ||
Comment 16•6 months ago
|
||
Got a try run working here, and I've triggered 100 sp3 tests on it: https://treeherder.mozilla.org/jobs?repo=try&revision=7982b009daaf3cfc84ad109c649b74b23adc00ee
| Reporter | ||
Comment 17•6 months ago
|
||
Doesn't seem like it has anything to do with the temperature (or at least the temp that I was tracking). I'll try again later tonight and see what that data looks like then. Right now, all the sp3 score data points I got are in the lower band except for 1 that was in the higher band. It's temperature was near the bottom end, but so were some of the others and those were in the lower band. Here's a graph of them: https://treeherder.mozilla.org/perfherder/graphs?highlightAlerts=1&highlightChangelogData=0&highlightCommonAlerts=0&highlightInitialDataPoints=0&replicates=0&series=try,5599836,1,13&series=try,5826105,1,13&series=try,5826104,1,13&timerange=86400
Comment 18•6 months ago
|
||
(In reply to Kash Shampur [:kshampur] ⌚EST from comment #9)
fwiw my own queries also support your findings that it is not machine specific (that is, multiple machines will appear in both the high/low score groups, checked over a 60 day time period with autoland data)
I vaguely remember less variance when the runners were still on 18.04, so maybe some governor has become more sensible to temperature with the new kernel?
going to loop in :aerickson who might be able to better answer this
Yeah, that’s plausible. Newer kernels/power management (schedutil, intel_pstate/HWP, thermal-aware scheduling, etc.) are much more dynamic than the old 18.04-era defaults, which can increase run-to-run variance on bursty CI workloads even if average performance is better. There have also been a lot of CPU security mitigations added since 18.04, like Spectre, Meltdown, Retbleed, and SMT-related mitigations, which can introduce additional scheduler/syscall jitter in some workloads.
Comment 19•6 months ago
|
||
So one thing that occurs to me: on my build machine I see a fair amount of noise when benchmarking... but I can appreciably improve that by running sudo cpupower frequency-set -g performance ahead of measurement.
Do we lock the CPU performance governors on our linux machines?
Comment 20•6 months ago
|
||
Can we get perf profiles from these machines? sudo perf record -g -a
Comment 21•6 months ago
|
||
(In reply to Matthew Gaudet (he/him) [:mgaudet] from comment #19)
So one thing that occurs to me: on my build machine I see a fair amount of noise when benchmarking... but I can appreciably improve that by running
sudo cpupower frequency-set -g performanceahead of measurement.Do we lock the CPU performance governors on our linux machines?
We don't. All cores on our 18.04 and 24.04 deployments on the Moonshot hardware in MDC1 use the powersave governor.
[aerickson@t-linux64-ms-001 ~]$ grep . /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
/sys/devices/system/cpu/cpu0/cpufreq/scaling_governor:powersave
/sys/devices/system/cpu/cpu1/cpufreq/scaling_governor:powersave
/sys/devices/system/cpu/cpu2/cpufreq/scaling_governor:powersave
/sys/devices/system/cpu/cpu3/cpufreq/scaling_governor:powersave
/sys/devices/system/cpu/cpu4/cpufreq/scaling_governor:powersave
/sys/devices/system/cpu/cpu5/cpufreq/scaling_governor:powersave
/sys/devices/system/cpu/cpu6/cpufreq/scaling_governor:powersave
/sys/devices/system/cpu/cpu7/cpufreq/scaling_governor:powersave
[aerickson@t-linux64-ms-001 ~]$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.6 LTS"
[aerickson@t-linux64-ms-001 ~]$
[aerickson@t-linux64-ms-016 ~]$ grep . /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
/sys/devices/system/cpu/cpu0/cpufreq/scaling_governor:powersave
/sys/devices/system/cpu/cpu1/cpufreq/scaling_governor:powersave
/sys/devices/system/cpu/cpu2/cpufreq/scaling_governor:powersave
/sys/devices/system/cpu/cpu3/cpufreq/scaling_governor:powersave
/sys/devices/system/cpu/cpu4/cpufreq/scaling_governor:powersave
/sys/devices/system/cpu/cpu5/cpufreq/scaling_governor:powersave
/sys/devices/system/cpu/cpu6/cpufreq/scaling_governor:powersave
/sys/devices/system/cpu/cpu7/cpufreq/scaling_governor:powersave
[aerickson@t-linux64-ms-016 ~]$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=24.04
DISTRIB_CODENAME=noble
DISTRIB_DESCRIPTION="Ubuntu 24.04.2 LTS"
[aerickson@t-linux64-ms-016 ~]$
(In reply to Markus Stange [:mstange] from comment #20)
Can we get perf profiles from these machines?
sudo perf record -g -a
Where can I find this tool?
Comment 22•6 months ago
|
||
Do you know if we're pinned to a specific frequency by this governor, or if we're scaling up and down?
I imagine the hope was that this would pin us to a specific low freq to avoid thermal throttling?
Comment 23•6 months ago
|
||
(In reply to Matthew Gaudet (he/him) [:mgaudet] from comment #22)
Do you know if we're pinned to a specific frequency by this governor, or if we're scaling up and down?
powersave doesn't usually pin the CPU to one fixed frequency. It generally allows the CPU to scale up and down dynamically, but with a bias toward lower frequencies / power savings.
The exact behavior depends on the driver (intel_pstate vs acpi-cpufreq) and hardware generation. On modern Intel systems with intel_pstate, powersave is often still quite dynamic and can turbo up under load, it just ramps less aggressively than performance.
I imagine the hope was that this would pin us to a specific low freq to avoid thermal throttling?
I'm not exactly sure why we haven't ever changed these settings. We didn't do any tweaking on the 16.04 deployment, so we didn't do it on 18.04, so we didn't do it on 24.04.
These are very power-dense server chassis (the HP Moonshot chassis), so perhaps setting them to performance will cause chassis/rack power issues or increased power spend. Perhaps we're trying to mimic what the average user is running (this hardware was specifically chosen because of the consumer-level GPU).
Comment 24•6 months ago
|
||
I would personally be super interested in an experiment where we set the governor to performance for a while. Unclear if we can do a
push governor
run test
pop governor
or if it requires a deploy or a pool or somesuch.
scaling_governor:powersave is what I get on my Ubuntu machine when i reboot it, but it's noticeable the difference between that and scaling_governor:performance when i run bechmarks on my machine.
Edit! I did some more experimentation: I get much better results if I even nail to a specific frequency:
sudo cpupower frequency-set -d 5Ghz -u 5Ghz
# Set Performance:
for i in `seq 0 9`; do /home/matthew/firefox/obj-opt-shell-nodebug-x86_64-pc-linux-gnu/dist/bin/js --strict-benchmark-mode cli.js doxbee-promise | grep 'Overall Score'; done;
Overall Score 77.59 pts
Overall Score 77.39 pts
Overall Score 77.46 pts
Overall Score 76.66 pts
Overall Score 74.88 pts
Overall Score 77.71 pts
Overall Score 77.50 pts
Overall Score 78.12 pts
Overall Score 78.25 pts
Overall Score 77.67 pts
# Set Powersave
matthew@ZenTower:~/perf/JetStream$ for i in `seq 0 9`; do /home/matthew/firefox/obj-opt-shell-nodebug-x86_64-pc-linux-gnu/dist/bin/js --strict-benchmark-mode cli.js doxbee-promise | grep 'Overall Score'; done;
Overall Score 78.85 pts
Overall Score 78.63 pts
Overall Score 78.03 pts
Overall Score 71.62 pts
Overall Score 76.95 pts
Overall Score 77.09 pts
Overall Score 77.87 pts
Overall Score 76.82 pts
Overall Score 75.13 pts
Overall Score 77.21 pts
# Set 5Ghz
matthew@ZenTower:~/perf/JetStream$ for i in `seq 0 9`; do /home/matthew/firefox/obj-opt-shell-nodebug-x86_64-pc-linux-gnu/dist/bin/js --strict-benchmark-mode cli.js doxbee-promise | grep 'Overall Score'; done;
Overall Score 72.99 pts
Overall Score 73.82 pts
Overall Score 73.97 pts
Overall Score 73.37 pts
Overall Score 74.59 pts
Overall Score 74.16 pts
Overall Score 72.17 pts
Overall Score 73.49 pts
Overall Score 72.95 pts
Overall Score 73.57 pts
| Assignee | ||
Comment 25•6 months ago
|
||
just following up with a discussion w/ :mgaudet, this could potentially be checked easily by SSHing into one of the workers and running these CPU frequency governor settings and seeing if that gets rid of the bimodality... e.g. what he witnessed on his machine
:aerickson would it be difficult to give :mgaudet SSH access?
Comment 26•6 months ago
|
||
I guess I'm confused about the goal... what is our goal? To just have the most consistency in results as possible?
We bought hardware with consumer GPUs because it behaves like a consumer's machine, but then we strip everything out of the OS and force performance mode... that's not like a user's system any longer. I guess we're really aiming to detect performance regressions (and variance delays detection), which isn't anything like the original 'consumer-like' use case (the goal was to get numbers on what typical users felt?). Has the use case for these systems changed or are we chasing two different goals?
We've run these systems like this for a long time... and the results were acceptable. What's changed? It seems across all platforms we're pushing to strip things out and make the platforms more consistent.
Will setting the performance CPU governor bring more consistent results... seems yes. Will it cause other issues (increased HW failure, power issues (many of these chassis are running on 2/3 power supplies), thermal issues, etc), probably.
(In reply to Kash Shampur [:kshampur] ⌚EST from comment #25)
just following up with a discussion w/ :mgaudet, this could potentially be checked easily by SSHing into one of the workers and running these CPU frequency governor settings and seeing if that gets rid of the bimodality... e.g. what he witnessed on his machine
:aerickson would it be difficult to give :mgaudet SSH access?
I don't know how you're going to compare if you're just setting performance on one node manually (or how you'd do that in time before the taskcluster client starts after each boot) on a few runs. I'd set up a dedicated pool with several hosts that make your desired setting in Puppet and then you can compare against the main pool.
Let me know the number of hosts and the setting you'd like and I'll set it up.
| Assignee | ||
Comment 27•6 months ago
|
||
(In reply to Andrew Erickson [:aerickson] from comment #26)
I guess I'm confused about the goal... what is our goal? To just have the most consistency in results as possible?
We bought hardware with consumer GPUs because it behaves like a consumer's machine, but then we strip everything out of the OS and force performance mode... that's not like a user's system any longer. I guess we're really aiming to detect performance regressions (and variance delays detection), which isn't anything like the original 'consumer-like' use case (the goal was to get numbers on what typical users felt?). Has the use case for these systems changed or are we chasing two different goals?
We've run these systems like this for a long time... and the results were acceptable. What's changed? It seems across all platforms we're pushing to strip things out and make the platforms more consistent.
Will setting the performance CPU governor bring more consistent results... seems yes. Will it cause other issues (increased HW failure, power issues (many of these chassis are running on 2/3 power supplies), thermal issues, etc), probably.
(In reply to Kash Shampur [:kshampur] ⌚EST from comment #25)
just following up with a discussion w/ :mgaudet, this could potentially be checked easily by SSHing into one of the workers and running these CPU frequency governor settings and seeing if that gets rid of the bimodality... e.g. what he witnessed on his machine
:aerickson would it be difficult to give :mgaudet SSH access?
I don't know how you're going to compare if you're just setting performance on one node manually (or how you'd do that in time before the taskcluster client starts after each boot) on a few runs. I'd set up a dedicated pool with several hosts that make your desired setting in Puppet and then you can compare against the main pool.
Let me know the number of hosts and the setting you'd like and I'll set it up.
Very valid points :aerickson, and thinking it over I have some thoughts after talking with :sparky.
Skimming arewefastyet.com, a few of these (not just sp3) happened around the timing of the OS upgrade from 18.04 to 24.04, so that seems like a likely root cause, plus the whole host of things you pointed out in comment 18.
Changing the CPU governor from powersave to performance doesn't actually address what changed. And given that this is already old, high-density chassis hardware, running performance mode may give us a short-term win but we'd likely end up with thermal degradation anyway. So I'll take back my previous suggestion... I don't think testing via SSH on a limited pool is the right path here, (at least not yet)
One thing that might help: suggestion from :mstange in comment 20 (sudo perf record -g -a) went unanswered. Someone on perf should review system profiles from fast vs. slow runs first. And to answer your question, perf tool should already be available by default on the Linux machines. I think a better next step is provisioning a couple of nodes and doing a --worker-override try push with the ability to run sudo perf record.
(Also worth considering: is there anything in Firefox (or even Chrome) that could be interacting poorly with Ubuntu 24? well, system profiles or Gecko profiles could help answer that)
Comment 28•6 months ago
|
||
(In reply to Markus Stange [:mstange] from comment #20)
Can we get perf profiles from these machines?
sudo perf record -g -aWhere can I find this tool?
It's part of the linux-tools package.
https://askubuntu.com/a/578618
Comment 29•6 months ago
•
|
||
We bought hardware with consumer GPUs because it behaves like a consumer's machine.
I am a bit confused, we are still on the moonshot HW here, or are there changes to that configuration ? I read there that those are (pretty old) Intel E3-1585LV5 with an equally old integrated GPU, and also 4 cores / 8GB memory sound not really mainstream to me anymore, more like budget (though this is a server CPU, so maybe a bit better) ?
I guess we're really aiming to detect performance regressions
... or improvements. IIUC that is what a single dev usually does when trying to validate a change with perf runs. Ending up to spin a lot of them in order to get some signal or even giving up on Linux numbers entirely.
However, I also think there is definitely value to test if things behave well performance-wise on an "out-of-the-box" setting of the OS. Which btw. would be "balanced", not "powersave", I think ? Seeing this bi-modal behavior could in theory also indicate that we are doing something wrong in how we load the CPU with threads, and how the OS scheduler handles them. However, the impact of our changes on the really interesting things on modern HW (like P-core/E-core scheduling) is not visible on this HW, AFAICT. Anyways, a first simple step could be to try out how the machines behave with the "balanced" governor. I would not expect to see much idle power consumption difference, but on load they might consume more, of course (and I assume those are often loaded).
To add some anecdotal data point: when doing some micro benchmarks locally on my Ubuntu (Ryzen AI 9 HX 370, probably above mainstream) I both had to clamp frequency and to force execution on the same core category (in this case I chose E cores as I have more of them) to get sharp results with little noise. This is surely no realistic setting for normal operations, but very useful for a faster, local feedback loop on optimizations (fast with normal, non-PGO opt builds). Just setting the governor to performance did not help much here.
Comment 30•6 months ago
•
|
||
(In reply to Andrew Erickson [:aerickson] from comment #26)
I guess I'm confused about the goal... what is our goal? To just have the most consistency in results as possible?
(Not on the perf team directly, so if what I say is disagreed with by them, listen to them)
In a sense we have two tasks we're trying to do
- We'd like to have a good feel of performance on "normal" machines. This is a good surrogate for user experience and is what we actually care about.
- We are trying to modify the browser to make it more performant, and to watch others' modifications of the browser, to ensure that it doesn't regress performance .
On point (1) yes, having normal machines normally set up totally makes sense.
On point (2) we want to find the smallest possible improvements and regressions. Tiny regressions can build up into large performance problems, and small wins aggregated together are large wins.
I don't think serving use-case (1) and use-case (2) with the same setup and same machines makes sense. If I were to prioritize those usecases, I'd prioritize (2) over (1), because we have expectations a priori that moving (2) will move (1), and we can be more sophisticated actors when working with (2).
There's a limit of course; to Jens' point, if we had a single threaded work load that we could nail to a specific core, at a specific frequency, we'd get more signal. The problem is that Firefox isn't a single threaded workload; we do end up hitting scheduler concerns, and the more artificial we make things, the less we're sure of that transfer from (2) to (1). Hyper optimization around the unique environment of CI will ultimately get us into trouble because we have a very diverse set of workloads (edit: and users).
There's been tons of work on computer performance measurement, and frankly everything's a mess out there. Some of the most fascinating work goes to truly insane lengths to try and reduce noise (Reboot between runs, new user between runs, disable networking, etc. etc. etc.), but I think we could move to some different choices that are more sensible for suiting use case (2).
| Assignee | ||
Comment 31•6 months ago
|
||
To move forward on mstange's suggestion, tried running perf record via a try push but hit the expected perf_event_paranoid=4 wall
[task 2026-05-20T19:50:49.079+00:00] 19:50:49 INFO - raptor-linux-perf Info: Starting perf record: perf record -a -g -F 999 -o /home/cltbld/tasks/task_177930595877547/build/blobber_upload_dir/linux-perf-speedometer3.data
[task 2026-05-20T19:50:50.069+00:00] 19:50:50 INFO - raptor-browsertime Warning: Failed to start Linux perf profiling: perf record exited immediately: Error:
[task 2026-05-20T19:50:50.070+00:00] 19:50:50 INFO - Access to performance monitoring and observability operations is limited.
[task 2026-05-20T19:50:50.071+00:00] 19:50:50 INFO - Consider adjusting /proc/sys/kernel/perf_event_paranoid setting to open
[task 2026-05-20T19:50:50.072+00:00] 19:50:50 INFO - access to performance monitoring and observability operations for processes
[task 2026-05-20T19:50:50.073+00:00] 19:50:50 INFO - without CAP_PERFMON, CAP_SYS_PTRACE or CAP_SYS_ADMIN Linux capability.
[task 2026-05-20T19:50:50.074+00:00] 19:50:50 INFO - More information can be found at 'Perf events and tool security' document:
[task 2026-05-20T19:50:50.075+00:00] 19:50:50 INFO - https://www.kernel.org/doc/html/latest/admin-guide/perf-security.html
[task 2026-05-20T19:50:50.076+00:00] 19:50:50 INFO - perf_event_paranoid setting is 4:
[task 2026-05-20T19:50:50.076+00:00] 19:50:50 INFO - -1: Allow use of (almost) all events by all users
[task 2026-05-20T19:50:50.077+00:00] 19:50:50 INFO - Ignore mlock limit after perf_event_mlock_kb without CAP_IPC_LOCK
[task 2026-05-20T19:50:50.078+00:00] 19:50:50 INFO - >= 0: Disallow raw and ftrace function tracepoint access
[task 2026-05-20T19:50:50.078+00:00] 19:50:50 INFO - >= 1: Disallow CPU event access
[task 2026-05-20T19:50:50.079+00:00] 19:50:50 INFO - >= 2: Disallow kernel profiling
[task 2026-05-20T19:50:50.079+00:00] 19:50:50 INFO - To make the adjusted perf_event_paranoid setting permanent preserve it
:aerickson, could we get a couple of machines (say 084 and 225, but more the merrier?) isolated with perf_event_paranoid set to 0 (or minimal sudo to run perf)? Picked those two based on sparky's query as machines that have been alternating between modes: https://sql.telemetry.mozilla.org/queries/119338#291440
Before making any changes to governor settings, we should check system profiles first
Also re: comment #29: from what I can find, there's no "balanced" governor on Linux. On systems using intel_pstate it seems the only options are performance and powersave?
Comment 32•6 months ago
|
||
Yeah, I've created a PR (https://github.com/mozilla-platform-ops/ronin_puppet/pull/1221) that installs the package and allows the test user to run perf via sudo and it's running on 084 and 225.
I guess the issue now is targetting... do you want these in a separate pool or do you just want to run enough jobs to hit these occasionally (and have code check if it can run the sudo command)?
[cltbld@t-linux64-ms-084 ~]$ sudo perf
usage: perf [--version] [--help] [OPTIONS] COMMAND [ARGS]
The most commonly used perf commands are:
annotate Read perf.data (created by perf record) and display annotated code
archive Create archive with object files with build-ids found in perf.data file
bench General framework for benchmark suites
buildid-cache Manage build-id cache.
buildid-list List the buildids in a perf.data file
c2c Shared Data C2C/HITM Analyzer.
config Get and set variables in a configuration file.
daemon Run record sessions on background
data Data file related processing
diff Read perf.data files and display the differential profile
evlist List the event names in a perf.data file
ftrace simple wrapper for kernel's ftrace functionality
inject Filter to augment the events stream with additional information
iostat Show I/O performance metrics
kallsyms Searches running kernel for symbols
kvm Tool to trace/measure kvm guest os
list List all symbolic event types
mem Profile memory accesses
record Run a command and record its profile into perf.data
report Read perf.data (created by perf record) and display the profile
script Read perf.data (created by perf record) and display trace output
stat Run a command and gather performance counter statistics
test Runs sanity tests.
top System profiling tool.
version display the version of perf binary
probe Define new dynamic tracepoints
trace strace inspired tool
kmem Tool to trace/measure kernel memory properties
kwork Tool to trace/measure kernel work properties (latencies)
lock Analyze lock events
sched Tool to trace/measure scheduler properties (latencies)
timechart Tool to visualize total system behavior during a workload
See 'perf help COMMAND' for more information on a specific command.
[cltbld@t-linux64-ms-084 ~]$```
| Assignee | ||
Comment 33•6 months ago
|
||
(In reply to Andrew Erickson [:aerickson] from comment #32)
I guess the issue now is targetting... do you want these in a separate pool or do you just want to run enough jobs to hit these occasionally (and have code check if it can run the sudo command)?
awesome thanks! yes a specific pool we could target with --worker-override would be great. then I can use my Try push from comment 31
| Assignee | ||
Comment 35•5 months ago
•
|
||
awesome, thanks :aerickson!
here's a try run so far...
https://treeherder.mozilla.org/jobs?repo=try&tier=1%2C2%2C3&searchStr=sp3&revision=9969734c93795b3243caf1c21303d11ce5329711
perf seems to be working...
markus/denis could you verify the system profiles look good? (the symbolicated one is available in the profiler zip e.g., and the raw perf.data is also available in the artifacts) if so then we can add more retriggers and/or change the perf flags
(this is the diff used https://gist.github.com/92kns/fadeb030bfa0194d6651fd516c5f1572)
| Reporter | ||
Updated•5 months ago
|
| Assignee | ||
Comment 36•5 months ago
|
||
(In reply to Kash Shampur [:kshampur] ⌚EST from comment #35)
awesome, thanks :aerickson!
here's a try run so far...
https://treeherder.mozilla.org/jobs?repo=try&tier=1%2C2%2C3&searchStr=sp3&revision=9969734c93795b3243caf1c21303d11ce5329711perf seems to be working...
markus/denis could you verify the system profiles look good? (the symbolicated one is available in the profiler zip e.g., and the raw perf.data is also available in the artifacts) if so then we can add more retriggers and/or change the perf flags
(this is the diff used https://gist.github.com/92kns/fadeb030bfa0194d6651fd516c5f1572)
(prelim look)
Used profiler-cli to examine the system profiles from the try push. Apparently the bimodality is caused by gnome-shell/Mutter consuming large amounts of CPU via software rendering (Mesa llvmpipe).
In low-scoring runs, Mutter's clutter compositor is active and competes with Firefox for CPU
In high-scoring runs, gnome-shell is nearly idle (~15s).
The suspect is GNOME's screen blank timeout (5 min default). When the screen blanks before the test, Mutter suspends compositing → Firefox gets full CPU → high score.
If it doesn't blank in time, Mutter composites every frame Firefox submits → low score.
Comment 37•5 months ago
•
|
||
Fun! Here are two imported profiles along with their numbers:
| Task ID | Worker | Profile | Run 1 | Run 2 | Run 3 | Run 4 | Run 5 | Mean |
|---|---|---|---|---|---|---|---|---|
diJwfJvBRm6v_0iopaZnJg |
t-linux64-ms-084 |
profile | 13.265 | 13.282 | 13.372 | 13.258 | 13.479 | 13.331 |
Mml6rueSQAS1Ge3QYcoEcA |
t-linux64-ms-084 |
profile | 12.711 | 12.610 | 12.661 | 12.754 | 12.752 | 12.698 |
In the slow profile, indeed there is a lot of CPU activity on llvm-pipe threads in the gnome-shell process.
Comment 38•5 months ago
|
||
That's a very interesting finding. I wonder how this ties into the time correlation from comment 8, not sure what makes the GNOME session become active. But maybe it is actually just random, there were not too many data points, after all.
It seems to me that we need to decide which scenario (compositor active or not) we want to be used for testing and ensure it is stably used the same (might be easier to enforce the screen to be always on?). My gut feeling is that we'd get cleaner signal about perf changes from runs without compositor. But it is probably also the less realistic mode of operation for "performance delivered to user" measurements. A realistic desktop scenario would probably have some HW-accelerated compositor being active ?
Comment 39•5 months ago
|
||
(In reply to Andrew Erickson [:aerickson] from comment #32)
Yeah, I've created a PR (https://github.com/mozilla-platform-ops/ronin_puppet/pull/1221) that installs the package and allows the test user to run
perfvia sudo and it's running on 084 and 225. ``
Hi Andrew, is it possible to enable this functionality on all the workers so we can generate these perf profiles regularly?
Updated•5 months ago
|
Comment 40•5 months ago
|
||
(In reply to Denis Palmeiro [:denispal] from comment #39)
(In reply to Andrew Erickson [:aerickson] from comment #32)
Yeah, I've created a PR (https://github.com/mozilla-platform-ops/ronin_puppet/pull/1221) that installs the package and allows the test user to run
perfvia sudo and it's running on 084 and 225. ``Hi Andrew, is it possible to enable this functionality on all the workers so we can generate these perf profiles regularly?
Yes, the branch has been merged. The tool and the sudoers rule will be present on all 1804 and 2404 linux moonshot hosts.
Comment 41•5 months ago
|
||
084 and 225 are back in the separate pool (https://firefox-ci-tc.services.mozilla.com/provisioners/releng-hardware/worker-types/gecko-t-linux-talos-2404-bug2031822?sortBy=Last%20Active&sortDirection=desc) running the openbox PR I just created (https://github.com/mozilla-platform-ops/ronin_puppet/pull/1227) per discussion with Kash.
| Assignee | ||
Comment 42•5 months ago
|
||
thank you :aerickson for the openbox PR. I've got some interesting results now:
using openbox --replace seems to help in the sense that there is one mode now and it is the higher mode.
(In reply to Jens Stutte [:jstutte] from comment #38)
It seems to me that we need to decide which scenario (compositor active or not) we want to be used for testing and ensure it is stably used the same (might be easier to enforce the screen to be always on?). My gut feeling is that we'd get cleaner signal about perf changes from runs without compositor. But it is probably also the less realistic mode of operation for "performance delivered to user" measurements. A realistic desktop scenario would probably have some HW-accelerated compositor being active ?
and based on this comment i also tried out always on and always off screen states using xset, but in either case it is in the low mode.
profiler-cli suggests that's exactly what's happening under the hood. in the openbox run the gnome-shell compositor is just... gone. no llvmpipe threads. but it's still present in the on/off xset runs, so screen state doesn't matter, Mutter stays active and keeps the runs in the lower mode.
which i think also answers your question :jstutte, openbox gives us the "no compositor" scenario you leaned toward and locks it in stably.
the tradeoff is it's the less realistic mode (no HW-acceleration)
https://mozilla-hub.atlassian.net/browse/RELOPS-2392 would attempt to address that.
Comment 43•5 months ago
|
||
In the openbox profiles, is there activity on llvmpipe threads in the Firefox GPU process? If so that means we're still not really taking advantage of the GPU.
| Assignee | ||
Comment 44•5 months ago
|
||
(In reply to Markus Stange [:mstange] from comment #43)
In the openbox profiles, is there activity on llvmpipe threads in the Firefox GPU process? If so that means we're still not really taking advantage of the GPU.
yep, good question, there's still a ton of llvmpipe activity in the Firefox GPU process. so we're definitely still not using the actual GPU. openbox just got rid of gnome-shell's software compositor, but Firefox itself is still rendering everything on the CPU via llvmpipe.
getting on the Intel GPU (https://mozilla-hub.atlassian.net/browse/RELOPS-2392) is what'd actually fix that part.
| Assignee | ||
Comment 45•5 months ago
•
|
||
:aerickson added config to use the intel gpu
try still running but so far looks good. examining the existing profiles suggests no more llvmpipe, gnomeshell is present but idle, and rendering is happening in the intel gpu
| Assignee | ||
Comment 46•5 months ago
|
||
(In reply to Kash Shampur [:kshampur] ⌚EST from comment #45)
:aerickson added config to use the intel gpu
try still running but so far looks good. examining the existing profiles suggests no more llvmpipe, gnomeshell is present but idle, and rendering is happening in the intel gpu
still looks bimodal, but with a tighter spread
| Assignee | ||
Comment 47•5 months ago
|
||
(In reply to Kash Shampur [:kshampur] ⌚EST from comment #46)
(In reply to Kash Shampur [:kshampur] ⌚EST from comment #45)
:aerickson added config to use the intel gpu
try still running but so far looks good. examining the existing profiles suggests no more llvmpipe, gnomeshell is present but idle, and rendering is happening in the intel gpu
still looks bimodal, but with a tighter spread
oh but it actually seems to be machine specific. that pool has 2 workers only, (084 and 225)
- ms-084: 14.00–14.13 (8 jobs)
- ms-225: 14.38–14.51 (9 jobs)
084 seems to be consistently a bit slower (it's actually consistently slower across every subtest, not just the rendering ones). so this maybe is not truly bimodal. either way we should probably roll out the intel config everywhere unless anyone has objections
| Assignee | ||
Comment 48•5 months ago
|
||
Doing a quick check on a few non-sp3 tests before requesting a full rollout from aerickson. Try runs still running but so far so good.
Comment 49•5 months ago
|
||
FWIW, I don't think we should switch to openbox unless we can show that gnome-shell is unworkable
| Assignee | ||
Comment 50•5 months ago
|
||
(In reply to Kash Shampur [:kshampur] ⌚EST from comment #48)
Doing a quick check on a few non-sp3 tests before requesting a full rollout from aerickson. Try runs still running but so far so good.
Try runs looked green. Requested :aerickson for this to be applied to all
| Assignee | ||
Comment 51•5 months ago
|
||
:aerickson merged https://github.com/mozilla-platform-ops/ronin_puppet/pull/1227
let's see how the autoland/m-c data looks over the next few days...
| Assignee | ||
Comment 52•5 months ago
•
|
||
Comment 53•5 months ago
|
||
Still pretty noisy but looks less bimodal
| Assignee | ||
Comment 54•5 months ago
|
||
some low-scorers are still appearing...
https://treeherder.mozilla.org/perfherder/graphs?highlightAlerts=1&highlightChangelogData=0&highlightCommonAlerts=0&highlightInitialDataPoints=0&replicates=0&selected=5691153,2556484167&series=autoland,5691153,1,13&timerange=604800
ran a system profile try push here
https://treeherder.mozilla.org/jobs?repo=try&tier=1%2C2%2C3&revision=66f3dc5b51d9b187093098e7b5e8352cc1c51761&selectedTaskRun=PgMc2ThPQpeyffCScImXMA.0
to look for some low scoring ones
and according to profiler-cli the low scoring ones seem to not be using intel gpu
in this Try push it seems to be ms-019, ms-026, and ms-027 (not an exhaustive list, i am sure there might be more...? and in one case ms-019, no sudo privileges on perf command so definitely doesn't have any of the recent deployed changes...)
:aerickson could you please take a look? is it possible the recently deployed changes haven't been applied to the entire linux fleet?
| Assignee | ||
Updated•5 months ago
|
| Assignee | ||
Comment 56•5 months ago
|
||
(In reply to Jeff Muizelaar [:jrmuizel] from comment #55)
Can you link to a low scoring profile?
Comment 57•5 months ago
|
||
Yes, I agree. Those are not using the Intel GPU
Comment 58•5 months ago
|
||
(In reply to Kash Shampur [:kshampur] ⌚EST from comment #54)
:aerickson could you please take a look? is it possible the recently deployed changes haven't been applied to the entire linux fleet?
Checked those hosts. They were 3 of 4 hosts we were testing another branch on and their branch hadn't been rebased on the new master. Fixed now.
Comment 59•4 months ago
|
||
Where are we at with this?
| Assignee | ||
Comment 60•4 months ago
|
||
(In reply to Andrew Erickson [:aerickson] from comment #59)
Where are we at with this?
just catching up post PTO.... i'd say bimodality is clearly gone so we can close this.
But data is still noisy but that should be a separate bug.
variance seems to be intermachine
https://performance.mozilla.org/speedometer_job_debug.html?os=linux&range=1month&repository=autoland&hide=Chrome
Description
•