Closed Bug 2031822 Opened 7 months ago Closed 4 months ago

Linux speedometer3 has a high amount of bi-modality

Categories

(Testing :: Raptor, defect, P2)

defect

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: sparky, Assigned: kshampur)

References

Details

(Whiteboard: [fxp][operational])

Attachments

(2 files)

The linux sp3 test is showing a lot of bimodality at the moment. See here for a sample graph of it: https://treeherder.mozilla.org/perfherder/graphs?highlightAlerts=1&highlightChangelogData=0&highlightCommonAlerts=0&highlightInitialDataPoints=0&replicates=0&series=autoland,5691152,1,13&timerange=1209600&zoom=1774982911573,1776203969593,11.033555555555555,15.059444444444445

It started to become much more prominent once we enabled the 12 task duplicates on autoland.

We should check if some machines are slower than others

Pretty sure it was bimodal before, but then we stopped reporting all the subscores briefly, then the bimodality came back.

(Doesn't -seem- to be machine though)

I made a graph here, and I don't see an obvious pattern related to specific machines being different from others: https://sql.telemetry.mozilla.org/queries/117597#287629

Duplicate of this bug: 2033096

https://quiet-haupia-492082.netlify.app/speedometer?os=linux&replicates=true&repository=autoland&colorByMachine=true&range=week also suggests that this is not correlated to the machine the test runs on. i.e. the same machine will have fast runs and slow runs.

The chrome runs are noisy too which further suggests that this might not be Firefox specific.

It might be worth trying to collect CPU clocks from fast runs and slow runs to see if there's a noticeable difference there.

Here are some example profiles that are from fast runs and slow runs. I haven't verified that the profile run also shows the same speed difference.

Linux profiles from CI: https://share.firefox.dev/3OmgoND and https://share.firefox.dev/4cVbsHV
corresponding to https://treeherder.mozilla.org/jobs?repo=autoland&revision=3b5c70d393508355636debff19de554c715aa4d6&selectedTaskRun=NbatlmoiSmiTpatOSWIjZA.0 and https://treeherder.mozilla.org/jobs?repo=autoland&revision=235c31a3441a5804b35a2d2bef0dc2662620cc4c&selectedTaskRun=WdfwCZ41TLayB2B8DUxkKw.0

not everyone is looking at the really long slack thread (understandably) so just want to summarize a few linux related details to consolidate info in this one bug. Maybe something will be useful in the future.

  • Hardware: All Linux perf machines run on HPE Moonshot Specs at https://wiki.mozilla.org/Performance/Platforms#HPE_Moonshot (confirmed current by aerickson).
  • Former Windows machines: Linux 24.04 workers were reclaimed from old Windows hardware. 18.04 and 24.04 share the same hardware.
  • PSU concern: (currently unknown) Since they were previously old Windows machines, there's an open question of whether they might also have under-specced PSUs
  • High-density chassis + age is a plausible explanation for the high Linux variance seen historically.

also just to be clear also this seems to affect all linux perftests not just sp3

It seems to depend only on push timing, no machine correlation, apparently. Note that ./mach try perf --rebuild X will apparently queue the base runs first, making it more likely to run the new revision on warm (slower) machines, IIUC. A separate batch of only new runs later did end up in the higher bucket for all runs across all touched machines.

I vaguely remember less variance when the runners were still on 18.04, so maybe some governor has become more sensible to temperature with the new kernel?

fwiw my own queries also support your findings that it is not machine specific (that is, multiple machines will appear in both the high/low score groups, checked over a 60 day time period with autoland data)

I vaguely remember less variance when the runners were still on 18.04, so maybe some governor has become more sensible to temperature with the new kernel?

going to loop in :aerickson who might be able to better answer this

Flags: needinfo?(aerickson)

Jens, can you make a graph that shows the time correlation?

Flags: needinfo?(jstutte)

Claude can. It is actually not the first processing but the retriggers via perfcompare that were skewed vs. base higher perf, but the result is probably the same. I'd read out of this that up to ten runs are likely to be faster than more running at the same time? Obviously I do not have datapoints of what else was going on on those machines here.

Flags: needinfo?(jstutte)

Is that just for one push? I was hoping for a graph of correlation over all of the jobs

That is just one push, I did not do more than looking at that. And honestly I do not know how to best do that, because any load of any test running on those machines would need to be tracked?

But already this small amount of data seems to show pretty clearly that the only possible correlation is time, not specific machines.

(In reply to Greg Mierzwinski [:sparky] from comment #4)

I made a graph here, and I don't see an obvious pattern related to specific machines being different from others: https://sql.telemetry.mozilla.org/queries/117597#287629

Maybe Greg can update/extend this query a bit (starting from the date range) ? IIUC this seems to look only at sp3 runs on those machines, and bi-modality is clearly visible there, too. What misses is correlation to general load and/or even better temperature, not sure if we have data for that.

Flags: needinfo?(gmierzwinski)

We don't have temperature's being tracked in our DB, but I tried to do something for overall load on the machines here (could be a bad query, so it would be good if someone could double-check): https://sql.telemetry.mozilla.org/queries/119338

I see that when we hit 110+ concurrent, the data seems to often get into the lower band, and below 110 we get into the upper band more. So it looks like there's a possibility that something related to overall load is happening here. It's difficult to say for certain since this pattern isn't always true, but that may be due to machine rack placement.

I'll see if I can start gathering temps on these machines.

Flags: needinfo?(gmierzwinski)

Got a try run working here, and I've triggered 100 sp3 tests on it: https://treeherder.mozilla.org/jobs?repo=try&revision=7982b009daaf3cfc84ad109c649b74b23adc00ee

Doesn't seem like it has anything to do with the temperature (or at least the temp that I was tracking). I'll try again later tonight and see what that data looks like then. Right now, all the sp3 score data points I got are in the lower band except for 1 that was in the higher band. It's temperature was near the bottom end, but so were some of the others and those were in the lower band. Here's a graph of them: https://treeherder.mozilla.org/perfherder/graphs?highlightAlerts=1&highlightChangelogData=0&highlightCommonAlerts=0&highlightInitialDataPoints=0&replicates=0&series=try,5599836,1,13&series=try,5826105,1,13&series=try,5826104,1,13&timerange=86400

(In reply to Kash Shampur [:kshampur] ⌚EST from comment #9)

fwiw my own queries also support your findings that it is not machine specific (that is, multiple machines will appear in both the high/low score groups, checked over a 60 day time period with autoland data)

I vaguely remember less variance when the runners were still on 18.04, so maybe some governor has become more sensible to temperature with the new kernel?

going to loop in :aerickson who might be able to better answer this

Yeah, that’s plausible. Newer kernels/power management (schedutil, intel_pstate/HWP, thermal-aware scheduling, etc.) are much more dynamic than the old 18.04-era defaults, which can increase run-to-run variance on bursty CI workloads even if average performance is better. There have also been a lot of CPU security mitigations added since 18.04, like Spectre, Meltdown, Retbleed, and SMT-related mitigations, which can introduce additional scheduler/syscall jitter in some workloads.

Flags: needinfo?(aerickson)

So one thing that occurs to me: on my build machine I see a fair amount of noise when benchmarking... but I can appreciably improve that by running sudo cpupower frequency-set -g performance ahead of measurement.

Do we lock the CPU performance governors on our linux machines?

Flags: needinfo?(aerickson)

Can we get perf profiles from these machines? sudo perf record -g -a

(In reply to Matthew Gaudet (he/him) [:mgaudet] from comment #19)

So one thing that occurs to me: on my build machine I see a fair amount of noise when benchmarking... but I can appreciably improve that by running sudo cpupower frequency-set -g performance ahead of measurement.

Do we lock the CPU performance governors on our linux machines?

We don't. All cores on our 18.04 and 24.04 deployments on the Moonshot hardware in MDC1 use the powersave governor.

[aerickson@t-linux64-ms-001 ~]$ grep . /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
/sys/devices/system/cpu/cpu0/cpufreq/scaling_governor:powersave
/sys/devices/system/cpu/cpu1/cpufreq/scaling_governor:powersave
/sys/devices/system/cpu/cpu2/cpufreq/scaling_governor:powersave
/sys/devices/system/cpu/cpu3/cpufreq/scaling_governor:powersave
/sys/devices/system/cpu/cpu4/cpufreq/scaling_governor:powersave
/sys/devices/system/cpu/cpu5/cpufreq/scaling_governor:powersave
/sys/devices/system/cpu/cpu6/cpufreq/scaling_governor:powersave
/sys/devices/system/cpu/cpu7/cpufreq/scaling_governor:powersave
[aerickson@t-linux64-ms-001 ~]$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.6 LTS"
[aerickson@t-linux64-ms-001 ~]$

[aerickson@t-linux64-ms-016 ~]$ grep . /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
/sys/devices/system/cpu/cpu0/cpufreq/scaling_governor:powersave
/sys/devices/system/cpu/cpu1/cpufreq/scaling_governor:powersave
/sys/devices/system/cpu/cpu2/cpufreq/scaling_governor:powersave
/sys/devices/system/cpu/cpu3/cpufreq/scaling_governor:powersave
/sys/devices/system/cpu/cpu4/cpufreq/scaling_governor:powersave
/sys/devices/system/cpu/cpu5/cpufreq/scaling_governor:powersave
/sys/devices/system/cpu/cpu6/cpufreq/scaling_governor:powersave
/sys/devices/system/cpu/cpu7/cpufreq/scaling_governor:powersave
[aerickson@t-linux64-ms-016 ~]$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=24.04
DISTRIB_CODENAME=noble
DISTRIB_DESCRIPTION="Ubuntu 24.04.2 LTS"
[aerickson@t-linux64-ms-016 ~]$

(In reply to Markus Stange [:mstange] from comment #20)

Can we get perf profiles from these machines? sudo perf record -g -a

Where can I find this tool?

Flags: needinfo?(aerickson)

Do you know if we're pinned to a specific frequency by this governor, or if we're scaling up and down?

I imagine the hope was that this would pin us to a specific low freq to avoid thermal throttling?

Flags: needinfo?(aerickson)

(In reply to Matthew Gaudet (he/him) [:mgaudet] from comment #22)

Do you know if we're pinned to a specific frequency by this governor, or if we're scaling up and down?

powersave doesn't usually pin the CPU to one fixed frequency. It generally allows the CPU to scale up and down dynamically, but with a bias toward lower frequencies / power savings.

The exact behavior depends on the driver (intel_pstate vs acpi-cpufreq) and hardware generation. On modern Intel systems with intel_pstate, powersave is often still quite dynamic and can turbo up under load, it just ramps less aggressively than performance.

I imagine the hope was that this would pin us to a specific low freq to avoid thermal throttling?

I'm not exactly sure why we haven't ever changed these settings. We didn't do any tweaking on the 16.04 deployment, so we didn't do it on 18.04, so we didn't do it on 24.04.

These are very power-dense server chassis (the HP Moonshot chassis), so perhaps setting them to performance will cause chassis/rack power issues or increased power spend. Perhaps we're trying to mimic what the average user is running (this hardware was specifically chosen because of the consumer-level GPU).

Flags: needinfo?(aerickson)

I would personally be super interested in an experiment where we set the governor to performance for a while. Unclear if we can do a

push governor
run test 
pop governor 

or if it requires a deploy or a pool or somesuch.

scaling_governor:powersave is what I get on my Ubuntu machine when i reboot it, but it's noticeable the difference between that and scaling_governor:performance when i run bechmarks on my machine.


Edit! I did some more experimentation: I get much better results if I even nail to a specific frequency:

sudo cpupower frequency-set -d 5Ghz -u 5Ghz

# Set Performance: 
for i in `seq 0 9`; do /home/matthew/firefox/obj-opt-shell-nodebug-x86_64-pc-linux-gnu/dist/bin/js --strict-benchmark-mode cli.js doxbee-promise | grep 'Overall Score'; done; 
Overall Score                                   77.59 pts
Overall Score                                   77.39 pts
Overall Score                                   77.46 pts
Overall Score                                   76.66 pts
Overall Score                                   74.88 pts
Overall Score                                   77.71 pts
Overall Score                                   77.50 pts
Overall Score                                   78.12 pts
Overall Score                                   78.25 pts
Overall Score                                   77.67 pts

# Set Powersave 
matthew@ZenTower:~/perf/JetStream$ for i in `seq 0 9`; do /home/matthew/firefox/obj-opt-shell-nodebug-x86_64-pc-linux-gnu/dist/bin/js --strict-benchmark-mode cli.js doxbee-promise | grep 'Overall Score'; done; 
Overall Score                                   78.85 pts
Overall Score                                   78.63 pts
Overall Score                                   78.03 pts
Overall Score                                   71.62 pts
Overall Score                                   76.95 pts
Overall Score                                   77.09 pts
Overall Score                                   77.87 pts
Overall Score                                   76.82 pts
Overall Score                                   75.13 pts
Overall Score                                   77.21 pts

# Set 5Ghz 
matthew@ZenTower:~/perf/JetStream$ for i in `seq 0 9`; do /home/matthew/firefox/obj-opt-shell-nodebug-x86_64-pc-linux-gnu/dist/bin/js --strict-benchmark-mode cli.js doxbee-promise | grep 'Overall Score'; done; 
Overall Score                                   72.99 pts
Overall Score                                   73.82 pts
Overall Score                                   73.97 pts
Overall Score                                   73.37 pts
Overall Score                                   74.59 pts
Overall Score                                   74.16 pts
Overall Score                                   72.17 pts
Overall Score                                   73.49 pts
Overall Score                                   72.95 pts
Overall Score                                   73.57 pts

just following up with a discussion w/ :mgaudet, this could potentially be checked easily by SSHing into one of the workers and running these CPU frequency governor settings and seeing if that gets rid of the bimodality... e.g. what he witnessed on his machine

:aerickson would it be difficult to give :mgaudet SSH access?

Flags: needinfo?(aerickson)

I guess I'm confused about the goal... what is our goal? To just have the most consistency in results as possible?

We bought hardware with consumer GPUs because it behaves like a consumer's machine, but then we strip everything out of the OS and force performance mode... that's not like a user's system any longer. I guess we're really aiming to detect performance regressions (and variance delays detection), which isn't anything like the original 'consumer-like' use case (the goal was to get numbers on what typical users felt?). Has the use case for these systems changed or are we chasing two different goals?

We've run these systems like this for a long time... and the results were acceptable. What's changed? It seems across all platforms we're pushing to strip things out and make the platforms more consistent.

Will setting the performance CPU governor bring more consistent results... seems yes. Will it cause other issues (increased HW failure, power issues (many of these chassis are running on 2/3 power supplies), thermal issues, etc), probably.

(In reply to Kash Shampur [:kshampur] ⌚EST from comment #25)

just following up with a discussion w/ :mgaudet, this could potentially be checked easily by SSHing into one of the workers and running these CPU frequency governor settings and seeing if that gets rid of the bimodality... e.g. what he witnessed on his machine

:aerickson would it be difficult to give :mgaudet SSH access?

I don't know how you're going to compare if you're just setting performance on one node manually (or how you'd do that in time before the taskcluster client starts after each boot) on a few runs. I'd set up a dedicated pool with several hosts that make your desired setting in Puppet and then you can compare against the main pool.

Let me know the number of hosts and the setting you'd like and I'll set it up.

Flags: needinfo?(aerickson)

(In reply to Andrew Erickson [:aerickson] from comment #26)

I guess I'm confused about the goal... what is our goal? To just have the most consistency in results as possible?

We bought hardware with consumer GPUs because it behaves like a consumer's machine, but then we strip everything out of the OS and force performance mode... that's not like a user's system any longer. I guess we're really aiming to detect performance regressions (and variance delays detection), which isn't anything like the original 'consumer-like' use case (the goal was to get numbers on what typical users felt?). Has the use case for these systems changed or are we chasing two different goals?

We've run these systems like this for a long time... and the results were acceptable. What's changed? It seems across all platforms we're pushing to strip things out and make the platforms more consistent.

Will setting the performance CPU governor bring more consistent results... seems yes. Will it cause other issues (increased HW failure, power issues (many of these chassis are running on 2/3 power supplies), thermal issues, etc), probably.

(In reply to Kash Shampur [:kshampur] ⌚EST from comment #25)

just following up with a discussion w/ :mgaudet, this could potentially be checked easily by SSHing into one of the workers and running these CPU frequency governor settings and seeing if that gets rid of the bimodality... e.g. what he witnessed on his machine

:aerickson would it be difficult to give :mgaudet SSH access?

I don't know how you're going to compare if you're just setting performance on one node manually (or how you'd do that in time before the taskcluster client starts after each boot) on a few runs. I'd set up a dedicated pool with several hosts that make your desired setting in Puppet and then you can compare against the main pool.

Let me know the number of hosts and the setting you'd like and I'll set it up.

Very valid points :aerickson, and thinking it over I have some thoughts after talking with :sparky.

Skimming arewefastyet.com, a few of these (not just sp3) happened around the timing of the OS upgrade from 18.04 to 24.04, so that seems like a likely root cause, plus the whole host of things you pointed out in comment 18.

Changing the CPU governor from powersave to performance doesn't actually address what changed. And given that this is already old, high-density chassis hardware, running performance mode may give us a short-term win but we'd likely end up with thermal degradation anyway. So I'll take back my previous suggestion... I don't think testing via SSH on a limited pool is the right path here, (at least not yet)

One thing that might help: suggestion from :mstange in comment 20 (sudo perf record -g -a) went unanswered. Someone on perf should review system profiles from fast vs. slow runs first. And to answer your question, perf tool should already be available by default on the Linux machines. I think a better next step is provisioning a couple of nodes and doing a --worker-override try push with the ability to run sudo perf record.

(Also worth considering: is there anything in Firefox (or even Chrome) that could be interacting poorly with Ubuntu 24? well, system profiles or Gecko profiles could help answer that)

(In reply to Markus Stange [:mstange] from comment #20)

Can we get perf profiles from these machines? sudo perf record -g -a

Where can I find this tool?

It's part of the linux-tools package.
https://askubuntu.com/a/578618

We bought hardware with consumer GPUs because it behaves like a consumer's machine.

I am a bit confused, we are still on the moonshot HW here, or are there changes to that configuration ? I read there that those are (pretty old) Intel E3-1585LV5 with an equally old integrated GPU, and also 4 cores / 8GB memory sound not really mainstream to me anymore, more like budget (though this is a server CPU, so maybe a bit better) ?

I guess we're really aiming to detect performance regressions

... or improvements. IIUC that is what a single dev usually does when trying to validate a change with perf runs. Ending up to spin a lot of them in order to get some signal or even giving up on Linux numbers entirely.

However, I also think there is definitely value to test if things behave well performance-wise on an "out-of-the-box" setting of the OS. Which btw. would be "balanced", not "powersave", I think ? Seeing this bi-modal behavior could in theory also indicate that we are doing something wrong in how we load the CPU with threads, and how the OS scheduler handles them. However, the impact of our changes on the really interesting things on modern HW (like P-core/E-core scheduling) is not visible on this HW, AFAICT. Anyways, a first simple step could be to try out how the machines behave with the "balanced" governor. I would not expect to see much idle power consumption difference, but on load they might consume more, of course (and I assume those are often loaded).

To add some anecdotal data point: when doing some micro benchmarks locally on my Ubuntu (Ryzen AI 9 HX 370, probably above mainstream) I both had to clamp frequency and to force execution on the same core category (in this case I chose E cores as I have more of them) to get sharp results with little noise. This is surely no realistic setting for normal operations, but very useful for a faster, local feedback loop on optimizations (fast with normal, non-PGO opt builds). Just setting the governor to performance did not help much here.

(In reply to Andrew Erickson [:aerickson] from comment #26)

I guess I'm confused about the goal... what is our goal? To just have the most consistency in results as possible?

(Not on the perf team directly, so if what I say is disagreed with by them, listen to them)

In a sense we have two tasks we're trying to do

  1. We'd like to have a good feel of performance on "normal" machines. This is a good surrogate for user experience and is what we actually care about.
  2. We are trying to modify the browser to make it more performant, and to watch others' modifications of the browser, to ensure that it doesn't regress performance .

On point (1) yes, having normal machines normally set up totally makes sense.

On point (2) we want to find the smallest possible improvements and regressions. Tiny regressions can build up into large performance problems, and small wins aggregated together are large wins.

I don't think serving use-case (1) and use-case (2) with the same setup and same machines makes sense. If I were to prioritize those usecases, I'd prioritize (2) over (1), because we have expectations a priori that moving (2) will move (1), and we can be more sophisticated actors when working with (2).

There's a limit of course; to Jens' point, if we had a single threaded work load that we could nail to a specific core, at a specific frequency, we'd get more signal. The problem is that Firefox isn't a single threaded workload; we do end up hitting scheduler concerns, and the more artificial we make things, the less we're sure of that transfer from (2) to (1). Hyper optimization around the unique environment of CI will ultimately get us into trouble because we have a very diverse set of workloads (edit: and users).

There's been tons of work on computer performance measurement, and frankly everything's a mess out there. Some of the most fascinating work goes to truly insane lengths to try and reduce noise (Reboot between runs, new user between runs, disable networking, etc. etc. etc.), but I think we could move to some different choices that are more sensible for suiting use case (2).

To move forward on mstange's suggestion, tried running perf record via a try push but hit the expected perf_event_paranoid=4 wall

[task 2026-05-20T19:50:49.079+00:00] 19:50:49     INFO -  raptor-linux-perf Info: Starting perf record: perf record -a -g -F 999 -o /home/cltbld/tasks/task_177930595877547/build/blobber_upload_dir/linux-perf-speedometer3.data
[task 2026-05-20T19:50:50.069+00:00] 19:50:50     INFO -  raptor-browsertime Warning: Failed to start Linux perf profiling: perf record exited immediately: Error:
[task 2026-05-20T19:50:50.070+00:00] 19:50:50     INFO -  Access to performance monitoring and observability operations is limited.
[task 2026-05-20T19:50:50.071+00:00] 19:50:50     INFO -  Consider adjusting /proc/sys/kernel/perf_event_paranoid setting to open
[task 2026-05-20T19:50:50.072+00:00] 19:50:50     INFO -  access to performance monitoring and observability operations for processes
[task 2026-05-20T19:50:50.073+00:00] 19:50:50     INFO -  without CAP_PERFMON, CAP_SYS_PTRACE or CAP_SYS_ADMIN Linux capability.
[task 2026-05-20T19:50:50.074+00:00] 19:50:50     INFO -  More information can be found at 'Perf events and tool security' document:
[task 2026-05-20T19:50:50.075+00:00] 19:50:50     INFO -  https://www.kernel.org/doc/html/latest/admin-guide/perf-security.html
[task 2026-05-20T19:50:50.076+00:00] 19:50:50     INFO -  perf_event_paranoid setting is 4:
[task 2026-05-20T19:50:50.076+00:00] 19:50:50     INFO -    -1: Allow use of (almost) all events by all users
[task 2026-05-20T19:50:50.077+00:00] 19:50:50     INFO -        Ignore mlock limit after perf_event_mlock_kb without CAP_IPC_LOCK
[task 2026-05-20T19:50:50.078+00:00] 19:50:50     INFO -  >= 0: Disallow raw and ftrace function tracepoint access
[task 2026-05-20T19:50:50.078+00:00] 19:50:50     INFO -  >= 1: Disallow CPU event access
[task 2026-05-20T19:50:50.079+00:00] 19:50:50     INFO -  >= 2: Disallow kernel profiling
[task 2026-05-20T19:50:50.079+00:00] 19:50:50     INFO -  To make the adjusted perf_event_paranoid setting permanent preserve it

:aerickson, could we get a couple of machines (say 084 and 225, but more the merrier?) isolated with perf_event_paranoid set to 0 (or minimal sudo to run perf)? Picked those two based on sparky's query as machines that have been alternating between modes: https://sql.telemetry.mozilla.org/queries/119338#291440

Before making any changes to governor settings, we should check system profiles first

Also re: comment #29: from what I can find, there's no "balanced" governor on Linux. On systems using intel_pstate it seems the only options are performance and powersave?

Flags: needinfo?(aerickson)

Yeah, I've created a PR (https://github.com/mozilla-platform-ops/ronin_puppet/pull/1221) that installs the package and allows the test user to run perf via sudo and it's running on 084 and 225.

I guess the issue now is targetting... do you want these in a separate pool or do you just want to run enough jobs to hit these occasionally (and have code check if it can run the sudo command)?


[cltbld@t-linux64-ms-084 ~]$ sudo perf

 usage: perf [--version] [--help] [OPTIONS] COMMAND [ARGS]

 The most commonly used perf commands are:
   annotate        Read perf.data (created by perf record) and display annotated code
   archive         Create archive with object files with build-ids found in perf.data file
   bench           General framework for benchmark suites
   buildid-cache   Manage build-id cache.
   buildid-list    List the buildids in a perf.data file
   c2c             Shared Data C2C/HITM Analyzer.
   config          Get and set variables in a configuration file.
   daemon          Run record sessions on background
   data            Data file related processing
   diff            Read perf.data files and display the differential profile
   evlist          List the event names in a perf.data file
   ftrace          simple wrapper for kernel's ftrace functionality
   inject          Filter to augment the events stream with additional information
   iostat          Show I/O performance metrics
   kallsyms        Searches running kernel for symbols
   kvm             Tool to trace/measure kvm guest os
   list            List all symbolic event types
   mem             Profile memory accesses
   record          Run a command and record its profile into perf.data
   report          Read perf.data (created by perf record) and display the profile
   script          Read perf.data (created by perf record) and display trace output
   stat            Run a command and gather performance counter statistics
   test            Runs sanity tests.
   top             System profiling tool.
   version         display the version of perf binary
   probe           Define new dynamic tracepoints
   trace           strace inspired tool
   kmem            Tool to trace/measure kernel memory properties
   kwork           Tool to trace/measure kernel work properties (latencies)
   lock            Analyze lock events
   sched           Tool to trace/measure scheduler properties (latencies)
   timechart       Tool to visualize total system behavior during a workload

 See 'perf help COMMAND' for more information on a specific command.

[cltbld@t-linux64-ms-084 ~]$```
Flags: needinfo?(aerickson)

(In reply to Andrew Erickson [:aerickson] from comment #32)

I guess the issue now is targetting... do you want these in a separate pool or do you just want to run enough jobs to hit these occasionally (and have code check if it can run the sudo command)?

awesome thanks! yes a specific pool we could target with --worker-override would be great. then I can use my Try push from comment 31

awesome, thanks :aerickson!

here's a try run so far...
https://treeherder.mozilla.org/jobs?repo=try&tier=1%2C2%2C3&searchStr=sp3&revision=9969734c93795b3243caf1c21303d11ce5329711

perf seems to be working...

markus/denis could you verify the system profiles look good? (the symbolicated one is available in the profiler zip e.g., and the raw perf.data is also available in the artifacts) if so then we can add more retriggers and/or change the perf flags

(this is the diff used https://gist.github.com/92kns/fadeb030bfa0194d6651fd516c5f1572)

Flags: needinfo?(mstange.moz)
Flags: needinfo?(dpalmeiro)
Assignee: nobody → kshampur
Status: NEW → ASSIGNED

(In reply to Kash Shampur [:kshampur] ⌚EST from comment #35)

awesome, thanks :aerickson!

here's a try run so far...
https://treeherder.mozilla.org/jobs?repo=try&tier=1%2C2%2C3&searchStr=sp3&revision=9969734c93795b3243caf1c21303d11ce5329711

perf seems to be working...

markus/denis could you verify the system profiles look good? (the symbolicated one is available in the profiler zip e.g., and the raw perf.data is also available in the artifacts) if so then we can add more retriggers and/or change the perf flags

(this is the diff used https://gist.github.com/92kns/fadeb030bfa0194d6651fd516c5f1572)

(prelim look)

Used profiler-cli to examine the system profiles from the try push. Apparently the bimodality is caused by gnome-shell/Mutter consuming large amounts of CPU via software rendering (Mesa llvmpipe).

In low-scoring runs, Mutter's clutter compositor is active and competes with Firefox for CPU
In high-scoring runs, gnome-shell is nearly idle (~15s).

The suspect is GNOME's screen blank timeout (5 min default). When the screen blanks before the test, Mutter suspends compositing → Firefox gets full CPU → high score.
If it doesn't blank in time, Mutter composites every frame Firefox submits → low score.

Fun! Here are two imported profiles along with their numbers:

Task ID Worker Profile Run 1 Run 2 Run 3 Run 4 Run 5 Mean
diJwfJvBRm6v_0iopaZnJg t-linux64-ms-084 profile 13.265 13.282 13.372 13.258 13.479 13.331
Mml6rueSQAS1Ge3QYcoEcA t-linux64-ms-084 profile 12.711 12.610 12.661 12.754 12.752 12.698

In the slow profile, indeed there is a lot of CPU activity on llvm-pipe threads in the gnome-shell process.

Flags: needinfo?(mstange.moz)

That's a very interesting finding. I wonder how this ties into the time correlation from comment 8, not sure what makes the GNOME session become active. But maybe it is actually just random, there were not too many data points, after all.

It seems to me that we need to decide which scenario (compositor active or not) we want to be used for testing and ensure it is stably used the same (might be easier to enforce the screen to be always on?). My gut feeling is that we'd get cleaner signal about perf changes from runs without compositor. But it is probably also the less realistic mode of operation for "performance delivered to user" measurements. A realistic desktop scenario would probably have some HW-accelerated compositor being active ?

(In reply to Andrew Erickson [:aerickson] from comment #32)

Yeah, I've created a PR (https://github.com/mozilla-platform-ops/ronin_puppet/pull/1221) that installs the package and allows the test user to run perf via sudo and it's running on 084 and 225. ``

Hi Andrew, is it possible to enable this functionality on all the workers so we can generate these perf profiles regularly?

Flags: needinfo?(dpalmeiro)
Flags: needinfo?(aerickson)

(In reply to Denis Palmeiro [:denispal] from comment #39)

(In reply to Andrew Erickson [:aerickson] from comment #32)

Yeah, I've created a PR (https://github.com/mozilla-platform-ops/ronin_puppet/pull/1221) that installs the package and allows the test user to run perf via sudo and it's running on 084 and 225. ``

Hi Andrew, is it possible to enable this functionality on all the workers so we can generate these perf profiles regularly?

Yes, the branch has been merged. The tool and the sudoers rule will be present on all 1804 and 2404 linux moonshot hosts.

Flags: needinfo?(aerickson)

thank you :aerickson for the openbox PR. I've got some interesting results now:

https://perf.compare/compare-results?baseRev=9969734c93795b3243caf1c21303d11ce5329711&baseRepo=try&newRev=6e2d023d8772e807642df7403d07cb9375dc022b&newRepo=try&framework=13&test_version=student-t&title=Use+openbox+--replace

using openbox --replace seems to help in the sense that there is one mode now and it is the higher mode.

(In reply to Jens Stutte [:jstutte] from comment #38)

It seems to me that we need to decide which scenario (compositor active or not) we want to be used for testing and ensure it is stably used the same (might be easier to enforce the screen to be always on?). My gut feeling is that we'd get cleaner signal about perf changes from runs without compositor. But it is probably also the less realistic mode of operation for "performance delivered to user" measurements. A realistic desktop scenario would probably have some HW-accelerated compositor being active ?

and based on this comment i also tried out always on and always off screen states using xset, but in either case it is in the low mode.

off : https://perf.compare/compare-results?baseRev=9969734c93795b3243caf1c21303d11ce5329711&baseRepo=try&newRev=c457b65cb0a74ced7cfc7c92f118b297db395b10&newRepo=try&framework=13&test_version=student-t&title=screen+state+off

on: https://perf.compare/compare-results?baseRev=9969734c93795b3243caf1c21303d11ce5329711&baseRepo=try&newRev=390316981fb289f0decd29b10d5c1b889dd14e96&newRepo=try&framework=13&test_version=student-t&title=screen+state+on

profiler-cli suggests that's exactly what's happening under the hood. in the openbox run the gnome-shell compositor is just... gone. no llvmpipe threads. but it's still present in the on/off xset runs, so screen state doesn't matter, Mutter stays active and keeps the runs in the lower mode.

which i think also answers your question :jstutte, openbox gives us the "no compositor" scenario you leaned toward and locks it in stably.
the tradeoff is it's the less realistic mode (no HW-acceleration)
https://mozilla-hub.atlassian.net/browse/RELOPS-2392 would attempt to address that.

In the openbox profiles, is there activity on llvmpipe threads in the Firefox GPU process? If so that means we're still not really taking advantage of the GPU.

(In reply to Markus Stange [:mstange] from comment #43)

In the openbox profiles, is there activity on llvmpipe threads in the Firefox GPU process? If so that means we're still not really taking advantage of the GPU.

yep, good question, there's still a ton of llvmpipe activity in the Firefox GPU process. so we're definitely still not using the actual GPU. openbox just got rid of gnome-shell's software compositor, but Firefox itself is still rendering everything on the CPU via llvmpipe.

getting on the Intel GPU (https://mozilla-hub.atlassian.net/browse/RELOPS-2392) is what'd actually fix that part.

:aerickson added config to use the intel gpu

https://perf.compare/compare-results?baseRev=9969734c93795b3243caf1c21303d11ce5329711&baseRepo=try&newRev=5a1f95d5fa49aca7344023e3339cbb92da5b0155&newRepo=try&framework=13&test_version=student-t&title=base+vs+intel+config

try still running but so far looks good. examining the existing profiles suggests no more llvmpipe, gnomeshell is present but idle, and rendering is happening in the intel gpu

(In reply to Kash Shampur [:kshampur] ⌚EST from comment #45)

:aerickson added config to use the intel gpu

https://perf.compare/compare-results?baseRev=9969734c93795b3243caf1c21303d11ce5329711&baseRepo=try&newRev=5a1f95d5fa49aca7344023e3339cbb92da5b0155&newRepo=try&framework=13&test_version=student-t&title=base+vs+intel+config

try still running but so far looks good. examining the existing profiles suggests no more llvmpipe, gnomeshell is present but idle, and rendering is happening in the intel gpu

still looks bimodal, but with a tighter spread

(In reply to Kash Shampur [:kshampur] ⌚EST from comment #46)

(In reply to Kash Shampur [:kshampur] ⌚EST from comment #45)

:aerickson added config to use the intel gpu

https://perf.compare/compare-results?baseRev=9969734c93795b3243caf1c21303d11ce5329711&baseRepo=try&newRev=5a1f95d5fa49aca7344023e3339cbb92da5b0155&newRepo=try&framework=13&test_version=student-t&title=base+vs+intel+config

try still running but so far looks good. examining the existing profiles suggests no more llvmpipe, gnomeshell is present but idle, and rendering is happening in the intel gpu

still looks bimodal, but with a tighter spread

oh but it actually seems to be machine specific. that pool has 2 workers only, (084 and 225)

  • ms-084: 14.00–14.13 (8 jobs)
  • ms-225: 14.38–14.51 (9 jobs)

084 seems to be consistently a bit slower (it's actually consistently slower across every subtest, not just the rendering ones). so this maybe is not truly bimodal. either way we should probably roll out the intel config everywhere unless anyone has objections

Doing a quick check on a few non-sp3 tests before requesting a full rollout from aerickson. Try runs still running but so far so good.

FWIW, I don't think we should switch to openbox unless we can show that gnome-shell is unworkable

(In reply to Kash Shampur [:kshampur] ⌚EST from comment #48)

Doing a quick check on a few non-sp3 tests before requesting a full rollout from aerickson. Try runs still running but so far so good.

Try runs looked green. Requested :aerickson for this to be applied to all

:aerickson merged https://github.com/mozilla-platform-ops/ronin_puppet/pull/1227

let's see how the autoland/m-c data looks over the next few days...

Still pretty noisy but looks less bimodal

See Also: → 2046664

some low-scorers are still appearing...
https://treeherder.mozilla.org/perfherder/graphs?highlightAlerts=1&highlightChangelogData=0&highlightCommonAlerts=0&highlightInitialDataPoints=0&replicates=0&selected=5691153,2556484167&series=autoland,5691153,1,13&timerange=604800

ran a system profile try push here
https://treeherder.mozilla.org/jobs?repo=try&tier=1%2C2%2C3&revision=66f3dc5b51d9b187093098e7b5e8352cc1c51761&selectedTaskRun=PgMc2ThPQpeyffCScImXMA.0

to look for some low scoring ones

and according to profiler-cli the low scoring ones seem to not be using intel gpu

in this Try push it seems to be ms-019, ms-026, and ms-027 (not an exhaustive list, i am sure there might be more...? and in one case ms-019, no sudo privileges on perf command so definitely doesn't have any of the recent deployed changes...)

:aerickson could you please take a look? is it possible the recently deployed changes haven't been applied to the entire linux fleet?

Flags: needinfo?(aerickson)

Can you link to a low scoring profile?

Flags: needinfo?(kshampur)

(In reply to Jeff Muizelaar [:jrmuizel] from comment #55)

Can you link to a low scoring profile?

here is one and here is another

Flags: needinfo?(kshampur)

Yes, I agree. Those are not using the Intel GPU

(In reply to Kash Shampur [:kshampur] ⌚EST from comment #54)

:aerickson could you please take a look? is it possible the recently deployed changes haven't been applied to the entire linux fleet?

Checked those hosts. They were 3 of 4 hosts we were testing another branch on and their branch hadn't been rebased on the new master. Fixed now.

Flags: needinfo?(aerickson)

Where are we at with this?

(In reply to Andrew Erickson [:aerickson] from comment #59)

Where are we at with this?

just catching up post PTO.... i'd say bimodality is clearly gone so we can close this.
But data is still noisy but that should be a separate bug.
variance seems to be intermachine
https://performance.mozilla.org/speedometer_job_debug.html?os=linux&range=1month&repository=autoland&hide=Chrome

Status: ASSIGNED → RESOLVED
Closed: 4 months ago
Resolution: --- → FIXED
See Also: → 2055283
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: