Open Bug 2004395 Opened 8 months ago Updated 1 month ago

[meta] Reduce noise in CI machines due to background tasks

Categories

(Testing :: Performance, task, P2)

Product:
Testing
Component:
Performance
Type:
task

Tracking

(Not tracked)

Status:
ASSIGNED

People

(Reporter: kshampur, Assigned: kshampur)

References

Details

(Keywords: meta, Whiteboard: [fxp][operational])

I'll try to use this bug to track updates across slack/element/bugs instead of independently updating each bug.

re pasting profiles...

Windows:

RuntimeBroker.exe (Bug 1990093)
https://share.firefox.dev/4nGgjQP
https://share.firefox.dev/4mDboiQ

backgroundTaskHost.exe (Bug 1990091)
https://share.firefox.dev/4mvJYel

Several unknown processes (Bug 1990098)
https://share.firefox.dev/42GaScv (pids 3084 and 3104.)

macOS:

mediaanalysisd and mdworker_shared (Bug 1990096)
https://share.firefox.dev/46tVP6S

This comment has an example Try with Florian's diff applied https://bugzilla.mozilla.org/show_bug.cgi?id=1990096#c10 to look at the profiles

:mcornmesser to look into the background processes after first working on reduction of the management overhead on these workers.

for Mac, we'll have to try one of these 2 strats https://bugzilla.mozilla.org/show_bug.cgi?id=1990096#c15
or a 3rd strat of renaming the directory the task runs out of to have the .noindex extension (which might work in sequoia?) but renaming the task folders might be not be advisable

Ryan's PR that is tracking some Mac changes https://github.com/mozilla-platform-ops/ronin_puppet/pull/958

https://bugzilla.mozilla.org/show_bug.cgi?id=1990096#c17

for Mac, potentially worked ... waiting on confirmation

still waiting for confirmation on macOS , but Ryan has merged in the meantime
https://github.com/mozilla-platform-ops/ronin_puppet/commit/dac293b56d8f3c0dfdb601adc84afc0771e6fb1c

(In reply to Kash Shampur [:kshampur] ⌚EST from comment #4)

https://bugzilla.mozilla.org/show_bug.cgi?id=1990096#c17

for Mac, potentially worked ... waiting on confirmation

Need to repush to Try due to https://bugzilla.mozilla.org/show_bug.cgi?id=1990096#c19

(In reply to Kash Shampur [:kshampur] ⌚EST from comment #6)

(In reply to Kash Shampur [:kshampur] ⌚EST from comment #4)

https://bugzilla.mozilla.org/show_bug.cgi?id=1990096#c17

for Mac, potentially worked ... waiting on confirmation

Need to repush to Try due to https://bugzilla.mozilla.org/show_bug.cgi?id=1990096#c19

for posterity https://github.com/mozilla-platform-ops/ronin_puppet/commit/e10afea8a15f74b57761167e457e77b6929f773d
had caused some issues, so reverted and waiting on fix

for windows:

mcornmesser is getting close on a new config for the Win HW workers.

windows still being looked at by mcornmesser...

mac might be resolved now with a PR from rcurran (still in staging) https://bugzilla.mozilla.org/show_bug.cgi?id=1990096#c20

Windows HW Workers Update...

mcornmesser has been working on reducing background processes on Windows hardware workers. The initial config changes were merged (PR and additional details in in RELOPS-1946) and attempted to deploy to production last Thursday, but ran into issues and had to roll back. Planning to try again early this/next week.

For now, focusing on getting what's working today into production first. There are still two processes under investigation, but he's stepping away from that for a bit since we're not sure if removing them would impact OS stability.

One thing we learned: anything dealing with audio can't be removed. When those processes were disabled, we hit test failures with HEVC playback (see bug 2012007).

Also worth noting - there's a new monitoring agent called nscp that will be running as a background service on the Windows HW workers. Might be worth keeping an eye on it in case in introduces any CPU overhead.

See Also: → 2010120

As pointed in Bug 2010120, this seems to have started in July. I am not sure if anyone has dug deep into changes from around July... maybe it is worth checking https://github.com/mozilla-releng/fxci-config history (i'll run this idea by Mark)

I pointed out that ref-hw sp3 scores looks more stable compared to non ref hw
Mark has pointed out software configuration should be same between the two, and only hardware is different. So for the moment nothing obvious here...

and regarding the change in behaviour observed in July as pointed out by Bug 2010120, there doesn't seem to be anything interested in that time period. Just a GW upgrade. After recent changes get deployed Mark will look into checking that on an older version

windows update: new deployment going out for cleaner test environment

copy pasting summary from slack...

https://mozilla-hub.atlassian.net/browse/RELOPS-1946
https://github.com/mozilla-platform-ops/ronin_puppet/pull/1012

(except for the Appx package removal)

windows change had to be temporarily backed out due to failures.

another thing (copy pasting my message from slack):

I was comparing ref vs non-ref Win11 machines and a few things stood out.
the ref-hw reports a single processor speed (2.96 GHz) and non-ref has the 1.9 / 4.6 GHz range. Though this feels more like a symptom than a cause.. The question is what’s actually tipping them into one mode vs another. (maybe it is or isn't the background processes)

Looking at some CPU-related metrics we already collect:

Windows 11 Reference

Windows 11 (non-ref)

Ref-hw looks relatively stable across cpuTime, powerUsage_cpu_cores, and powerUsage_cpu_package (aside from a brief blip around Jan 23–28).
Non-ref hw shows noticeably more variation, and cpuTime in particular looks almost bimodal.

not sure yet what conclusion to draw (if there is any at all) but i thought this was be some interesting information

Mark has done some investigation as well and found that there is different BIOS versions in the non-ref workers (and less heterogeneity in the ref workers)

Mark/Jmoss are currently looking into some sort of temporary solution to elevated permissions so that Denis can collect ETW profiles of the win machines to dig deeper

There seems to be an improvement on intel macs (where we have the most noise anyway)

https://bugzilla.mozilla.org/show_bug.cgi?id=1990096#c25

even though bg processes have not completely disappeared.

See Also: → 2014786
Severity: -- → S2
Priority: -- → P1
Whiteboard: [fxp][operational]

few noteable updates with windows...

an xperf workaround has been set up on a small pool by :mcornmesser, and :mstange has been examining profiles.
nothing yet stands out from profile except there is clearly a "high" and "low" consistent mode

I did an analysis over the last 30 days and with mozilla-central data (regular pool) and found this as well
https://gist.github.com/92kns/26aaed34ab2de01c6693428a25edd4b7#notes

next step: new xperf command is being rolled out soon which might give more insight into the bimodal machines

have not updated this in a while..

windows:
so it was found that the power supply unit was failing. The units supplied have a mix of 120 W 20 V and 90 W 19 V, plus it seems some were failing

in the debug pool it was tested on a few NUCs and the replacement seems to have "restored" the sp3 scores

the ETA to check/swap all PSU's is in June due to scheduling

macOS x64.
We aren't too interested in x64 mac data these days and in fact platform wide seems like we are slowly moving over to the M4s (i have a bug here filed for perf https://bugzilla.mozilla.org/show_bug.cgi?id=2022278)
it is possible mac devices dont have OEM power units (currently unknown) but the old OS is also an issue

IMO we should just forget about intel macs (for the time being, and no one has been pushing for it either)

for Mac, https://bugzilla.mozilla.org/show_bug.cgi?id=1990096#c35

and Windows, until PSUs are replaced, mcornmesser is working on removing suspect nodes from the main pool so that in the interim we just run on "good" NUCs

Depends on: 2038340

newish update since last...

at the moment we are at about ~60 nodes and CI data has improved quite a bit.
Talking with Mark, we've decided to essentially now removed nodes with Fx sp3 scores below 22

We are still seeing suspected "good nodes" still degrading in real time, but for the most part things seem okay now with limited NUCs

:mcornmesser is working with the vendor and securing a batch of 20 new PSU's and then a systematic roll out process will begin in the following few weeks

Priority: P1 → P2
You need to log in before you can comment on or make changes to this bug.