1285483 - t-yosemite-r7 machines have high coreaudiod and launchd CPU usage

Status: RESOLVED INCOMPLETE

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Nick Thomas [:nthomas] (UTC+12)]

coreaudiod is running at ~80% CPU continuously, launchd at ~40%. Originally Ithought this was just t-yosemite-r7-0022 (bug 1260442 comment #7 though #10) but it persisted after a reimage and hardware diagnostics. On inspection of other machines in the same pool it seems they are also affected.

It may be related to busted plist files:
 http://alexborisov.org/osx-high-cpu-by-coreaudiod-and-launchd/
 https://discussions.apple.com/thread/6605981?start=0&tstart=0

We have bluetooth disabled via puppet.

Comment 1

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

We have now for all t-yosemite-r7 : coreaudiod  running at ~80% CPU continuously and launchd running at ~40%.
I made some test and found that enabling Bluetooth will solve this issue and reduce to 0% the CPU.
We can simply enabling Bluetooth with this command: 
sudo launchctl load -w /system/library/launchdaemons/com.apple.blued.plist.

Comment 2

[Kim Moir [:kmoir] ET]

So I'm not sure we want to enable bluetooth. See bug 786679

That being said, if there are changes that need to be made the configuration changes would have to be made in puppet.

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

I agree with Kim, enabling it will probably have consequences. What if we do the load -w, then unload, let puppet do what we have configured (https://dxr.mozilla.org/build-central/source/puppet/modules/disableservices/manifests/common.pp#53, #102 #109) and see how the CPU usage is ?

Comment 4

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

As soon as we unload  bluetooth the CPU will be back at ~80% for coreaudiod and ~40% for launchd.I have also tested that.As bluetooth is missing these 2 processes will try to find it and for this will use the CPU.

Comment 5

[Kim Moir [:kmoir] ET]

Not sure what it happening here.  It looks like this is a bug with os x from the reports on the web, and Apple's bug tracking is all closed to the public iirc so not sure how we can proceed other than disabling services and see if that brings down the load.  As a side note, it looks like these preferences have changed in the latest version of os x

http://hg.mozilla.org/build/puppet/file/tip/modules/disableservices/manifests/common.pp#l102

(no longer in this plist)

and if you run 
defaults read > defaults.txt 
and look at the file 
there are other new bluetooth defaults that might need to be disabled via puppet

Comment 6

[Kim Moir [:kmoir] ET]

You could try submitting a bug to apple

https://developer.apple.com/bug-reporting/

Not sure if you will hear anything from them, I have not heard any feedback from them in the past

Comment 7

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

I submitted some bugs to apple but did not get any feedback until today.

Comment 8

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Amy do you know if we have any special agreament with Apple  and if we can open a ticket with this issue?

Comment 9

[Amy Rich [:arr] [:arich]]

They're just consumer machines with applecare, no special contract.

Comment 10

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

By unloading coreaudio,launchd will be using 0 CPU but when coreaudio is again loaded,the CPU is back at (80% for coreaudio and 40% at launchd) if the bluetooth is disabled.
Also I did not received any answer from Apple about this issue on OSX and didn't found any configuration link between coreaudiod and bluetooth. 
Any new idea or suggestion is welcome here.

Comment 11

[Hal Wine [:hwine] use NI!]

Do we know if 'coreaudio' is required for any of the tests being run?

I just started having a similar issue with my work laptop, and researched this morning serendipitous finding:
  http://blog.viktorpetersson.com/post/100148585299/how-to-fix-kerneltask-cpu-usage-on-yosemite
Which has the deeper link:
  http://www.rdoxenham.com/?p=259
which suggests that the load may be purposeful to cool the CPU. 

I can't tell from comment 10 if the overall load drops to 0, or just the coreaudio load does. See the display in the first article linked above -- if we just see the cpu load shift to another service under kernel_task, we may be hitting this CPU cooling issue.

If it is a thermal issue, :arr probably has some suggestions ;)

Comment 12

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Attachment: bug1285483_bluetooth_off.PNG

The processes when bluetooth is off.

Comment 13

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Attachment: bug1285483_bluetooth_on.PNG

The processes when bluettoth is on.

As can be seen only coreaudio and launchd are changed,all other processes use the same CPU,so I don't think is the same problem as in those links.

Comment 14

[Kim Moir [:kmoir] ET]

Since we are getting new macs in the near future, we can close this bug.

Comment 15

[Amy Rich [:arr] [:arich]]

FYI, we are not getting new versions of macs or upgrading the OS. The additional macs that we have now will not increase the pool size but will be the percentage that will be offline for the datacenter moves. If we need to figure this out to stay within capacity for tests, we should keep this bug open and solve it.

Comment 16

[Kim Moir [:kmoir] ET]

Okay sorry for the misunderstanding on my part.

Comment 17

[Kim Moir [:kmoir] ET]

Andrei

Maybe ask in #sysadmins if anyone has any suggestions for this bug, perhaps they might know someone who has better mac knowledge

Comment 18

[Nick Thomas [:nthomas] (UTC+12)]

I originally filed this with 'if you see something, file a bug' in mind, but maybe it's not an issue that's a big deal or worth putting any more energy in to ? Specifically, if it's deliberate (per comment #10) but backs off under other load it may have no impact on our tests, and only show up when idle. We could monitor the CPU usage during a test to check that.

Comment 19

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Monitored the activity of some machines from this pool during some tests,seems coreaudiod and launchd have constant CPU usage also when is not idle but have no impact on tests.