I was on an OSX build slave and wondered why it was sluggish. It looks like there's a service called 'fseventsd' taking up quite a bit of CPU time.
[email@example.com ~]# find /.fseventsd/ -type f | wc -l 295503 [firstname.lastname@example.org ~]# du -ms /.fseventsd/ 11708 /.fseventsd/
also noticed this: Jun 12 13:15:47 bld-lion-r5-025 fseventsd: Logging disabled completely for device:1: /Volumes/Boot OS X Looks like '/Volumes/Boot OS X' is mounted for some time after boot, and then unmounted.... maybe from this: Jun 12 06:37:20 bld-lion-r5-027 com.apple.kextcache: / locked; waiting for lock. Jun 12 06:37:33 bld-lion-r5-027 com.apple.kextcache: Created prelinked kernel /System/Library/Caches/com.apple.kext.caches/Startup/kernelcache. Jun 12 06:37:33 bld-lion-r5-027 com.apple.kextcache: Lock acquired; proceeding. Jun 12 06:37:33 bld-lion-r5-027 com.apple.kextcache: /System/Library/Caches/com.apple.kext.caches/Startup/kernelcache out of date. Jun 12 06:37:36 bld-lion-r5-027 fseventsd: Logging disabled completely for device:1: /Volumes/Boot OS X Jun 12 06:37:57 bld-lion-r5-027 com.apple.kextcache: Successfully updated helper partition disk1s3. Jun 12 06:37:59 bld-lion-r5-027 fseventsd: Logging disabled completely for device:1: /Volumes/Boot OS X Jun 12 06:38:16 bld-lion-r5-027 com.apple.kextcache: Successfully updated helper partition disk0s3.
opendirectoryd also seems to take a lot of CPU from time to time
I've had this running on bld-lion-r5-025 for a few days. I'm not 100% sure if it's working, but it doesn't seem to be hurting either. I'd like to deploy this and let it bake for a few days and see if it makes a meaningful dent in build times.
Attachment #8622664 - Flags: review?(dustin)
Attachment #8622664 - Flags: review?(dustin) → review+
This didn't seem to make any difference. We should back this out.
I can't find that there's any way to globally disable fseventsd. That's a bummer.
so I took a look at fseventds time across a bunch of our machines, and there's a very strong correlation with total directory size. I think we could see improvement by purging data out of this directory, on boot, or maybe as part of runner
my hypothesis here is that we do a lot of operations that generate a huge amount of filesystem events that need to be logged (e.g. clobbers). as the directory grows, the OS needs to spend more time adding/removing entries from the growing directory file, and then writing it back to disk. the directory sizes for /.fseventsd on our build machines range from 28,084 to 9,936,670 bytes, with the median being 1,773,678 bytes NB this is the size of the directory file itself (ls -ld /.fseventsd), not the size of the directory's contents. I'm purging entries older than 7 days, and will see if this improves fseventsd time.
prior to purging old entries, the median total cpu time of fseventsd was 488s over 3,696 samples. post purging, the median time is 172s, over 11,284 samples. I'd like to purge this directory regularly, either as part of runner or puppet.
I wonder if we could file a radar bug with Apple to find out if there's a way to disable fseventsd or limit the amount of data it saves?
Subsequent cleanups didn't seem to have the same impact on cpu time. I'm not really sure what's going on here, but I can't fix it.
Assignee: catlee → nobody
Status: NEW → RESOLVED
Last Resolved: a year ago
Resolution: --- → INCOMPLETE
Component: Platform Support → Buildduty
Product: Release Engineering → Infrastructure & Operations
You need to log in before you can comment on or make changes to this bug.