844204 - Monitor OS X Finder CPU usage while building

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Gregory Szorc [:gps]]

I noticed that my build times started becoming much longer on my MacBook Pro. I opened up Activity Monitor.app and noticed that when I build, the Finder process (what OS X uses for indexing) is consuming anywhere from 80% to 200% CPU during the course of the build. This adversely impacts my build times on the order of ~7 minutes.

I recently installed OS X 10.8 when I installed a fresh SSD into my machine. I'm not sure if the SSD is bad or what.

I just disabled Spotlight indexing of my source tree. I guess we'll see if that has any effect.

It might be worth adding some monitoring to mach or mozbuild to detect high Finder CPU usage and offer to fix the problem (assuming we identify one).

Comment 1

[Richard Newman [:rnewman]]

Perhaps we should be giving our objdir a ".noindex" suffix…

Comment 2

[Seth Fowler [:seth] [:s2h]]

Puzzling. I'm also on OS X 10.8, and I don't see this behavior.

Comment 3

[Gregory Szorc [:gps]]

Attachment: Monitor finder CPU usage during build, v1

This annoyed me so much I created a patch!

Things will only work if you have psutil available (optionally part of virtualenv) and if you are building on Darwin.

I confirmed locally that this patch works. It's a little hard to unit test, so no tests are included.

Eventually all of this logic for building the tree and performing background tasks during building will likely be moved out of mach_commands.py. I don't have the will to do this yet. Some day.

Comment 4

[Gregory Szorc [:gps]]

I've been doing some experimentation.

The md* processes are also using a fair amount of CPU during builds. However, these are running as root, so we can't get their CPU counters through psutil unless we run as root. I'm not comfortable trying to do that.

If I exclude my object directory from Spotlight, Finder.app stops hammering my CPU during builds.

Finder.app seems to act up only when doing incremental builds. If I do a clobber, it doesn't get out of control. Who knows what's going on under the hood.

Comment 5

[Gregory Szorc [:gps]]

Saw a related post on planet: http://blog.pearce.org.nz/2013/03/reducing-windows-background-cpu-load.html

We should consider expanding scope of this bug to cover services in Windows. At least we should file a follow-up bug to handle them. Slow builds anywhere because of OS magic is a bad developer experience! And who knows, maybe our RelEng infra is misbehaving too!

Comment 6

[(not currently active) Ted Mielczarek]

I think if we know of specific things that will cause slowdowns we should either programmatically disable them or prompt the user to disable them. Specifically, it sounds like Spotlight indexing of the objdir, Windows indexing, and Microsoft Defender scanning are all things we ought to disable if possible.

Comment 7

[Gregory Szorc [:gps]]

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #6)
> I think if we know of specific things that will cause slowdowns we should
> either programmatically disable them or prompt the user to disable them.
> Specifically, it sounds like Spotlight indexing of the objdir, Windows
> indexing, and Microsoft Defender scanning are all things we ought to disable
> if possible.

I agree. However, I'm not sure how much we'll be able to disable automatically. I'm pretty sure Microsoft Defender won't let you selectively turn it off without a UAC prompt :)

Surely there are APIs we could query to determine if an exclusion is present and prompt/warn to install them?

Comment 8

[(not currently active) Ted Mielczarek]

Comment on attachment 717300 [details] [diff] [review]
Monitor finder CPU usage during build, v1

Review of attachment 717300 [details] [diff] [review]:
-----------------------------------------------------------------

This looks fine, but I feel like you ought to have these methods be more generically-named so we could apply this on other platforms. It's not a particularly big deal, and maybe we can just cross that bridge when we come to it.

::: python/mozbuild/mozbuild/mach_commands.py
@@ +33,5 @@
> +during the build - an average of %f%% (100%% is 1 core). This made your build
> +slower.
> +
> +Consider adding ".noindex" to the end of your object directory name to have
> +Finder ignore it. Or, add an indexing exclusion through the Spotlight System

I said this before, but we should probably just make the default object directory do this on Mac. (Either add .noindex, or figure out how to programmatically exclude it.)

Comment 9

[Gregory Szorc [:gps]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/c994692d1ea8

Yes, all the tree building code likely needs to be factored out of mach_commands.py and some care needs to be given to the method naming so we do intelligent things on other platforms.

Comment 10

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/c994692d1ea8