925864 - collectd df module needs to filter out non fixed disk mounts

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: General, task)

Description

[Jake Watkins [:dividehex]]

Whenever a DMG gets mounted on an OSX releng system, it registers as a mount point and collectd picks it up.  This wastes space and other resources on graphite6.  Especially since new creates are rate limited and could slow down the creation of new (real) metrics wsp dbs.

e.g.
creating database file /var/lib/carbon/whisper/hosts/bld-lion-r5-050_build_releng_scl3_mozilla_com/df/private-tmp-FirefoxInstaller/df_complex/used.wsp (archive=[(60, 789120)] xff=None agg=None)

Comment 1

[Jake Watkins [:dividehex]]

I attempted to fiddle with the df config yesterday to get it to only report certain /dev/disk* devices but it just wasn't picking up the filter.  I'll need to look at the source to determine how the mount list is being read on OSX and test a filter based on that.  It might also be easier to filter by Volume name as long as we have a known list of names we use.  Keeping Volume names consistent in current and future OSX platforms would be come more essential in this case

Comment 2

[Jake Watkins [:dividehex]]

As far as I can see, there is no easy way to identify fixed disks vs mounted dmgs. There might be some differences in the values of the returned statfs struct [1] but for OSX, collectd only cares about three of them (f_mntonname, f_mntfromname, f_fstypename). I added some extra debug info, recompiled collectd-5.3.0 with debugging enabled and captured the values it was reading from the getfsstat() call.

I think the best fix(workaround?) here is to filter for "/"
... and "/Volumes/Build" since talos-mntlion-r5-* mount the second hdd to that mount point.

Root of course
[2013-10-28 17:05:59] [debug] dir: /
[2013-10-28 17:05:59] [debug] device: /dev/disk0s2
[2013-10-28 17:05:59] [debug] type: hfs

[2013-10-28 17:05:59] [debug] dir: /dev
[2013-10-28 17:05:59] [debug] device: devfs
[2013-10-28 17:05:59] [debug] type: devfs

[2013-10-28 17:05:59] [debug] dir: /net
[2013-10-28 17:05:59] [debug] device: map -hosts
[2013-10-28 17:05:59] [debug] type: autofs

[2013-10-28 17:05:59] [debug] dir: /home
[2013-10-28 17:05:59] [debug] device: map auto_home
[2013-10-28 17:05:59] [debug] type: autofs

This is a DVD installer disk
[2013-10-28 17:05:59] [debug] dir: /Volumes/Mac OS X Install DVD
[2013-10-28 17:05:59] [debug] device: /dev/disk1s3
[2013-10-28 17:05:59] [debug] type: hfs

This is a dmg mounted
[2013-10-28 17:05:59] [debug] dir: /Volumes/Testing
[2013-10-28 17:05:59] [debug] device: /dev/disk2s2
[2013-10-28 17:05:59] [debug] type: hfs

[1] https://developer.apple.com/library/mac/documentation/Darwin/Reference/ManPages/man2/statfs.2.html

Comment 3

[Jake Watkins [:dividehex]]

Attachment: bug925864.patch

Adding MountPoint config values causes explicit filtering for mount point names
http://collectd.org/documentation/manpages/collectd.conf.5.shtml#plugin_df

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 824386 [details] [diff] [review]
bug925864.patch

Review of attachment 824386 [details] [diff] [review]:
-----------------------------------------------------------------

Looks good.  So it sounds like the "Device" config doesn't work, while the "MountPoint" does.  Is that worth reporting upstream?

Comment 5

[Jake Watkins [:dividehex]]

(In reply to Dustin J. Mitchell [:dustin] (I read my bugmail; don't needinfo me) from comment #4)

> Looks good.  So it sounds like the "Device" config doesn't work, while the
> "MountPoint" does.  Is that worth reporting upstream?

I can't say for sure if the device config worked or not.  It might have been a regex bug or me just being dumb.  I just decided to go down the path of filtering by mountpoint since it is more definitive.  rootfs == "/" should always be true. Since cd/dvd drives appear as /dev/disk* also, filtering for /dev/disk2 will pickup the second disk or cd/dvd depending on if a second disk exists or not.  We have both hardware configurations.

Not to mention the minis with RAID configured!