974220 - Monitoring for junction mounts on product delivery

Status: RESOLVED FIXED

(Infrastructure & Operations :: Storage, task)

Description

[Nick Thomas [:nthomas] (UTC+12)]

[Filing in Storage while requirements get sorted out]

I would like to have "first class" monitoring & logging of the disk usage on the product delivery partitions, now that junction mounts on the netapp have obscured some information. This is to maintain the coverage of pre-junction setup, which lets us catch out of disk situations before they happen.

Specifically, for all the separate partitions on the NetApp:
 * free+used disk space in graphite
 * free space alerts in nagios
We get the NFS mounts for free in graphite, but everything else is going to be manually managed (AIUI). As we add more junctions we should add monitoring.

How does that sound gcox ?

Comment 1

[Greg Cox [:gcox]]

I know zilch about graphite.

From what I can see, there weren't nagios watches on this part of the tree to begin with (mistake? possibly to avoid the spam of things lighting off in unison?  I dunno), just an "is this nfs mount still working?" check.  I've been cueing off the filer monitoring tools when I gripe about space usage.

We'll probably have to script something for nagios ourselves, as the mounts (beyond the top ones) don't/won't appear in /etc/fstab, and that freaks out the regular nagios checks.  The junctions DO appear in /proc/mounts, so we should be able to do a simple iteration* and make something happen there without having to nanny a list each time we touch a volume.

* for i in `cat /proc/mounts | awk '{print $2}' | grep ^/path/to/topmount` ; do  bundle up the alert status of $i; done return bundle;

Space is a bit of a mess; we should have that cleaned up in a week or so as we destroy old volumes from the TCW and add our final shelves.

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

That sounds fine, assuming we can map it over to graphite OK (or make netapp trending available to a few RelEngers ?) It appears that nested mounts don't show up in /proc/mounts at the second level though, eg firefox/tinderbox-builds appears, but firefox/tinderbox-builds/old doesn't. Yay ?

Comment 3

[Greg Cox [:gcox]]

Unused junction mounts atrophy out of /proc/mounts naturally.  Any file access down their path would make them reappear.

Comment 4

[Greg Cox [:gcox]]

Sketched out a nagios check a while back; need to bulletproof it and roll it out.

We have a talk later this week, on Baby's First Graphite, so maybe we can get some kind of work there in the medium future.

Comment 5

[Greg Cox [:gcox]]

graphite intro was good, I grok the basics.  Once I can automate the data pulls it should be doable to put together a dashboard.

Comment 6

[Greg Cox [:gcox]]

Rolled out my nagios check.

* ftp/upload/rsync/uploadcron have had their old "check_nfs_mounts" check replaced with my new "check_junction_mounts" check.  The check_nfs_mounts check wasn't providing much value: if the root of the junction filled, both would've lit off; if any subjunction filled, the old one wouldn't notice.
* Nagios warning on this check is at 91%, and critical is at 95%.  Our internal monitoring lights off at 80 and 90%, so we (Storage) will get a chance to catch things before it ends up lighting off the oncall's phone.  While 91% is high, it's against large volumes, so, that 9% is still a good chunk of space.
* Tested against hung mounts; does self-cleanup and reports a single hung mount: doesn't leave zombie df's, at least in general-purpose testing.

* Made a mention of the change in the Nagios mana on NFS Mounts.

TODO:
Will circle back when there's some graphite news.  Got a dedicated login, and my scripts are coming along, so it should be just a matter of putting all the pieces together and making some graphs that make sense.

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

I took a gander at the WIP in graphite, and it looks great. Thanks for setting it up.

To help me remember, there are lots graphs in gcox's userspace, and the partial path to the raw data is storage.netapp.scl3-nc1.vfilers.proddist.volumes.

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

All ready to close this now ?

Comment 9

[Greg Cox [:gcox]]

I'm holding it as a reminder to myself to finish some other graphs related to the data that is now being collected (which is now blocked on some unrelated filer changes), and to be consistent in both scl3 and phx1.

Effectively, though, yeah, it's done unless I have some spanking new ideas on what to graph.  Otherwise it's just some spit-and-polish stuff, and what you have in comment 7 is the crux of it.

Comment 10

[Nick Thomas [:nthomas] (UTC+12)]

Okie doke. For my use case it would be helpful for the graphite graphs to default to a one or two week time span, rather than 24 hours. I can duplicate into my user space if 24 hours works better for you though.

Comment 11

[Greg Cox [:gcox]]

:nthomas showed me how to fish; expanded the graphs out into the months range to show more trending.

Comment 12

[Greg Cox [:gcox]]

Cloned my esx stuff to phx1; good enough until someone wants new graphs.