377043 - Statistical analysis tool for cron job mail

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations: Projects, task)

Description

[Dave Miller [:justdave]]

I was gone on vacation for a week, and cleared out my mailbox that the cron job mail going to sysalerts filters into the day I left.  Upon my return, that mailbox had 26373 messages in it.  That's a hell of a lot of cron job mail in a week.

A lot of it is probably nuisance mail that we should find a way to eliminate (that's covered on another bug I believe).  But even so, important stuff seems to be getting missed...

I'm thinking it might be good to set up a program somewhere and have it process a mailbox that is fed by the sysalerts list.  This program would have a database and track things like the number of repeat mails from the same server/cron job and basically give us an overview on a web page or something of which cron jobs are causing output how often, let us tag specific messages as nuisance (based on the output content, not the cron job).  We could perhaps have it submit notices into nagios for paging when an unknown output happens.  If it works well, we could get all the cron mail off the sysalerts list and ease the burden on our mailboxes (and the oncall person can just work with the webtool).

Comment 1

[J. Paul Reed [:preed]]

(In reply to comment #0)

> I'm thinking it might be good to set up a program somewhere and have it process
> a mailbox that is fed by the sysalerts list.  This program would have a
> database and track things like the number of repeat mails from the same
> server/cron job and basically give us an overview on a web page or something of
> which cron jobs are causing output how often, let us tag specific messages as
> nuisance (based on the output content, not the cron job).  We could perhaps
> have it submit notices into nagios for paging when an unknown output happens. 
> If it works well, we could get all the cron mail off the sysalerts list and
> ease the burden on our mailboxes (and the oncall person can just work with the
> webtool).

Splunk?

Comment 2

[Dave Miller [:justdave]]

oremj is currently working on this (I saw a demo of what he had set up so far yesterday)

Comment 3

[Jeremy Orem [:oremj]]

Splunk setup is complete and mail is already being processed.