604397 - Change Socorro collector monitors for 1.7.5

Status: RESOLVED WONTFIX

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[Laura Thomson :laura]

When we push 1.7.5 (anticipated in a week or so), collectors will begin logging to syslog.  Since collector monitoring is I think partly based on logs, this monitor will need to change accordingly.

We should work out the details of the new monitor before the release.

Comment 1

[Justin Dow [:jabba]]

I think collector logs aren't being monitored. Currently I think collectors are being monitored by nagios submitting crash reports only. Rather than WONTFIXing this bug, I wonder if I should instead implement a monitor on the syslog that searches for a specific string. The question is: Is there a specific string that would show up in the log that should have nagios page the oncall sysadmin?

Comment 2

[K Lars Lohn [:lars] [:klohn]]

perhaps watching the syslog for the strings "ERROR" or "CRITICAL" on a line that also contains "Socorro Collector" would do.

Comment 3

[Justin Dow [:jabba]]

I talked to aravind about it and we decided that wontfix is probably the best route here. I don't think we have a nagios script that we can just plug in here, so it'd take some time to write one. Given that the collector monitors work and work well, I think this would be extra noise and have low value to add additional monitors.