Closed Bug 581126 Opened 15 years ago Closed 15 years ago

Create nagios alert to warn when Weave scripts provided by Metrics fail

Categories

(mozilla.org Graveyard :: Server Operations, task)

Product:
mozilla.org Graveyard
Component:
Server Operations
Platform:
x86
All
Type:
task
Priority:
Not set
Severity:
major

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: nelson.sousa, Assigned: aravind)

Details

The scripts provided by the Metrics team to extract Weave data from Phoenix have failed a few times, causing the Weave Summary dashboard to break. Although a safeguard is in place now that prevents this, whenever the scripts fail no new data is provided to the dashboard and it's necessary that someone within Metrics is informed and the script logs emailed to the Metrics team so that we can track the causes and solve them as fast as possible.
We need more information here. What exactly do you want us to monitor? Who needs to be alerted? Nagios isn't going to email with logs, you'll have to work on something else to do that.
Assignee: server-ops → ayounsi
Cc'ing zandr. He's very familiar with the process and can give valuable input here
CC'ing Aravind as well, he's been our contact person when we detect problems. The purpose of the alert is to let you know when something goes wrong, so that you can then inform us. The main problem with these scripts is that we only find out something went wrong when we look at the dashboard and see there's data missing. Then we must contact someone from Weave or IT to send us the logs and then find a solution. The goal is to avoid one of these interactions, streamlining the whole process of determining the causes and implementing a solution as quickly as possible.
Summary: Create nagio alert to warn when Weave scripts provided by Metrics fail → Create nagios alert to warn when Weave scripts provided by Metrics fail
Assignee: ayounsi → server-ops
Assignee: server-ops → aravind
Nagios isn't meant for this sort of stuff. I am going to put in a check in the script so it e-mails nelson if stuff goes wrong. Please let me know if that's not adequate.
not adequate. Email mozilla@webdetails.pt . You'll make more noise and Nelson will appreciate having more people able to answer that call
Done. Script will e-mail mozilla@webdetails.pt if it exits with a non-zero status.
Status: NEW → RESOLVED
Closed: 15 years ago
Resolution: --- → FIXED
Thanks.
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.