411360 - Add nagios script to monitor Socorro collector

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Samuel Sidler (old account; do not CC)]

Reported by bsmedberg, Jul 10, 2007

Provide a method for nagios to monitor the status of the collector. We
should attempt to write to the filestore during it, instead of merely
returning an "I'm alive" result.

Sayrer, can you take this?

Comment 1

[Carsten Book [:Tomcat]]

dupe of bug 411360 ?

Comment 2

[Carsten Book [:Tomcat]]

(In reply to comment #1)
> dupe of bug 411360 ?
> 

err i mean bug 401810

Comment 3

[Michael Morgan [:morgamic]]

Not a dupe, they are separate tiers.  Aravind -- do you have a monitor already in place for this?  If not, how do you want to go about checking the collector?

Comment 4

[(not currently active) Ted Mielczarek]

You used to be able to GET https://crash-reports.mozilla.com/submit and get back a "405 method not allowed", but I guess we changed that? Looks like you can still HEAD it and get that, so maybe that's sufficient?

Comment 5

[Samuel Sidler (old account; do not CC)]

Aravind, any update here?

Comment 6

[Michael Morgan [:morgamic]]

Aravind, is there any nagios check for the collector?

Comment 7

[Michael Morgan [:morgamic]]

This isn't fixed, monitoring isn't there yet.

Comment 8

[Michael Morgan [:morgamic]]

Aravind - can you post an update here and give an ETA for when a check could be put in place?

Comment 9

[Michael Morgan [:morgamic]]

Aravind - can you post an update here and give an ETA for when a check could be
put in place?

Comment 10

[Aravind Gottipati [:aravind]]

I will try and get something going this week.

Comment 11

[Michael Morgan [:morgamic]]

All we need is a HEAD request on /submit to verify the 405 exists.

Comment 12

[Michael Morgan [:morgamic]]

-> server-ops so this can get done

Comment 13

[Michael Morgan [:morgamic]]

This needs some support from webdev:
1) collector configuration to ignore the "nagios" product
2) sample dump file with post parameters to mimic a submission
3) curl command to submit said post to verify that a valid UUID is sent back

Lars - could you provide instructions on 1) and I can provide the rest.  Thanks.

Comment 14

[Michael Morgan [:morgamic]]

Attachment: curl test

Comment 15

[Michael Morgan [:morgamic]]

Attachment: test dump for curl script

Comment 16

[Michael Morgan [:morgamic]]

So, to run this:
1) unpack the test dump somewhere
2) cron to run the .sh in same cwd as the test dump

At some point we can configure the collector to ignore submissions with Product=Nagios.

Expected output is easy to parse:
macnugget:td morgamic$ ./test.sh 
CrashID=bp-0b0ec860-85d7-438c-9a54-f8d720081119

Comment 17

[K Lars Lohn [:lars] [:klohn]]

to get collector to avoid queuing any submissions with "Nagios" as the product, add the line below to the list of throttle conditions in collectorconfig.py.  
This will cause the collector to place 100% of entries where Product == "Nagios" into deferred storage.  It will still, however, return a uuid for the submission.

("Product", "Nagios", 0), # queue nothing where Product == "Nagios"

#BEFORE:
throttleConditions.default = [
  # other conditions ...
  (None, True, 10) # queue 10% of what's left
]

#AFTER:
throttleConditions.default = [
  ("Product", "Nagios", 0),
  # other conditions ...
  (None, True, 10) # queue 10% of what's left
]

Comment 18

[Michael Morgan [:morgamic]]

s/Nagios/nagios/

Comment 19

[Michael Morgan [:morgamic]]

Lars, is this a code change we can make in the default settings?  If so it might be easier to deploy.

Comment 20

[K Lars Lohn [:lars] [:klohn]]

Morgamic: are you suggesting that I put it into the collectorconfig.py.dist file?

Comment 21

[Michael Morgan [:morgamic]]

Eh, maybe not a good idea.  I guess IT can figure out how to add the new rule.  I was just remember we added the non-release version condition last week as well, but that's just another list member so it shouldn't be hard.

Comment 22

[Aravind Gottipati [:aravind]]

Done, now have the test script submitting a crash report and nagios verifying that we get a uuid back.

Comment 23

[Michael Morgan [:morgamic]]

Sweet!  Thanks Aravind -- have a good weekend.