838925 - Add monitoring for stuck timeout loops

Status: RESOLVED FIXED

(Testing Graveyard :: Mozpool, defect)

Description

[Dustin J. Mitchell [:dustin] (he/him)]

Bug 817762 seems to be back again, at least (thankfully) in staging.

We should monitor for this in production, too.

I think the easiest way will be for mozpool to touch a file every time it runs the timeout loop.  Then, nagios can check the age of that file.

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: bug838925.patch

Easy.  I'll add this config item via puppet, and then monitor it with nagios.

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: bug838925-puppet.patch

corresponding patch for puppet

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: infrapuppet.patch

And the change to add the monitoring in nagios.  check_file_age already exists on the imaging servers and in nrpe.cfg.

Comment 4

[Ashish Vijayaram [:ashish]]

Comment on attachment 711954 [details] [diff] [review]
infrapuppet.patch

Looks good!

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 711943 [details] [diff] [review]
bug838925.patch

D'oh! I just landed this one instead of the puppet patch.  I will back out if it's not OK.

Comment 6

[Mark Côté [:mcote]]

Comment on attachment 711943 [details] [diff] [review]
bug838925.patch

Looks good.

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

The puppetagain and mozpool patches are landed.  I'll land the infra puppet patch when the others are in production.

Comment 8

[Dustin J. Mitchell [:dustin] (he/him)]

infra puppet patch landed, although it had at least three bugs in it (wrong nagios server, wrong hostgroup, and in the wrong file)!