1273005 - Nagios alerts says 'Backlog Age is UNKNOWN: min() arg is an empty sequence' when nothing pending

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Nick Thomas [:nthomas] (UTC+12)]

Looks like it happens at 
  https://hg.mozilla.org/build/braindump/file/default/nagios-related/check_backlog_age.py#l47

BTW, it would be good to check the change made at bug 1248589 comment #4 is also in braindump, to keep the code in sync with the private repo for IT puppet.

Comment 1

[Alin Selagea [:aselagea]]

Attachment: bug_1273005.patch

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8753898 [details] [diff] [review]
bug_1273005.patch

>diff -r 2a36e340a0b2 nagios-related/check_backlog_age.py
>--- a/nagios-related/check_backlog_age.py	Fri May 13 16:15:18 2016 -0400
>+++ b/nagios-related/check_backlog_age.py	Wed May 18 18:48:18 2016 +0300
>@@ -1,4 +1,4 @@
>-#!/usr/bin/env python
>+#!/usr/bin/python

env is generally preferred, was was there some reason for this change ? 

>     response = urllib2.urlopen(pending_url)

We could add timeout here while we're modifying this file, eg by passing timeout=30 to urlopen.

>+    if len(result['pending']) == 0:
>+        submitted_at.append(get_unix_time())

We can end up with a status of 'OK Backlog Age: -1h:59m:59s, Ignored: 0' if it takes more than 0.5 seconds to retrieve builds-pending.js. I suggest using unix_time instead of calling get_unix_time() again.

Fix those up in a new patch and I'll put an r+ on it.

Comment 3

[Alin Selagea [:aselagea]]

Attachment: bug_1273005.patch

Updated the patch.

Comment 4

[Alin Selagea [:aselagea]]

The check was also updated, marking this as resolved.