627825 - review nagios alerts for builds-running, builds-pending

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P5)

Description

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

To reduce nagios spam, these alerts were intentionally over-relaxed until bug#625978 was fixed and, we have some idea of how quickly we can expect the files to now being posted. 

Once bug#625978, we should update this bug with the new nagios threshold we want, and kick this bug over to ServerOps. For now, filing to track, and leaving in RelEng.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

So bug 627821 (relaxing the checks) was WONTFIX so we may never have eased them.

Amy, what are the age thresholds for the these nagios checks on dm-wwwbuild01:
 http_age - build-4hr
 http_age - builds-pending
 http_age - builds-running

Comment 2

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to Nick Thomas [:nthomas] from comment #1)
> So bug 627821 (relaxing the checks) was WONTFIX so we may never have eased
> them.
> 
> Amy, what are the age thresholds for the these nagios checks on
> dm-wwwbuild01:
>  http_age - build-4hr
>  http_age - builds-pending
>  http_age - builds-running

arr: ping? what is the current threshold on these alerts?


ed: also, as sheriffs closed the trees because of these build json files being stale over the weekend (bug#926245), any opinions on what threshold you'd be looking for?

Comment 3

[Ed Morley [:emorley]]

(In reply to John O'Duinn [:joduinn] from comment #2)
> ed: also, as sheriffs closed the trees because of these build json files
> being stale over the weekend (bug#926245), any opinions on what threshold
> you'd be looking for?

I requested "http_age - build-4hr" be adjusted in bug 914686 to...
* check_interval: 300s
* file age threshold: 300s

...I'm presuming this affected both our email alert and the #releng IRC alert (arr: would be good to confirm they are linked?).

Something similar for builds-running and builds-pending would be ideal :-)

Comment 4

[Amy Rich [:arr] [:arich]]

The SRE team takes care of nagios, now, so tagging ashish to answer comment 1, since he'll probably be up soon.

Comment 5

[Ashish Vijayaram [:ashish]]

Given the age of the original request, these checks don't exist anymore, likely gone with dm-wwwbuild01:

> http_age - builds-pending
> http_age - builds-running

From what I gather, "http_age - builds-4hr" is now "http file age - /buildjson/builds-4hr.js.gz". I'm positive the other two checks were not lost in migration (to the current Nagios infrastructure).

I'll be glad to add these back at priority. Please let me know which server to check these on and other parameters - check_interval, thresholds, contact groups.

Comment 6

[Ashish Vijayaram [:ashish]]

Okay, in the general interest of keeping things monitored, I've gone ahead and added checks for builds-pending.js and builds-running.js. Same thresholds and config as for builds-4hr.js.gz:

Check interval: 300s
Check failures before alert: 3
File age threshold: 300s
URL: https://nagios.mozilla.org/releng-scl3/cgi-bin/status.cgi?navbarsearch=1&host=builddata.pub.build.mozilla.org

Please reopen this bug for any changes.