1261710 - Nagios check for Backlog Age should ignore Win10 testing

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Nick Thomas [:nthomas] (UTC+12)]

Win64 testing isn't ready yet, but RyanVM is pushing to try to work on it. No problem with that, but there's only 1 slave (out of the two for testing) and it takes many hours to clear the backlog out. In the meantime nagios is bleating like this:
 Sun 13:50:46 PDT [4458] nagios1.private.releng.scl3.mozilla.com:Backlog Age is CRITICAL: CRITICAL Backlog Age: 24h:09m:25s

So, can we somehow exclude jobs names that begin with 'Windows 10 64-bit try' until we have a full-size test pool ?

Comment 1

[Ryan VanderMeulen [:RyanVM]]

(In reply to Nick Thomas [:nthomas] from comment #0)
> Win64 testing isn't ready yet, but RyanVM is pushing to try to work on it.

To be clear, Windows 10 testing is the issue here, not Win64 :)

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Add support to ignore prefixes

Comment 3

[Chris AtLee [:catlee]]

Comment on attachment 8737945 [details] [diff] [review]
Add support to ignore prefixes

Review of attachment 8737945 [details] [diff] [review]:
-----------------------------------------------------------------

just a minor nit to fix up. looks good otherwise!

::: nagios-related/check_backlog_age.py
@@ +35,2 @@
>  # Get the earliest 'submitted_at' value
> +def get_min_submitted_at(ignore_prefixes):

This argument name shadows the global variable name above, which is a bit confusing.

Can you either rename the global (e.g. IGNORE_PREFIXES), or have this function simply use the global variable?

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8737945 [details] [diff] [review]
Add support to ignore prefixes

Landed with fix to use ignore_prefixes as a global:
https://hg.mozilla.org/build/braindump/rev/0a3b9bec2572

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

ashish, how do we deploy this change to nagios1.private.releng.scl3.mozilla.com ?

Comment 6

[Ashish Vijayaram [:ashish]]

Passing on ni? to :ryanc, who's point for Nagios now.

Comment 7

[Ryan C [:ryanc] (UTC-4)]

Hey Nick,

You'd commit this in Puppet via,

modules/nrpe/files/plugins/check_backlog_age

Let me know if you need further assistance with this.

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

ryanc, I don't have access to that part of SVN, would you mind landing it for me ? Attachment 8737945 [details] [diff] is the diff but note that the file path and name differ.

Comment 9

[Ryan C [:ryanc] (UTC-4)]

Nick,

Committed in r116593. Still looks to function the same,

17:26:42 <nagios-releng> ryanc: nagios1.private.releng.scl3.mozilla.com:Backlog Age is WARNING - WARNING Backlog Age: 11h:22m:59s, Ignored: 6 Last Checked: 2016-04-07 17:26:11 PDT

Let me know if you should need anything else.

Comment 10

[Nick Thomas [:nthomas] (UTC+12)]

Thanks Ryan, the "Ignore: <N>" part is new so all is well.