627821 - Relax nagios checks on dm-wwwbuild01 file age

Status: RESOLVED WONTFIX

(Infrastructure & Operations :: RelOps: General, task)

Description

[Zandr Milewski [:zandr]]

These checks are very noisy these days, since the files aren't always getting built and uploaded in time.

bug 625978 is to fix the root cause.

In the mean time, we should relax the age check to 15 minutes to reduce nagios chatter.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

Did this already happen ?

Comment 2

[Rob Tucker [:rtucker]]

These service checks are commented out:
#http_age&contact_groups build::dm-wwwbuild01:build.mozilla.org:/builds/builds-running.js!7m
#http_age&contact_groups build::dm-wwwbuild01:build.mozilla.org:/builds/builds-pending.js!7m
#http_age&contact_groups build::dm-wwwbuild01:build.mozilla.org:/builds/builds-4hr.js.gz!7m

Would you like one of them to be enabled and set to a 15 minute threshold?

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

https://nagios.mozilla.org/nagios/cgi-bin/status.cgi?navbarsearch=1&host=dm-wwwbuild01
has other ideas, more than one set of definitions ?

Comment 4

[Rob Tucker [:rtucker]]

I found them. They are hardcoded into the services.cfg file as opposed to how we usually do these in a generated way. Do you want me to change all of them to a different value? If so what value?

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

We haven't had much flapping recently, but it's somewhat dependent on our VM and the DB server in ways we don't yet understand. And load on the buildbot cluster has been low while we're all frozen for the 4.0 releases, so we'll have to see what happens as everything cranks back up.

Which is a long way of saying it depends what value they all have at the moment. Could you look it up ?

Comment 6

[Rob Tucker [:rtucker]]

They are currently set to 7 minutes.

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

Lets leave them at 7 minutes. They haven't been flapping recently, and anyway we want this information to be timely. Plus, we know there's a leak somewhere in the buildapi code which provides the files we're monitoring, and the sooner we know that's slowing things down the better.