674974 - [bouncer] Daily run of sentry must have (partially) failed

Status: RESOLVED DUPLICATE of bug 716440

(mozilla.org Graveyard :: Server Operations, task)

Description

[Nick Thomas [:nthomas] (UTC+12)]

These two bugs are examples where we had bogus locations were enabled in bouncer, for products where Check Now was not set:
 https://bugzilla.mozilla.org/show_bug.cgi?id=670110
 https://bugzilla.mozilla.org/show_bug.cgi?id=668682#c6

Please check what is happening with the sentry run which runs once a day.

To flesh out the 2nd case:
Was:
Product 				OS 			Available 
Thunderbird-3.1.10-Complete		win			5000	

10 mins after turning on Check Now:
Product 				OS 			Available 
Thunderbird-3.1.10-Complete		osx			75
Thunderbird-3.1.10-Complete		linux			75
Thunderbird-3.1.10-Complete		win			75
Thunderbird-3.1.10-Complete		solaris-sparc		75
Thunderbird-3.1.10-Complete		solaris-i386		75
Thunderbird-3.1.10-Complete		opensolaris-sparc	75
Thunderbird-3.1.10-Complete		opensolaris-i386	75

So not only did we not have all platforms, we have a mirror which didn't actually have the win files.

Comment 1

[Dave Miller [:justdave]]

So here's the current theory on this:

A while back we added a timeout to the sentry script so that if it took over 5 or 6 minutes to check a host (cumulative, across all of the files being checked), it failed out the host assuming it was too slow.

There was an exemption to this timeout added to the non-checknow version of the script which runs overnight.  The current theory is that this exemption is apparently not working correctly.

I haven't gotten a chance to debug it any yet, adding the details in case someone else gets to it before I do.

Comment 2

[Dave Miller [:justdave]]

FWIW, this will be a code issue in the sentry scripts, which are in version control, so anyone could probably poke at it.

Comment 3

[Corey Shields [:cshields]]

Is this still an issue?  what's the action needed to close this out?

Adding a couple of our coders to cc in case this still needs attention.  Please point out where "in version control" they are.  thx!

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

The current issues with sentry's daily run are in bug 716440.