978956 - Improve nagios alerts for pending job backlog

Status: RESOLVED DUPLICATE of bug 1220191

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Ryan VanderMeulen [:RyanVM]]

Right now, we only receive alerts based on the number of pending jobs being too high. However, this misses a critical time component when the issue isn't the number of builds pending but the duration of time they've been pending for. Note that this is about pending builds, not tests.

Right now, the only process we have in place for reporting builds falling behind is "sheriffs happen to notice when not running in only unstarred mode on TBPL that AWS builds have been pending for an hour, close the trees, and proceed to scream in #releng about it" (this just happened). We've seen other instances in the last week of AWS builds falling 15-20min behind across the board, even during low load times of day (in one case, it was legitimate bustage from a RelEng change).

Builds falling behind by 30+ minutes, especially during high-load parts of the day, is a tree-closing event because the bustage pile-up consequences can be worse if not. Therefore, we need better ways of monitoring for these pending build delays so they can be discovered in a more orderly fashion and investigated before things get out of control.

Comment 1

[Ryan VanderMeulen [:RyanVM]]

Seriously, I can recount multiple recent instances where having this alert would have prevented issues from blowing up into much worse ones.

Comment 2

[Ryan VanderMeulen [:RyanVM]]

Adjusting the summary to make it clear that this applies to builds and tests both.