[Nagios] Add queue size alerts on pushapkworker

NEW
Unassigned

Status

P2
normal
2 years ago
3 months ago

People

(Reporter: jlorenzo, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

2 years ago
Like in bug 1314840, we'd like some alerts on the queue size for pushapk-scriptworker (another instance of scriptworker). In our case 2 tasks in the queue should trigger an alert. The worker usually handles 1 task per day.

The other check described in 1314840 (the about file age) is not needed. We don't persist anything from a run to another.

For the record, first nagios checks are added in bug 1321513.
Did this get done?
Flags: needinfo?(jlorenzo)
(Reporter)

Comment 2

a year ago
Sorry for the response delay, I wasn't sure of the current state of the alerts.

Thanks to Simon's work in [1] and the fact that pushapk_scriptworker uses the scriptworker puppet module[2], having alerts on pushapkworker-* is just a matter of enabling them on Nagios. Simon has paved the way for signing-linux-* in [3]. I'll wait until bug 1332640 lands, so the patch should be fairly trivial.

By the way, if that helps testing alerts, we can first set the number of pending jobs to 1|2. Now pushapk_scriptworker handles aurora and beta and is sometimes triggered by the date branch as well. Hence, alerts at 2, should be easy to trigger.

[1] https://hg.mozilla.org/build/puppet/pushloghtml?changeset=57dedcf28c4b
[2] bug 1329944
[3] attachment 8828846 [details] (bug 1332640)
Depends on: 1332640
Flags: needinfo?(jlorenzo)
Priority: -- → P2
(Assignee)

Updated

3 months ago
Component: General Automation → General
Product: Release Engineering → Release Engineering
Component: General → Release Automation: Pushapk
QA Contact: catlee → jlorenzo
You need to log in before you can comment on or make changes to this bug.