1328258 - [Nagios] Add queue size alerts on pushapkworker

Status: RESOLVED WONTFIX

(Release Engineering :: Release Automation: Other, defect, P2)

Description

[Johan Lorenzo [:jlorenzo]]

Like in bug 1314840, we'd like some alerts on the queue size for pushapk-scriptworker (another instance of scriptworker). In our case 2 tasks in the queue should trigger an alert. The worker usually handles 1 task per day.

The other check described in 1314840 (the about file age) is not needed. We don't persist anything from a run to another.

For the record, first nagios checks are added in bug 1321513.

Comment 1

[Chris AtLee [:catlee]]

Did this get done?

Comment 2

[Johan Lorenzo [:jlorenzo]]

Sorry for the response delay, I wasn't sure of the current state of the alerts.

Thanks to Simon's work in [1] and the fact that pushapk_scriptworker uses the scriptworker puppet module[2], having alerts on pushapkworker-* is just a matter of enabling them on Nagios. Simon has paved the way for signing-linux-* in [3]. I'll wait until bug 1332640 lands, so the patch should be fairly trivial.

By the way, if that helps testing alerts, we can first set the number of pending jobs to 1|2. Now pushapk_scriptworker handles aurora and beta and is sometimes triggered by the date branch as well. Hence, alerts at 2, should be easy to trigger.

[1] https://hg.mozilla.org/build/puppet/pushloghtml?changeset=57dedcf28c4b
[2] bug 1329944
[3] attachment 8828846 [details] (bug 1332640)

Comment 3

[Johan Lorenzo [:jlorenzo]]

pushapkworker has been migrated to GCP since then and we don't use nagios to monitor queue sizes. Closing bug.