If you think a bug might affect users in the 57 release, please set the correct tracking and status flags for Release Management.

Establish new Socorro Collectors disk usage alerts

RESOLVED FIXED

Status

mozilla.org Graveyard
Server Operations
RESOLVED FIXED
5 years ago
3 years ago

People

(Reporter: lonnen, Assigned: ericz)

Tracking

Details

(Reporter)

Description

5 years ago
Socorro collectors should alert at 30% disk usage. This should give us a buffer to investigate what is causing collectors to fill up.
(Reporter)

Comment 1

5 years ago
Alerts should report to #socorro-alerts, and cron-socorro@m.c
Disk alerts for Collectors already alert sooner than the typical 10%/5% for the rest of the infra and do alert in #socorro-alerts and email cron-socorro@m.c.
(Reporter)

Comment 3

5 years ago
How much earlier? We'd like to give ourselves more lead time if they're filling up.
(Assignee)

Comment 4

5 years ago
It will currently warn at 15% free space left and alert as critical with 11% left.  We can bump those up if you'd like, just let us know what values to put in.
(Reporter)

Comment 5

5 years ago
70% free space should do it. That may sound aggressive, but it should be enough to avoid false positives and give us a long lead time to diagnose and fix the issue.
Assignee: server-ops → eziegenhorn
(Assignee)

Comment 6

5 years ago
As per :ashish, I need to make check_disk_early take arguments in order to support this.
Status: NEW → ASSIGNED
(Assignee)

Comment 7

5 years ago
check_disk_all_early modified and set to warn at 65% full and alert critical at 70% full for the socorro processors.
Status: ASSIGNED → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → FIXED
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.