New Relic alert: Fullest disk > 85% on 'th-stage-db{1,2}'

RESOLVED DUPLICATE of bug 1179223

Status

P1
normal
RESOLVED DUPLICATE of bug 1179223
4 years ago
4 years ago

People

(Reporter: emorley, Assigned: emorley)

Tracking

Details

Sigh, this time of the quarter again.

https://rpm.newrelic.com/accounts/677903/incidents/16293097

Investigating. (Though presume bug 1179223 plays a part)
Summary: Alert open: Fullest disk > 85% on 'th-stage-db{1,2}' → New Relic alert: Fullest disk > 85% on 'th-stage-db{1,2}'
We're seeing some timeouts during cycle-data:
https://rpm.newrelic.com/accounts/677903/applications/5585473/traced_errors?utf8=%E2%9C%93&search[name]=cycle-data&search[sort_by]=count&search[type]=all
(increase the time range to a few days, since it only runs once a day)

However most of the problem is just the binlogs (only 60GB taken up by the DB):

[emorley@treeherder2.stage.db.scl3 mysql]$ sudo du -ch | tail -n 1
340G    total

[emorley@treeherder2.stage.db.scl3 mysql]$ du -ch treeherder*-bin.* | tail -n 1
281G    total
Status: ASSIGNED → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → DUPLICATE
Duplicate of bug: 1179223
You need to log in before you can comment on or make changes to this bug.