Closed Bug 1182171 Opened 9 years ago Closed 9 years ago

New Relic alert: Fullest disk > 85% on 'th-stage-db{1,2}'

Categories

(Tree Management :: Treeherder: Infrastructure, defect, P1)

Product:
Tree Management
Component:
Treeherder: Infrastructure
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED DUPLICATE of bug 1179223

People

(Reporter: emorley, Assigned: emorley)

References

Details

Sigh, this time of the quarter again.

https://rpm.newrelic.com/accounts/677903/incidents/16293097

Investigating. (Though presume bug 1179223 plays a part)
Summary: Alert open: Fullest disk > 85% on 'th-stage-db{1,2}' → New Relic alert: Fullest disk > 85% on 'th-stage-db{1,2}'
We're seeing some timeouts during cycle-data:
https://rpm.newrelic.com/accounts/677903/applications/5585473/traced_errors?utf8=%E2%9C%93&search[name]=cycle-data&search[sort_by]=count&search[type]=all
(increase the time range to a few days, since it only runs once a day)

However most of the problem is just the binlogs (only 60GB taken up by the DB):

[emorley@treeherder2.stage.db.scl3 mysql]$ sudo du -ch | tail -n 1
340G    total

[emorley@treeherder2.stage.db.scl3 mysql]$ du -ch treeherder*-bin.* | tail -n 1
281G    total
Status: ASSIGNED → RESOLVED
Closed: 9 years ago
Resolution: --- → DUPLICATE
You need to log in before you can comment on or make changes to this bug.