Closed
Bug 1394555
Opened 6 years ago
Closed 6 years ago
prod-ro database filled up, but prod seemed ok
Categories
(Tree Management :: Treeherder: Infrastructure, enhancement, P1)
Tree Management
Treeherder: Infrastructure
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: camd, Assigned: fubar)
References
Details
Attachments
(1 file)
28.76 KB,
image/png
|
Details |
Free storage space went down to near zero. Unclear what filled it up at this time.
Comment 1•6 years ago
|
||
Looks like a black swan event
Assignee | ||
Comment 2•6 years ago
|
||
It's leveled off at 250gb free. However, prod-ro is currently showing zero connections (and nothing appears to have gone sideways last night). Do we still need this instance? Leaving NI on myself to set up some cloudwatch alarms.
Assignee: nobody → klibby
Flags: needinfo?(klibby)
Flags: needinfo?(emorley)
Comment 3•6 years ago
|
||
The prod-ro instance is being used - by redash and activedata. Depending on whether they are connected at the time there are not always connections present. I'd been tracking the disk space consumption on prod and trying to decide whether we should bump or else increase efforts to stop ingesting certain types of data (eg bug 1342296), given bumping the disk size is irreversible. Perhaps for the prod-ro instance there were complex adhoc redash queries running (or stuck) at the time, and so temporary tables for them consumed enough of the space to make it run out unlike RW prod?
Flags: needinfo?(emorley)
Updated•6 years ago
|
Priority: -- → P1
Comment 4•6 years ago
|
||
This is fine now that the disk allowances were increased in bug 1397712.
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
![]() |
||
Updated•3 years ago
|
Flags: needinfo?(klibby)
You need to log in
before you can comment on or make changes to this bug.
Description
•