Closed Bug 1394555 Opened 6 years ago Closed 6 years ago

prod-ro database filled up, but prod seemed ok

Categories

(Tree Management :: Treeherder: Infrastructure, enhancement, P1)

enhancement

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: camd, Assigned: fubar)

References

Details

Attachments

(1 file)

Free storage space went down to near zero.  Unclear what filled it up at this time.
Looks like a black swan event
It's leveled off at 250gb free. 

However, prod-ro is currently showing zero connections (and nothing appears to have gone sideways last night). Do we still need this instance?

Leaving NI on myself to set up some cloudwatch alarms.
Assignee: nobody → klibby
Flags: needinfo?(klibby)
Flags: needinfo?(emorley)
The prod-ro instance is being used - by redash and activedata. Depending on whether they are connected at the time there are not always connections present.

I'd been tracking the disk space consumption on prod and trying to decide whether we should bump or else increase efforts to stop ingesting certain types of data (eg bug 1342296), given bumping the disk size is irreversible.

Perhaps for the prod-ro instance there were complex adhoc redash queries running (or stuck) at the time, and so temporary tables for them consumed enough of the space to make it run out unlike RW prod?
Flags: needinfo?(emorley)
Blocks: 1215587
Depends on: 1397712
Priority: -- → P1
This is fine now that the disk allowances were increased in bug 1397712.
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
Flags: needinfo?(klibby)
You need to log in before you can comment on or make changes to this bug.