1394555 - prod-ro database filled up, but prod seemed ok

Status: RESOLVED FIXED

(Tree Management :: Treeherder: Infrastructure, enhancement, P1)

Description

[Cameron Dawson [:camd]]

Free storage space went down to near zero.  Unclear what filled it up at this time.

Comment 1

[Jake Watkins [:dividehex]]

Attachment: Screenshot 2017-08-28 13.59.01 (2).png

Looks like a black swan event

Comment 2

[Kendall Libby [:fubar] (he/him)]

It's leveled off at 250gb free. 

However, prod-ro is currently showing zero connections (and nothing appears to have gone sideways last night). Do we still need this instance?

Leaving NI on myself to set up some cloudwatch alarms.

Comment 3

[Ed Morley [:emorley]]

The prod-ro instance is being used - by redash and activedata. Depending on whether they are connected at the time there are not always connections present.

I'd been tracking the disk space consumption on prod and trying to decide whether we should bump or else increase efforts to stop ingesting certain types of data (eg bug 1342296), given bumping the disk size is irreversible.

Perhaps for the prod-ro instance there were complex adhoc redash queries running (or stuck) at the time, and so temporary tables for them consumed enough of the space to make it run out unlike RW prod?

Comment 4

[Ed Morley [:emorley]]

This is fine now that the disk allowances were increased in bug 1397712.