1028545 - backup1.db.corp filling up because of objectcache table in db

Status: RESOLVED FIXED

(Data & BI Services Team :: DB: MySQL, task)

Description

[Brandon Johnson [:cyborgshadow]]

backup1.db.corp has been alerting disk usage since thursday. I've been clearing it because of backups, but I found the root cause this morning.

The mysql objectcache table has grown from 2.8 GB to 4.3 GB in the last 3 days.

It's a 3 column table with an expdate column, 480k rows of mediumblob, and 479k of those 480k are set to expire in 2038. This is simply bad practice.

We'll need to contact whoever made changes to that recently and get them to undo it.

For now, I'm wiping all the cache from this table that expires in 2038 in order to save the disk from crashing.

Comment 1

[Brandon Johnson [:cyborgshadow]]

make that 4.79 Million rows expiring in 2038, out of 4.8 million rows total. I missed a number apparently, but caught it when I was removing rows by the 100,000 at a time and was able to do this more than 5 times.

Comment 2

[Brandon Johnson [:cyborgshadow]]

This is an ongoing issue blocked by webops bug 1028547. 

It will recur every 5 days until fixed. We could (if webops can't contain it) throw in a cron script to run once a day to do the following as a workaround:
- delete from objectcache where exptime > '2038-01-01' LIMIT 1000000;
- REPAIR TABLE objectcache;

Comment 3

[Brandon Johnson [:cyborgshadow]]

Cylia wanted to do the prod change for 1028547 on Monday. Ran a full purge to make it through the weekend.

Comment 4

[Brandon Johnson [:cyborgshadow]]

Cyliang pushed the change today. I did a purge and tested.


Fixed, resolved, purged, and verified. :)