582183 - bm-l10n-db has no space left on the device

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Zibi Braniecki [:zbraniecki][:gandalf]]

[root@bm-l10n-db var]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                       13G   12G   47M 100% /
/dev/sda1              99M   19M   76M  20% /boot
tmpfs                1014M     0 1014M   0% /dev/shm

Comment 1

[Zibi Braniecki [:zbraniecki][:gandalf]]

It seems that mysql-slow.log eats half of the drive.

Comment 2

[Shyam Mani [:fox2mike]]

So this is something you can take care of?

Comment 3

[Zibi Braniecki [:zbraniecki][:gandalf]]

yeah.


Strategy:
 - stop collecting binary logs (we don't use them anyway)
 - archive the slow-log
 - investigate methods to avoid slow-log eating all available space

This should save us ~8GB.

Comment 4

[Shyam Mani [:fox2mike]]

Cool, assigning to you then, let me know if you need any help.

Comment 5

[Zibi Braniecki [:zbraniecki][:gandalf]]

I reclaimed 3.6GB from mysql-bin logs and adjusted /etc/my.cnf not to build binary logs.

Comment 6

[Zibi Braniecki [:zbraniecki][:gandalf]]

done
 - 9G free space
 - slow query log will only log queries that take more than 5 seconds (comparing to 1 second used before)

hopefully this will let us focus on really slow queries that will appear there without filling disk space.

Comment 7

[Dave Miller [:justdave]]

(In reply to comment #3)
>  - stop collecting binary logs (we don't use them anyway)

You do use them.  You busted your backups.  Correct fix here is to set it to automatically discard them after they've been used (set expire-logs-days to 2 or something).  The backup server is going to have to be re-synced from scratch now, which will require downtime on the DB server.

Comment 8

[Dave Miller [:justdave]]

You need these:

log-bin=mysql-bin
expire_logs_days=2

Once that's up, let us know when we can re-sync it.

Comment 9

[Zibi Braniecki [:zbraniecki][:gandalf]]

oh, thnx. There was no info in the my.cnf doc about why it's there. I'll add it.

Comment 10

[timellis]

Also, you have slow log rotation in your /etc/logrotate.d/mysql, but apparently you're not pruning the slow logs themselves, which filled up your disk. I always create a cron job that does a "find /var/log/mysql/mysql-slow... -ctime +3 -exec rm -f \{\} \;" sort of thing to clear out the old logs.

Comment 11

[Zibi Braniecki [:zbraniecki][:gandalf]]

added

expire_logs_days=2

should be fixed now. Sorry for the trouble guys.

Comment 12

[Dave Miller [:justdave]]

ok, when's a good time to take it down for the re-sync?  We'll need to clone the whole thing to the backup server to get the replication started again.

Comment 13

[matthew zeier [:mrz]]

ping?

Comment 14

[matthew zeier [:mrz]]

Tempted to close this bug for lack of comments - will let it sit another week.

Comment 15

[Dave Miller [:justdave]]

Just FYI, this means you have no database backups.

Comment 16

[Zibi Braniecki [:zbraniecki][:gandalf]]

Sorry guys, I was on holidays.

The only remaining thing is to do the re-sync, right?

Comment 17

[Dave Miller [:justdave]]

(In reply to comment #16)
> The only remaining thing is to do the re-sync, right?

Yep, just need a time from you when it's convenient to take it down for a re-sync.

Comment 18

[Zibi Braniecki [:zbraniecki][:gandalf]]

Ok. Suggested downtime is 9am PDT (in little less than 2 hours). Will it work for you?

I'm online in case you want to sync further.

Comment 19

[Dave Miller [:justdave]]

Sorry, I was out yesterday, well past then by now.  Try again? :)  I'm going to be gone the first part of Saturday Eastern time as well, but I should be around in the late afternoon or evening, or we can try during the week sometime.

Comment 20

[Zibi Braniecki [:zbraniecki][:gandalf]]

Today, 10am PDT sounds good?

Comment 21

[Dave Miller [:justdave]]

Assuming you mean PDT, sounds good to me.

Comment 22

[Dave Miller [:justdave]]

Does 6am PDT work on Thursday the 2nd?  Build is having a maintenance window then, and it would be convenient to get this in at the same time.

Comment 23

[Zibi Braniecki [:zbraniecki][:gandalf]]

yes. that should work. Thanks!

Comment 24

[Dave Miller [:justdave]]

This is done.