bm-l10n-db has no space left on the device

RESOLVED FIXED

Status

mozilla.org Graveyard
Server Operations
RESOLVED FIXED
7 years ago
3 years ago

People

(Reporter: gandalf, Assigned: justdave)

Tracking

Details

(Reporter)

Description

7 years ago
[root@bm-l10n-db var]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                       13G   12G   47M 100% /
/dev/sda1              99M   19M   76M  20% /boot
tmpfs                1014M     0 1014M   0% /dev/shm
(Reporter)

Comment 1

7 years ago
It seems that mysql-slow.log eats half of the drive.
Status: NEW → ASSIGNED
So this is something you can take care of?
(Reporter)

Comment 3

7 years ago
yeah.


Strategy:
 - stop collecting binary logs (we don't use them anyway)
 - archive the slow-log
 - investigate methods to avoid slow-log eating all available space

This should save us ~8GB.
Cool, assigning to you then, let me know if you need any help.
Assignee: server-ops → gandalf
(Reporter)

Comment 5

7 years ago
I reclaimed 3.6GB from mysql-bin logs and adjusted /etc/my.cnf not to build binary logs.
(Reporter)

Comment 6

7 years ago
done
 - 9G free space
 - slow query log will only log queries that take more than 5 seconds (comparing to 1 second used before)

hopefully this will let us focus on really slow queries that will appear there without filling disk space.
Status: ASSIGNED → RESOLVED
Last Resolved: 7 years ago
Resolution: --- → FIXED
(In reply to comment #3)
>  - stop collecting binary logs (we don't use them anyway)

You do use them.  You busted your backups.  Correct fix here is to set it to automatically discard them after they've been used (set expire-logs-days to 2 or something).  The backup server is going to have to be re-synced from scratch now, which will require downtime on the DB server.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
You need these:

log-bin=mysql-bin
expire_logs_days=2

Once that's up, let us know when we can re-sync it.
(Reporter)

Comment 9

7 years ago
oh, thnx. There was no info in the my.cnf doc about why it's there. I'll add it.

Comment 10

7 years ago
Also, you have slow log rotation in your /etc/logrotate.d/mysql, but apparently you're not pruning the slow logs themselves, which filled up your disk. I always create a cron job that does a "find /var/log/mysql/mysql-slow... -ctime +3 -exec rm -f \{\} \;" sort of thing to clear out the old logs.
Status: REOPENED → RESOLVED
Last Resolved: 7 years ago7 years ago
Resolution: --- → FIXED
(Reporter)

Comment 11

7 years ago
added

expire_logs_days=2

should be fixed now. Sorry for the trouble guys.
ok, when's a good time to take it down for the re-sync?  We'll need to clone the whole thing to the backup server to get the replication started again.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
ping?
Tempted to close this bug for lack of comments - will let it sit another week.

Updated

7 years ago
Status: REOPENED → RESOLVED
Last Resolved: 7 years ago7 years ago
Resolution: --- → INCOMPLETE
Just FYI, this means you have no database backups.
(Reporter)

Comment 16

7 years ago
Sorry guys, I was on holidays.

The only remaining thing is to do the re-sync, right?
Status: RESOLVED → REOPENED
Resolution: INCOMPLETE → ---

Updated

7 years ago
Assignee: gandalf → justdave
(In reply to comment #16)
> The only remaining thing is to do the re-sync, right?

Yep, just need a time from you when it's convenient to take it down for a re-sync.
(Reporter)

Comment 18

7 years ago
Ok. Suggested downtime is 9am PDT (in little less than 2 hours). Will it work for you?

I'm online in case you want to sync further.
Sorry, I was out yesterday, well past then by now.  Try again? :)  I'm going to be gone the first part of Saturday Eastern time as well, but I should be around in the late afternoon or evening, or we can try during the week sometime.
(Reporter)

Comment 20

7 years ago
Today, 10am PDT sounds good?
Assuming you mean PDT, sounds good to me.
Does 6am PDT work on Thursday the 2nd?  Build is having a maintenance window then, and it would be convenient to get this in at the same time.
(Reporter)

Comment 23

7 years ago
yes. that should work. Thanks!
This is done.
Status: REOPENED → RESOLVED
Last Resolved: 7 years ago7 years ago
Resolution: --- → FIXED
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.