1023128 - DB Disk - All on treeherder2.db.scl3.mozilla.com is WARNING: DISK WARNING - free space: / 3630 MB (10% inode=97%):

Status: RESOLVED DUPLICATE of bug 1023060

(Data & BI Services Team :: DB: MySQL, task)

Description

[MOC Nagios API]

Automated alert report from nagios1.private.scl3.mozilla.com:

Hostname: treeherder2.db.scl3.mozilla.com
Service:  DB Disk - All
State:    WARNING
Output:   DISK WARNING - free space: / 3630 MB (10% inode=97%):

Runbook:  http://m.allizom.org/DB+Disk+-+All

Comment 1

[Ryan Watson [:w0ts0n]]

Just paged for critical

treeherder2.db.scl3.mozilla.com:DB Disk - All is CRITICAL: DISK CRITICAL - free space: / 2023 MB (5% inode=97%):

Comment 2

[Ryan Watson [:w0ts0n]]

Moved the log file from /var/log/mysql/mysql.err to /data/log/mysql/other/mysql.err to buy us some time.

Comment 3

[Sheeri Cabral [:sheeri]]

There's a lot of "unsafe statement written to binary log" here. The solution is to change treeherder to use binlog_format=MIXED. 

There are no owners at https://mana.mozilla.org/wiki/display/websites/treeherder.mozilla.org?src=search#treeherder.mozilla.org-Hardware nor are there any dev/stage db's listed (I tried the dev and stage clusters in scl3, they don't have any treeherder db's).

We should set binlog_format=MIXED on dev/stage first, and if that is fine, change production. But we need to know who to contact...

Comment 4

[Sheeri Cabral [:sheeri]]

See also bug 1023060....

Comment 5

[Sheeri Cabral [:sheeri]]

I've removed /data/log/mysql/other/mysql.err