Closed Bug 767617 Opened 13 years ago Closed 13 years ago

puppetdashboard[1-2].db.phx1 change binlog management

Categories

(Data & BI Services Team :: DB: MySQL, task)

Product:
Data & BI Services Team
Component:
DB: MySQL
Platform:
x86
macOS
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: mpressman, Assigned: mpressman)

Details

the my.cnf has the setting as: expire_logs_days=10 This is probably just a fat finger ;)
Assignee: server-ops-database → mpressman
checked the logs and not a fat finger, but intentionally set for servers that do not have expire_logs_days set. Does this affect any other hosts than puppetdashboard[1-2]? We'll want to specifically set those to 1.
1 day of purge isn't enough to keep from triggering nagios disk alerts hitting 90% capacity. After an alert this morning, I removed all binlogs with the exception of the current days logs and we've hit the 10% free threshold. This could get worse with more data added unless we address the > 20GB of binlogs in a single day.
Summary: puppetdashboard1.db.phx1 my.cnf expire_logs_days should be set to 1 → puppetdashboard[1-2].db.phx1 change binlog management
[10:02:50] <nagios-phx1> [111] puppetdashboard2.db.phx1.mozilla.com:DB Disk - All is WARNING: DISK WARNING - free space: / 11654 MB (9% inode=98%):
[15:02:33] <nagios-phx1> [125] puppetdashboard1.db.phx1.mozilla.com:DB Disk - All is WARNING: DISK WARNING - free space: / 12826 MB (10% inode=98%):
[10:29:55] <nagios-phx1> [179] puppetdashboard2.db.phx1.mozilla.com:DB Disk - All is WARNING: DISK WARNING - free space: / 12748 MB (10% inode=98%):
puppetdashboard2 has only 11GB of binlogs - we are growing the data at a clip which must either be reigned in, or get more disk.
the size of the database is taking up 86GB [root@puppetdashboard2.db.phx1 mysql]# du -sh puppet_dashboard/ 86G puppet_dashboard/
interestingly the size of the puppet_dashboard db on puppetdashboard1 is only 56GB [root@puppetdashboard1.db.phx1 mysql]# du -sh puppet_dashboard/ 56G puppet_dashboard/
I have defrag'd the backup, it's now 34G, and am rebuilding puppetdashboard2 from it.
puppetdashboard2 is rebuilt.
sheeri, thanks for handling to rebuild - this looks mucho better!
Status: NEW → RESOLVED
Closed: 13 years ago
Resolution: --- → FIXED
Product: mozilla.org → Data & BI Services Team
You need to log in before you can comment on or make changes to this bug.