771923 - backup1.db.scl3.mozilla.com:MySQL DB Backups is CRITICAL

Status: RESOLVED FIXED

(Data & BI Services Team :: DB: MySQL, task)

Description

[Matt Pressman [:mpressman]]

[09:31:08] <nagios-scl3> [565] backup1.db.scl3.mozilla.com:MySQL DB Backups is CRITICAL: CRITICAL: b01 is not currently running, bedrock is not currently running, buildbot is not currently running, builder-addons is not currently running, developer1 is not currently running, dev.scl3 is not currently running, l10n is not currently running, mysql1.metrics is not currently running, stage is not currently running, getpersonas is not currently runn

Comment 1

[Matt Pressman [:mpressman]]

By the time of the page, only developer1 and buildbot were stopped in replication. Both had a corrupt binlog on the master:

# Warning: this binlog is either in use or was not closed properly.

I found the next log position and changed to that on the slave and restarted replication.

Comment 2

[Adrian J Fernandez [:Aj]]

(In reply to Matt Pressman [:mpressman] from comment #1)
> By the time of the page, only developer1 and buildbot were stopped in
> replication. Both had a corrupt binlog on the master:
> 
For reference, the slaves weren't running so I started them.
Then paged you for the below;

> # Warning: this binlog is either in use or was not closed properly.
> 
> I found the next log position and changed to that on the slave and restarted
> replication.