759957 - intranet2.db.phx1 needs to be reseeded from intranet1.db.phx1

Status: RESOLVED FIXED

(Data & BI Services Team :: DB: MySQL, task)

Description

[Matt Pressman [:mpressman]]

intranet2.db.phx1 was blocked in replication running an optimize on puppet_dashboard.resource_statuses for around 7 hours. Either an over-aggressive logrotate removed the logs necessary, or more likely some config change and a related daemon restart on intranet1.db.phx1 caused it's binlog naming to be changed and intranet2 no longer has state for the necessary binlogs and needs to be reseeded.

Comment 1

[Matt Pressman [:mpressman]]

intranet2.db.phx1 has been reseeded and is now replicating.

Comment 2

[Sheeri Cabral [:sheeri]]

For the same reason, the backup for intranet has to be re-seeded. In progress now:

on the backup server, killed the mysql instance for intranet, deleted the old db (but kept the conf and the master.info file) and then:

cd /data/intranet
nc -l 9999 | tar xfi -

and on intranet:

[[e-mail]1 ~]# time innobackupex --stream=tar /var/lib/mysql | nc backup2.db.phx1.mozilla.com 9999

Comment 3

[Sheeri Cabral [:sheeri]]

That took 77 minutes, now I did this on the backup server:

innobackupex --apply-log . --ibbackup xtrabackup_51


And applied the logs. I'm going to chown -R mysql:mysql . and get replication going again.

Comment 4

[Sheeri Cabral [:sheeri]]

backup slave is replicating; closing.