intranet2.db.phx1 needs to be reseeded from intranet1.db.phx1

RESOLVED FIXED

Status

RESOLVED FIXED
7 years ago
4 years ago

People

(Reporter: mpressman, Assigned: mpressman)

Tracking

Details

(Assignee)

Description

7 years ago
intranet2.db.phx1 was blocked in replication running an optimize on puppet_dashboard.resource_statuses for around 7 hours. Either an over-aggressive logrotate removed the logs necessary, or more likely some config change and a related daemon restart on intranet1.db.phx1 caused it's binlog naming to be changed and intranet2 no longer has state for the necessary binlogs and needs to be reseeded.
(Assignee)

Updated

7 years ago
Assignee: server-ops-database → mpressman
(Assignee)

Comment 1

7 years ago
intranet2.db.phx1 has been reseeded and is now replicating.
Status: NEW → RESOLVED
Last Resolved: 7 years ago
Resolution: --- → FIXED
For the same reason, the backup for intranet has to be re-seeded. In progress now:

on the backup server, killed the mysql instance for intranet, deleted the old db (but kept the conf and the master.info file) and then:

cd /data/intranet
nc -l 9999 | tar xfi -

and on intranet:

[root@intranet1.db.phx1 ~]# time innobackupex --stream=tar /var/lib/mysql | nc backup2.db.phx1.mozilla.com 9999
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
That took 77 minutes, now I did this on the backup server:

innobackupex --apply-log . --ibbackup xtrabackup_51


And applied the logs. I'm going to chown -R mysql:mysql . and get replication going again.
backup slave is replicating; closing.
Status: REOPENED → RESOLVED
Last Resolved: 7 years ago7 years ago
Resolution: --- → FIXED
Product: mozilla.org → Data & BI Services Team
You need to log in before you can comment on or make changes to this bug.