589181 - amo01 database backups are failing

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Wil Clouser [:clouserw]]

All the AMO database dumps on khan are empty:

/data/backup-drop/mrdb04/mysql/addons_remora

Comment 1

[Dave Miller [:justdave]]

They're supposed to be.  They moved to cm-webdev01-master01 about a year ago.

Comment 2

[Dave Miller [:justdave]]

and they'd be under amo01 not mrdb04 :)  but otherwise the same pathname over there.

Comment 3

[Jorge Villalobos [:jorgev] (he/him)]

Actually, this bug was filed because the remora DB at cm-webdev01-master01 doesn't have any data from this week, it's out of sync. I mentioned this to Wil and he filed this bug.
So, I guess the cause for the bug is incorrect, but the actual bug is still there.

Comment 4

[Wil Clouser [:clouserw]]

Yeah, just the wrong host.  The db dumps on cm-webdev01-master01 were missing

Comment 5

[Dave Miller [:justdave]]

OK, looking into it.

Comment 6

[Dave Miller [:justdave]]

They look there to me...

cm-webdev01-master01:/data/backup-drop/amo01/mysql/addons_remora/

Comment 7

[Wil Clouser [:clouserw]]

They are empty

Comment 8

[Dave Miller [:justdave]]

ooohh...  communication here... you have the files, but the files are empty.  gotcha.  I'm also seeing that, didn't look to see that they had data in them, I just looked that they were there.  These are directly pulled from the backups, this also means we have no backups of this specific database because the backups are busted :|

Comment 9

[Dave Miller [:justdave]]

Trying to run the backup by hand gives me this:

mysqldump: Got error: 1146: Table 'addons_remora.stats_firefoxcup' doesn't exist when using LOCK TABLES

looks like the backup replicant is corrupted, I'll queue up a re-sync from one of the other slaves.

Comment 10

[Dave Miller [:justdave]]

According to the mail the cron job has been sending, it's been broken since August 12.  Yay for noisy cron job output mailing lists that you can never find anything in :/  (jabba is working on fixing that)

Comment 11

[Dave Miller [:justdave]]

I cloned slave03 to the backup server, and it still had the same error.  I suspect slave03 is where the backup server was last cloned from and this wasn't caught before.  Re-cloned from slave02, and that one appears to be NOT corrupted so far (yay).  Production backup is running now, I'll push back to slave03 to replace the corrupted copy there after the backup completes.

Comment 12

[Dave Miller [:justdave]]

Production backup successfully completed.  It's rsyncing out both to dracula (for push to webdev) and to slave03 (to reload it since it was the source of the corruption) currently.  You guys should have a replaced copy of today's backup with the full data sometime in the next 15 minutes or so.

Comment 13

[Dave Miller [:justdave]]

ok, slave03 restore completed, I also verified the new backup did make it to cm-webdev01-master01, you guys should be good to go.

Comment 14

[Wil Clouser [:clouserw]]

Looks good.  I think nagios is making sure the files are there - maybe we should adjust it to make sure the filesize is larger than 500M.