586723 - tm-amo01-webdev02 no longer replicating

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Wil Clouser [:clouserw]]

tm-amo01-webdev01 is more out of date than usual.  Fligtar says we should be using tm-amo01-webdev02 instead, but it's not replicating either.  I didn't know -01 was old news, but whichever one we are using, please make it work again. thx. :)

Comment 1

[Justin Scott [:fligtar]]

This has been broken since Tuesday night's database work, and all of the daily reports that are run off of it have failed. Can it please be fixed tomorrow morning?

Comment 2

[timellis]

I'm doing tm-amo01-webdev02, then.

Can one of you comment on the bug whether or not we need to do tm-amo01-webdev01? If it isn't necessary to rebuild it, I'd rather not.

Comment 3

[Justin Scott [:fligtar]]

We were told early this year (or last year?) to stop using webdev01 after it couldn't keep up because of performance problems. I haven't used it since then.

Comment 4

[timellis]

Got it. tm-amo01-webdev02 has been rebuilt and is currently up-to-date! Use it. Love it. Reopen this bug if either (1) you see problems with webdev02 or (2) you realise webdev01 needs also to be rebuilt.

Comment 5

[Justin Scott [:fligtar]]

Great, thanks!

Comment 6

[Wil Clouser [:clouserw]]

can we get rid of webdev01 if it's not going to have valid data?

Comment 7

[timellis]

I've sent Dave Miller a note about it. We may be reclaiming the machine as a spare, but it may have other responsibilities besides the database.

Comment 8

[Justin Scott [:fligtar]]

This stopped replicating again 2 days ago.

Comment 9

[Dave Miller [:justdave]]

I corrected the replication error and resumed it.  It's *WAY* behind, but making decent progress, so I'll let it catch up on its own instead of reloading it.  I expect it'll probably be caught up in 2 or 3 hours at the rate it's going.

Comment 10

[Dave Miller [:justdave]]

I just added a nagios check for this as well, so we'll get a better heads-up if this happens again.  Since it's not a production db server it'll only alert on IRC and won't actually page anyone, but with it alerting on IRC every half hour it's sure to still get attention eventually.