938263 - Failover git.mozilla.org to phx1 and back to scl3

Status: RESOLVED FIXED

(Developer Services :: General, task)

Description

[Kendall Libby [:fubar] (he/him)]

Tracking bug - Fail git.m.o over to PHX1 for duration of SCL3 outage, and then back again.

Comment 1

[Hal Wine [:hwine] use NI!]

I assume the fail over setup will be r/o during tree closure. If not, I'm interested in how replication and fail over occurs, as we have a number of processes that automate writing.

Comment 2

[Kendall Libby [:fubar] (he/him)]

I'm waiting to hear back from :fox2mike about whether or not PHX1 will be RW or RO. The plan thus far is:

- global RO on git.m.o and git1.dmz.phx1
- final rsync to phx1 (a couple minutes, I think; will have real numbers today)
- flip DNS (which, unfortunately, will take ~10 minutes)
- enable RW on git1.dmz.phx1

Rollback will be the same method. Of note, rsync --delete is use to ensure any files removed during failover are removed back in SCL3.

Comment 3

[Kendall Libby [:fubar] (he/him)]

git.m.o failed over to PHX1 and returned to SCL3.