Closed Bug 726733 Opened 13 years ago Closed 13 years ago

Migrate quality.mozilla.org to genericrhel6 cluster in PHX1

Categories

(mozilla.org Graveyard :: Server Operations, task)

Product:
mozilla.org Graveyard
Component:
Server Operations
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
VERIFIED FIXED

People

(Reporter: nmaul, Assigned: bburton)

References

(
URL
)

Details

(Whiteboard: [4/2 9AM])

+++ This bug was initially created as a clone of Bug #723591 +++ This is currently hosted on a single server in SJC1 (dm-qmo01), and should be migrated to the genericrhel6 cluster in PHX1. This should help with performance limitations of the current setup (1-core VM w/ 512MB RAM), as well as being far more redundant and better supported by IT.
Blocks: 726739
Assignee: server-ops → bburton
When it's no longer used, please mark dm-qmo01 as spare as described in https://mana.mozilla.org/wiki/display/websites/Killing+sjc1+hosts
Brandon, This is a fairly standard WP install on dm-qmo01, docs on Mana are up to date for the location of the install + the DBs etc.
Also, please co-ordinate with Craig on the actual move.
I am working on this this afternoon, will ping Craig when it's ready to test
Status: NEW → ASSIGNED
This is in progress, we'll need a new db in phx1, which I've filed bug 727649 to get, once that's in place, I'll update the wp-config.php settings and do some testing, then coordinate with devs to do some testing via /etc/hosts
This is ready to test via /etc/hosts, please add the following line and let me know how things look, the DB should be current as this morning and I did a git pull and configure the recaptcha stuff, like in current prod, so that should work as well 63.245.217.86 quality.mozilla.org Let me know how it looks for you and we can coordinate a move/cutover for Tuesday
Assignee: bburton → craigcook.bugz
Craig, any update here? We'd like to get this wrapped up soon.
(In reply to Dustin J. Mitchell [:dustin] from comment #7) > Craig, any update here? We'd like to get this wrapped up soon. Sorry, I totally missed this. I'm not sure what it is you need me to do at this point? I added the above to my hosts file as in comment #6 and QMO seems to load just fine (still), but I think the move happened a month ago so that's not surprising. If the move already happened, the site has been trucking along just fine since then so I think we can call this resolved. If there's something still to do just let me know what it is.
If this is completely cut over, that's great! Otherwise, it sounds like we can make that change.
Assignee: craigcook.bugz → bburton
This cutover did not happen because I did not get positive confirmation that it looked good on the new IP and because we need to be able to schedule a window of 30 minutes where we can redirect the site to a maintenance page, while a DBA moves the database to the new server in PHX. Based on your comment above, Craig, it sounds like QMO does look good, so the next step is to agree on a time to put up the maintenance page and perform the database and DNS migrations. What's involved in scheduling the maintenance?
Why don't we simply disable DB writes to the old database (ex: remove all but SELECT privs, or disable commenting and refrain from making new posts), dump/import the DB in the new location, and change DNS? No hard downtime, and can probably get away with not announcing anything... just the QMO guys need to know about it. The new one will be writable right away, so as soon as DNS is changed the site would be back to normal.
That's fine with me. Craig, is this something we could plan for Monday morning?
As a point of order, Monday is sjc1 train C, and likely to be hectic - maybe Tuesday would be better?
I'm off Tuesday, if someone else wants to work with Sheeri to do the mysql bits (I'd talk to her about this previously) and do the DNS, great, otherwise I can do it first thing PST on Wed.
Wednesday AM sounds good to me, and I'd like it if Raymond could be available too (he's taken ownership of QMO since Al moved from QA to security). There's a big release on mozilla.org planned for Tuesday (if all goes well), so Wednesday may be a bit less hectic for everyone. We can easily disable user registration and the forums during the move window and just tell the team not to post anything during that period. The site automatically syndicates external posts (a la Planet) and we can temporarily switch that off as well. How does that plan sound?
That sounds good. Let me know Raymond's availability on Wed and we'll make it happen.
Whiteboard: [3/28 9AM]
Do we still wanna try to get this done this morning? Let me know
(In reply to Brandon Burton [:solarce] from comment #17) > Do we still wanna try to get this done this morning? > > Let me know Yes please do
Brandon: Ping me on IRC whenever you're ready, if you still want to do it today. Sorry for dropping the ball this morning. But it sounds like we can do this pretty much whenever so if we don't get to it today no worries, we'll reschedule, maybe for Friday.
I'd like to coordinate with one our DBAs to do this, so I'll ping you tomorrow AM PST and we'll go from there
Sorry, I got caught up in some stuff this morning and now it is 5PM, I'll plan for first thing Monday and reach out to folks tomorrow to nail down the details
Whiteboard: [3/28 9AM] → [4/2 9AM]
Blocks: 741489
Database has been migrated. Old database user has been put to SELECT and EXECUTE only. DNS has been updated and is resolving the new IP via mozilla DNS, google dns, and verizon dns.
Status: ASSIGNED → RESOLVED
Closed: 13 years ago
Resolution: --- → FIXED
verified fixed host quality.mozilla.org quality.mozilla.org has address 63.245.217.86 quality.mozilla.org mail is handled by 10 dm-mail01.mozilla.org. quality.mozilla.org mail is handled by 10 dm-mail02.mozilla.org.
Status: RESOLVED → VERIFIED
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.