Closed
Bug 726733
Opened 13 years ago
Closed 13 years ago
Migrate quality.mozilla.org to genericrhel6 cluster in PHX1
Categories
(mozilla.org Graveyard :: Server Operations, task)
mozilla.org Graveyard
Server Operations
Tracking
(Not tracked)
VERIFIED
FIXED
People
(Reporter: nmaul, Assigned: bburton)
References
()
Details
(Whiteboard: [4/2 9AM])
+++ This bug was initially created as a clone of Bug #723591 +++
This is currently hosted on a single server in SJC1 (dm-qmo01), and should be migrated to the genericrhel6 cluster in PHX1. This should help with performance limitations of the current setup (1-core VM w/ 512MB RAM), as well as being far more redundant and better supported by IT.
Updated•13 years ago
|
Assignee: server-ops → bburton
Comment 1•13 years ago
|
||
When it's no longer used, please mark dm-qmo01 as spare as described in https://mana.mozilla.org/wiki/display/websites/Killing+sjc1+hosts
Comment 2•13 years ago
|
||
Brandon,
This is a fairly standard WP install on dm-qmo01, docs on Mana are up to date for the location of the install + the DBs etc.
Comment 3•13 years ago
|
||
Also, please co-ordinate with Craig on the actual move.
Assignee | ||
Comment 4•13 years ago
|
||
I am working on this this afternoon, will ping Craig when it's ready to test
Status: NEW → ASSIGNED
Assignee | ||
Comment 5•13 years ago
|
||
This is in progress, we'll need a new db in phx1, which I've filed bug 727649 to get, once that's in place, I'll update the wp-config.php settings and do some testing, then coordinate with devs to do some testing via /etc/hosts
Assignee | ||
Comment 6•13 years ago
|
||
This is ready to test via /etc/hosts, please add the following line and let me know how things look, the DB should be current as this morning and I did a git pull and configure the recaptcha stuff, like in current prod, so that should work as well
63.245.217.86 quality.mozilla.org
Let me know how it looks for you and we can coordinate a move/cutover for Tuesday
Assignee | ||
Updated•13 years ago
|
Assignee: bburton → craigcook.bugz
Comment 7•13 years ago
|
||
Craig, any update here? We'd like to get this wrapped up soon.
Comment 8•13 years ago
|
||
(In reply to Dustin J. Mitchell [:dustin] from comment #7)
> Craig, any update here? We'd like to get this wrapped up soon.
Sorry, I totally missed this. I'm not sure what it is you need me to do at this point? I added the above to my hosts file as in comment #6 and QMO seems to load just fine (still), but I think the move happened a month ago so that's not surprising. If the move already happened, the site has been trucking along just fine since then so I think we can call this resolved. If there's something still to do just let me know what it is.
Comment 9•13 years ago
|
||
If this is completely cut over, that's great! Otherwise, it sounds like we can make that change.
Assignee: craigcook.bugz → bburton
Assignee | ||
Comment 10•13 years ago
|
||
This cutover did not happen because I did not get positive confirmation that it looked good on the new IP and because we need to be able to schedule a window of 30 minutes where we can redirect the site to a maintenance page, while a DBA moves the database to the new server in PHX.
Based on your comment above, Craig, it sounds like QMO does look good, so the next step is to agree on a time to put up the maintenance page and perform the database and DNS migrations.
What's involved in scheduling the maintenance?
Reporter | ||
Comment 11•13 years ago
|
||
Why don't we simply disable DB writes to the old database (ex: remove all but SELECT privs, or disable commenting and refrain from making new posts), dump/import the DB in the new location, and change DNS? No hard downtime, and can probably get away with not announcing anything... just the QMO guys need to know about it. The new one will be writable right away, so as soon as DNS is changed the site would be back to normal.
Assignee | ||
Comment 12•13 years ago
|
||
That's fine with me. Craig, is this something we could plan for Monday morning?
Comment 13•13 years ago
|
||
As a point of order, Monday is sjc1 train C, and likely to be hectic - maybe Tuesday would be better?
Assignee | ||
Comment 14•13 years ago
|
||
I'm off Tuesday, if someone else wants to work with Sheeri to do the mysql bits (I'd talk to her about this previously) and do the DNS, great, otherwise I can do it first thing PST on Wed.
Comment 15•13 years ago
|
||
Wednesday AM sounds good to me, and I'd like it if Raymond could be available too (he's taken ownership of QMO since Al moved from QA to security). There's a big release on mozilla.org planned for Tuesday (if all goes well), so Wednesday may be a bit less hectic for everyone.
We can easily disable user registration and the forums during the move window and just tell the team not to post anything during that period. The site automatically syndicates external posts (a la Planet) and we can temporarily switch that off as well. How does that plan sound?
Assignee | ||
Comment 16•13 years ago
|
||
That sounds good. Let me know Raymond's availability on Wed and we'll make it happen.
Assignee | ||
Updated•13 years ago
|
Whiteboard: [3/28 9AM]
Assignee | ||
Comment 17•13 years ago
|
||
Do we still wanna try to get this done this morning?
Let me know
Comment 18•13 years ago
|
||
(In reply to Brandon Burton [:solarce] from comment #17)
> Do we still wanna try to get this done this morning?
>
> Let me know
Yes please do
Comment 19•13 years ago
|
||
Brandon: Ping me on IRC whenever you're ready, if you still want to do it today. Sorry for dropping the ball this morning. But it sounds like we can do this pretty much whenever so if we don't get to it today no worries, we'll reschedule, maybe for Friday.
Assignee | ||
Comment 20•13 years ago
|
||
I'd like to coordinate with one our DBAs to do this, so I'll ping you tomorrow AM PST and we'll go from there
Assignee | ||
Comment 21•13 years ago
|
||
Sorry, I got caught up in some stuff this morning and now it is 5PM, I'll plan for first thing Monday and reach out to folks tomorrow to nail down the details
Whiteboard: [3/28 9AM] → [4/2 9AM]
Assignee | ||
Comment 22•13 years ago
|
||
Database has been migrated. Old database user has been put to SELECT and EXECUTE only.
DNS has been updated and is resolving the new IP via mozilla DNS, google dns, and verizon dns.
Status: ASSIGNED → RESOLVED
Closed: 13 years ago
Resolution: --- → FIXED
Comment 23•13 years ago
|
||
verified fixed
host quality.mozilla.org
quality.mozilla.org has address 63.245.217.86
quality.mozilla.org mail is handled by 10 dm-mail01.mozilla.org.
quality.mozilla.org mail is handled by 10 dm-mail02.mozilla.org.
Status: RESOLVED → VERIFIED
Updated•10 years ago
|
Product: mozilla.org → mozilla.org Graveyard
You need to log in
before you can comment on or make changes to this bug.
Description
•