Migrate quality.mozilla.org to genericrhel6 cluster in PHX1

VERIFIED FIXED

Status

mozilla.org Graveyard
Server Operations
VERIFIED FIXED
6 years ago
3 years ago

People

(Reporter: jakem, Assigned: solarce)

Tracking

Details

(Whiteboard: [4/2 9AM], URL)

(Reporter)

Description

6 years ago
+++ This bug was initially created as a clone of Bug #723591 +++

This is currently hosted on a single server in SJC1 (dm-qmo01), and should be migrated to the genericrhel6 cluster in PHX1. This should help with performance limitations of the current setup (1-core VM w/ 512MB RAM), as well as being far more redundant and better supported by IT.
(Reporter)

Updated

6 years ago
Blocks: 726739
Blocks: 716047
Assignee: server-ops → bburton
When it's no longer used, please mark dm-qmo01 as spare as described in https://mana.mozilla.org/wiki/display/websites/Killing+sjc1+hosts
Brandon,

This is a fairly standard WP install on dm-qmo01, docs on Mana are up to date for the location of the install + the DBs etc.
Also, please co-ordinate with Craig on the actual move.
(Assignee)

Comment 4

6 years ago
I am working on this this afternoon, will ping Craig when it's ready to test
Status: NEW → ASSIGNED
(Assignee)

Comment 5

6 years ago
This is in progress, we'll need a new db in phx1, which I've filed bug 727649 to get, once that's in place, I'll update the wp-config.php settings and do some testing, then coordinate with devs to do some testing via /etc/hosts
(Assignee)

Comment 6

6 years ago
This is ready to test via /etc/hosts, please add the following line and let me know how things look, the DB should be current as this morning and I did a git pull and configure the recaptcha stuff, like in current prod, so that should work as well

63.245.217.86     quality.mozilla.org

Let me know how it looks for you and we can coordinate a move/cutover for Tuesday
(Assignee)

Updated

6 years ago
Assignee: bburton → craigcook.bugz
Craig, any update here?  We'd like to get this wrapped up soon.
(In reply to Dustin J. Mitchell [:dustin] from comment #7)
> Craig, any update here?  We'd like to get this wrapped up soon.

Sorry, I totally missed this. I'm not sure what it is you need me to do at this point? I added the above to my hosts file as in comment #6 and QMO seems to load just fine (still), but I think the move happened a month ago so that's not surprising. If the move already happened, the site has been trucking along just fine since then so I think we can call this resolved. If there's something still to do just let me know what it is.
If this is completely cut over, that's great!  Otherwise, it sounds like we can make that change.
Assignee: craigcook.bugz → bburton
(Assignee)

Comment 10

6 years ago
This cutover did not happen because I did not get positive confirmation that it looked good on the new IP and because we need to be able to schedule a window of 30 minutes where we can redirect the site to a maintenance page, while a DBA moves the database to the new server in PHX.

Based on your comment above, Craig, it sounds like QMO does look good, so the next step is to agree on a time to put up the maintenance page and perform the database and DNS migrations.

What's involved in scheduling the maintenance?
(Reporter)

Comment 11

6 years ago
Why don't we simply disable DB writes to the old database (ex: remove all but SELECT privs, or disable commenting and refrain from making new posts), dump/import the DB in the new location, and change DNS? No hard downtime, and can probably get away with not announcing anything... just the QMO guys need to know about it. The new one will be writable right away, so as soon as DNS is changed the site would be back to normal.
(Assignee)

Comment 12

6 years ago
That's fine with me. Craig, is this something we could plan for Monday morning?
As a point of order, Monday is sjc1 train C, and likely to be hectic - maybe Tuesday would be better?
(Assignee)

Comment 14

6 years ago
I'm off Tuesday, if someone else wants to work with Sheeri to do the mysql bits (I'd talk to her about this previously) and do the DNS, great, otherwise I can do it first thing PST on Wed.
Wednesday AM sounds good to me, and I'd like it if Raymond could be available too (he's taken ownership of QMO since Al moved from QA to security). There's a big release on mozilla.org planned for Tuesday (if all goes well), so Wednesday may be a bit less hectic for everyone.

We can easily disable user registration and the forums during the move window and just tell the team not to post anything during that period. The site automatically syndicates external posts (a la Planet) and we can temporarily switch that off as well. How does that plan sound?
(Assignee)

Comment 16

6 years ago
That sounds good. Let me know Raymond's availability on Wed and we'll make it happen.
(Assignee)

Updated

6 years ago
Whiteboard: [3/28 9AM]
(Assignee)

Comment 17

6 years ago
Do we still wanna try to get this done this morning?

Let me know
(In reply to Brandon Burton [:solarce] from comment #17)
> Do we still wanna try to get this done this morning?
> 
> Let me know

Yes please do
Brandon: Ping me on IRC whenever you're ready, if you still want to do it today. Sorry for dropping the ball this morning. But it sounds like we can do this pretty much whenever so if we don't get to it today no worries, we'll reschedule, maybe for Friday.
(Assignee)

Comment 20

6 years ago
I'd like to coordinate with one our DBAs to do this, so I'll ping you tomorrow AM PST and we'll go from there
(Assignee)

Comment 21

6 years ago
Sorry, I got caught up in some stuff this morning and now it is 5PM, I'll plan for first thing Monday and reach out to folks tomorrow to nail down the details
Whiteboard: [3/28 9AM] → [4/2 9AM]
Blocks: 741489
(Assignee)

Comment 22

6 years ago
Database has been migrated. Old database user has been put to SELECT and EXECUTE only.

DNS has been updated and is resolving the new IP via mozilla DNS, google dns, and verizon dns.
Status: ASSIGNED → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → FIXED
verified fixed 

host quality.mozilla.org
quality.mozilla.org has address 63.245.217.86
quality.mozilla.org mail is handled by 10 dm-mail01.mozilla.org.
quality.mozilla.org mail is handled by 10 dm-mail02.mozilla.org.
Status: RESOLVED → VERIFIED
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.