726733 - Migrate quality.mozilla.org to genericrhel6 cluster in PHX1

Status: VERIFIED FIXED

Description

[Jake Maul [:jakem]]

+++ This bug was initially created as a clone of Bug #723591 +++

This is currently hosted on a single server in SJC1 (dm-qmo01), and should be migrated to the genericrhel6 cluster in PHX1. This should help with performance limitations of the current setup (1-core VM w/ 512MB RAM), as well as being far more redundant and better supported by IT.

Comment 1

[Dustin J. Mitchell [:dustin] pronoun: he]

When it's no longer used, please mark dm-qmo01 as spare as described in https://mana.mozilla.org/wiki/display/websites/Killing+sjc1+hosts

Comment 2

[Shyam Mani [:fox2mike]]

Brandon,

This is a fairly standard WP install on dm-qmo01, docs on Mana are up to date for the location of the install + the DBs etc.

Comment 3

[Shyam Mani [:fox2mike]]

Also, please co-ordinate with Craig on the actual move.

Comment 4

[Brandon Burton [:solarce]]

I am working on this this afternoon, will ping Craig when it's ready to test

Comment 5

[Brandon Burton [:solarce]]

This is in progress, we'll need a new db in phx1, which I've filed bug 727649 to get, once that's in place, I'll update the wp-config.php settings and do some testing, then coordinate with devs to do some testing via /etc/hosts

Comment 6

[Brandon Burton [:solarce]]

This is ready to test via /etc/hosts, please add the following line and let me know how things look, the DB should be current as this morning and I did a git pull and configure the recaptcha stuff, like in current prod, so that should work as well

63.245.217.86     quality.mozilla.org

Let me know how it looks for you and we can coordinate a move/cutover for Tuesday

Comment 7

[Dustin J. Mitchell [:dustin] pronoun: he]

Craig, any update here?  We'd like to get this wrapped up soon.

Comment 8

[Craig Cook (:craigcook)]

(In reply to Dustin J. Mitchell [:dustin] from comment #7)
> Craig, any update here?  We'd like to get this wrapped up soon.

Sorry, I totally missed this. I'm not sure what it is you need me to do at this point? I added the above to my hosts file as in comment #6 and QMO seems to load just fine (still), but I think the move happened a month ago so that's not surprising. If the move already happened, the site has been trucking along just fine since then so I think we can call this resolved. If there's something still to do just let me know what it is.

Comment 9

[Dustin J. Mitchell [:dustin] pronoun: he]

If this is completely cut over, that's great!  Otherwise, it sounds like we can make that change.

Comment 10

[Brandon Burton [:solarce]]

This cutover did not happen because I did not get positive confirmation that it looked good on the new IP and because we need to be able to schedule a window of 30 minutes where we can redirect the site to a maintenance page, while a DBA moves the database to the new server in PHX.

Based on your comment above, Craig, it sounds like QMO does look good, so the next step is to agree on a time to put up the maintenance page and perform the database and DNS migrations.

What's involved in scheduling the maintenance?

Comment 11

[Jake Maul [:jakem]]

Why don't we simply disable DB writes to the old database (ex: remove all but SELECT privs, or disable commenting and refrain from making new posts), dump/import the DB in the new location, and change DNS? No hard downtime, and can probably get away with not announcing anything... just the QMO guys need to know about it. The new one will be writable right away, so as soon as DNS is changed the site would be back to normal.

Comment 12

[Brandon Burton [:solarce]]

That's fine with me. Craig, is this something we could plan for Monday morning?

Comment 13

[Dustin J. Mitchell [:dustin] pronoun: he]

As a point of order, Monday is sjc1 train C, and likely to be hectic - maybe Tuesday would be better?

Comment 14

[Brandon Burton [:solarce]]

I'm off Tuesday, if someone else wants to work with Sheeri to do the mysql bits (I'd talk to her about this previously) and do the DNS, great, otherwise I can do it first thing PST on Wed.

Comment 15

[Craig Cook (:craigcook)]

Wednesday AM sounds good to me, and I'd like it if Raymond could be available too (he's taken ownership of QMO since Al moved from QA to security). There's a big release on mozilla.org planned for Tuesday (if all goes well), so Wednesday may be a bit less hectic for everyone.

We can easily disable user registration and the forums during the move window and just tell the team not to post anything during that period. The site automatically syndicates external posts (a la Planet) and we can temporarily switch that off as well. How does that plan sound?

Comment 16

[Brandon Burton [:solarce]]

That sounds good. Let me know Raymond's availability on Wed and we'll make it happen.

Comment 17

[Brandon Burton [:solarce]]

Do we still wanna try to get this done this morning?

Let me know

Comment 18

[raymond [:retornam] (inactive)]

(In reply to Brandon Burton [:solarce] from comment #17)
> Do we still wanna try to get this done this morning?
> 
> Let me know

Yes please do

Comment 19

[Craig Cook (:craigcook)]

Brandon: Ping me on IRC whenever you're ready, if you still want to do it today. Sorry for dropping the ball this morning. But it sounds like we can do this pretty much whenever so if we don't get to it today no worries, we'll reschedule, maybe for Friday.

Comment 20

[Brandon Burton [:solarce]]

I'd like to coordinate with one our DBAs to do this, so I'll ping you tomorrow AM PST and we'll go from there

Comment 21

[Brandon Burton [:solarce]]

Sorry, I got caught up in some stuff this morning and now it is 5PM, I'll plan for first thing Monday and reach out to folks tomorrow to nail down the details

Comment 22

[Brandon Burton [:solarce]]

Database has been migrated. Old database user has been put to SELECT and EXECUTE only.

DNS has been updated and is resolving the new IP via mozilla DNS, google dns, and verizon dns.

Comment 23

[raymond [:retornam] (inactive)]

verified fixed 

host quality.mozilla.org
quality.mozilla.org has address 63.245.217.86
quality.mozilla.org mail is handled by 10 dm-mail01.mozilla.org.
quality.mozilla.org mail is handled by 10 dm-mail02.mozilla.org.