Closed Bug 609956 Opened 14 years ago Closed 14 years ago

Migrate pulse.mozilla.org to Phoenix

Categories

(mozilla.org Graveyard :: Server Operations, task)

Product:
mozilla.org Graveyard
Component:
Server Operations
Platform:
x86
All
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: bc, Unassigned)

Details

In order to support increased work loads, we would like to reconfigure pulse.mozilla.org to have:

2    CPU
4G   RAM
150G disk

A thin disk would be great if at all possible. Note that I will be freeing up quite a bit of storage in the various datastores (d-sata-build-00{1,2,3}, d-fcal-build-002{1,2} used by the qa cluster in bug 609727.

fox2mike: If you would like me remove the vms on the qa cluster prior to reconfiguring pulse, let me know. Let us know when you would like to do this so we can send out a notice.
Assignee: server-ops → shyam
(In reply to comment #0)
> In order to support increased work loads, we would like to reconfigure
> pulse.mozilla.org to have:
> 
> 2    CPU
> 4G   RAM
> 150G disk

Just a note, VMWare officially doesn't recommend we change the processors once the OS is installed. FWIW, I've done that before and we've not had issues. In case we do run into wonky issues *after* the upgrade, do let me know :)
 
> A thin disk would be great if at all possible. Note that I will be freeing up
> quite a bit of storage in the various datastores (d-sata-build-00{1,2,3},
> d-fcal-build-002{1,2} used by the qa cluster in bug 609727.

Thin disk?

> fox2mike: If you would like me remove the vms on the qa cluster prior to
> reconfiguring pulse, let me know. Let us know when you would like to do this so
> we can send out a notice.

How does Tuesday 9pm sound? Downtime of maybe bout 30-60 mins. VM will have to be rebooted multiple times (for the disk) and I'll take the opportunity to upgrade the kernel and system software while I'm at it.
Whiteboard: 11/09/2010 @ 9pm
Flags: needs-downtime+
(In reply to comment #1)
> 
> How does Tuesday 9pm sound? Downtime of maybe bout 30-60 mins. VM will have to
> be rebooted multiple times (for the disk) and I'll take the opportunity to
> upgrade the kernel and system software while I'm at it.

Works fine. There's no rush on the downtime as there aren't many users, I just want to make sure to post to the mailing list as a heads up.
This VM should move to Phoenix.

@legnitto - how hard is it to install your app from scratch?  Would be easier to make a fresh VM and then just switch DNS vs. copying a multi MB VM around.
Shouldn't be a big deal to install from scratch
Great, a new VM allows some flexibility. A couple of observations:

This should *not* go on the qp-vmware01 host.
I think 64bit Linux is the way to go.
A thin disk would be nice as well.

Christian: anything else ?
Great - building a fresh 64bit host.
Flags: needs-downtime+ → needs-downtime-
Summary: Reconfigure pulse.mozilla.org/dm-pulse01.mozilla.org vm → Migrate pulse.mozilla.org to Phoenix
Whiteboard: 11/09/2010 @ 9pm
dp-pulse01 / 10.8.74.40 is online.  Needs to be shoved into puppet so LegNeato can login.
(In reply to comment #5)
> Christian: anything else ?

Nope, I think that's good.
From irc convo, 

/ is 15GB.  Need an NFS mount on /data such that it can grow as needed.
Can we make / 50gb and still have the nfs mount? That will give us the most flexibility. I need to see the impact of NFS on message throughput with durable queues, and having 50gb locally would let pulse work fine while investigating that. Once NFS is found sufficient we could dial it back to 15gb
Nothing's actually "local" - the whole datastore is on NFS.  You'd have the same throughput regardless.
Ah, good to know.
Is this new VM ready so that I can start setting it up? If not, can it become ready this week?
(In reply to comment #13)
> Is this new VM ready so that I can start setting it up? If not, can it become
> ready this week?

I'll have it ready before the end of my day.
Key is on dp-pulse01, as soon as the access bug 612491 is solved, you can go ahead and setup what's needed.

You have only the / 15GB for now, I'll setup NFS when Aravind is around in the morning.
/data/pulse with bout 64GB on it for now and owned by the apache user is now mounted on dm-pulse01.
Christian, punt back to me when this is ready to go live and I'll make DNS changes etc and shut out the old VM. Also, please let us know if you need firewall holes, I'll file bugs with netops for that too. I remember it needed some holes when I set it up the last time around.
Assignee: shyam → clegnitto
Whiteboard: Waiting on Setup
Wrote bug 614029 for the firewall bits
Setup is done, sweet. Still a couple more holes that need be opened up in bug 614029, but nothing that would prevent us from switching over.

Can you give me a good time it works for you to switch over? I want to notify people in advance (we don't need > 1 day notice).
Whiteboard: Waiting on Setup
Shoot for Tuesday's maintenance window?
Flags: needs-downtime- → needs-downtime+
Whiteboard: 12/14/2010
Sounds good. Tuesday it is.
Oooh, I thought by Tuesday you meant this Tuesday. If it's going to be on the 14th I'd rather do it sooner. Does tomorrow work? I've notified the list it will be tomorrow so it could happen anytime that day. Apologies for me not double-checking the dates
Whiteboard: 12/14/2010
I did meant 12/07.  Any blockers to changing dns tonight?
Assignee: clegnitto → server-ops
Christine, we can move this any time this week - let me know when works for you.
Anytime is fine, the sooner the better.
Do you need to be around for this?  It's just a matter of changing DNS right now.
Nope, I do not need to be around for this.
This is done and DNS has been migrated.  pulse is now running from PHX.
Status: NEW → RESOLVED
Closed: 14 years ago
Resolution: --- → FIXED
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.