501077 - Move dm-mailman01 to a physical hardware server

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Reed Loden [:reed]]

dm-mailman01 has been a VM for a long time... it's our only mail-related box that isn't a real machine. I think we should move it to a blade, as it's having a hard time keeping up lately with its load, and it just seems like it should be a real machine anyway. A better CPU and more RAM would both be good help for it.

Comment 1

[Dave Miller [:justdave]]

We could give it one of the old DL360 G4s, too...  a blade might be a waste of resources/overkill.  The thing to watch out for here is disk space.  The mailing list archives for @mozilla.org lists are on the main drive on this machine, and we've had to grow the hard drive twice in the last 2 years to make room for their continued growth.  If we put this on physical hardware, it'd probably be a good idea to move the mailman archives to networked storage so it can still be grown easily in the future.

Comment 2

[matthew zeier [:mrz]]

chizu, can you get this lined up for Thursday's window?

Comment 3

[Dave Miller [:justdave]]

If we're going to make tonight's window, we need an NFS or iSCSI share in place ASAP to have time to rsync the content over before the window.

Comment 4

[matthew zeier [:mrz]]

shyam, something you can look at?

Comment 5

[Shyam Mani [:fox2mike]]

Sure, are we targetting this thursday?

Comment 6

[matthew zeier [:mrz]]

If possible, sure.

Comment 7

[matthew zeier [:mrz]]

The 360 in the URL would be a good host for this but there don't appear to be any drives in there.  

Needs an onsite visit.

Comment 8

[Phong Tran [:phong]]

That server should have a pair of 72 GB drives.  They just need to be inserted in all the way.

Comment 9

[Zandr Milewski [:zandr]]

Will swing by on the way in.

Comment 10

[Zandr Milewski [:zandr]]

Drives are seated, chassis is connected but not powered on. Fire at will.

Comment 11

[matthew zeier [:mrz]]

dm-mailman02 is up - where's the nfs (or iscsi?) volume you made for this?

Comment 12

[matthew zeier [:mrz]]

Started an rsync from dm-mailmail01 -> dm-mailman02 (/usr/lib/mailman & /var/lib/mailman).

Comment 13

[matthew zeier [:mrz]]

I'm not sure  how you guys want to best schedule this.  I've copied over all the data but it takes a long time to get the file list to pick up deltas.

Comment 14

[Dave Miller [:justdave]]

Only "clean" way to do it is going to be to take an extended outage window.  And if "du -sh" took 8 hours to calculate how much disk space it was using, I'm sure rsync will take at least that long to figure out what's changed.  If we disable postfix on the box, the mail will get queued.  It'll be offline long enough for people to get 4 hour delay notifications though, so it'll best be announced.

Comment 15

[Dave Miller [:justdave]]

This is probably the kind of thing that would be good to do on a Sunday afternoon or a holiday.  I'll quite likely be around on Monday, most of my holiday activity is planned for Saturday and Sunday, if you'd like to schedule this for late morning/early afternoon on Monday.

Comment 16

[matthew zeier [:mrz]]

Timed rsync:

real    344m36.094s
user    3m29.479s
sys     5m4.212s

About 6 hours.  Dave, you want to tackle this on Sunday/Monday?  I'll send something out if you'll be able to handle this and do the actual switch.

Comment 17

[Dave Miller [:justdave]]

Yeah, late morning Monday is probably good.  I won't be around much on Sunday (during the day at least).  10am pacific is probably a good time for it.  Schedule 8 just to be safe.

Comment 18

[Dave Miller [:justdave]]

all done.

There was a MESS of stuff we forgot about that I discovered as I was moving things...

- multiple local user accounts (complete with accompanying cron jobs) for autoresponder scripts running procmail
- configuration for htdig
- nrpe config and check scripts
- lots of perl modules for majordomo (which are all available via rpmforge these days) that had been installed via CPAN on the old box, and thus didn't show up in the package list.
- apache configuration
- cgi scripts and web apps related to the bugzilla.org website

I'm *pretty* sure we got everything.  I even copied over the ssh host keys just to be safe, since I ended up swapping the IPs on the two servers in lieu of having to screw with DNS and firewall.