If you think a bug might affect users in the 57 release, please set the correct tracking and status flags for Release Management.

Move dm-mailman01 to a physical hardware server

RESOLVED FIXED

Status

mozilla.org Graveyard
Server Operations
RESOLVED FIXED
8 years ago
3 years ago

People

(Reporter: reed, Assigned: justdave)

Tracking

Details

(URL)

(Reporter)

Description

8 years ago
dm-mailman01 has been a VM for a long time... it's our only mail-related box that isn't a real machine. I think we should move it to a blade, as it's having a hard time keeping up lately with its load, and it just seems like it should be a real machine anyway. A better CPU and more RAM would both be good help for it.
We could give it one of the old DL360 G4s, too...  a blade might be a waste of resources/overkill.  The thing to watch out for here is disk space.  The mailing list archives for @mozilla.org lists are on the main drive on this machine, and we've had to grow the hard drive twice in the last 2 years to make room for their continued growth.  If we put this on physical hardware, it'd probably be a good idea to move the mailman archives to networked storage so it can still be grown easily in the future.

Updated

8 years ago
Depends on: 501133
(Reporter)

Updated

8 years ago
Assignee: server-ops → justdave

Updated

8 years ago
Summary: Move dm-mailman01 to a blade → Move dm-mailman01 to a server

Updated

8 years ago
Assignee: justdave → thardcastle

Comment 2

8 years ago
chizu, can you get this lined up for Thursday's window?
If we're going to make tonight's window, we need an NFS or iSCSI share in place ASAP to have time to rsync the content over before the window.

Comment 4

8 years ago
shyam, something you can look at?
Sure, are we targetting this thursday?

Comment 6

8 years ago
If possible, sure.

Comment 7

8 years ago
The 360 in the URL would be a good host for this but there don't appear to be any drives in there.  

Needs an onsite visit.
Flags: colo-trip+

Comment 8

8 years ago
That server should have a pair of 72 GB drives.  They just need to be inserted in all the way.
Will swing by on the way in.
Assignee: thardcastle → zandr
Status: NEW → ASSIGNED
Drives are seated, chassis is connected but not powered on. Fire at will.
Group: mozilla-confidential

Updated

8 years ago
Assignee: zandr → mrz
dm-mailman02 is up - where's the nfs (or iscsi?) volume you made for this?
Assignee: mrz → thardcastle
Started an rsync from dm-mailmail01 -> dm-mailman02 (/usr/lib/mailman & /var/lib/mailman).
I'm not sure  how you guys want to best schedule this.  I've copied over all the data but it takes a long time to get the file list to pick up deltas.
Only "clean" way to do it is going to be to take an extended outage window.  And if "du -sh" took 8 hours to calculate how much disk space it was using, I'm sure rsync will take at least that long to figure out what's changed.  If we disable postfix on the box, the mail will get queued.  It'll be offline long enough for people to get 4 hour delay notifications though, so it'll best be announced.
This is probably the kind of thing that would be good to do on a Sunday afternoon or a holiday.  I'll quite likely be around on Monday, most of my holiday activity is planned for Saturday and Sunday, if you'd like to schedule this for late morning/early afternoon on Monday.
Timed rsync:

real    344m36.094s
user    3m29.479s
sys     5m4.212s

About 6 hours.  Dave, you want to tackle this on Sunday/Monday?  I'll send something out if you'll be able to handle this and do the actual switch.
Yeah, late morning Monday is probably good.  I won't be around much on Sunday (during the day at least).  10am pacific is probably a good time for it.  Schedule 8 just to be safe.
all done.

There was a MESS of stuff we forgot about that I discovered as I was moving things...

- multiple local user accounts (complete with accompanying cron jobs) for autoresponder scripts running procmail
- configuration for htdig
- nrpe config and check scripts
- lots of perl modules for majordomo (which are all available via rpmforge these days) that had been installed via CPAN on the old box, and thus didn't show up in the package list.
- apache configuration
- cgi scripts and web apps related to the bugzilla.org website

I'm *pretty* sure we got everything.  I even copied over the ssh host keys just to be safe, since I ended up swapping the IPs on the two servers in lieu of having to screw with DNS and firewall.
Assignee: thardcastle → justdave
Status: ASSIGNED → RESOLVED
Last Resolved: 8 years ago
Resolution: --- → FIXED
Group: mozilla-confidential, infra
No longer depends on: 501133
Summary: Move dm-mailman01 to a server → Move dm-mailman01 to a physical hardware server
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.