726806 - migrate mailman hosts out of sjc1

Status: RESOLVED FIXED

(Infrastructure & Operations :: Infrastructure: Other, task)

Description

[Dustin J. Mitchell [:dustin] (he/him)]

Three hosts are doing mailman stuff in scj1:

    dm-mailman01
    mailman1
    notorious

The last plan I heard of was to move these, with some downtime, to scl3.

Comment 1

[Justin Dow [:jabba]]

I'd vote to move these to HCI as mailman1, mailman2 and mailman3.mail.corp.phx1.mozilla.com.

Comment 2

[Rick Bryce [:rbryce]]

(In reply to Justin Dow [:jabba] from comment #1)
> I'd vote to move these to HCI as mailman1, mailman2 and
> mailman3.mail.corp.phx1.mozilla.com.

I agree.  Before we do any moving of mail servers.  We need to identify the public IP addresses mail will be sent from. If the IPs change we need to get public DNS setup before hand, at least 2 weeks prior.  Our volume, combined with an instant DNS change will cause new IPs to be blacklisted.

Comment 3

[Dave Miller [:justdave]]

SPF records for the domains may need to be updated.  If they don't exist we should probably create them, specifically listing the new IPs.  That'll probably help them validate.

   Domain name        Currently on               Target host
   =================  =========================  =================================
1) mozilla.org        dm-mailman01 (mailman)     mailman1.mail.corp.phx1
2) lists.mozilla.org  notorious                  mailman2.mail.corp.phx1
3) mozilla.com        mailman1.dmz.sjc1          mailman3.mail.corp.phx1
4) bugzilla.org       dm-mailman01 (majordomo2)  mailman4.mail.corp.phx1 (mailman)

Normally I'd say #1 and #2 should be physical hosts and #3 and #4 can be VMs.  But given how fast our VMs are in the HCI infra, they may all do just fine on VMs.

The majordomo2 install on dm-mailman01 is not really maintained, and we've had security bugs found in it.  So we should migrate bugzilla.org off of that onto mailman.  We've been meaning to for a long time, but there are namespace collisions with mozilla.org that made it impossible while it was on the same box because of the way mailman handles virtual domains (list names are global across all domains, you just get to pick which domain each list associates with).

Comment 4

[Justin Dow [:jabba]]

(In reply to Rick Bryce [:rbryce] from comment #2)
> (In reply to Justin Dow [:jabba] from comment #1)
> > I'd vote to move these to HCI as mailman1, mailman2 and
> > mailman3.mail.corp.phx1.mozilla.com.
> 
> I agree.  Before we do any moving of mail servers.  We need to identify the
> public IP addresses mail will be sent from. If the IPs change we need to get
> public DNS setup before hand, at least 2 weeks prior.  Our volume, combined
> with an instant DNS change will cause new IPs to be blacklisted.

Since we are moving the services wholesale to new boxes, obviously internal IPs will change, but perhaps we can port the existing public IPs to just point at the new location in HCI? Then we don't need to worry about any of that.

CC'ing ravi to help determine the feasibility of moving public IPs from sjc1 to corp.phx1.

Comment 5

[Ravi Pina [:ravi]]

Technically possible, but not realistic or practical in the short term.

I'd love to see mailman consolidated to a single mailman instance, but as Dave points out our HCI infra is quite capable so there may be an advantage to having an instance for .com and .org.

Comment 6

[Dave Miller [:justdave]]

(In reply to Ravi Pina [:ravi] from comment #5)
> I'd love to see mailman consolidated to a single mailman instance

Not possible with the current version of mailman, either, due to namespace collisions.  Mailman 3 is supposed to allow it, but that's still vaporware at this point.

lists.mozilla.org is also hacked up a bit to support the newsgroup gateway and associated features, so it'd need to stay separate anyway.

Comment 7

[Rob Tucker [:rtucker]]

Any word on the status of this migration?

Comment 8

[Dave Miller [:justdave]]

Hasn't been touched because we've been busy with Zimbra.  With the exception of bugzilla.org, though, the other three are pretty straightforward and could probably all be done in a day (save for rsync time to move the data).

dm-mailman01 has an interdependency on dm-mail{01,02,03}.

Nothing knows about mailman1.dmz.sjc1 except zimbra, and it'll be an easy tweak to tell it where it moves to.

notorious is a standalone box, not dependent on anything else in the email ecosystem, so it will be easy to move.

Comment 9

[Dave Miller [:justdave]]

VMs in HCI have been requested on bug 744688

Comment 10

[Dave Miller [:justdave]]

Mailman 3 just hit beta a couple weeks ago, and we could try it....  on the other hand, our timetable is pretty short here, not sure if we have time to test it out thoroughly...

Comment 11

[Ravi Pina [:ravi]]

The mailman1.dmz instance may be a suitable one to beta the new
Mailman.  It runs only internal lists.

Comment 12

[Dave Miller [:justdave]]

mailman1.dmz.sjc1 has just been shutdown.  Everything that used to be on it has been moved to mailman3.mail.corp.phx1.mozilla.com.

I left it on mailman 2.1.x...  I poked around a bit at mailman 3b1, and it seems to be vastly lacking in documentation, and I was in too much of a hurry to try to reverse-engineer it enough to install it properly.

Comment 13

[Dave Miller [:justdave]]

I'm going to drop the confidential flag on this bug - I don't see anything confidential in the existing comments, and it affects public-facing services, so there's no reason not to let people who use those services follow along.

I'm spinning off new bugs for each server just to help keep track of what's going on a little more closely.  No new bug for mailman1(mozilla.com) just because it's already done. :)

dm-mailman01(bugzilla.org)   -> bug 628085
dm-mailman01(mozilla.org)    -> bug 752129
notorious(lists.mozilla.org) -> bug 752130

Comment 14

[Dave Miller [:justdave]]

We're done!!!!!  yay.