628085 - Migrate bugzilla.org mailing lists from majordomo2 to mailman

Status: RESOLVED FIXED

(Bugzilla :: bugzilla.org, defect)

Description

[Reed Loden [:reed]]

Bugzilla is using majordomo2 for its mailing lists. For all other project mailing lists, mailman is used. All the majordomo2 mailing lists should be converted to mailman. This will cut down on maintenance hassles by centralizing mailing list solutions into one platform.

Comment 1

[Corey Shields [:cshields]]

Will have to wait until after Firefox4.

Comment 2

[Dave Miller [:justdave]]

Mailman can handle multiple domain names just fine as long as all of the list names are globally unique across the entire instance.  In our case, they're not.  Both bugzilla.org and mozilla.org have a security@ mailing list.  The existence of this name conflict precludes us from being able to run both domains on the same instance of mailman.

I've just been chatting with one of the upstream mailman devs, and we basically have three options here.

1) Install a second copy of mailman on the server from source instead of using the RPMs.  Install it to a new independent location so it can be run in parallel with the existing installation.  Configure postfix and apache to use this other instance when dealing with bugzilla.org

2) Set up a new machine and move bugzilla.org to it so it can have its own Mailman installed from RPM.

3) Wait for Mailman 3.0 (which has real virtual host support) to hit beta status and then upgrade to that.  Mailman 3.0 is currently in alpha status, and the lack of a complete upgrade path from 2.1 is apparently the last remaining blocker to it becoming beta.  The last alpha release was 3.0a6 and was released on September 20th 2010.  Development is pretty slow.  No idea when the first beta will hit.

Comment 3

[Reed Loden [:reed]]

... or just patch mailman to support multiple vhosts with one as the master (mozilla.org). I've done this multiple times for other places. It's not that difficult. :)

Comment 4

[Dave Miller [:justdave]]

Everything I've read about that patch says it has issues and isn't really production-ready (and that the current maintainer isn't interesting in maintaining it - mainly because MM 3.0 already has it).

Comment 5

[Dave Miller [:justdave]]

bug 726806 is our internal tracking bug for migrating a slew of mailman hosts into.

We need to replace dm-mailman01 as part of the move out of the SJC1 facility.  Might as well do this migration as part of the move, one less thing to set up at the destination.

bug 744688 is requesting the VM be created that will host this.

Comment 6

[Gervase Markham [:gerv]]

At the moment, AIUI, the [e-mail] mailing list is gatewayed in to the mozilla.dev.apps.bugzilla discussion forum, which has its own mailing list version ([e-mail]). Would this be a good opportunity to just switch to having the standard setup in this case?

Gerv

Comment 7

[Dave Miller [:justdave]]

Target host for this is mailman4.mail.corp.phx1.

Comment 8

[Dave Miller [:justdave]]

dm-mailman01 will be removed from the SJC1 facility on May 14, so this needs to be completed before then.

Comment 9

[Dave Miller [:justdave]]

OK, this has successfully migrated from dm-mailman01 to mailman4.mail.corp.phx1.

It has *not* migrated from majordomo2 to mailman yet, however.

I do have majordomo2 and mailman installed side-by-side on the new box with mailman set up for bugzilla.org, and we're no longer sharing with mozilla.org.

We'll be able to shift the lists over one at a time at our leisure now instead of being in a big rush.

Leaving this open since we haven't moved into mailman yet, but breaking the dependency on the sjc1 move-out, since we've moved out. :)

Comment 10

[Dave Miller [:justdave]]

One of the blockers to this previously was figuring out how to migrate the list archives out of majordomo2 into mailman.  Well, it turns out that majordomo2 stores the archives in generic mbox files, which is also how mailman stores them.  However, mailman keeps one monolithic mbox file for the list and then parses that out into multiple html files based on ${archive_rotation_interval}.  Majordomo2 keeps a separate mbox file for each ${archive_rotation_interval}.  So basically, you concatenate all of the majordomo2 archive files for a list into a single file, drop that into Mailman's archive directory, and run /usr/lib/mailmna/bin/arch ${listname}.

I was all set to start moving lists tonight...  and then realized how hacky the bugzilla.org email setup is currently, and that the current setup doesn't even include the mailman lists in the alias file that it pushes to the mail relay.  So even if we did convert a list to mailman, you wouldn't be able to send mail to it, which makes it pointless.

-----------------
The current setup
-----------------

Externally:
MX records for bugzilla.org point at mx1/mx2.
Internally:
MX records for mx.bugzilla.org point at mx1/mx2
MX records for lists.bugzilla.org point at mailman4

All three hosts think they're authoritative for "bugzilla.org"

mailman4 generates a virtual alias file for postfix which maps ${address}@bugzilla.org to ${address}@lists.bugzilla.org for each list and all of the administrative aliases for that list, and then pushes that alias file to mx1/mx2.

mx1/mx2 have a separate virtual alias file that contains all of the forwarding addresses for users with bugzilla.org email addresses

mx1/mx2 have a single catch-all entry for @mx.bugzilla.org that maps back to @bugzilla.org

mailman4 has a single catch-all entry for @lists.bugzilla.org that maps back to @bugzilla.org.

mailman4 has single catch-all entry for @bugzilla.org (matched for anything that doesn't have a more-specific alias like the lists and admin aliases) which maps to @mx.bugzilla.org

This allows users with bugzilla.org email addresses to receive mail from bugzilla.org lists even though they aren't on the same box.

And that's hella complicated and confusing.

----------------
How it should be
----------------

This should mirror how mozilla.org is set up on mailman1:

- The users with bugzilla.org email addresses should be defined on mailman4, not mx1/mx2.
- mailman4 should push a transport map, not an alias file, to mx1/mx2 which contains all of the user addresses, majordomo lists, and mailman lists.  This file simply states that each address exists, and that it's handled by mailman4, so that mx1/mx2 can reject mail for unknown users.
- mx.bugzilla.org and lists.bugzilla.org can just go away (as MX targets) and all of the hacky forwarding between them can go away, too.

Comment 11

[Dave Miller [:justdave]]

I filed bug 812418 to get a maintenance window scheduled for mx1/mx2 to make the changes outlined in comment 10.

Comment 12

[Dave Miller [:justdave]]

Comment 9 has repeated itself.  (majordomo2 made the move to cps.bugzilla.org intact instead of migrating to mailman first because of impending deadlines on vacating mailman4).  Fixing dependencies appropriately.

Comment 13

[Ed Lim [:limed]]

Moving this out of infra's component since we don't have anything we can do here

Comment 14

[Dave Miller [:justdave]]

And comment 12 again repeated itself. majordomo2 made the move to barry.bugzilla.org intact instead of migrating to mailman first.

We're toying with the idea of going to mailman3 now. And I have a volunteer to investigate. :-)

(and removing a bunch of Mozilla people off the CC since Mozilla IT isn't involved with this anymore)

Comment 15

[Corey Shields [:cshields]]

(In reply to Dave Miller [:justdave] ([e-mail]) from comment #14)

And comment 12 again repeated itself. majordomo2 made the move to barry.bugzilla.org intact instead of migrating to mailman first.

We're toying with the idea of going to mailman3 now. And I have a volunteer to investigate. :-)

(and removing a bunch of Mozilla people off the CC since Mozilla IT isn't involved with this anymore)

Hey justdave, wb! fyi we'll be moving Mozilla's discussion forums and lists off of mailman this quarter, communities will go to ggroups and discourse (discourse.mozilla.org) which looks to have a Thunderbird main category setup already.

Comment 16

[Dave Miller [:justdave]]

(In reply to Corey Shields [:cshields] from comment #15)

which looks to have a Thunderbird main category setup already.

Nifty. This isn't about Thunderbird of course. And I sort of fear that if we put Bugzilla stuff in Mozilla's Discourse that people will get confused between upstream and BMO again.

How are you dealing with the list archives? Are they migrating somewhere, or a read-only copy getting put someplace, or?

Comment 17

[Dave Miller [:justdave]]

Hmm, Bugzilla could always set up their own Discourse instance. It's not exactly hard to set up (done it before other places). Could run it on carly since all it has on it are IRC bots. I have an existing Discourse set up elsewhere on that same class of Linode server and it runs fine (even though it's technically under Discourse's recommended server specs).

Comment 18

[Michelle Goossens [:masterwayz]]

Had a chat with Dave about this.
Currently it looks like we are going to set up a Discourse instance for Bugzilla and set up integrations with other platforms. A plan list is being worked on right now on what steps we are going to take to do this and migrate everything over.

Comment 19

[Dave Miller [:justdave]]

Not entirely sold on Discourse yet... finally got a chance to see a live Mailman 3 server recently and it's actually not bad...

Comment 20

[Dave Miller [:justdave]]

We moved almost all of the mailing lists from majordomo2 to mailman2 about two months ago.

Comment 21

[Dave Miller [:justdave]]

OK, this is done. I just moved the last two straggler internal lists to mailman and uninstalled Majordomo2.