812418 - Fix hacky bugzilla.org email setup to use a transport map like mozilla.org

Status: RESOLVED FIXED

(Infrastructure & Operations :: Infrastructure: Mail, task)

Description

[Dave Miller [:justdave]]

bugzilla.org's email is currently set up in an extremely hacky way between mx1/mx2 and mailman4.  In order to make bugzilla.org's transition from majordomo2 to mailman for their mailing lists run smoothly, we need to vastly simplify the way it's done.

The changes outlined below will require configuration changes to mx1/mx2, which falls within the HCI Maintenance Policy at https://mana.mozilla.org/wiki/display/IT/HCI+Maintenance and thus should occur within a scheduled maintenance window.

-----------------
The current setup
-----------------

Externally:
MX records for bugzilla.org point at mx1/mx2.
Internally:
MX records for mx.bugzilla.org point at mx1/mx2
MX records for lists.bugzilla.org point at mailman4

All three hosts think they're authoritative for "bugzilla.org"

mailman4 generates a virtual alias file for postfix which maps ${address}@bugzilla.org to ${address}@lists.bugzilla.org for each list and all of the administrative aliases for that list, and then pushes that alias file to mx1/mx2.

mx1/mx2 have a separate virtual alias file that contains all of the forwarding addresses for users with bugzilla.org email addresses

mx1/mx2 have a single catch-all entry for @mx.bugzilla.org that maps back to @bugzilla.org

mailman4 has a single catch-all entry for @lists.bugzilla.org that maps back to @bugzilla.org.

mailman4 has single catch-all entry for @bugzilla.org (matched for anything that doesn't have a more-specific alias like the lists and admin aliases) which maps to @mx.bugzilla.org

This allows users with bugzilla.org email addresses to receive mail from bugzilla.org lists even though they aren't on the same box.

And that's hella complicated and confusing.

----------------
How it should be
----------------

This should mirror how mozilla.org is set up on mailman1:

- The users with bugzilla.org email addresses should be defined on mailman4, not mx1/mx2.
- mailman4 should push a transport map, not an alias file, to mx1/mx2 which contains all of the user addresses, majordomo lists, and mailman lists.  This file simply states that each address exists, and that it's handled by mailman4, so that mx1/mx2 can reject mail for unknown users.
- mx.bugzilla.org and lists.bugzilla.org can just go away (as MX targets) and all of the hacky forwarding between them can go away, too.

---------
prep work
---------

The scripts on mailman4 to generate the transport map can be set up in advance in parallel to the existing scripts pushing an alias file.  They can use different filenames, and push to mx1/mx2 as usual.  This won't affect anything until main.cf on mx1/mx2 are adjusted to actually use it.

The alias file for bugzilla.org from mx1/mx2 can similarly be copied to mailman4 in advance, and we just wait until the window to plug it into the postfix config there as well.

In order to slide this into this weekend's HCI Maintenance slot, this would need to be communicated by Noon PST on Friday, November 16.

If everything goes right, there will be no user-facing downtime.

Services potentially affected in case something does go wrong:
- smtp.mozilla.org for outbound mail
- Inbound mail for every domain hosted by Mozilla EXCEPT:
    mozilla.org, lists.mozilla.org, mozilla.com, mozillafoundation.org
    NOTE: Those are the domains that are NOT affected.

Comment 1

[Dave Miller [:justdave]]

Changes to be made to the main.cf on mx1/mx2:

remove from virtual_alias_domains:
    bugzilla.org

add to relay_domains:
    bugzilla.org

remove from virtual_alias_maps:
    hash:/etc/postfix/domains/bugzilla.org
    hash:/etc/postfix/domains/lists.bugzilla.org

add to transport_maps:
    hash:/etc/postfix/domains/bugzilla.org-new

After we've made sure it's working, we can remove the old alias files and rename bugzilla.org-new back to bugzilla.org so it still matches (and change transport_maps accordingly).

Comment 2

[Dave Miller [:justdave]]

Backout procedure in case of problems is to restore the original config (revert the changes made in comment 1).

Comment 3

[Ed Lim [:limed]]

(In reply to Dave Miller [:justdave] from comment #1)
> Changes to be made to the main.cf on mx1/mx2:
> 
> remove from virtual_alias_domains:
>     bugzilla.org
> 
> add to relay_domains:
>     bugzilla.org
> 
> remove from virtual_alias_maps:
>     hash:/etc/postfix/domains/bugzilla.org
>     hash:/etc/postfix/domains/lists.bugzilla.org
> 
> add to transport_maps:
>     hash:/etc/postfix/domains/bugzilla.org-new
> 
> After we've made sure it's working, we can remove the old alias files and
> rename bugzilla.org-new back to bugzilla.org so it still matches (and change
> transport_maps accordingly).

Remember that this is all done in puppet now :)

Comment 4

[Dave Miller [:justdave]]

config updated in puppet and pushed.

push scripts on mailman4 updated.  transport map manually pushed in order to get it there.  We're missing host keys, so it's never worked since it moved.  We've apparently never changed any email addresses in all that time, so that's why we never got any cron errors.

The only functional hangup was bugbot's email forwards to the community box that he runs on, and mailman4 can't reach it.  I fixed this by putting in a transport map for that domain name pointing it back at mx1 (since mx1 could obviously reach it before).

The only remaining task here before closing this bug out is to get the ssh keys in place to allow mailman4 to push to mx1/mx2.  I'll need assistance from jabba on that because those are all puppet-managed as well now.

Comment 5

[Dave Miller [:justdave]]

Discussed with with jabba, instead of using ssh+rsync to push the transport map to mx1/mx2, we're going to set up an rsync server on mailman4, and have mx1/mx2 pull it via rsync.

Comment 6

[Amy Rich [:arr] [:arich]]

deleted bogus whiteboard tag

Comment 7

[Dave Miller [:justdave]]

This was essentially done, except for the auto-push of the transport files.  They change so infrequently I've just been pushing them by hand every time they change.  This host is going away Really Soon Now™ so there's no point in fixing it.  The transport files can continue to be pushed by hand for the short remaining life it has, and the new home of the mail services won't need this hack.