Closed Bug 240169 Opened 16 years ago Closed 5 years ago

add SPF (Sender Policy Framework) name server entries to mozilla.org for mail server use

Categories

(Infrastructure & Operations :: Infrastructure: Mail, task, minor)

task
Not set
minor

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: peter+bmo, Assigned: limed)

References

(Blocks 1 open bug, )

Details

Attachments

(1 obsolete file)

see above URL for more information
Assignee: mitchell → endico
Component: Miscellaneous → Server Operations
QA Contact: mitchell → myk
infinitepenguins.net's nameservers appear to be down at the moment.  I managed
to find enough references to it to track down an IP address and successfully got
to the site that way.  http://spftools.inifinitepenguins.net (when their
nameservers start working again) is a collection of tools for dealing with SPF.

Changing the URL on the bug to point at a page which actually tells you what SPF
is (which is better for our purposes at this point, though the url in the above
paragraph may come in handy for implementation if we do it).

I think this will be extremely difficult for mozilla.org to implement.  With the
exception of bugmail and mailing lists, nobody with an @mozilla.org email
address uses mozilla.org's servers to send mail.  And with the number of people
that have @mozilla.org email addresses, and the variety of places that they send
mail from, I'm willing to bet if we do this we will have literally hundreds of
SPF entries (and we'd have to be updating them constantly).  This sounds like an
administrative nightmare.

Of course, getting said folks to actually use the mozilla.org mail servers for
sending mail would solve that, but that in itself is going to be a nightmare to
make happen.  Most of the folks with @mozilla.org addresses don't actually have
accounts on mozilla.org machines.  Those addresses all forward somewhere.  And
change is hard. :)

We could always go with a "softfail" record...

 	v=spf1	This identifies the TXT record as an SPF string.
	a	mozilla.org's IP address is 207.126.111.202 
                (h-207-126-111-202-mozilla.sv.meer.net).  That server is
                allowed to send mail from mozilla.org.
	ptr	Any server whose name ends in mozilla.org is allowed to send
                mail from mozilla.org.
	~all	SPF queries that do not match any other mechanism will return
                "softfail". Messages that are not sent from an approved server
                should still be accepted but may be subjected to greater 
                scrutiny.


If you run BIND
Paste this into your zone file:

mozilla.org. IN TXT "v=spf1 a ptr include:domain1.com ~all"
Summary: add SPF name server entries to mozilla.org → add SPF (Sender Policy Framework) name server entries to mozilla.org for mail server use
oh, the domain1.com was an example of adding another permitted domain, and we'd
need an include: for each one.  With the softfail, it's not really necessary,
and I forgot to nuke it.
maybe also add an 'mx' to the spf record for allowing smtp.oregonstate.edu to
also send mails on behalf of mozilla.org
Reassigning to default owner/qa on endico's open bugs so I don't have to watch
her to get mail on them.
Assignee: endico → myk
QA Contact: myk → justdave
(In reply to comment #3)
> maybe also add an 'mx' to the spf record for allowing smtp.oregonstate.edu to
> also send mails on behalf of mozilla.org

I don't believe smtp.oregonstate.edu has any reason to send mail with an
@mozilla.org return address.  They only handle our incoming mail, not the
outgoing.  The only thing they'd be sending on our behalf is bounce notices, and
those will probably all actually say oregonstate.edu on the From line, and have
an empty SMTP envelope sender anyway.
Myk used to be the default assignee for this component.  Reassigning all remaining open bugs owned by him that he doesn't appear actively involved in to the default assignee/qa for later reassignment.
Assignee: myk → server-ops
QA Contact: justdave → justin
Attached file MozillaOrgTrim module definition (obsolete) —
Comment on attachment 206942 [details]
MozillaOrgTrim module definition

Sorry for the bugspam, wrong bug. Do not know what happened there.
Attachment #206942 - Attachment is obsolete: true
Component: Server Operations → Server Operations Projects
-> default assignee

FYI, this should be much easier to implement these days.  We have dedicated outbound mail servers now, and no reason not to force people to use them. :)
Assignee: server-ops → nobody
Even without forcing everyone to use the central outbound mail servers, we could still implement SPF by adding people's home servers as additional authorized senders of mozilla.org mail.

We should mail @mozilla.org personal addresses and let people know that we're going to be implementing SPF.  Then, folks who want to special-case a server can provide us with its IP address, and we can add that to the list.

And then we can turn on SPF.

I'm getting dozens of bounces and pseudo-bounces (auto-responses from customer support addresses, notices that an address is no longer valid, confirmations that I really did intend to subscribe to some mailing list, etc.) a day, and justdave says (in bug 368004, comment 2), "there's apparently a large number of spammers actually spoofing your address, there's literally a few thousand messages 'from' you rejected for a whole host of reasons in the log."

So implementing SPF would be more than just a minor convenience, it could make a significant impact on the time I spend deleting these messages from my Inbox, and it might even have a positive impact on server load.
Sending "you have a virus" bounces is already seen as socially irresponsible, how many of these problem servers are likely to support SPF?
We can also do "SRS" which puts a hash on the return address that must be present on any bounce notices or the bounces are rejected as "this message wasn't sent from here".  However, doing this also depends on everyone sending their mail through our server, because if they don't, the bounces won't make it back when things do bounce.
I'm not sure SRS would solve the problem, since the problem is not just with bounces.  I get lots of other types of messages because of spam with a forged return address, like list subscription confirmations and auto-responses from support addresses (assigning me a ticket, assuring me they'll look at my issue soon, etc.).  And then there are the "you have a virus" messages.

Would SRS handle those situations, and can we do SRS on a per-user basis?
Changing QA Contact.
QA Contact: justin → mrz
So we're still blocked on postfix not supporting SRS.  If we really want this is looks like we're going to need to switch back to sendmail. :(

The real reason we need SRS which hasn't been discussed in this bug, yet, it appears, is for people with @mozilla.org addresses that are forwarded elsewhere. The hosts they forward to have been starting to implement SPF checking, and have started to block mail from people with SPF records who send mail to mozilla.org because mozilla.org isn't authorized to send mail on behalf of the sender.  If we had SRS, mozilla.org would be sending mail on behalf of itself instead of using the MAIL FROM of the original sender when forwarding to an offsite destination.
How many people have and actually use @mozilla.org addresses?  What's the
breakdown of those addresses that are forwarded vs. hosted?  It seems like
there's enough value here that it's worth discussing accepting a constraint of
"all @mozilla.org accounts must be hosted".
(In reply to comment #16)
> How many people have and actually use @mozilla.org addresses?  What's the
> breakdown of those addresses that are forwarded vs. hosted?  It seems like
> there's enough value here that it's worth discussing accepting a constraint of
> "all @mozilla.org accounts must be hosted".

We currently don't support hosting (as in pop3/imap) @mozilla.org addresses for users at all.
That's not precisely correct, in the sense that I get and send all of my @mozilla.org mail at my @mozilla.com address.  So part of the question, perhaps, is "what's the breakdown of active @mozilla.org users who have vs. don't have existing @mozilla.com addresses?".
Often, our email address is our only proof-of-identity when interacting with people on the Net. The more easily that malicious parties can spoof our addresses, the harder we make our lives. I'd rather have the small inconvenience of needing to send mail from Mozilla servers than the large inconvenience of having someone spoof my Mozilla email address.
I agree.  I'm totally down with using Mozilla servers to send my mail, which I've been doing for years anyway.
I routinely send mail from my mozilla.org or mozilla.com addresses via other mail servers, when I'm on a crappy network that doesn't let me get out to SMTP or SMTPS, or when I can't get to smtp.mozilla.org because of LDAP glitches.  Mostly that's mozilla.com, but not exclusively.
(In reply to comment #21)
> I routinely send mail from my mozilla.org or mozilla.com addresses via other
> mail servers, when I'm on a crappy network that doesn't let me get out to SMTP
> or SMTPS

meer.net has found a nice fix for this: simply listen for SMTP(s) on a non-standard port that ISPs don't tend to block, and recommend that users configure their clients for that.

> or when I can't get to smtp.mozilla.org because of LDAP glitches. 
> Mostly that's mozilla.com, but not exclusively.

Agreed that there is a tradeoff.  I'm suggesting that cutting down on the amount of spam the world sees that appears to come from mozilla.org as well as all the backscatter that we get to our accounts is worth taking this convenience hit.
(In reply to comment #22)
> meer.net has found a nice fix for this: simply listen for SMTP(s) on a
> non-standard port that ISPs don't tend to block, and recommend that users
> configure their clients for that.

Yeah, that's what I've been doing on my personal mail server for years -- it's why I mostly have my laptop configured for a (that) non-smtp.mozilla.org mail server.
(In reply to comment #21)
> I routinely send mail from my mozilla.org or mozilla.com addresses via other
> mail servers, when I'm on a crappy network that doesn't let me get out to SMTP
> or SMTPS

That's what port 587 (submission) has been "invented" for ;-)
The crappy networks I end up on often block liberally below 1024.  I've had best luck with 2525 as alternate port.
(In reply to comment #25)
> The crappy networks I end up on often block liberally below 1024.  I've had
> best luck with 2525 as alternate port.

Meer does it, Pair does it, even shaver on his laptop does it,
Let's do it too, let's listen on a non-standard port: bug 475543.
DKIM might also be another thing we could implement that would help here.  Dave, what would need to happen for this bug to become high enough priority that someone inside of MoCo IT would own and drive it forward?
Here's a thought: what we really need is for mozilla.org to publish an SPF record which contains all the mail servers @mozilla.org users want it to, without IT admin having to configure it manually every time there's a change. I was wondering if we could build a system where there was a magic email address - e.g. spf-enable@mozilla.org. Sending an email there, with the body containing a magic password, meant that the email got parsed, the first hop extracted, and its IP added to the SPF-permitted list.

So if you had to use a new mail server, all you would have to do is send a mail using that server to spf-enable@mozilla.org, and then you could send mozilla.org mail using that server without worrying that it would get bounced.

Would that work, or have I missed something?

Gerv
That'd be hard to prevent abuse of.  Any spammer could get themselves whitelisted by mailing it, since we'd have no way to prevent them from forging a From line (other than enforcing SPF, which is the entire problem - chicken and the egg and all that).
Personally, it's been a long time since I've seen an email client that didn't let you set a different SMTP server on each account/personality...  we should just make people start using smtp.mozilla.org for their mozilla.org addresses.
Dave: make it so!
> Any spammer could get themselves whitelisted by mailing it

Except for the bit where I said "with the body containing a magic password" :-)

Dave: I thought shaver had an objection to making people use smtp.mozilla.org. 
But if it listens on a variety of ports, I personally am cool with it.

Gerv
Shaver appears to be CCed on this bug, hopefully he'll reply.

But yeah, we have STARTTLS available on ports 25 and 587 and SSL available on 465 and 2525.
I'm not sure what I'm being asked that I haven't already addressed in my previous comments, but if this is really the big deal that it's made out to be, then I'm sure I'll live.
One other potential "gotcha" that I just noticed...   you need to have a *mozilla.com* email account in order to authenticate to smtp.mozilla.org to send mail. There's a number of people with mail forwarded through *mozilla.org* who don't have mozilla.com accounts. I don't know how many of them actually use them though.
Infrasec:

Can you advise here..  should we enforce everyone to send mail through smtp.mozilla.org (creating a policy) and enable SPF or WONTFIX this?
I think justdave's comment 35 is no longer true; I send mail through smtp.mozilla.org. Although I still value the ability to use other servers when it's down or unreachable.

Looks like smtp.mozilla.org:2525 is no longer working...

Gerv
We should go ahead and do this now.  There's too many scams going around claiming to be from mozilla.org.  We should do DKIM-signing, too, while we're at it (not sure if there's another bug for that or not).
Assignee: nobody → server-ops-infra
Component: Server Operations: Projects → Server Operations: Infrastructure
QA Contact: mrz → jdow
mozilla.com and mozillafoundation.org, too.
I can't find a bug for DKIM (except for one wanting mailman to remove DKIM headers added by the sender before remailing messages which it alters to add footers and whatnot).  Should we just make this cover both or file one?
For outbound mail, we have a pair of relays acting as smtp.mozilla.org, three relays in Phoenix and one in Beijing acting as outbound for Zimbra, 3 Mailman servers that send their own outbound mail, and then various web clusters have their own outbound relays for mail sent by webapps in those clusters.
Note that, the dkim key should be at least 1024bits. I would recommend 2048 as a safety measure in case RSA gets troubles in the future. This is also what google uses (they used to have 512 bits which is not secure enough).
Dave/Jabba,

This needs to be done asap. Please let us know what we can do to assist.
After a discussion with :reed it seems safer to start with 1024bit keys (nothing lower) as 2048bit keys may cause issues with some MTAs/DNS resolvers due to the field length required.

It's also probably a good idea to have a selector that states the generation date (like s20121029 or something similar, like s2012-q4) and regenerate the key every ~quarter or so.

Opsec can produce "official" documentation for this

Note that, all our servers have to send through a proper smtp first (and, clients should too)

Hence depending on bug 801220 at least for our internal stuff

We should also communicate this to users, such as moco employees, in case they do not use our smtp.
Depends on: 801220
limed@jayne:~/scm/mozilla-dnsconfig/external$ dig mozilla.com txt +short
"v=spf1 ip4:63.245.208.0/20  ?all"
limed@jayne:~/scm/mozilla-dnsconfig/external$ dig mozilla.org txt +short
"v=spf1 ip4:63.245.208.0/20 ?all"

We have spf records published now, so far this record probably doesn't do anything. But this is a way we can actually have the records published first until we can verify what other hosts are sending mail.

The next step is to change ?all to ~all
Had to revert this, services nameserver is throwing out dns error emails, seeing that its 6pm here I think we'll enable this tomorrow when people are around to make sure there is no breakage
To help making tighter rules, here's a list that has been compiled using the netflow dump michal` made in bug 807106

I found 23 unique source ips from within our IP space sending email for the past ~30 days (63.245.208.0/20). It excludes internal communication, ie from 63.245.208.0/20, 10.0.0.0/8 to 63.245.208.0/20, 10.0.0.0/8 

63.245.208.248
63.245.214.131
63.245.214.162
63.245.216.163
63.245.216.221
63.245.216.223
63.245.216.225
63.245.216.227
63.245.216.244
63.245.216.245
63.245.216.248
63.245.218.7
63.245.218.72
63.245.220.10
63.245.220.219
63.245.220.224
63.245.220.240
63.245.223.129
63.245.223.13
63.245.223.25
63.245.223.26
63.245.223.28
63.245.223.9

Some of these may not be legitimate hosts, as in, they're probably not supposed to send email at all (such as guest-224.mv.mozilla.com - 63.245.220.224)
limed@jayne:~$ dig mozilla.org txt +short
"v=spf1 ip4:63.245.208.0/20 ~all"
limed@jayne:~$ dig mozilla.com txt +short
"v=spf1 ip4:63.245.208.0/20  ~all"

All good
Note: 223.202.6.0/27 is the china range. We don't have flow data for it right now.
Update to the dns record

limed@jayne:~/scm/mozilla-dnsconfig/external$ dig @ns1.mozilla.org mozilla.org txt +short
"v=spf1 ip4:63.245.208.0/20 ip4:223.202.6.0/27 ~all"
limed@jayne:~/scm/mozilla-dnsconfig/external$ dig @ns1.mozilla.org mozilla.com txt +short
"v=spf1 ip4:63.245.208.0/20 ip4:223.202.6.0/27  ~all"
note, record has been disabled at my request
this shouldn't be deployed even in soft fail until dependencies are fullfiled and acks + testing done
Assignee: server-ops-infra → limed
Component: Server Operations: Infrastructure → Infrastructure: Other
Product: mozilla.org → Infrastructure & Operations
Component: Infrastructure: Other → Infrastructure: Mail
QA Contact: jdow → limed
Duplicate of this bug: 972206
Is there a plan to address this?
looks like we're just waiting on bug 801220.
(In reply to Dave Miller [:justdave] (justdave@bugzilla.org) from comment #54)
> looks like we're just waiting on bug 801220.

...which I notice is secured because it has internal network info on it.

That bug is for setting up outbound relay hosts and firewalling off outbound port 25 from anywhere that's not the relay hosts.
See Also: → 1051184
See Also: → 991715
Duplicate of this bug: 1077960
Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/3037]
Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/3037] → [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/3041]
Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/3041] → [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/3045]
deleted bogus whiteboard tag
Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/3045]
In order for us to make use of bug 1102364 on bugzilla.mozilla.org, Google requires that senders make use of either DKIM or SPF (https://developers.google.com/gmail/markup/registering-with-google).

I imagine the switch from Zimbra to GMail will block any further work on this in the meantime, but it would be great if we could still do this after :-)
I strongly advise DKIM rather than SPF for the purpose of Bugzilla emails, fwiw.
(In reply to Richard Soderberg [:atoll] from comment #60)
> I strongly advise DKIM rather than SPF for the purpose of Bugzilla emails,
> fwiw.

For what reasons, out of curiosity? (I gather DKIM is inherently more secure, but wondered if there was some other issue with deploying SPF that I wasn't aware of). From what I can tell I'm after bug 807013 then :-)
(In reply to Ed Morley (moved to Treeherder) [:edmorley] from comment #61)
> (In reply to Richard Soderberg [:atoll] from comment #60)
> > I strongly advise DKIM rather than SPF for the purpose of Bugzilla emails,
> > fwiw.
> 
> For what reasons, out of curiosity? (I gather DKIM is inherently more
> secure, but wondered if there was some other issue with deploying SPF that I
> wasn't aware of). From what I can tell I'm after bug 807013 then :-)

Entirely for the security; Signatures > Source IPs, for a variety of reasons. (SPF was an excellent first step, though.)
(In reply to Richard Soderberg [:atoll] from comment #60)
> I strongly advise DKIM rather than SPF for the purpose of Bugzilla emails, fwiw.

No reason we can't or shouldn't do both.
Both are designed for different purposes, and both should be done.
Agreed, both should be done :)
Blocks: 1081574
Blocks: 1102364
Blocks: 1139840
No longer blocks: 1102364
limed@river:~$ dig @8.8.8.8 mozilla.com txt +short
"v=spf1 include:_spf.mozilla.com include:_spf.google.com ?all"
limed@river:~$ dig @8.8.8.8 mozillafoundation.org txt +short
"v=spf1 include:_spf.mozilla.com include:_spf.google.com ?all"

Although its just ?all i believe this is considered publishing spf records
limed: I believe this bug is about mozilla._org_. But it seems it has it too:

gerv@hare:/$ dig @8.8.8.8 mozilla.org txt +short
"google-site-verification=Lo_B34AJAe70BQVNF1Fo1zGGJudPmw9bLTnP2C8lV-s"
"v=spf1 include:_spf.mozilla.com include:_spf.google.com ~all"

Stricter than the other two, in fact.

Gerv
Ah I didn't actually check, that may work (for bug 1139840), thank you :-)
So can this bug be marked FIXED?

Gerv
(In reply to Gervase Markham [:gerv] from comment #67)
> limed: I believe this bug is about mozilla._org_. But it seems it has it too:
> 
> gerv@hare:/$ dig @8.8.8.8 mozilla.org txt +short
> "google-site-verification=Lo_B34AJAe70BQVNF1Fo1zGGJudPmw9bLTnP2C8lV-s"
> "v=spf1 include:_spf.mozilla.com include:_spf.google.com ~all"
> 
> Stricter than the other two, in fact.
> 
> Gerv

Yep at some point we will switch mozilla.com and mozillafoundation.org to be stricter, and yes i forgot to add mozilla.org's in the list of dig's apologies for the confusion
And I can say yes this is fixed
Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
Duplicate of this bug: 1142481
Duplicate of this bug: 1090120
You need to log in before you can comment on or make changes to this bug.