Closed Bug 1363039 Opened 8 years ago Closed 7 years ago

Add spf records for amazon SES

Categories

(Infrastructure & Operations :: Infrastructure: Mail, task)

Product:
Infrastructure & Operations
Component:
Infrastructure: Mail
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: gozer, Unassigned)

References

(Blocks 1 open bug)

Details

Bugzilla in AWS is using SES to send mail successfully, but I discovered it would be better for anti-spam operations if we also included SES's spf records. So from: "v=spf1 include:_spf.mozilla.com include:_spf.google.com ~all" To: "v=spf1 include:_spf.mozilla.com include:_spf.google.com include:amazonses.com ~all"
For which domain(s)? bugzilla.mozilla.com? This allows anyone using AWS to spoof email, I assume that has been considered?
(In reply to Peter Radcliffe [:pir] from comment #1) > For which domain(s)? bugzilla.mozilla.com? > > This allows anyone using AWS to spoof email, I assume that has been > considered? mozilla.org, but a good point regarding spoofing. Ideally, it would be nice if bugzilla sent e-mails from its own domain, but right now, it's using bugzilla-deamon@mozilla.org
Infosec feels that bugzilla should be sending mail from it's own domain, and that what is being requested should not be done due the risks associated.
There are a couple problems that we have with doing this: - We're already up against our maximum limit for DNS records (10) contained within the SPF record for mozilla.org and this would cause it to go over and possibly break a number of things - As things currently stand, our DMARC setup will probably break Bugzilla emails from SES should we ever actually enable DMARC: > _dmarc.mozilla.com. 59 IN TXT "v=DMARC1\; p=none\; rua=mailto:dmarc@mozilla.com" So we should probably fix that if we are actually sending emails from SES, since I imagine they all fail SPF checks as it is. Probably our best bet is to send emails from notifications@bugzilla.mozilla.org (or whatever). If we sent from @bugzilla.mozilla.org, we could set it up with SPF/DKIM/DMARC on the subdomain and limit mail sending from a less broad list than @mozilla.org, such as simply SES and SCL3. Note that it's perfectly acceptable to use a broad SPF list, especially if you use DKIM with your mail. As a side note, I'm pretty sure that SES/Gmail already don't let people send emails with your Return Sender, so spoofing isn't generally a concern. Note that SPF isn't typically not enforced by any major email provider (except as a signal for spam protection) unless you enable DMARC.
(In reply to Alicia Smith [:phrozyn] (use NEEDINFO) from comment #3) > Infosec feels that bugzilla should be sending mail from it's own domain, and > that what is being requested should not be done due the risks associated. Understood, but going to point out that it's currently sending out mail from @mozilla.org
(In reply to April King [:April] from comment #4) > There are a couple problems that we have with doing this: > > [...] > > Probably our best bet is to send emails from > notifications@bugzilla.mozilla.org (or whatever). If we sent from > @bugzilla.mozilla.org, we could set it up with SPF/DKIM/DMARC on the > subdomain and limit mail sending from a less broad list than @mozilla.org, > such as simply SES and SCL3. Note that it's perfectly acceptable to use a > broad SPF list, especially if you use DKIM with your mail. Sure does feel like moving bugzilla e-mails to its own subdomain would probably be a good thing in the long run. Given the historical nature of bugzilla-daemon@mozilla.org however, that might be a tricky thing to pull off. Thanks for the info!
If there's concern about spoofing we should not use SES, send mail from our own MTAs, establish SPF DKIM and a DMARC record and assert in DMARC that email shouldn't be accepted if it isn't signed and comes from our IP range. Otherwise we should include SES's SPF in our DNS as otherwise we're just giving a signal to receiving MTAs that our bugzilla emails may be spoofed.
CCing CloudOps folks, too, since this will eventually end up in their bucket. Do note that changing the email address or domain that Bugzilla uses will be a very painful thing, as it will break whatever filtering users have set up.
BMO's user base would be tremendously inconvenienced by moving bugmail to a different domain.
This will affect thousands of users, and require a communications strategy ahead of time to make sure as many of our users as possible are notified. We would also have to confirm there's no conflicts in BMO code (because the codebase is 20 years old and this is a thing to check for and not assume.) If we were to do this, we'd have re-prioritize other work. From :gene's comment above, it sounds like there is an alternative to SES which does not require a disruptive change.
(In reply to Emma Humphries ☕️ (she/her) [:emceeaich] (UTC-8) +needinfo me from comment #10) > This will affect thousands of users, and require a communications strategy > ahead of time to make sure as many of our users as possible are notified. > > We would also have to confirm there's no conflicts in BMO code (because the > codebase is 20 years old and this is a thing to check for and not assume.) > > If we were to do this, we'd have re-prioritize other work. > > From :gene's comment above, it sounds like there is an alternative to SES > which does not require a disruptive change. Gene's comment above is not practical in AWS right now. As this discussion is around sending mail from AWS it is not a necessary enhancement IMHO (Caveat #1: always willing to discuss of course) (Caveat #2: to quote gozer "Anything is possible, it is a simple matter of engineering). For the moment the SPF records are INFO only and not set to enforce. This was my team attempting to be proactive about security in AWS. As it is not required, and we all have other things to work on, I am going to wontfix this for now. It sounds like InfoSec has some other work around email security and when that comes around we can discuss options at that time.
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → WONTFIX
(In reply to Emma Humphries ☕️ (she/her) [:emceeaich] (UTC-8) +needinfo me from comment #10) > This will affect thousands of users, and require a communications strategy > ahead of time to make sure as many of our users as possible are notified. I would say "thousands or more, we have no way of knowing." > We would also have to confirm there's no conflicts in BMO code (because the > codebase is 20 years old and this is a thing to check for and not assume.) This is not and should not be a concern. The code isn't that bad, and we don't make many assumptions about that.
Bugzilla is now in running AWS and it's been having problems with emails being blocked etc. For this reason, this is now a requirement, no longer a nice to have. What's "a" path forward on this? Changing the from: is not an option for the above reasons.
Blocks: bmo-email-sending
Status: RESOLVED → REOPENED
Resolution: WONTFIX → ---
Additionally, the spf record is currently referencing two other records that do not exist, namely _netblocks.mozilla.com and _netblocks3.mozilla.com
actually, I'm finding that the spf records work intermittantly from google: dig @8.8.8.8 -t txt _netblocks.mozilla.com +short "v=spf1 ip4:63.245.208.0/20 ip4:203.192.141.162/32 ip4:64.213.97.194/32 ip4:64.213.68.131/32 ip4:60.234.66.18/32 ip4:89.202.203.51/32 ip4:146.82.185.165 ip4:206.15.76.125 ip6:2620:101:8000::/40 ~all" dig @8.8.8.8 -t txt _netblocks3.mozilla.com +short "v=spf1 ip4:173.247.198.38/32 ip4:207.126.102.129/32 ip4:206.169.237.184/32 ip4:207.218.72.65/32 ip4:118.163.10.190/32 ip4:60.234.54.74/32 ip4:206.169.237.178/32 ~all" $ dig @208.206.222.222 -t txt _netblocks3.mozilla.com +short ; <<>> DiG 9.13.0 <<>> @208.206.222.222 -t txt _netblocks3.mozilla.com +short ; (1 server found) ;; global options: +cmd ;; connection timed out; no servers could be reached $ dig @208.206.220.220 -t txt _netblocks3.mozilla.com +short ; <<>> DiG 9.13.0 <<>> @208.206.220.220 -t txt _netblocks3.mozilla.com +short ; (1 server found) ;; global options: +cmd ;; connection timed out; no servers could be reached So... maybe not related, but a problem for OpenDNS anyway
cc'ing rtucker for comment/context
I would think that adding the /32 to either netblocks or netblocks3 would resolve the issue, outside of the intermittent OpenDNS issues.
:bobm can you determine the /32's we've been allocated from AWS so we can move forward on this?
Flags: needinfo?(bobm)
(In reply to Chris Kolosiwsky [:ckolos] from comment #18) > :bobm can you determine the /32's we've been allocated from AWS so we can > move forward on this? Our allocated IP addresses are: 54.240.58.224 54.240.58.225 54.240.58.226
Flags: needinfo?(bobm)
currently: netblocks3 is: "v=spf1 ip4:173.247.198.38/32 ip4:207.126.102.129/32 ip4:206.169.237.184/32 ip4:207.218.72.65/32 ip4:118.163.10.190/32 ip4:60.234.54.74/32 ip4:206.169.237.178/32 ~all" I would update it to be: "v=spf1 ip4:173.247.198.38/32 ip4:207.126.102.129/32 ip4:206.169.237.184/32 ip4:207.218.72.65/32 ip4:118.163.10.190/32 ip4:60.234.54.74/32 ip4:206.169.237.178/32 54.240.58.224/32 54.240.58.225/32 54.240.58.226/32 ~all" The RFC states the DATA portion must be 255 characters or less, we're close at 214. I don't see any reason to not do this.
whoops, i forgot the ip4: leaders "v=spf1 ip4:173.247.198.38/32 ip4:207.126.102.129/32 ip4:206.169.237.184/32 ip4:207.218.72.65/32 ip4:118.163.10.190/32 ip4:60.234.54.74/32 ip4:206.169.237.178/32 ip4:54.240.58.224/32 ip4:54.240.58.225/32 ip4:54.240.58.226/32 ~all" 228 characters should still be fine
per irc conversations with ckolos, i'm making the change now.
changes made
I am seeing the SES IP's in the correct place for SPF for resolv in 1.1.1.1 8.8.8.8 8.8.4.4 > do > dig @${resolv} -t txt _netblocks3.mozilla.com +short | grep -o 54.240.58.22[4-6] && echo > done 54.240.58.224 54.240.58.225 54.240.58.226 54.240.58.224 54.240.58.225 54.240.58.226 54.240.58.224 54.240.58.225 54.240.58.226 lgtm!
Status: REOPENED → RESOLVED
Closed: 8 years ago → 7 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.