r2 says he'll talk with amazon about the concerns of using amazon SES in a DR situation
Summary: Discuss scenario with Amazon and understand the likelihood of them cutting off our ability to send emails do to bounces → Discuss scenario with Amazon and understand the likelihood of them cutting off our ability to send emails due to bounces
Gozer is going to get me the number of bounces we experience withing a set period of time.
Looking at mailman1.mail.scl3:/var/spool/mail/bugzilla First message recieved on: Thu, 7 Apr 2016 17:46:11 +0000 (UTC) Total mail count in there : 2,665,646
I've reach out to Amazon about this and am waiting to hear back.
Amazon wants to know the bounce rate as a percentage of bmo's total outgoing mail. Do we know how many messages bmo sends each day? At some point we came up with 500,000 as a limit we needed for SES, but I don't know if that reflects the actual amount we send.
I have some additional information: Amazon's guidelines call for us to stay below 5% of our total mail. Additionally, from the FAQ: "Q3. What types of bounces count toward my bounce rate? "Your bounce rate includes only hard bounces to domains you have not verified. Hard bounces are permanent delivery failures such as "address does not exist." Temporary and intermittent failures such as "mailbox full," or bounces due to blocked IP addresses, do not count toward your bounce rate." If a significant portion of our bounced mail is to mozilla.com or mozilla.org email addresses, it seems likely that we are will under the 5% threshold. However, I have a getinfo open to :dylan to verify the number of messages the system sends each day.
gozer did some analysis of the bounced mail and learned that 7 addresses account for the vast majority of bounces. In the short term, I suggest disabling mail to those addresses. This should greatly lessen the number of bounces. We can easily get back in the same situation, so bmo still needs a process to deal with bounced mail. This process should be automated, but even a manual process is better than none in this case. As of Thu Apr 7 17:46:43 2016, the stats of: mailman1.mail.scl3.mozilla.com:/var/spool/mail/bugzilla total size: 20G total bounces: 2,770,164 distinct email addresses bounced: 3,935 Top 85: 470356 firstname.lastname@example.org 428793 email@example.com 419241 firstname.lastname@example.org 320843 email@example.com 295364 firstname.lastname@example.org 232245 email@example.com 231694 firstname.lastname@example.org 81643 email@example.com 81643 firstname.lastname@example.org 61578 email@example.com 53213 firstname.lastname@example.org 18246 email@example.com 8751 firstname.lastname@example.org 4143 email@example.com 4069 firstname.lastname@example.org 3780 email@example.com 3239 firstname.lastname@example.org 2821 email@example.com 2468 firstname.lastname@example.org 1829 email@example.com 1366 firstname.lastname@example.org 1079 email@example.com 998 firstname.lastname@example.org 890 email@example.com 822 Sergioegomez1980@gmail.com 821 firstname.lastname@example.org 821 email@example.com 821 firstname.lastname@example.org 821 email@example.com 821 firstname.lastname@example.org 821 email@example.com 820 firstname.lastname@example.org 819 email@example.com 819 firstname.lastname@example.org 819 email@example.com 816 firstname.lastname@example.org 816 email@example.com 816 firstname.lastname@example.org 761 email@example.com 666 firstname.lastname@example.org 625 email@example.com 616 firstname.lastname@example.org 573 email@example.com 477 firstname.lastname@example.org 464 email@example.com 451 firstname.lastname@example.org 441 email@example.com 418 firstname.lastname@example.org 363 email@example.com 350 firstname.lastname@example.org 322 email@example.com 299 firstname.lastname@example.org 284 email@example.com 277 firstname.lastname@example.org 275 email@example.com 245 firstname.lastname@example.org 211 email@example.com 193 firstname.lastname@example.org 192 email@example.com 190 firstname.lastname@example.org 183 pbylenga@QAnalydocs.com 178 email@example.com 178 firstname.lastname@example.org 174 email@example.com 165 firstname.lastname@example.org 157 email@example.com 155 firstname.lastname@example.org 155 email@example.com 151 firstname.lastname@example.org 146 email@example.com 140 firstname.lastname@example.org 139 email@example.com 138 firstname.lastname@example.org 137 email@example.com 134 firstname.lastname@example.org 133 email@example.com 130 firstname.lastname@example.org 116 email@example.com 104 firstname.lastname@example.org 104 email@example.com 104 firstname.lastname@example.org 102 email@example.com 100 firstname.lastname@example.org 100 email@example.com 100 firstname.lastname@example.org
We're handling these specific emails now. We'll want to automate this, and automating it can be specific to AWS. There is a good perl client for Amazon's SNS, which is the preferred way of handling bounces in SES.
I've spoken again with Amazon. They will not immediately shut down our access to SES for excessive email bounces. They will send a warning email to the email address associated with the account. If we don't resolve it they will then escalate it to our account team who will attempt to contact us. They will only shut it down if we are completely unresponsive to their warnings and only with the permission of our own account team.
mcote: is the r2's's info sufficient to R/F this?
From my perspective, this is sufficient. However we should file a bug for comment 7. Dylan can you do this?
Flags: needinfo?(mcote) → needinfo?(dylan)
See Also: → bug 1317399
Any update here? According to bug 1308228 we're blocked by this.
This is fine with comment 11, ill close it out now
Status: NEW → RESOLVED
Last Resolved: 8 months ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.