Discuss scenario with Amazon and understand the likelihood of them cutting off our ability to send emails due to bounces




2 years ago
8 months ago


(Reporter: dylan, Assigned: r2)


Dependency tree / graph




2 years ago
r2 says he'll talk with amazon about the concerns of using amazon SES in a DR situation


2 years ago
Assignee: nobody → riweiss


2 years ago
Summary: Discuss scenario with Amazon and understand the likelihood of them cutting off our ability to send emails do to bounces → Discuss scenario with Amazon and understand the likelihood of them cutting off our ability to send emails due to bounces

Comment 1

2 years ago
Gozer is going to get me the number of bounces we experience withing a set period of time.
Flags: needinfo?(gozer)
Flags: needinfo?(gozer)
Looking at mailman1.mail.scl3:/var/spool/mail/bugzilla
First message recieved on: Thu,  7 Apr 2016 17:46:11 +0000 (UTC)

Total mail count in there : 2,665,646

Comment 3

2 years ago
I've reach out to Amazon about this and am waiting to hear back.

Comment 4

2 years ago
Amazon wants to know the bounce rate as a percentage of bmo's total outgoing mail. Do we know how many messages bmo sends each day? At some point we came up with 500,000 as a limit we needed for SES, but I don't know if that reflects the actual amount we send.
Flags: needinfo?(dylan)

Comment 5

2 years ago
I have some additional information: 

Amazon's guidelines call for us to stay below 5% of our total mail. Additionally, from the FAQ:
"Q3. What types of bounces count toward my bounce rate?

"Your bounce rate includes only hard bounces to domains you have not verified. Hard bounces are permanent delivery failures such as "address does not exist." Temporary and intermittent failures such as "mailbox full," or bounces due to blocked IP addresses, do not count toward your bounce rate."

If a significant portion of our bounced mail is to mozilla.com or mozilla.org email addresses, it seems likely that we are will under the 5% threshold. However, I have a getinfo open to :dylan to verify the number of messages the system sends each day.

Comment 6

2 years ago
gozer did some analysis of the bounced mail and learned that 7 addresses account for the vast majority of bounces. In the short term, I suggest disabling mail to those addresses. This should greatly lessen the number of bounces. We can easily get back in the same situation, so bmo still needs a process to deal with bounced mail. This process should be automated, but even a manual process is better than none in this case.

As of Thu Apr  7 17:46:43 2016, the stats of:


total size: 20G
total bounces: 2,770,164
distinct email addresses bounced: 3,935

Top 85:

 470356 gavin.bugzilla@gmail.com
 428793 breakthesystem+bugzilla@gmail.com
 419241 blackholebugzilla@gmail.com
 320843 anaconda614@hotmail.com
 295364 jaime.bugzilla@gmail.com
 232245 aw.bulk@gmail.com
 231694 iraztest@gmail.com
  81643 martinknapton@googlemail.com
  81643 diormoscow@aol.com
  61578 fabulousaleyia@live.com
  53213 zzaaqqss@hotmail.com
  18246 dom_bug_listener@mozilla.jstenback.com
   8751 benc@meer.net
   4143 lloyd@flock.com
   4069 channig2@student.fitchburgstate.edu
   3780 gaiatest@doliver.net
   3239 dentyliu@21cn.com
   2821 aesteve@estudioii.com
   2468 sharif.mra@gmail.com
   1829 asop.ah@live.it
   1366 apaiseaman@gmail.com
   1079 jasun50@live.com
    998 claudemir@flycrm.com.br
    890 misscpluff@hotmail.com
    822 Sergioegomez1980@gmail.com
    821 zeelshah67@yahoo.in
    821 subbumagam2012@gmail.com
    821 renesolis29@gmail.com
    821 rawand_kamal52@yahoo.com
    821 beetlearound@live.com
    821 ankush@m-techindia.com
    820 manishwalia85@yahoo.com
    819 pepinsexy75@hotmail.com
    819 bugzilla_seamon@sina.com
    819 al-mojaheden@hotmail.com
    816 supanutotun@hotmail.co.th
    816 shera831@hotmail.com
    816 b_o_a_t_6@hotmail.com
    761 formbugs@johnkeiser.com
    666 njtan11@aol.com
    625 jrgmorrison@aol.com
    616 rasika.shahane@cumminscollege.in
    573 nidaisfree@hotmail.com
    477 cvevans@users.sourceforge.net
    464 niteshtiwari@hotmail.com
    451 katarzyna1212.77@o2.pl
    441 aja+bugzilla@spamcop.net
    418 yoshiyuki@hpc.cs.ehime-u.ac.jp
    363 nsstrack-ext@sun.com
    350 sdwilsh@forerunnerdesigns.com
    322 tongkatali188@myopera.com
    299 acoustikat@yahoo.com
    284 admin@angosso.net
    277 22644@rss.xfruits.com
    275 khokhar_cth@live.com
    245 paul.pasca@softvision.ro
    211 sekundes@gmail.com
    193 stevem@webcommons.biz
    192 cohesion@sleepyhead.org
    190 shawnlandden@gmail.com
    183 pbylenga@QAnalydocs.com
    178 mihai.morar@softvision.ro
    178 kyferez@hotmail.com
    174 rboard2@roadrunner.com
    165 karla.merza@softvision.ro
    157 magicdice@hotmail.com
    155 psokol@jabster.pl
    155 cyborg42@indamail.hu
    151 gaurav.a.sarin@capgemini.com
    146 remus.pop@softvision.ro
    140 stoopidwalrus@gmail.com
    139 soso651988@hotmail.com
    138 kevinpetersen87@yahoo.com
    137 thajsta@gmail.com
    134 tmptgr@hotmail.com
    133 1006682961@qq.com
    130 vashistshashank@yahoo.in
    116 iwritecopy@mac.com
    104 xanthor+bz@nuinet.com
    104 grodno@ymail.com
    104 bugzilla@leeching.net
    102 patrickbp@boydcomputersvcs.com
    100 user277536@aol.com
    100 cocomoo@cox.net
    100 ax5ovr4i6p@sxipper.net

Comment 7

2 years ago
We're handling these specific emails now. We'll want to automate this, and automating it can be specific to AWS.
There is a good perl client for Amazon's SNS, which is the preferred way of handling bounces in SES.
Flags: needinfo?(dylan)

Comment 8

2 years ago
I've spoken again with Amazon. They will not immediately shut down our access to SES for excessive email bounces. They will send a warning email to the email address associated with the account. If we don't resolve it they will then escalate it to our account team who will attempt to contact us. They will only shut it down if we are completely unresponsive to their warnings and only with the permission of our own account team.


2 years ago
Blocks: 1308228
mcote: is the r2's's info sufficient to R/F this?
Flags: needinfo?(mcote)
From my perspective, this is sufficient.  However we should file a bug for comment 7.  Dylan can you do this?
Flags: needinfo?(mcote) → needinfo?(dylan)

Comment 11

a year ago
Filed as https://bugzilla.mozilla.org/show_bug.cgi?id=1317399
Flags: needinfo?(dylan)
See Also: → bug 1317399

Comment 12

8 months ago
Any update here? According to bug 1308228 we're blocked by this.

Comment 13

8 months ago
This is fine with comment 11, ill close it out now
Last Resolved: 8 months ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.