1438206 - (bmo-cloud-emails) Process SES email bounces properly

Status: RESOLVED FIXED

(bugzilla.mozilla.org :: Email Notifications, enhancement)

Description

[Dylan Hardison [:dylan] (he/him)]

We need to respond to SES bounces by disabling the account ( or at least disabling bugmail ).

Comment 1

[Dylan Hardison [:dylan] (he/him)]

Adding a link to the SES docs

Comment 2

[Dylan Hardison [:dylan] (he/him)]

Is this a hard requirement for https://bugzilla.mozilla.org/show_bug.cgi?id=bmo-cloud-migration ?

Comment 3

[Chris Kolosiwsky [:ckolos] (ckolos has left the building)]

In as much that violation of their bounce metrics means having SES disabled for the entire IAM, yes.

Comment 4

[:glob ✱]

what we need to do is hook up ses bounce messages to sns queues.

we'll need two topic/queue pairs: one to handle bounce notifications, and another to handle complaint notifications.
in both cases we'll receive a json message containing the email address - https://docs.aws.amazon.com/ses/latest/DeveloperGuide/notification-examples.html

we'd also need a daemon which subscribes to the queues and takes action depending on the queue/message.


complaints are where someone has complained to their isp about receiving email; for simplicity these should probably be forwarded to bmo-admin for manually processing.


bounces can be either transient or permanent.

transient bounces should just be logged.
permanent bounces should result in the account being disabled, either outright or by just disabling bugmail.

the big question is if we should immediately disable an account when we get a bounce, or if we should track the number of bounces over a set period of time and only deactivate when a threshold is crossed.

ses does differentiate between hard and soft bounces - https://aws.amazon.com/blogs/ses/email-definitions-bounces/
hard bounces include mailbox/server doesn't exist.
a example of a soft bounce is when a mailbox is full; ses will retry and eventually fall back to a hard bounce.  i suspect ses treats greylisting as a soft bounce.

given resilience/queuing/retry is built into email i find it hard to argue for adding what could be a reasonable amount of effort to handle misconfigured servers.


the options are:


(1) disable the account upon first bounce

ensure the disable-text is clear to both the user what happened, when, why, and what actions they need to take to reopen their account.

this is my preferred option.


(2) disable bugmail upon first bounce, expose to user

this is a softer approach that outright account disabling, however it would require more development work than (1) as there isn't any ability to store a reason for bugmail being disabled, nor is this setting exposed to the user in any way.

if implemented ideally this should add an overlay icon on the user's avatar in the header, with a new menu item to indicate their bugmail is disabled.  selecting this menu item should show on the email preferences screen when and why bugmail was disabled, with the ability for them to re-enable it.

of note disabling bugmail in this way will not stop bugzilla from sending all emails to that address - password reset and other non-bug-related emails will still be sent.


(3) disable the account when it passes N bounces within a set time period

track bounces for each account, and only disable once we pass a set threshold (eg. more than 3 bounces over 3 days).

this would involve some work on the backend to manage bounce rate data, and ideally a similar warning to (2) on a user's avatar to indicate mail delivery issues.

this option is much more lenient towards misconfigured email systems than (1).

Comment 5

[:glob ✱]

(In reply to Byron Jones ‹:glob› from comment #4)
> we'd also need a daemon which subscribes to the queues and takes action
> depending on the queue/message.

nope, we'll need a https endpoint that sns can post to - https://docs.aws.amazon.com/sns/latest/dg/SendMessageToHttp.html

Comment 6

[:glob ✱]

after a discussion with dylan we're going with option (1) disable bugmail upon first bounce.

Comment 7

[:glob ✱]

Attachment: pr

Comment 8

[:glob ✱]

setup steps:

1. set values for `ses_username` and `ses_password` in localconfig
2. create a single sns topic and configure ses so both bounces and complaints are delivered to that topic
3. configure the sns topic to https post json to the endpoint:
      https://$ses_username:$[e-mail]/ses/
4. watch the logs

the ses endpoint should automatically confirm the topic subscription.

https://docs.aws.amazon.com/ses/latest/DeveloperGuide/mailbox-simulator.html has test email addresses that will generate specific ses responses.

a complaint should result in an email to bugzilla-admin@
a bounce should disable the bugzilla account