1472483 - Questions forum email notifications are not sent

Status: VERIFIED FIXED

(support.mozilla.org :: Questions, task)

Description

[Chris Ilias [:cilias]]

Users are not getting email notifications when someone replies to a question in https://support.mozilla.org/questions .

STR:
1. Go to https://support.mozilla.org/en-US/questions/1223947
2. Click on Get Email Updates > When anybody replies, or just post a reply which will automatically subscribe you to the thread.
3. Wait until someone replies (or use a test account to post a reply).


Result: No email notification.

The latest email I've received is June 26th at 7:35PM Eastern time, but I'm not active enough. The first report of this was June 28th at 8:21pm Eastern time.
https://support.mozilla.org/en-US/forums/contributors/713073

NOTE: Emails for private messages are still working.

I've marked this as critical, because unlike bug 1204515, this bug is preventing users from getting notified of answers to their questions. Unless they know the thread URL and manually check it, they are not seeing answers.

Comment 1

[Noah (NoahSUMO/SolidSnake) [:Noah]]

The awesome & mighty Pmac is investigating our troublesome email notifications issue. I'll just CC him to this bug so he can have a place to track his work, if not already. Thanks again for looking into this Pmac! :)

Maybe some of these email improvements scoped out by Giorgos could help Paul here ;)
Improve email sending #3038
https://github.com/mozilla/kitsune/issues/3038

If that's super future work, apologies. Just bringing it up in case now is the right time to implement some of those tasks because of the email outage.

Comment 2

[Josh Mize [:jgmize]]

Attachment: Screenshot_2018-06-30 SES Management Console.png

The SES console showed that that email deliveries had dropped to near 0 since the 6/26, and when I checked the celery logs I saw a bunch of "HTTP Error 429: Too Many Requests" messages from sentry. I restarted all the celery worker pods in all of the kubernetes clusters, and a few minutes later saw that emails are being delivered again according to the SES console (graph attached).

Comment 3

[Chris Ilias [:cilias]]

Thank you Josh! I have received a couple of emails already.

Comment 4

[Alice Wyman]

See also bug 1204515 comment 52 (missing emails for KB articles ready to localize have now been received as well)

Comment 5

[Paul [:pmac] McLanahan]

Nice work Josh! Thanks. I'm calling this one fixed for now. Please reopen if you see this issue again in the next couple days.

Comment 6

[Chris Ilias [:cilias]]

One last test for latency at https://support.mozilla.org/en-US/questions/1223947#answer-1129116, and I can verify that the bug is fixed. The email arrived within a minute after my post.

Comment 7

[guigs [:guigs] PST (please needinfo me!)]

Hey there, I was talking to the team this morning about possibly having an alert for email notifications. (other than email)
What are your thoughts on having an alert sent out if this happens >= 24 hours or >=48 hours and no email notifications are sent out?

The use case here would be support does not have much of a back up notifier if the email feature is delayed. I can think of work arounds:
1. If a user keeps this filter open:(my contributions, ordered by updates) https://support.mozilla.org/en-US/questions/firefox?owner=mine&order=updated 
2. An alert on the site if more that (time passed since email updates sent)
3. Pinging this bug when it happens again

From what I observed these methods of communication did work: 
1. Contributors notice no email after 24 hours and email an admin or another #sumo irc contributor
2. Pinging in Slack from NDA contributors with Slack access or a SUMO admin 
3. Posts in the Contributor Forum updating the issue and updates here in this bug. 

What do you think?

Comment 8

[Paul [:pmac] McLanahan]

Agreed. We are currently discussing ways of adding better monitoring and alerting for this part of the site.