869588 - mdn: update EMAIL_HOST_* settings

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task, P3)

Description

[Luke Crouch [:groovecoder]]

MDN will soon start sending lots of emails. Currently, it would use the localhost smtp server on the mdn cluster(?). We need to change the:

EMAIL_HOST
EMAIL_HOST_USER
EMAIL_HOST_PASSWORD

values to the socketlabs values as currently used by SUMO.

Comment 1

[Chris Turra [:cturra]]

i have pushed these settings/local.py config changes as requested (using sumo as a template).

Comment 2

[Luke Crouch [:groovecoder]]

I don't think it's working - server error emails have stopped and when I tried to edit a page that sends an email notification it hung. (which means we need to finish bug 766627 too!)

We need to revert it unless/until we're sure it works since broken EMAIL_HOST kills server error emails AND any edits on pages with email watchers.

Comment 3

[Luke Crouch [:groovecoder]]

Also, let's make sure we get this working on dev and stage first, then do prod.

Comment 4

[Brandon Burton [:solarce]]

I've commented out these settings and deployed them

[[e-mail]3 ~]# /data/developer/deploy developer.mozilla.org
[2013-05-07 19:02:10] Running rsync_project
[2013-05-07 19:02:10] [localhost] running: /usr/bin/rsync -aq --include '.gitkeep' --exclude '.git*' --exclude '.hg*' --exclude '.svn*' --exclude 'CVS' --exclude '.bzr*' --delete /data/developer/src/developer.mozilla.org/ /data/developer/www/developer.mozilla.org/
[2013-05-07 19:02:19] [localhost] finished: /usr/bin/rsync -aq --include '.gitkeep' --exclude '.git*' --exclude '.hg*' --exclude '.svn*' --exclude 'CVS' --exclude '.bzr*' --delete /data/developer/src/developer.mozilla.org/ /data/developer/www/developer.mozilla.org/ (8.358s)
[2013-05-07 19:02:19] Finished rsync_project (8.359s)
[2013-05-07 19:02:19] Running push_www
[2013-05-07 19:02:19] [developer1.webapp.scl3.mozilla.com] running: /data/bin/update-www.sh developer.mozilla.org
[2013-05-07 19:02:19] [developer2.webapp.scl3.mozilla.com] running: /data/bin/update-www.sh developer.mozilla.org
[2013-05-07 19:02:19] [developer3.webapp.scl3.mozilla.com] running: /data/bin/update-www.sh developer.mozilla.org
[2013-05-07 19:02:20] [developer2.webapp.scl3.mozilla.com] finished: /data/bin/update-www.sh developer.mozilla.org (0.982s)
[2013-05-07 19:02:20] [developer1.webapp.scl3.mozilla.com] finished: /data/bin/update-www.sh developer.mozilla.org (1.536s)
[2013-05-07 19:02:20] [developer3.webapp.scl3.mozilla.com] finished: /data/bin/update-www.sh developer.mozilla.org (1.744s)
[2013-05-07 19:02:20] Finished push_www (1.751s)

Please confirm the email functionality works again

Comment 5

[Brandon Burton [:solarce]]

(In reply to Luke Crouch [:groovecoder] from comment #3)
> Also, let's make sure we get this working on dev and stage first, then do
> prod.

Are you ready to test this on dev?

Comment 6

[Luke Crouch [:groovecoder]]

Yup, we can make the EMAIL_HOST_* settings changes on both dev and stage now and I'll test on both.

Comment 7

[Brandon Burton [:solarce]]

(In reply to Luke Crouch [:groovecoder] from comment #6)
> Yup, we can make the EMAIL_HOST_* settings changes on both dev and stage now
> and I'll test on both.

Pushed to dev and stage

[[e-mail]3 ~]# /data/developer-dev/deploy developer-dev.allizom.org
[2013-05-08 14:37:56] Running rsync_project
[2013-05-08 14:37:56] [localhost] running: /usr/bin/rsync -aq --include '.gitkeep' --exclude '.git*' --exclude '.hg*' --exclude '.svn*' --exclude 'CVS' --exclude '.bzr*' --delete /data/developer-dev/src/developer-dev.allizom.org/ /data/developer-dev/www/developer-dev.allizom.org/
[2013-05-08 14:38:00] [localhost] finished: /usr/bin/rsync -aq --include '.gitkeep' --exclude '.git*' --exclude '.hg*' --exclude '.svn*' --exclude 'CVS' --exclude '.bzr*' --delete /data/developer-dev/src/developer-dev.allizom.org/ /data/developer-dev/www/developer-dev.allizom.org/ (4.122s)
[2013-05-08 14:38:00] Finished rsync_project (4.123s)
[2013-05-08 14:38:00] Running push_www
[2013-05-08 14:38:00] [developer1.dev.webapp.scl3.mozilla.com] running: /data/bin/update-www.sh developer-dev.allizom.org
[2013-05-08 14:38:13] [developer1.dev.webapp.scl3.mozilla.com] finished: /data/bin/update-www.sh developer-dev.allizom.org (13.713s)
[developer1.dev.webapp.scl3.mozilla.com] out: nodejs-kumascript: stopped
[developer1.dev.webapp.scl3.mozilla.com] out: celeryd-kuma: stopped
[developer1.dev.webapp.scl3.mozilla.com] out: celeryd-kuma: started
[developer1.dev.webapp.scl3.mozilla.com] out: nodejs-kumascript: started
[2013-05-08 14:38:13] Finished push_www (13.714s)
[[e-mail]3 ~]# /data/developer-stage/deploy developer.allizom.org
[2013-05-08 14:38:41] Running rsync_project
[2013-05-08 14:38:41] [localhost] running: /usr/bin/rsync -aq --include '.gitkeep' --exclude '.git*' --exclude '.hg*' --exclude '.svn*' --exclude 'CVS' --exclude '.bzr*' --delete /data/developer-stage/src/developer.allizom.org/ /data/developer-stage/www/developer.allizom.org/
[2013-05-08 14:38:47] [localhost] finished: /usr/bin/rsync -aq --include '.gitkeep' --exclude '.git*' --exclude '.hg*' --exclude '.svn*' --exclude 'CVS' --exclude '.bzr*' --delete /data/developer-stage/src/developer.allizom.org/ /data/developer-stage/www/developer.allizom.org/ (6.491s)
[2013-05-08 14:38:47] Finished rsync_project (6.492s)
[2013-05-08 14:38:47] Running push_www
[2013-05-08 14:38:47] [developer1.stage.webapp.scl3.mozilla.com] running: /data/bin/update-www.sh developer.allizom.org
[2013-05-08 14:38:49] [developer1.stage.webapp.scl3.mozilla.com] finished: /data/bin/update-www.sh developer.allizom.org (1.220s)
[2013-05-08 14:38:49] Finished push_www (1.221s)

Comment 8

[Luke Crouch [:groovecoder]]

Nope, socketlabs doesn't work. Do we need to open a network flow for outbound SMTP?

Comment 9

[Brandon Burton [:solarce]]

(In reply to Luke Crouch [:groovecoder] from comment #8)
> Nope, socketlabs doesn't work. Do we need to open a network flow for
> outbound SMTP?

Flows have been added, please test again

Comment 10

[Luke Crouch [:groovecoder]]

Looks good. Let's do stage now.

Comment 11

[Luke Crouch [:groovecoder]]

nvm, tested stage already. now we can try prod.

Comment 12

[Luke Crouch [:groovecoder]]

note: looks like we need to update prod EMAIL_HOST_* settings AND CELERY_ALWAYS_EAGER.

Comment 13

[Brandon Burton [:solarce]]

(In reply to Luke Crouch [:groovecoder] from comment #12)
> note: looks like we need to update prod EMAIL_HOST_* settings AND
> CELERY_ALWAYS_EAGER.

I've pushed the email settings and celery settings live

Comment 14

[Luke Crouch [:groovecoder]]

Hmm ... prod test didn't work? And I can't see the /var/log/celeryd-kuma.log file to see if it was celery or smtp that broke. :/

Comment 15

[Brandon Burton [:solarce]]

(In reply to Luke Crouch [:groovecoder] from comment #14)
> Hmm ... prod test didn't work? And I can't see the /var/log/celeryd-kuma.log
> file to see if it was celery or smtp that broke. :/

Yeah, I forgot developer-celery1.webapp in the flow needs, I've opened bug 870862 to get this flow in place

I've disabled the settings and pushed that out, in the mean time

Comment 16

[Brandon Burton [:solarce]]

This was successfully pushed and tested

12:50:32     @groovecoder | solarce: w00t!
12:50:49          solarce | [2013-05-16 09:48:37,783: DEBUG/PoolWorker-7] Sending edited notification email for document (id=4967)
12:50:49          solarce | [2013-05-16 09:48:38,091: INFO/MainProcess] Task tidings.events._fire_task[844ecf6d-ccbe-4dd4-b33e-55116bc2fe52] succeeded in 0.630956888199s: None
12:51:02     @groovecoder | yup, I got the email! interestingly, I also got the previous email notifications! :)