1015037 - Deploy production PulseGuardian

Status: RESOLVED FIXED

(Webtools :: Pulse, defect, P1)

Description

[Mark Côté [:mcote]]

Tracking bug for the work items required to get PulseGuardian[1] into production.

[1] https://wiki.mozilla.org/Auto-tools/Projects/Pulse/PulseGuardian

Comment 1

[Mark Côté [:mcote]]

SecReview was completed in bug 1011628 and all issues raised there have been fixed.  Can we deploy this to production pulse now?

There isn't a huge rush here, since I think we'll want to switch over to the new permissions model first, or at least have PulseGuardian create users with the correct permissions first.  I'll be filing separate bugs for rolling out the permissions, to pulse-dev first.

Comment 2

[Mark Côté [:mcote]]

Now that bug 1011633 is fixed, we can go ahead and roll out to pulse.mozilla.org!

I think we can take down the current pulse.mozilla.org website, which is a django app (http://hg.mozilla.org/automation/pulsewebsite/).  Not all the info has been transferred over from the old website to the pulseguardian website (see bug 1017957), but I don't think anything there is so valuable that we need to transfer it over first.

Comment 3

[Mark Côté [:mcote]]

Don't think there's much point in creating a separate blocker bug for actual deployment, so assigning this over.

Comment 4

[C. Liang [:cyliang]]

The Pulse Guardian process appears to be running on the production Pulse app server and I've switched over the web site to run the Pulse Guardian WSGI script.  However, I'm getting an error when trying to log in via Persona:

  Login error: Couldn't connect to the Persona verifier (https://verifier.login.persona.org/verify)

I verified that Persona login does work in Dev.

Is there something I need to do RE: setting up flask?  I don't think I did anything put install it on dev so there may be a step or two that I'm missing that Ahmed may have performed.

Comment 5

[Mark Côté [:mcote]]

Hm, yeah, probably the persona_audience variable in pulseguardian/config.py.  Might want to check what it's set to on dev.

Comment 6

[C. Liang [:cyliang]]

The actual persona_audience line is the same; I don't know if more has to be done RE: generation of flask keys on production.


Dev:

# Web app
flask_host = 'pulse-dev.allizom.org'
flask_port = 443
flask_secret_key = '<REDACTED>'
flask_debug_mode = True

# Persona
persona_verifier = 'https://verifier.login.persona.org/verify'
persona_audience = 'https://{0}:{1}'.format(flask_host, flask_port)


Prod:

# Web app
flask_host = 'pulse.mozilla.org'
flask_port = 443
flask_secret_key = '<REDACTED>'
flask_debug_mode = True

# Persona
persona_verifier = 'https://verifier.login.persona.org/verify'
persona_audience = 'https://{0}:{1}'.format(flask_host, flask_port)

Comment 7

[Mark Côté [:mcote]]

Ah, maybe it's a routing issue?  Did we have to open something specific for pulse-dev to talk to persona.org?

Comment 8

[Mark Côté [:mcote]]

Apparently we ran into the same problem when setting up dev (bug 1011633 comment 9).  Isn't clear exactly how we resolved it though.

Comment 9

[C. Liang [:cyliang]]

I think the resolution was a combination of:
   1. getting the server to talk to the verifier via the datacenter proxy and
   2. fixing an issue with the blank identifier being fed to format()

I did two things to address #1: 
  - add specific http[s]_proxy environment variables to pulseguardian.wsgi file (already done in prod)
  - set system-wide use of the data-center proxy [1]

Enforcing the system-wide proxy will require a reboot of pulse-app1.dmz.phx1.  I'm assuming that a reboot of this server will cause an outage of the Pulse service; if that's not the case, I should be able to reboot the server today.  Otherwise, we'll need to arrange for a reboot.

[1] https://mana.mozilla.org/wiki/display/SECURITY/Configuring+Your+Servers+to+Use+Proxies

Comment 10

[Mark Côté [:mcote]]

Hm okay.  This shouldn't be a big problem, but we should give an heads-up to the regular consumers.  I'll send out an email and maybe we can reboot tomorrow morning.

Comment 11

[C. Liang [:cyliang]]

Further testing shows that Persona (correctly) works if you visit https://pulse.mozilla.org and (also correctly) fails if you visit http://pulse.mozilla.org.  Opened a separate bug (1067690) to address that issue.

Comment 12

[Mark Côté [:mcote]]

Also note that Puppet is still controlling users on Pulse, so any users created via Pulse Guardian will be deleted by Puppet.  We just need to make sure Pulse Guardian can somehow recreate users before shutting Puppet off.

Comment 13

[Mark Côté [:mcote]]

I'd actually like to keep this open until bug 1068419 is fixed, since it prevents Pulse Guardian from doing one of its main jobs (registering users).  Bug 1067690 is definitely nice to have but not technically a blocker.

Comment 14

[C. Liang [:cyliang]]

As per IRC conversation this morning, I'm closing this bug as DONE.