1162286 - Unable to log in to Persona (production/staging)

Status: RESOLVED FIXED

(Cloud Services :: Server: Identity, defect)

Description

[Stephen Donner [:stephend] Not actively reading bugmail]

Attachment: persona-staging-issues.png

Web QA started noticing Persona login issues around 2:29pm PDT, beginning with our One and Done staging test-automation, and continuing as this bug is being written up.

STR:

1. Load https://login.persona.org/ or http://beta.123done.org/ and sign in
2. Enter your Mozilla LDAP email address, and click "Next"

Actual:

See screenshots (from staging and production) - you likely won't be able to sign in

Comment 1

[Stephen Donner [:stephend] Not actively reading bugmail]

Attachment: production failure to log in

Comment 2

[Wesley Dawson [:whd]]

This is a known issue that is being actively worked on, see http://status.persona.org/.

Comment 3

[Daniel Thorn [:relud]]

regarding logging in to staging with @mozilla.com and @mozilla.org addresses, that's a separate issue. I believe that login.mozilla.org does not allow anosrep.org to access it.

Comment 4

[Daniel Thorn [:relud]]

An unidentified failure caused persona to stop working in prod. Prod was rolled back to a previous deployment, and is now working again. User signups and password changes since 3pm PDT 4/20/15 were lost in the process.

Comment 5

[Stephen Donner [:stephend] Not actively reading bugmail]

(In reply to Daniel Thornton [:relud] from comment #4)
> An unidentified failure caused persona to stop working in prod. Prod was
> rolled back to a previous deployment, and is now working again. User signups
> and password changes since 3pm PDT 4/20/15 were lost in the process.

1) What followup has or will be done to investigate this "unidentified failure"?  Is there and/or will there be a post-mortem meeting scheduled to try to ascertain what the real problem(s) was/were?

(In reply to Daniel Thornton [:relud] from comment #3)
> regarding logging in to staging with @mozilla.com and @mozilla.org
> addresses, that's a separate issue. I believe that login.mozilla.org does
> not allow anosrep.org to access it.

2) That's interesting, because according to the non-permalinked entry on http://status.persona.org/, it states, "Our database server has started misbehaving, it's under investigation. This is affecting users with the email fallback, as well as users on the mozilla idp."  That seems to differ from your assertion.  Can we dig into this, to be sure?

I and my team -- and others, I'm sure, are more than willing/ready, to help dig into/understand a nearly 3-hour production/staging-facing issue

3) None of these issues were readily known to #moc (because https://status.mozilla.org/ and http://status.persona.org/ are separate -- and the latter didn't mention a bug #) until I filed a bug -- nearly an hour later(!) -- which is something we should definitely try to address (happy to file a separate bug!) - see logs from #moc for examples

I really do want to try to help close the gap on both the information/reporting/time-to-fix side, so please don't assume any snark!  <3

Comment 6

[Daniel Thorn [:relud]]

The unidentified 1&2) A config change meant for stage landed in prod. The change was located, and removed. No data has been lost. Persona is working again. Production has not been rolled back to any earlier stacks. It was not related to our database. It affected all users.

3) status.mozilla.org isn't normally used for persona, nor is #moc. please watch status.persona.org or subscribe to the [e-mail] list for updates regarding persona.