Unable to log in to Persona (production/staging)

RESOLVED FIXED

Status

Cloud Services
Server: Identity
--
critical
RESOLVED FIXED
3 years ago
3 years ago

People

(Reporter: stephend, Assigned: relud)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [fromAutomation], URL)

Attachments

(2 attachments)

Created attachment 8602374 [details]
persona-staging-issues.png

Web QA started noticing Persona login issues around 2:29pm PDT, beginning with our One and Done staging test-automation, and continuing as this bug is being written up.

STR:

1. Load https://login.persona.org/ or http://beta.123done.org/ and sign in
2. Enter your Mozilla LDAP email address, and click "Next"

Actual:

See screenshots (from staging and production) - you likely won't be able to sign in
Created attachment 8602375 [details]
production failure to log in

Comment 2

3 years ago
This is a known issue that is being actively worked on, see http://status.persona.org/.

Updated

3 years ago
Component: Operations → Server: Identity
(Assignee)

Updated

3 years ago
Assignee: nobody → dthornton
(Assignee)

Comment 3

3 years ago
regarding logging in to staging with @mozilla.com and @mozilla.org addresses, that's a separate issue. I believe that login.mozilla.org does not allow anosrep.org to access it.
(Assignee)

Comment 4

3 years ago
An unidentified failure caused persona to stop working in prod. Prod was rolled back to a previous deployment, and is now working again. User signups and password changes since 3pm PDT 4/20/15 were lost in the process.
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
(In reply to Daniel Thornton [:relud] from comment #4)
> An unidentified failure caused persona to stop working in prod. Prod was
> rolled back to a previous deployment, and is now working again. User signups
> and password changes since 3pm PDT 4/20/15 were lost in the process.

1) What followup has or will be done to investigate this "unidentified failure"?  Is there and/or will there be a post-mortem meeting scheduled to try to ascertain what the real problem(s) was/were?

(In reply to Daniel Thornton [:relud] from comment #3)
> regarding logging in to staging with @mozilla.com and @mozilla.org
> addresses, that's a separate issue. I believe that login.mozilla.org does
> not allow anosrep.org to access it.

2) That's interesting, because according to the non-permalinked entry on http://status.persona.org/, it states, "Our database server has started misbehaving, it's under investigation. This is affecting users with the email fallback, as well as users on the mozilla idp."  That seems to differ from your assertion.  Can we dig into this, to be sure?

I and my team -- and others, I'm sure, are more than willing/ready, to help dig into/understand a nearly 3-hour production/staging-facing issue

3) None of these issues were readily known to #moc (because https://status.mozilla.org/ and http://status.persona.org/ are separate -- and the latter didn't mention a bug #) until I filed a bug -- nearly an hour later(!) -- which is something we should definitely try to address (happy to file a separate bug!) - see logs from #moc for examples

I really do want to try to help close the gap on both the information/reporting/time-to-fix side, so please don't assume any snark!  <3
Blocks: 1162410
Blocks: 1162391
(Assignee)

Comment 6

3 years ago
The unidentified 1&2) A config change meant for stage landed in prod. The change was located, and removed. No data has been lost. Persona is working again. Production has not been rolled back to any earlier stacks. It was not related to our database. It affected all users.

3) status.mozilla.org isn't normally used for persona, nor is #moc. please watch status.persona.org or subscribe to the dev-identity@mozilla.org list for updates regarding persona.
You need to log in before you can comment on or make changes to this bug.