820446 - issue with bugzilla and persona login 2012/12/10

Status: RESOLVED FIXED

(Cloud Services :: Server: Identity, defect)

Description

[John Morrison [:jrgm]]

Monday 2012-12-10 while rolling out train-2012.11.23 to production there was
an issue with login to bugzilla with persona for ~5 minutes. We rolled back
immediately.

The error returned (to me) said 'Invalid Username Or Password'.

I saved off the failing POST data to b.m.o, and check-assertion says it
appears valid.

I chatted with Byron Jones (glob) on #bmo.  They just use the standard /verify
service (code here
http://bzr.mozilla.org/bmo/4.0/annotate/head:/extensions/BrowserID/lib/Login.pm#L63)

I'm thinking that the bugzilla code is not happy with the verifier change, but
I'll know better when I can get ahold of some additional log info from
production.

Comment 1

[John Morrison [:jrgm]]

From some log information, there were 8 (5 bugzilla) failed logins in that time window, of which 5, possibly 6 were my failures. There were two other RP's affected.
I'm seeing an error on 'port mismatch'.

Comment 2

[Jed Parsons (use needinfo, please) [:jedp, :jparsons]]

jrgm theorized that this was a string/int comparison error 
(http://irclog.gr/#show/irc.mozilla.org/identity/187967)

git bisecting confirms that he is correct, and that the problem is in this commit:

https://github.com/mozilla/browserid/commit/d494afbbce176697998f3b39764200d8732ad256#L28L77

I have a fix locally; I am adding test cases and will submit a PR asap.

Comment 3

[John Morrison [:jrgm]]

From production, we chirped the below, here:
https://github.com/mozilla/browserid/blob/dev/lib/verifier/certassertion.js#L127

"verification failure, audience mismatch: 'https://bugzilla.mozilla.org' != 'https://bugzilla.mozilla.org:443': port mismatch"

Comment 4

[Austin King [:ozten]]

Good call, matches your hypothesis.

Comment 5

[Jed Parsons (use needinfo, please) [:jedp, :jparsons]]

:ozten merged https://github.com/mozilla/browserid/pull/2855 - hopefully this fixes the problem

Comment 6

[Austin King [:ozten]]

This is fixed, no?