Please deploy browserid-verifier 0.2.3 to stage



5 years ago
5 years ago


(Reporter: rfkelly, Assigned: mostlygeek)


Firefox Tracking Flags

(Not tracked)


(Whiteboard: [qa+])



5 years ago
This version of browserid-verifier includes additional logging for failed verification attempts, which should help us to track down what's going on in Bug 1045502 and Bug 1059787.

Please deploy it to stage for both tokenserver-local and standalone verifier stacks.  Since our loadtest includes some proportion of bad signatures, we should be able to verify whether the logs are coming through correctly. should be merged as part of this update to pull in some logging fixes.
Adding some Cloud QA people for next week's training session(s).
Whiteboard: [qa+]
Double deploy! Awesome.
I did not even know the standalone Bid-Verifier in Stage (or Prod) was still live.
We will work on this Wednesday and Thursday.
Assignee: nobody → bwong
QA Contact: jbonacci

Comment 5

5 years ago
verifier 0.2.3-1 deployed on 

- fxa-bv-stage stack
- tokenserver-with-verifier stack

Both stacks looks like they are running and initialized correctly. Ready for some QA.
Last Resolved: 5 years ago
Resolution: --- → FIXED
Not exactly "Resolved" since there is still config work going on ;-)

I will start with the tokenserver-with-verifier stack since it seems to be most ready...

Comment 8

5 years ago
OK both stacks are ready for QA now. and were both merged and deployed.
OK, thanks.

I did my initial checks of the Tokenserver+Verifier environment:
Verified one new instance in the tokenserver-with-verifier CF stack: 

Verified the following code is running:
puppet-config-tokenserver 20140905164304-1 x86_64 14013
fxa-browserid-verifier-svcops 0.2.3-1 x86_64 47459269
tokenserver-svcops 1.2.9-1 x86_64 73530502

Verified the processes, files, and logs.

{"services": {"sync": ["1.5"]}, "auth": ""}

Quick test was successful.

Moving over to BrowserID-Verifier next...
Ran into a non-OPs/non-deployment blocker in the browserid-verifier repo.
Debugging with :rfkelly...
The usual npm crap.
Past that and building/testing now...
OK, went back and verified some configs and yamls on TokenServer+Verifier.

For the standalone Verifier:
Verified a single instance deployed to the same CF stack: fxa-bv-stage

Code version: fxa-browserid-verifier-svcops 0.2.3-1 x86_64 47459269

Verified the processes, files, and logs.

Verified configs and yamls.

Quick test was successful.

So, all load testing will begin tomorrow.
Apparently Stage DB is itty-bitty, so going to "ease into" load...
Short load tests against Verifier and TokenServer+Verifier look good:

Trying a short parallel load test before I go with two 30min load tests...
Just started up parallel 45min load tests.
Then, I will troll the logs...
Stand-alone Verifier:
Nginx access.log:
There are just a few stray 404s/405s on the Verifier, which appear to be bot-related.

The verifier_err.log has one line in it: "express: res.json(obj, status): Use res.json(status, obj) instead"
The timestamp on that is about 4 hours old.

The verifier_out.log contains some entries that look like this:
{"op":"bid.v2","name":"bid.v2","time":"2014-09-12T00:07:30.991Z","pid":2716,"v":1,"hostname":"ip-10-51-168-73","message":"verify { result: 'failure',\n  reason: 'untrusted issuer, expected \\'\\', got \\'\\'',\n  assertion: '...etc...
...etc...',\n  trustedIssuers: [],\n  rp: '' }"}
I believe this is ok.

The nginx access log has the usual 200s and 401s.
And a couple of stray 404s that look like bot activity.

The token.error.log shows the following:
Exception KeyError: KeyError(27265776,)...etc...

The token.log has the expected 200s, 401s, and connection pool messages.

The verifier_err.log file has express: res.json(obj, status): Use res.json(status, obj) instead
same as with the standalone Verifier (and about 4 hours old)

The verifier_out.log file has failures that look like this:
{"op":"bid.v2","name":"bid.v2","time":"2014-09-12T00:04:37.705Z","pid":3335,"v":1,"hostname":"ip-10-165-100-194","message":"verify { result: 'failure',\n  reason: 'expired',\n  assertion: '...etc...
...etc...',\n  trustedIssuers: \n   [ '',\n     '',\n     '',\n     '',\n     '',\n     '',\n     '' ],\n  rp: '' }"}

I need these double-checked by Dev and OPs.

Comment 18

5 years ago
Excellent; these are the usual "bad assertion" errors generated by the loadtest, and they show the extra logging info that we want here.  This looks good to me.
Whoops, also some like this:
{"op":"bid.v2","name":"bid.v2","time":"2014-09-12T00:04:35.518Z","pid":3335,"v":1,"hostname":"ip-10-165-100-194","message":"verify { result: 'failure',\n  reason: 'algorithms do not match',\n  assertion: ...etc...',\n  trustedIssuers: \n   [ '',\n     '',\n     '',\n     '',\n     '',\n     '',\n     '' ],\n  rp: '' }"}

and this:
{"op":"bid.v2","name":"bid.v2","time":"2014-09-12T00:04:34.853Z","pid":3335,"v":1,"hostname":"ip-10-165-100-194","message":"verify { result: 'failure',\n  reason: 'audience mismatch: scheme mismatch',\n  assertion: '...etc...',\n  trustedIssuers: \n   [ '',\n     '',\n     '',\n     '',\n     '',\n     '',\n     '' ],\n  rp: '' }"}

and these of course (somewhat similar to the standalone Verifier):
{"op":"bid.v2","name":"bid.v2","time":"2014-09-12T00:04:30.171Z","pid":3335,"v":1,"hostname":"ip-10-165-100-194","message":"verify { result: 'failure',\n  reason: 'untrusted issuer, expected \\'\\', got \\'\\'',\n  assertion: '...etc...',\n  trustedIssuers: \n   [ '',\n     '',\n     '',\n     '',\n     '',\n     '',\n     '' ],\n  rp: '' }"}
I will update my documentation...

We can move forward with Production.
You need to log in before you can comment on or make changes to this bug.