Closed Bug 1063350 Opened 10 years ago Closed 10 years ago

Please deploy browserid-verifier 0.2.3 to stage

Categories

(Cloud Services :: Operations: Deployment Requests - DEPRECATED, task)

Product:
Cloud Services
Component:
Operations: Deployment Requests - DEPRECATED
Platform:
x86_64
Windows 7
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
VERIFIED FIXED

People

(Reporter: rfkelly, Assigned: mostlygeek)

References

Details

(Whiteboard: [qa+]) 

This version of browserid-verifier includes additional logging for failed verification attempts, which should help us to track down what's going on in Bug 1045502 and Bug 1059787.

Please deploy it to stage for both tokenserver-local and standalone verifier stacks.  Since our loadtest includes some proportion of bad signatures, we should be able to verify whether the logs are coming through correctly.
https://github.com/mozilla-services/puppet-config/pull/851 should be merged as part of this update to pull in some logging fixes.
Adding some Cloud QA people for next week's training session(s).
Whiteboard: [qa+]
Double deploy! Awesome.
I did not even know the standalone Bid-Verifier in Stage (or Prod) was still live.
We will work on this Wednesday and Thursday.
Assignee: nobody → bwong
Status: NEW → ASSIGNED
QA Contact: jbonacci
verifier 0.2.3-1 deployed on 

- fxa-bv-stage stack
- tokenserver-with-verifier stack

Both stacks looks like they are running and initialized correctly. Ready for some QA.
Status: ASSIGNED → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
Not exactly "Resolved" since there is still config work going on ;-)

I will start with the tokenserver-with-verifier stack since it seems to be most ready...
OK both stacks are ready for QA now. https://github.com/mozilla-services/puppet-config/pull/865 and https://github.com/mozilla-services/svcops/pull/236 were both merged and deployed.
OK, thanks.

I did my initial checks of the Tokenserver+Verifier environment:
Verified one new instance in the tokenserver-with-verifier CF stack: 
i-e5f5050b
ec2-54-167-157-233

Verified the following code is running:
puppet-config-tokenserver 20140905164304-1 x86_64 14013
fxa-browserid-verifier-svcops 0.2.3-1 x86_64 47459269
tokenserver-svcops 1.2.9-1 x86_64 73530502

Verified the processes, files, and logs.

curl https://token.stage.mozaws.net
{"services": {"sync": ["1.5"]}, "auth": "https://token.stage.mozaws.net"}

Quick test was successful.

Moving over to BrowserID-Verifier next...
Ran into a non-OPs/non-deployment blocker in the browserid-verifier repo.
Debugging with :rfkelly...
The usual npm crap.
Past that and building/testing now...
OK, went back and verified some configs and yamls on TokenServer+Verifier.

For the standalone Verifier:
Verified a single instance deployed to the same CF stack: fxa-bv-stage
i-5175f97c
ec2-54-80-51-114


Code version: fxa-browserid-verifier-svcops 0.2.3-1 x86_64 47459269

Verified the processes, files, and logs.

Verified configs and yamls.


Quick test was successful.

So, all load testing will begin tomorrow.
Apparently Stage DB is itty-bitty, so going to "ease into" load...
Short load tests against Verifier and TokenServer+Verifier look good:
Verifier: https://loads.services.mozilla.com/run/c9be8224-e679-45ca-91bb-71f3fbc0c21d
TS/V: https://loads.services.mozilla.com/run/c407c139-13af-4b51-82bf-bc23ef6331bd

Trying a short parallel load test before I go with two 30min load tests...
Just started up parallel 45min load tests.
Then, I will troll the logs...
Blocks: 1066370
Stand-alone Verifier:
Nginx access.log:
There are just a few stray 404s/405s on the Verifier, which appear to be bot-related.

The verifier_err.log has one line in it: "express: res.json(obj, status): Use res.json(status, obj) instead"
The timestamp on that is about 4 hours old.

The verifier_out.log contains some entries that look like this:
{"op":"bid.v2","name":"bid.v2","time":"2014-09-12T00:07:30.991Z","pid":2716,"v":1,"hostname":"ip-10-51-168-73","message":"verify { result: 'failure',\n  reason: 'untrusted issuer, expected \\'login.mozilla.org\\', got \\'mockmyid.s3-us-west-2.amazonaws.com\\'',\n  assertion: '...etc...
...etc...',\n  trustedIssuers: [],\n  rp: 'https://secret.mozilla.com' }"}
I believe this is ok.


TS+Verifier:
The nginx access log has the usual 200s and 401s.
And a couple of stray 404s that look like bot activity.

The token.error.log shows the following:
Exception KeyError: KeyError(27265776,)...etc...

The token.log has the expected 200s, 401s, and connection pool messages.

The verifier_err.log file has express: res.json(obj, status): Use res.json(status, obj) instead
same as with the standalone Verifier (and about 4 hours old)

The verifier_out.log file has failures that look like this:
{"op":"bid.v2","name":"bid.v2","time":"2014-09-12T00:04:37.705Z","pid":3335,"v":1,"hostname":"ip-10-165-100-194","message":"verify { result: 'failure',\n  reason: 'expired',\n  assertion: '...etc...
...etc...',\n  trustedIssuers: \n   [ 'api.accounts.firefox.com',\n     'api-accounts.stage.mozaws.net',\n     'api-accounts-dev.stage.mozaws.net',\n     'api-accounts.dev.lcip.org',\n     'api-accounts-latest.dev.lcip.org',\n     'nightly.dev.lcip.org',\n     'latest.dev.lcip.org' ],\n  rp: 'https://token.stage.mozaws.net' }"}

I need these double-checked by Dev and OPs.
Excellent; these are the usual "bad assertion" errors generated by the loadtest, and they show the extra logging info that we want here.  This looks good to me.
Whoops, also some like this:
{"op":"bid.v2","name":"bid.v2","time":"2014-09-12T00:04:35.518Z","pid":3335,"v":1,"hostname":"ip-10-165-100-194","message":"verify { result: 'failure',\n  reason: 'algorithms do not match',\n  assertion: ...etc...',\n  trustedIssuers: \n   [ 'api.accounts.firefox.com',\n     'api-accounts.stage.mozaws.net',\n     'api-accounts-dev.stage.mozaws.net',\n     'api-accounts.dev.lcip.org',\n     'api-accounts-latest.dev.lcip.org',\n     'nightly.dev.lcip.org',\n     'latest.dev.lcip.org' ],\n  rp: 'https://token.stage.mozaws.net' }"}

and this:
{"op":"bid.v2","name":"bid.v2","time":"2014-09-12T00:04:34.853Z","pid":3335,"v":1,"hostname":"ip-10-165-100-194","message":"verify { result: 'failure',\n  reason: 'audience mismatch: scheme mismatch',\n  assertion: '...etc...',\n  trustedIssuers: \n   [ 'api.accounts.firefox.com',\n     'api-accounts.stage.mozaws.net',\n     'api-accounts-dev.stage.mozaws.net',\n     'api-accounts.dev.lcip.org',\n     'api-accounts-latest.dev.lcip.org',\n     'nightly.dev.lcip.org',\n     'latest.dev.lcip.org' ],\n  rp: 'https://token.stage.mozaws.net' }"}

and these of course (somewhat similar to the standalone Verifier):
{"op":"bid.v2","name":"bid.v2","time":"2014-09-12T00:04:30.171Z","pid":3335,"v":1,"hostname":"ip-10-165-100-194","message":"verify { result: 'failure',\n  reason: 'untrusted issuer, expected \\'\\', got \\'mockmyid.s3-us-west-2.amazonaws.com\\'',\n  assertion: '...etc...',\n  trustedIssuers: \n   [ 'api.accounts.firefox.com',\n     'api-accounts.stage.mozaws.net',\n     'api-accounts-dev.stage.mozaws.net',\n     'api-accounts.dev.lcip.org',\n     'api-accounts-latest.dev.lcip.org',\n     'nightly.dev.lcip.org',\n     'latest.dev.lcip.org' ],\n  rp: 'https://token.stage.mozaws.net' }"}
I will update my documentation...

We can move forward with Production.
Status: RESOLVED → VERIFIED
You need to log in before you can comment on or make changes to this bug.