Closed Bug 1069748 Opened 11 years ago Closed 11 years ago

Please deploy browserid-verifier 0.2.4 to stage

Categories

(Cloud Services :: Operations: Deployment Requests - DEPRECATED, task)

Product:
Cloud Services
Component:
Operations: Deployment Requests - DEPRECATED
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
VERIFIED FIXED

People

(Reporter: rfkelly, Unassigned)

References

Details

(Whiteboard: [qa+])

This version of browserid-verifier includes a change to log output formatting, to put potential PII as top-level fields on the JSON log rather than as fields of an embedded JSON string: https://github.com/mozilla/browserid-verifier/pull/63 It should help with Bug 1053959 Comment 9 and similar. Please deploy v0.2.4 to stage for both tokenserver-local and standalone verifier stacks. As it's just a logging format change, a light load and sanity test should be sufficient for deployment.
Does this require a change to the log parsing?
IIUC it should work correctly with the existing parser, but part of Bug 1053959 is to do extra stuff using the new format.
https://github.com/mozilla-services/puppet-config/pull/879 should be thrown in here to fix the bid.summary issue, but no log parsing changes are required (in fact, this change allows me to write less custom log parsing code).
Whatever you all decide, please hold this till early next week (9/22 or later). QA will pick it up then...
Well, more specifically, deploy to Stage when you have time, QA may not start testing till next week... And #879 looks useful - how easy is that to address and merge?
Status: NEW → ASSIGNED
QA Contact: jbonacci
Hey, who gets to do this? If we go to Stage this week, I can have Karl help out (he will be covering Sync/TS/Verifier while I am out on PTO)...
QA Contact: jbonacci → kthiessen
FYI, I'm in TRIBE on Wed-Thu 2014-09-24 and 25, so if we can get this into Stage on Friday, jbonacci and I can bang on it before he escapes for three weeks.
Updated fxa-bv-stage and the tokenserver-with-verifier stacks in stage with browserid-verifier 0.2.4
Status: ASSIGNED → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
:mostlygeek thanks. I might peak at these stacks tomorrow. But the big testing will start Friday...
This is instance i-3a04b5d4, a c3.large in us-east-1c Standard tags are: app => tokenserver, type => tokenserver, env => stage, stack => tokenserver-with-verifier Public IP is 54.205.111.86 / ec2-54-205-111-86.compute-1.amazonaws.com [kthiessen@ip-10-101-180-144 ~]$ ps auxwww | grep circus 10005 312 0.0 0.0 103248 824 pts/0 S+ 18:20 0:00 grep circus root 3344 0.0 0.5 263064 19488 ? Ssl Sep23 1:09 /usr/bin/python /usr/bin/circusd --log-output /var/log/circus.log --pidfile /var/run/circusd.pid /etc/circus.ini [kthiessen@ip-10-101-180-144 ~]$ ps aux | grep -E 'node|heka|nginx|circus' 10005 314 0.0 0.0 103248 840 pts/0 S+ 18:20 0:00 grep -E node|heka|nginx|circus root 3344 0.0 0.5 263064 19488 ? Ssl Sep23 1:09 /usr/bin/python /usr/bin/circusd --log-output /var/log/circus.log --pidfile /var/run/circusd.pid /etc/circus.ini root 3367 0.3 0.8 686592 31120 ? Ssl Sep23 13:41 /usr/bin/hekad -config=/etc/heka.d root 3557 0.0 0.0 52968 1396 ? Ss Sep23 0:00 nginx: master process /opt/openresty/nginx/sbin/nginx -c /etc/nginx/nginx.conf nginx 3558 0.0 0.0 54676 3552 ? S Sep23 0:10 nginx: worker process nginx 3559 0.0 0.0 54676 3548 ? S Sep23 0:09 nginx: worker process app 3563 0.0 1.0 672176 39936 ? Ssl Sep23 0:13 /usr/bin/node server.js app 30967 0.0 0.6 668128 26016 ? Sl Sep24 0:00 /usr/bin/node /data/fxa-browserid-verifier/lib/ccverifier/worker.js [kthiessen@ip-10-101-180-144 ~]$ rpm -qa | grep -E 'token|browser' puppet-config-tokenserver 20140905164304-1 x86_64 14013 fxa-browserid-verifier-svcops 0.2.4-1 x86_64 47458488 tokenserver-svcops 1.2.9-1 x86_64 73530502 This is instance i-5a6e0077, a c3.large in us-east-1a Standard tags are: app => fxa, type => browserid_verifier, env => stage, stack => fxa-bv-stage Public IP is 54.235.4.100 / ec2-54-235-4-100.compute-1.amazonaws.com [kthiessen@ip-10-144-176-88 ~]$ rpm -qa | grep verifier fxa-browserid-verifier-svcops 0.2.4-1 x86_64 47458488 [kthiessen@ip-10-144-176-88 ~]$ ps aux | grep -E 'node|heka|nginx|circus' root 2692 0.0 0.5 262736 19476 ? Ssl Sep23 0:44 /usr/bin/python /usr/bin/circusd --log-output /var/log/circus.log --pidfile /var/run/circusd.pid /etc/circus.ini app 2711 0.0 1.5 674088 57572 ? Ssl Sep23 0:18 /usr/bin/node server.js root 2761 0.2 0.5 515784 18764 ? Ssl Sep23 8:39 /usr/bin/hekad -config=/etc/heka.d root 2819 0.0 0.0 52968 1396 ? Ss Sep23 0:00 nginx: master process /opt/openresty/nginx/sbin/nginx -c /etc/nginx/nginx.conf nginx 2820 0.0 0.0 54676 3552 ? S Sep23 0:09 nginx: worker process nginx 2821 0.0 0.0 54676 3548 ? S Sep23 0:07 nginx: worker process app 31435 0.0 0.4 665568 15680 ? Sl 17:32 0:00 /usr/bin/node /data/fxa-browserid-verifier/lib/ccverifier/worker.js 10005 31513 0.0 0.0 103248 836 pts/0 R+ 18:23 0:00 grep -E node|heka|nginx|circus
Logs on both systems look clean. On tokenserver-with-verifier: 4 cd /var/log 5 ls -lrt 6 cd hekad 7 ls -l 8 cd /media/ephemeral0/ 9 ls -l 10 cd logs 11 ls -lrt 12 cd tokenserver/ 13 ls -lrt 14 less token.log 15 ls -l 16 less process_account_deletions.error.log 17 less purge_old_records.error.log 18 sort token.log | uniq -w16 -c | sort -rn On fxa-bv-stage: 3 cd /var/log 4 ls -l 5 ls -lrt 6 cd hekad 7 ls -l 8 less *err.log 9 cd ../puppet/ 10 cd /media/ephemeral0/ 11 ls -l 12 cd fxa-browserid-verifier/ 13 ls -l 14 less verifier_out.log 15 cd ../nginx/ 16 ls -l 17 cd logs 18 ls -l 19 fgrep 405 fxa-browserid-verifier.access.log 20 fgrep "GET / " fxa-browserid-verifier.access.log 21 cd /data/ 22 ls -l 23 cd fxa-browserid-verifier/ 24 ls -l ... and nothing looked out of the ordinary. Please let me know if there are logs I missed looking at.
[kthiessen@ip-10-144-176-88 app]$ cat /etc/motd This is instance i-5a6e0077, a c3.large in us-east-1a Standard tags are: app => fxa, type => browserid_verifier, env => stage, stack => fxa-bv-stage Public IP is 54.235.4.100 / ec2-54-235-4-100.compute-1.amazonaws.com [kthiessen@ip-10-144-176-88 app]$ pwd /etc/puppet/yaml/app [kthiessen@ip-10-144-176-88 app]$ fgrep app_version * fxa.dev.yaml:fxa::auth_server::app_version: 'latest' fxa.dev.yaml:fxa::content_server::app_version: 'latest' fxa.yaml:fxa::browserid_verifier::app_version: '1.0.0-1snap201403111716git02f8d5' fxa.yaml:fxa::auth_server::app_version: '0.9.3-20140325git139131f.el6' fxa.yaml:fxa::content_server::app_version: 'latest' I would expect the 'app-_version' tags to be broken out by environment rather than in the common yaml -- dev version in the dev yaml, stage in stage, etc. Am I missing something about the scheme here?
Tokenserver has been redeployed to include the shellshock and nss security fixes from RedHat. Also notable changes: - new AMI with security fixes and updated puppet config code - uses GP2 (ssd-ebs) backed storage instead of the slower standard EBS - updated puppet-config version for deploy to: 20140926180605 - updated CFN so tokenserver is autoscaling. - adds one server when average CPU is > 60% - removes one server when average CPU < 40% Verifier stack: - new AMI with security fixes and updated puppet config code - uses GP2 (ssd-ebs) backed storage instead of the slower standard EBS - updated puppet-config version for deploy to: 20140926180605
More about the autoscaling: - if average CPU is > 60% for 1 minute, it will add a server up to the max (21) - if average CPU is < 40% for 1 minute, it will remove a server down to the min (defined in CFN) The stage stack has been deployed with 1 server. When testing it will be good to hit it with enough load to see if the number of servers increases until CPU is < 60% average. After the load test is stopped servers should dropped back to the minimum, which is 1 for stage.
10-minute load tests on both tokenserver and verifier look good. Continuing on to 60-minute load tests.
60-minute load tests look good. More to come on Monday.
:kthiessen Regarding app version: We pass those in from cloudformation now and they are written to /etc/puppet/yaml/local.yaml which overrides everything. This way we can change the RPM versions of apps without affecting the puppet versioning.
What is the status of this? can I deploy the new tokenserver / verifier now?
Ran one slightly more-intense 60-minute test on Monday. Looks good. Please deploy the new tokenserver / verifier, and I will verify the deployment.
Filing prod deploy ticket now ... will update this ticket with the prod ticket #.
Production ticket is above, https://bugzilla.mozilla.org/show_bug.cgi?id=1075170
Status: RESOLVED → VERIFIED
You need to log in before you can comment on or make changes to this bug.