1024640 - Deploy the loop-client 0.1.0-1 to stage

Status: VERIFIED FIXED

(Cloud Services :: Operations: Deployment Requests - DEPRECATED, task)

Description

[James Bonacci [:jbonacci]]

Dev and OPs to add specific details here.

Comment 1

[Benson Wong [:mostlygeek]]

this one is waiting on the cache busting changes that are being done in 1023217. Once that is complete we'll deploy it w/out the CDN for now.

Comment 2

[Benson Wong [:mostlygeek]]

After some discussion we'll deploy this w/out the CDN / cache busting code. It'll just be nginx serving static content for now.

Comment 3

[James Bonacci [:jbonacci]]

OK. That's fine.

Comment 4

[Benson Wong [:mostlygeek]]

OK deployed. 

- 2x m3.medium
- only running nginx 
- heka is sending out nginx logs 
- looking at https://call.stage.mozaws.net/config.js ... looks right

Ready for QA verification

Comment 5

[James Bonacci [:jbonacci]]

So, here are the results: https://loads.services.mozilla.com/run/8b7cb33d-1752-46b3-be5d-6475a4bb3488

Run Id     8b7cb33d-1752-46b3-be5d-6475a4bb3488
Duration     30 min and 29 sec.
Started     2014-06-19 20:52:33 UTC
Ended     2014-06-19 21:23:02 UTC
State     Ended

Users     [20]
Hits     None
Agents     5
Duration     1800
Server URL     https://token.stage.mozaws.net

Results
Tests over     428408 
Successes     428024
Failures     0
Errors     0
TCP Hits     437191
Opened web sockets     0
Total web sockets     0
Bytes/websockets     0
Requests / second (RPS)     239

addFailure     384

I will look into the failures...

Comment 6

[James Bonacci [:jbonacci]]

Sorry copy/paste error - ignore previous entry...
(Bugzilla needs a comment edit/delete feature)

Comment 7

[James Bonacci [:jbonacci]]

OK. Everything looks good except we need hekad hooked up ;-)

Comment 8

[James Bonacci [:jbonacci]]

Forgot to move this back to Reopened yesterday - waiting on the hekad additions/changes...

Comment 9

[Benson Wong [:mostlygeek]]

- Added heka
- Changed from nginx to use new openresty module (openresty = nginx w/ 3rd party modules)
- logs are now in /media/ephemeral0/nginx/log
- enabled gzip for a performance bump
- everything else is the same

Comment 10

[James Bonacci [:jbonacci]]

:mostlygeek most everything checks out ok, except there are current errors in this log:
/var/log/hekad/loop-client.stderr.log

2014/06/23 18:18:30 Decoder 'LoopWebserver-LoopNginxDecoder' error: Failed parsing: 10.50.203.24 - - [23/Jun/2014:17:58:22 +0000] "GET / HTTP/1.1" 200 1935 "-" "ELB-HealthChecker/1.0" "-"
2014/06/23 18:24:52 Decoder 'LoopWebserver-LoopNginxDecoder' error: Failed parsing: 1403547892.362 "-" "GET / HTTP/1.1" 200 1935 "-" "ELB-HealthChecker/1.0" 0.000 - "-"

--> which I think went away



...etc...
2014/06/23 18:18:33 Plugin 'AggregatorOutput' error: dial tcp :5565: connection refused
...etc...
2014/06/23 20:29:38 Plugin 'AggregatorOutput' error: dial tcp :5565: connection refused
--> which continue to show up

Comment 11

[Benson Wong [:mostlygeek]]

OK I fixed the bug in puppet-config (https://github.com/mozilla-services/puppet-config/pull/620) and heka is no longer erroring on the AggregatorOutput

Comment 12

[James Bonacci [:jbonacci]]

OK, assuming you restarted the services on this instance, I will check again shortly...

Comment 13

[James Bonacci [:jbonacci]]

I am still seeing errors:

1: /media/ephemeral0/nginx/logs/loop-client.error.log
2014/06/23 20:14:55 [error] 10945#0: *409....etc...
These may be a part of some config change and restart to get logging to happen with heka?


2: /var/log/hekad/loop-client.stderr.log
2014/06/23 22:06:31 Plugin 'AggregatorOutput' error: dial tcp 10.183.191.147:5565: connection timed out


Also, tangentially related, what are these logs for?
/media/ephemeral0/nginx/logs/default.access.log
/media/ephemeral0/nginx/logs/default.error.log

Comment 14

[Benson Wong [:mostlygeek]]

My bad, I should have mentioned I created a new stack (loop-stage) - poor naming? yes. I'll create a new stack w/ better naming after (same CFN/settings). 

https://call.stage.mozaws.net is pointed at this new stack. 
I'm only running a single server: ec2-54-87-69-192.compute-1.amazonaws.com

Comment 15

[Benson Wong [:mostlygeek]]

Also

/media/ephemeral0/nginx/logs/default.access.log
/media/ephemeral0/nginx/logs/default.error.log

Just ignore these. I switched to using a new nginx (openresty) and the puppet-config will generate these as the catchall log locations. They're really for dev to help us notice when we don't set up specific logging correctly.

Comment 16

[James Bonacci [:jbonacci]]

OK. Yea, I see also that /var/log/nginx is gone, /media/ephemeral/nginx/logs are in place.

Comment 17

[James Bonacci [:jbonacci]]

I think we are in good shape now.
I see this deployed to the one instance:
loop-client-svcops 0.1.0-1 noarch 5484963
puppet-config-loop 20140623204304-1 x86_64 13045

I was able to hit the Stage client via the URL: https://call.stage.mozaws.net/

I will test this some more tomorrow by pointing Nightly to Stage maybe.

Comment 18

[James Bonacci [:jbonacci]]

Also:
https://call.stage.mozaws.net/config.js
var loop = loop || {};
loop.config = {serverUrl: 'https://loop.stage.mozaws.net'};

curl -I https://call.stage.mozaws.net
HTTP/1.1 200 OK
Accept-Ranges: bytes
Content-length: 1935
Content-Type: text/html
Date: Mon, 23 Jun 2014 23:04:24 GMT
ETag: "539220d7-78f"
Last-Modified: Fri, 06 Jun 2014 20:13:11 GMT
Server: openresty/1.7.0.1
Vary: Accept-Encoding
Connection: keep-alive