Closed Bug 1082188 Opened 6 years ago Closed 6 years ago

Loop-Push — Please deploy pushgo 1.4 to Stage

Categories

(Cloud Services :: Operations: Deployment Requests - DEPRECATED, task)

task
Not set
normal

Tracking

(Not tracked)

VERIFIED FIXED

People

(Reporter: rpapa, Assigned: oremj)

References

Details

Please deploy the 1.4 pre-release to the staging environment.
https://github.com/mozilla-services/pushgo/releases/tag/1.4rc
Assignee: nobody → oremj
I've deployed 1.4rc, but need to fire up an etcd cluster and switch the discovery type before calling this FIXED. I should be able to finish that up by tomorrow afternoon.
Thanks, Jeremy!
Depends on: 1072495
This is blocked on Amazon raising our stack limit. They say it may take up to two days, but we are trying to expedite the process.
Stage has been updated to 1.4rc

https://push.stage.mozaws.net
https://updates-push.stage.mozaws.net/
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
I'm working on getting smoke/load tests to run without too much duct tape and will be noting issues in this ticket as I come across them.

Here's the smoke/load tests I'm aware of.  Please let me know if:
1. there are other tests not listed here that QA should be aware of
2. if the desire is to run them all, or pick a few best candidates.

--------------
Smoke Tests 
--------------

1. https://github.com/jrconlin/simplepush_test/blob/master/smoke_test.py 
* per JR, good smoke test candidate
2. https://github.com/mozilla-services/pushgo/

--------------
Load Tests 
--------------

1. https://github.com/bbangert/push-tester/ (Haskell test)
* uses loads, v2?
2. https://github.com/mozilla-services/simplepush-testpod (oremj/mostlygeek)
3. https://github.com/kitcambridge/push-tester-go
4. https://github.com/edmoz/load-test/tree/master/_loads/pushgo
Crap, I mispoke to rpapa earlier and confused push against fmd. 

The go test does test the interfaces sufficiently. The smoke_test.py is an interaction test only.
(In reply to JR Conlin [:jrconlin,:jconlin] from comment #6)
> Crap, I mispoke to rpapa earlier and confused push against fmd. 
> 
> The go test does test the interfaces sufficiently. The smoke_test.py is an
> interaction test only.

JR, thanks for the clarification.  I still need to look through the tests more thoroughly to understand better what each does, but I may circle back to you later.

In regards to the pushgo test, I'm seeing a dep error when I do a 'make simplepush'
Should I be installing memcached first?


https://github.com/mozilla-services/pushgo/ 

1. Installed latest go version go1.3.3 darwin/amd64
2. ran make
** GOPATH wasn't set by Makefile (Kit to fix)
** JR helped me set GOPATH
3. $ make simplepush

rm -f simplepush
Building simplepush
go build -o simplepush github.com/mozilla-services/pushgo
# github.com/ianoshen/gomc
.godeps/src/github.com/ianoshen/gomc/const.go:4:10: fatal error: 'libmemcached/memcached.h' file not found
#include <libmemcached/memcached.h>
         ^
1 error generated.
make: *** [simplepush] Error 2
Flags: needinfo?(jrconlin)
I've not tried it on a mac, but you should be able to run

make libmemcached-1.0.18 

to build a compatible version of libmemcached
Flags: needinfo?(jrconlin)
(In reply to JR Conlin [:jrconlin,:jconlin] from comment #8)
> I've not tried it on a mac, but you should be able to run
> 
> make libmemcached-1.0.18 
> 
> to build a compatible version of libmemcached

correction;
make memcached will install the version to your system, but will require sudo to do so. If possible, that would probably work better.
Thanks, JR.  I was able to install memcached with make libmemcached-1.0.18, but it looks like the header files aren't being installed where they need to be.  Is there something I need to do to get memcached to install into the .godeps/src dir?

$ make simplepush
rm -f simplepush
Building simplepush
go build -o simplepush github.com/mozilla-services/pushgo
# github.com/ianoshen/gomc
.godeps/src/github.com/ianoshen/gomc/const.go:4:10: fatal error: 'libmemcached/memcached.h' file not found
#include <libmemcached/memcached.h>
         ^
1 error generated.
make: *** [simplepush] Error 2
Flags: needinfo?(jrconlin)
Richard, you might need to set some environment variables for Cgo to find libmemcached:

export CGO_CFLAGS="-I/usr/local/include"
export CGO_CPPFLAGS="-I/usr/local/include"
export CGO_CXXFLAGS="-I/usr/local/include"
export CGO_LDFLAGS="-L/usr/local/lib"

Does the build still fail if those are set?
when doing make libmemcached-1.0.18, memcached.h gets installed into local dir: 
/libmemcached-1.0.18/libmemcached
however, per :spindrift, you can remove dependency by renaming src/github.com/mozilla-services/pushgo/simplepush/emcee_store.go to emcee_store.no
Once this is renamed, pushgo successfully compiles and it's possible to run a test:

$ make test
go test github.com/mozilla-services/pushgo/id github.com/mozilla-services/pushgo/simplepush
ok  	github.com/mozilla-services/pushgo/id	0.006s
ok  	github.com/mozilla-services/pushgo/simplepush	1.371s
Flags: needinfo?(jrconlin)
heh, in one of my forks, I have a make rule that excludes emcee_store.go and libmemcache because of just that sort of reason.

Eventually, this will be addressed.
Awesome.  Kit said he's going to merge that change/rule into the pushgo test and will also schedule time w/ QA next week to review load test (should be ready by mid-week).  

After meeting today, we decided the following tests would be sufficient for verifying a STAGE build:
SMOKE: https://github.com/mozilla-services/pushgo/
LOAD: https://github.com/bbangert/push-tester/

Still need to consider best way to test deployment on PROD.
Blocks: 1090504
No longer blocks: 1055139
Summary: SimplePush — Please deploy pushgo 1.4 to Stage → Loop-Push — Please deploy pushgo 1.4 to Stage
We still need to determine a prod config.
sounds like we need the following:
1. loads,v2 up and running
2. gather loadtest data for stage (from cloudwatch?)
3. deploy single cluster to prod for 34, and maybe 35 
Currently blocked by: Bug 1075654 

[12:35]  benbanger: rpapa: the staging/prod environment for loop-push
[12:36]  benbangert: there's a ticket for the prod, but its unknown what config to run it with
[12:36]  benbangert: since we don't have data from load-testing yet
[12:36]  benbangert: oremj: is there any monitoring or data for stage?
[12:38]  oremj: benbangert: cloudwatch should have some data
[12:38]  benbangert: rpapa: my thinking is, we make sure prod loop-push can survive 34, and maybe 35 with a single cluster
[12:39]  benbangert: or at least, we know where the death point is, so we don't feature-flag too many
[12:39]  benbangert: unfortunately given the way the wss caches the DNS name, we can't move ppl until client implements redirect, and we have redirect working.
no loadtesting on this release.

soft-release of 1.4 rc2 planned for next Mon.,11/17
so closing this ticket
Status: RESOLVED → VERIFIED
You need to log in before you can comment on or make changes to this bug.