969190 - Figure out node-allocation limits and management on stage

Status: VERIFIED FIXED

(Cloud Services Graveyard :: Server: Token, defect, P1)

Description

[James Bonacci [:jbonacci]]

Running a "combined" (TokenServer/Sync) load test in Stage, we encountered a number of problems while debugging the failed test and the ephemeral nginx and tokenserver logs.

Details are in IRC fro 2/6/2014, 4:30pm PST and onward

These issues will probably turn into at least 2 bugs perhaps in GitHub repos for node allocation fixes and also for more python workers to avoid those 111 connection refused errors...

Comment 1

[Ryan Kelly [:rfkelly]]

The sync loadtests currently use 1000000 unique accounts, while stage TS only has around 10000 node-allocation slots.  So we can easily exhaust the available allocation and cause TS to start giving us 503s.

Comment 2

[Bob Micheletto [:bobm]]

It appears that the load test users require ~1/3 MB of storage space apiece.  The staging token server node table has been adjusted to reflect this space, resulting in a current total capacity for stage of 1,580,000 users.  That should accommodate the unique accounts required by the stress.py script.

Comment 3

[Ryan Kelly [:rfkelly]]

Awesome :-)

Is there anything we should be doing when we discover that the allocations have been exhausted, apart from just erroring out with a 503?  I presume we'll monitor for this in prod, not sure there's anything else that can be done from within the app.

Comment 4

[James Bonacci [:jbonacci]]

OK, so it sounds like we are ready to do some TS load testing against Stage next week...

Comment 5

[Ryan Kelly [:rfkelly]]

So we ran out of nodes in stage again yesterday, but I think that's because one of them was marked as "down" in the tokenserver nodes db.

Comment 6

[Ryan Kelly [:rfkelly]]

Another P1 bug?  Luckily it's already closed! :-)

:jbonacci do you want to re-open or will we assume it's fixed now that all stage nodes are un-downed?

Comment 7

[James Bonacci [:jbonacci]]

See bug 982176

Comment 8

[Ryan Kelly [:rfkelly]]

So I asked James re-open this to sanity-check that the tokenserver loadtests are using the same set of user accounts as the sync loadtests, which they are.  They both use "user[1-1000000]@mockmyid.s3-us-west-2.amazonaws.com" for the core of the test, which should be under the limit of available node allocations.

Tokenserver loadtests to 4% new user requests, which will very slowly exhaust available node allocations.  That's an important part of the test so we'll just have to cope with that via monitoring etc.

Comment 9

[James Bonacci [:jbonacci]]

OK. Cool. We can write more specific bugs down the road if things need to get tweaked...