1145302 - Please deploy shavar v0.6.1 to STAGE

Status: VERIFIED FIXED

(Cloud Services :: Operations: Deployment Requests - DEPRECATED, task)

Description

[Ryan Tilder [:rtilder]]

The relevant tag: https://github.com/mozilla-services/shavar/releases/tag/0.6.1

Comment 1

[Chris Kolosiwsky [:ckolos] (ckolos has left the building)]

Deployed a "prod like" stack at shavar-061-201503192146.stage.mozaws.net . Non-functional testing shows the stack live and responding as expected.

:rpapa, I will deploy the stand alone host on friday morning.

Comment 2

[Richard Pappalardo [:rpapa][:rpappalardo]]

Thank you, Chris!

Comment 3

[Richard Pappalardo [:rpapa][:rpappalardo]]

posting here for Reference:

Bug 1145371 - Shavar server needs to be ready for beta and production load
Bug 1145373 - [tracking bug] Shavar server needs load, fuzz, end to end testing

Comment 4

[Richard Pappalardo [:rpapa][:rpappalardo]]

more reference:
Bug 1147462 - Clone safebrowsing list into second list to test client-server multiple-list synchronization

Comment 5

[Richard Pappalardo [:rpapa][:rpappalardo]]

Sizing activity is currently being done to determine an optimal deployment configuration for shavar server.

Objective is to determine:
[1]. Max. number of requests shavar can handle with:
    a) quickest response time
    b) minimal error
[2]. Optimal deployment configuration to maximize [1]

All testing is being performed using mostlygeek's fork of beeswithmachineguns:
https://github.com/mostlygeek/beeswithmachineguns/

See results below:

------------------------------------
SUMMARY
------------------------------------

Loadtesting was first done on a standalone m3.medium shavar instance to determine the capacity of a single node.
This instance size can handle approx. 200-350 requests/second.

Testing is still underway to determine the optimal instance size and configuration for a shavar stack with both ELB and auto-scaling.

Thus far it's been fairly easy to bring shavar up to 3500 requests/second with negligible error.  Max. reached thus far has been ~6500 requests/second with a c4.large stack of shavar servers @ a 0.1% error rate (9 auto-scaled instances). 

In both cases, CPU remained pegged at 80% over the course of both half hour and hour-long tests (auto-scaling reached a total of 9 instances).

The following configurations are being tested (updates to follow):
* standalone m3.medium node - no ELB, no AG

ELB stacks w/ autoscaling (on: @60%cpu / off: @40%cpu) 
* c4.large instances 
* c3.xlarge instances
* c3.large instances
* m3.large instances

Comment 6

[Richard Pappalardo [:rpapa][:rpappalardo]]

------------------------------------
standalone node - single instance: m3.medium
(no ELB, no auto-scaling) 
------------------------------------

SUMMARY

A standalone (single) m3.medium shavar instance was loadtested without ELB nor auto-scaling enabled.
The following results were obtained using the test profiles indicated:

# BEE SERVERS: 1
BEE INSTANCE SIZE: t2.micro

CONCURRENT CONNECTIONS: 1
RPS: ~150
FAILURES: 0
RESPONSE TIME: 10 ms/request
CPU: maxed @50% (just below the would-be auto-scaling threshold)

CONCURRENT CONNECTIONS: 2
RPS: ~265
FAILURES: 0
RESPONSE TIME: 10 ms/request
CPU: maxed @85% (just below the would-be auto-scaling threshold)

CONCURRENT CONNECTIONS: up to 32 
RPS: 380 
FAILURES: 0
RESPONSE TIME: 80-130 ms/request
CPU: maxed @100%

Comment 7

[Richard Pappalardo [:rpapa][:rpappalardo]]

Deployment update -

While sizing exercise continues on stage, we're going to go ahead and deploy 0.6.1 with an ELB stack of m3.medium instances next Monday (4/20).  This size should be more than adequate for current traffic.

I will be out, but :sphilp will do the verification in my place.

Any further sizing results for 0.6.1 will be continue noted in this ticket.

------------------------------------
E2E TEST
------------------------------------

Stage has been verified using the following methods for e2e testing:
[1]. https://wiki.mozilla.org/Services/TrackingProtection/ShavarTesting
[2]. https://github.com/rpappalax/shavar-e2etest
these are essentially redundant, but just want to make sure the e2e test script is reliable.