584800 - Set up load test infra

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Toby Elliott [:telliott]]

Two part bug:

Tracking the setup of the fully-hardwared sync stage cluster, where we'll be deploying load testing against.

Tracking the setup of the clients. My current understanding is that stage will be behind firewalls, making EC2 a challenge. mrz suggested using some available mac minis.

The number we'll need depends on how many 'users' we want to test for. A laptop client can generate approximately 20 transactions per second (minis may be higher). 1M active users (which is a bit under 50% of total users) requires 325 transactions a second. I believe that's higher than our spec (which, as I last recall, was 1M total users per db, suggesting we need somewhere on the order of 165 tps, or 9 machines)

Getting a single mini set up would let us test tps, as well as give us an image we can sync to the others.

Comment 1

[Zandr Milewski [:zandr]]

(In reply to comment #0)
> Two part bug:

Please don't do that unless you're filing bugs for each component and making them block this one. (Which is what you should do now)

Odds are any "two part bug" will have different groups working on each part.

[/me stops channeling reed now]

Comment 2

[matthew zeier [:mrz]]

> Tracking the setup of the clients. My current understanding is that stage will
> be behind firewalls, making EC2 a challenge. mrz suggested using some available
> mac minis.

I said that before I understood what you were doing and specifically suggested doing some sort of job on the talos servers but that doesn't seem practical now.

Comment 3

[matthew zeier [:mrz]]

This bug is really about standing up a load generation "cluster".

Comment 4

[Toby Elliott [:telliott]]

(In reply to comment #1)
> 
> Please don't do that unless you're filing bugs for each component and making
> them block this one. (Which is what you should do now)

Good point. Done.

Comment 5

[Justin Fitzhugh]

whats the eta on this?  this is blocking getting load tests going.

Comment 6

[Justin Dow [:jabba]]

I'll take this bug, as I'm sure I'll be setting up the hardware end of it.

I need to know what kind of hardware is needed and it needs to get ordered.

Is the "client" a desktop app? I noticed someone mention macbooks. Could we set up a cluster of old/used/intern MacBooks to do this? I'm sure we have a pool of used laptops somewhere, as interns are ending their internships?

Comment 7

[Justin Dow [:jabba]]

After talking to zandr, it sounds like the "hardware" we'll be using for the client side load generation will be virtual machines running on the weave-kvm.mv.mozilla.com cluster.

Comment 8

[Justin Dow [:jabba]]

I think my end of this is done. Punting to zandr in case there is anything left for this.

Comment 9

[Zandr Milewski [:zandr]]

Man, there are a lot of bugs for this task. Done, running, Share and Enjoy.