514611 - Set up extra capacity for 0.8 release

Status: RESOLVED FIXED

(Cloud Services Graveyard :: Server: Sync, defect, P1)

Description

[Ragavan S [:rags]]

As discussed in the post-mortem meeting today, let's make sure we have some extra capacity lined up and ready to go to handle spikes for the 0.7 release.

Let's use this bug to track what hardware we have available in our hands, our plan for getting it installed and ready to go and how we plan to use it, if needed.

As a guiding number, 0.7 should handle at least 250K users comfortably.

Comment 1

[Zandr Milewski [:zandr]]

I think we need three more pm-weavefs## class machines. This will bring us to a total of 8 boxes, 4 master/slave pairs.

mrz: I think we have the BL460c's (need 3) but we don't have any SB40c's. Let's get those ordered.

Comment 2

[Zandr Milewski [:zandr]]

OK, based on the conversation today, we're talking about much bigger numbers.

We want 3 chassis, populated as follows:

2 BL2x220c's as webheads.
6 BL460c's with 6 SB40c's 
Drives in the SB40c's should be 15000RPM SAS drives. drives in the BL460c's aren't performance critical.

Toby and I looked at some IO stats and we think that the current load peaks reach about 2/3rds capacity on the current clusters. Those peaks work out to approximately 4500 concurrent users per cluster. So let's pad a little and say 6000 concurrent users per cluster with the current system.

6 clusters/chassis = 36,000 concurrent users/chassis
3 chassis = 108,000 concurrent users. At 3:1, that's 324,000 users.

That's more than the 250k number we're targeting, so assume I'll steal a cluster or two for the stage environment.

This also leaves 2 slots per chassis in case I'm wrong about the number of webheads, but the current webheads are very lightly loaded.

Comment 3

[Ragavan S [:rags]]

We have extra hardware on the way that should give us at least 2 new clusters (double our current capacity). 

Matthew/Zandr are figuring out the optimal hardware platform for 1.0 GA.

Comment 4

[Ragavan S [:rags]]

Zandr mentioned that for whatever reason the extra hardware mentioned in comment #3 wasn't ordered until Friday afternoon (Sep 25). 
That said, this is not a blocker for 0.7 anymore, so moving this to 0.8.

Comment 5

[matthew zeier [:mrz]]

For clarification, the only thing ordered was a storage blade for Mike to do his testing work on.

Comment 6

[Zandr Milewski [:zandr]]

2 additional nodes (=4 servers) ordered, ETA end of next week

Comment 7

[Mike Connor [:mconnor]]

moving beta blockers to block 1.0 beta

Comment 8

[Ragavan S [:rags]]

Zandr thinks we are OK with current capacity for Beta.

Comment 9

[Zandr Milewski [:zandr]]

I'll light these up next week, but configuring them will depend on the results of bug 526804

Comment 10

[Zandr Milewski [:zandr]]

There's more than enough capacity available for the near term, I'm tracking the bad blade in bug 532313.

Closing this so it doesn't block b3