983466 - let's stand up 2 more Bedrock nodes in PHX1

Status: RESOLVED FIXED

(Infrastructure & Operations :: Virtualization, task)

Description

[Jake Maul [:jakem]]

Hi all,

It seems like we've got enough power to host things under optimal (no failures) conditions, but I don't think we have enough to handle the load in PHX1 should SCL3 fail. I'm also a little twitchy about the load we have on 2 of the 3 nodes in PHX1, if one of them fails.

If we bump this up to 5 nodes, we'll have approximately equal per-node loading on all nodes across SCL3 and PHX1, which seems like a good thing in itself (although the VMs are less powerful, at least they'll be doing the same amount of work).

This seems like it would be a better overall scenario as compared to allocating more cores and more RAM to the existing nodes. However if anyone disagrees, we could go that route instead.

bedrock4.webapp.phx1
bedrock5.webapp.phx1
2 cores, 8GB RAM, 40GB disk should be fine

Thanks!

Comment 1

[Chris Knowles [:cknowles]]

Alright, I've got this, and will have them for you tomorrow.

CJK

Comment 2

[Chris Knowles [:cknowles]]

Alright, those are spun up - ticket 983644 has been filed for netflows.  (mod if you need to) 

puppet/nagios initial work has been done, ready for your customization.

CJK

Comment 3

[Josh Mize [:jgmize]]

Will someone please add me to the cc list for https://bugzilla.mozilla.org/show_bug.cgi?id=983664 so that I can track the progress?