991372 - add additional DEA node to stackato 2.10.6 dev cluster

Status: RESOLVED FIXED

(Infrastructure & Operations :: IT-Managed Tools, task)

Description

[Chris Turra [:cturra]]

per the email i sent out earlier today, we're running very close to capacity in the dev stackato cluster. this bug is to track to the work required to spin up an additional dea node.

  https://mail.mozilla.org/pipermail/paas-users/2014-April/000230.html

Comment 1

[Chris Turra [:cturra]]

i discussed this with stackato support last night. here is a really good explanation of the situation we're in:

"""
So there's a few things that are at play here.  The main thing is the difference between provisioned memory and actual memory usage.  When an app is deployed, the amount requested in the mem: key is promised to that app.  Stackato doesn't allow over-provisioning of resources.  If you push an app requesting 128MB of memory, once it's deployed that memory isn't available for provisioning any more, even if the app is only using half of it.

Another factor here is the maximum amount of DEA memory that you've configured to be available to apps.  The default value here is 80%, as in 80% system memory available for apps and 20% reserved for system.  That's a hard cap and sounds like what you're running in to here. 
"""


what i am going to propose in the mail-list this morning is to take an outage window this evening to increase the available memory on the DEA nodes from 8G -> 16G. this should give us about 2x the capacity we have now. tho, i am not certain we will get linear growth, it should give us quite a bit of head room. once we hit that ceiling, we'll add a fourth DEA server into the mix.

Comment 2

[Chris Turra [:cturra]]

https://mail.mozilla.org/pipermail/paas-users/2014-April/000232.html

Comment 3

[Chris Turra [:cturra]]

this work is being pushed off until tomorrow (04/04) at 16:00 pacific so it doesn't impact the moderator application which is being used for tomorrows town hall.

Comment 4

[Chris Turra [:cturra]]

this maintenance is now complete \o/