1462169 - Request for more RAM on syslog-proxy1.dmz.mdc1.mozilla.com

Status: RESOLVED FIXED

(Infrastructure & Operations :: Virtualization, task)

Description

[Alicia Smith [:phrozyn] (use NEEDINFO)]

We'd like to request more RAM on syslog-proxy1 in mdc1 as this host has difficulty keeping up if we do maintenance or some other issue comes up that prevents the consumers from connecting since it has a hard memory limit of 40% of system RAM in rabbitmq due to how erlang buffers events, (Roughly 1.6GB) We take in about 5k eps which means that limit hits fast as a result of queueing events. 

The purpose is not to allow indefinite queueing, but to at least allow some time for queueing should problems arise to give us time to resolve them and prevent losing those events due to a crash.

Right now the host has 4GB of ram, we'd like to increase that to 8GB.

Let me know if you need any further information!

Thanks!

Comment 1

[Greg Cox [:gcox]]

Sounds reasonable.  Ping Chris or me on IRC with a time you can take a 5min blip outage to upsize.  Any US-biz-hour is workable, outside-biz-hours is "give us some warning".

Comment 2

[Alicia Smith [:phrozyn] (use NEEDINFO)]

:michal` we'll need to coordinate with you on when we can stop sending events to syslog-proxy1 for about 10 mins.

Can  you provide some times that this outage of event processing will be acceptable?

Comment 3

[Michal Purzynski [:michal`] (use NEEDINFO)]

(In reply to Alicia Smith [:phrozyn] (use NEEDINFO) from comment #2)
> :michal` we'll need to coordinate with you on when we can stop sending
> events to syslog-proxy1 for about 10 mins.
> 
> Can  you provide some times that this outage of event processing will be
> acceptable?

Let tackle it while I'm still in US, ideally this week.

How about tomorrow, Texas and Pacific work hours? 1PM PDT? I bet I will have to restart all syslog-ng processes, but that's nothing a pssh cannot do ;-)

Comment 4

[Alicia Smith [:phrozyn] (use NEEDINFO)]

I believe that's doable for me as well.

Comment 5

[Greg Cox [:gcox]]

Working hours for both me and Chris; ping us in-channel, first of us to respond loses^wgets to help. :)

Comment 6

[Michal Purzynski [:michal`] (use NEEDINFO)]

(In reply to Greg Cox [:gcox] from comment #5)
> Working hours for both me and Chris; ping us in-channel, first of us to
> respond loses^wgets to help. :)

Ping me on IRC or through Signal. Slack is for slackers and sends pink unicorns for notifications. Pink unicorns are dropped on the edge firewall here.

Comment 7

[Chris Knowles [:cknowles]]

working with :phrozyn, took it down, added 4G RAM to it - updated in our tracking sheets.  

It's back in her hands - let me know if you need anything else.