850200 - increase the swap on the scl3 bmo webheads from 2g to 8g

Status: RESOLVED WONTFIX

(mozilla.org Graveyard :: Server Operations, task)

Description

[:glob ✱]

rom bug 849721:

- the nagios is alerts (low swap) may not be a new issue
  - phx webheads had 8g swap, scl3 has 2g
  - ashish noticed that phx was pretty heavy on swap consumption (swap avg 1.1G over a year), however this wasn't a high enough percentage to trigger nagios

would it be possible to increase the swap on the scl3 bmo webheads from 2g to 8g?

Comment 1

[Shyam Mani [:fox2mike]]

Great catch, not sure I even realized this.

Comment 2

[Reed Loden [:reed]]

If the box is swapping so much on a regular basis, why not just add more RAM (if the problem can't be solved in other ways)?

Comment 3

[Shyam Mani [:fox2mike]]

(In reply to Reed Loden [:reed] from comment #2)
> If the box is swapping so much on a regular basis, why not just add more RAM
> (if the problem can't be solved in other ways)?

As always, you don't have visibility into everything :)

The new webheads have 16GB of RAM. The previous webheads only had 8...so this is a simple step to see if this fixes the issue we're seeing.

Comment 4

[Shyam Mani [:fox2mike]]

Going to move this to server-ops.

Comment 5

[Adrian J Fernandez [:Aj]]

Is this still needed? If so can it be done on Tuesday (2013/10/16)?

Comment 6

[Shyam Mani [:fox2mike]]

(In reply to Adrian Fernandez [:Aj] from comment #5)
> Is this still needed? If so can it be done on Tuesday (2013/10/16)?

I guess we could do it. Will it be a downtime-less change? If not, we need to go to the CAB for this.

Comment 7

[Adrian J Fernandez [:Aj]]

Well there are two main ways of doing it.

1. Offline, resize the partitions so that swap (in this case /dev/sda2) gets bigger and / (/dev/sda3) gets a smaller.
2. Online, Add an actual swap file.

I am more of a fan of #1 as to keep things clean and simple but can do 2 if we don't want to schedule any possible downtime (technically there shouldn't be as we would drain in Zeus then perform the changes).

Comment 8

[Shyam Mani [:fox2mike]]

We should do 1 and do it on a weekend. This means we should drain a node from zeus, do the work and then undrain. Good opportunity to do kernel upgrades too. Before we do that, I'd cycle back to :glob and see if he still feels it's good to have this.

Comment 9

[:glob ✱]

imho we've run with the current configuration for long enough that it probably isn't worth the effort to change this.

Comment 10

[Shyam Mani [:fox2mike]]

Okay then :)