yesterday, and again today, nagios has alerted that cruncher was having high load spikes. yesterday we looked and found a lot (17!) of reporter.py jobs running and I noticed that swap space was very low and a number of processes were deadlocked because of swap. killing the reporter.py jobs allowed the load to recover. today I looked and found again a lot of reporter.py jobs again and swap space at zero! I have killed the 17 running jobs and load is again reducing. Also noted that by killing those jobs swap space free went from 0 to 1883520k. until the glorious future that is scl3 arrives, and we can move the various memory hungry processes that are all running on cruncher (redis, rabbit, web api's, json-building reports) I would like to increase the memory. Doubling it would be sweet, but I imagine any significant increase will be enough of a bandaid to keep the patient alive during transport to the trauma center otherwise known as scl3. thanks
In order to increase the RAM, we need to shut down the virtual machine. When is a good time?
I've upped the memory in vmware, but it's still only seeing 4G. It doesn't look it has a PAE kernel, so I'm going to pass this off to someone more comfortable with mucking with the kernel to schedule that.
Okay, so it was as easy as I thought. PAE kernel installed, machine rebooted, and we're at 8G now.