If you think a bug might affect users in the 57 release, please set the correct tracking and status flags for Release Management.

increase memory for VM that is cruncher.build.mozilla.org and install PAE kernel

RESOLVED FIXED

Status

Infrastructure & Operations
RelOps
RESOLVED FIXED
6 years ago
4 years ago

People

(Reporter: bear, Assigned: arr)

Tracking

Details

(Reporter)

Description

6 years ago
yesterday, and again today, nagios has alerted that cruncher was having high load spikes.

yesterday we looked and found a lot (17!) of reporter.py jobs running and I noticed that swap space was very low and a number of processes were deadlocked because of swap.  killing the reporter.py jobs allowed the load to recover.

today I looked and found again a lot of reporter.py jobs again and swap space at zero!  I have killed the 17 running jobs and load is again reducing.  Also noted that by killing those jobs swap space free went from 0 to 1883520k.

until the glorious future that is scl3 arrives, and we can move the various memory hungry processes that are all running on cruncher (redis, rabbit, web api's, json-building reports) I would like to increase the memory.

Doubling it would be sweet, but I imagine any significant increase will be enough of a bandaid to keep the patient alive during transport to the trauma center otherwise known as scl3.

thanks
(Assignee)

Comment 1

6 years ago
In order to increase the RAM, we need to shut down the virtual machine.  When is a good time?
(Assignee)

Comment 2

6 years ago
I've upped the memory in vmware, but it's still only seeing 4G.  It doesn't look it has a PAE kernel, so I'm going to pass this off to someone more comfortable with mucking with the kernel to schedule that.
Summary: increase memory for VM that is cruncher.build.mozilla.org → increase memory for VM that is cruncher.build.mozilla.org and install PAE kernel
(Assignee)

Comment 3

6 years ago
Okay, so it was as easy as I thought.  PAE kernel installed, machine rebooted, and we're at 8G now.
Assignee: server-ops-releng → arich
Status: NEW → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → FIXED
Component: Server Operations: RelEng → RelOps
Product: mozilla.org → Infrastructure & Operations
You need to log in before you can comment on or make changes to this bug.