Releng puppetmasters are heavily loaded

RESOLVED FIXED

Status

RESOLVED FIXED
4 years ago
4 years ago

People

(Reporter: dustin, Assigned: dustin)

Tracking

Details

Since taking down the AWS puppetmasters, the two onsite puppetmasters are heavily loaded.  Even if they can handle the load together, we don't have N+1 redundancy.

rail, do you have a ballpark figure for the annual cost of a master in AWS?  It's about $1600/yr for vmware.  Just trying to decide where this would best be deployed.
Ah, the IT TCO calculator I used to get that figure has estimates for RHEL on AWS, too -- about half of that ($800/yr), and that's including the RHEL license.  So if we were to spin up a new one, AWS would be the place to do it.

That said, maybe just putting more CPUs on these nodes is a better idea, particularly since we just did a hell of a lot of work to tear down the four AWS puppetmasters.
virt folks, can we get the CPU count on releng-puppet{1,2}.srv.releng.scl3 upped?  3 each is a good place to start, although for N+1 four each is probably better.
Component: RelOps: Puppet → Virtualization
QA Contact: dustin → cshields
Summary: Spin up a new puppetmaster → Releng puppetmasters are heavily loaded
I think we'll need to up the RAM a little bit, too - bug 1096734 had us suffering OOMs
Blocks: 1102822

Comment 4

4 years ago
went from 2/4 to 4/6 (vCPU/vRAM) with :dustin handing us the reboots.  2 had (apparently?) OOM'ed so hard that vSphere didn't even know it had tools; verified clean after reboot.
Status: NEW → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.