501255 - Upgrade virtual hardware for production-master

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Nick Thomas [:nthomas] (UTC+12)]

p-m is a busy vm that we needs more grunt to complete its work. It currently has 2GB of RAM, a single CPU, and runs on the Intel01 cluster (ESX host). We should upgrade it 3GB, two CPUs, and the faster Intel02 cluster. This will require a downtime because the VM will need to be shutdown, but it could be combined with updating VMWare Tools for the ESX upgrade.

I'm going to make the same changes to staging-master today to make sure that it works.

Comment 1

[[e-mail] (:bhearsum)]

We up'ed production-master to 3GB of ram and 2 CPUs in the downtime today. We tried to move it to the INTEL-02 cluster but it refused to power on, saying "Insufficient resources to satisfy configured failover level for HA". This is probably due to the 2GB ram reservation on it, which I did not want to fiddle with.

Phong, do you have any ideas about how to get production-master on INTEL-02?

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

Phong said on IRC to try powering it off, removing the reservation, migrating the VM, then re-adding it.

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

We had an outage on the master (bug 505669, oom-killer terminated buildbot processes) so I went ahead and migrated production-master over to Intel02. 

It was the 2GB RAM reservation that was causing the "Insufficient resources to satisfy configured failover level for HA" messages, so I've removed it for now. We need to find out what VMs are currently making reservations and confirm they need it.

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

Also removed changes.pck, which we know speeds up waterfall generation, and we really don't need every change for months to populate the recent changes column.

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

I think we're done here.