470462 - Setup VMWare reservations for buildbot master VMs

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

In bug#467322, creating a VMWare reservation on production-master VM helped prevent overload, and then dropped connections from slaves.

Can you do the same for our other production buildbot master VMs:

production-1.8-master
production-1.9-master
qm-rhel02
qm-buildbot01
sm-try-master

Comment 1

[[e-mail] (:bhearsum)]

Have we seen connections dropping on these? I'm a little weary to active this unnecessarily and possibly steal cycles from machines that do need them.

Comment 2

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to comment #1)
> Have we seen connections dropping on these? I'm a little weary to active this
> unnecessarily and possibly steal cycles from machines that do need them.

Not yet, but I'd like to be proactive with this, given the pain of the problems we had when it happened to production-master. 

Adding VMware reservations will prevent other VMs from stealing cycles/memory from our master VMs, with the resulting dropped connections and tree woes... which seems like the right priority imho. If there are any VMs more important then our masters, maybe they should be added to this list also?

Comment 3

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

any update?

Comment 4

[Phong Tran [:phong]]

Resource allocation has been set to "high" for the 5 listed VM's.  I don't think this is necessary anymore since we have enough resources in the build cluster now.

Comment 5

[[e-mail] (:bhearsum)]

The past couple of days we've noticed timeouts during the compile phase of win32 builds. The Intel group doesn't look overloaded at all, so I wonder if this is related to these reservations. Is there any way to get data on this?

Comment 6

[[e-mail] (:bhearsum)]

ping? Maybe we can remove reservations from production-1.8-master and production-1.9-master to test the theory in comment #5?

Comment 7

[Phong Tran [:phong]]

resource allocation changed to normal.

Did you know that production-1.8-master only has 512 MB of RAM?

Comment 8

[[e-mail] (:bhearsum)]

Thanks Phong, we'll watch for hanging builds now. Hopefully this will prevent them.