734393 - Evenly distribute tegras across masters

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task, P3)

Description

[Armen [:armenzg]]

The load on bm19 and bm20 is high (from checking ganglia) while bm22 has no slaves connected to it.

If one of those two masters go down we will have half of our tegras up.

Comment 1

[Aki Sasaki (not active)]

This will take manual editing of buildbot.tac files and a restart of the affected tegra buildslaves.

Comment 2

[Armen [:armenzg]]

Is there a way to easily shut down all of the CP instances on a foopy?
I know I could have a for loop calling stop_cp.sh for each tegra.

Would that work? or is there an easier way that you know?

Comment 3

[Armen [:armenzg]]

I will be doing this on Monday.

Comment 4

[Mike Taylor [:bear]]

(In reply to Armen Zambrano G. [:armenzg] - Release Engineer from comment #3)
> I will be doing this on Monday.

stop_cp.sh will do that by default if you don't give it a parameter IIRC

Comment 5

[Armen [:armenzg]]

I will know in few days when I can get to this and adjust back the priority to P2 when working on it. P3 means I am not touching it and might be grabbing it back in 2 weeks.

Comment 6

[Armen [:armenzg]]

I believe Callek/bear already this did from looking at:
http://buildbot-master22.build.mtv1.mozilla.com:8201/buildslaves?no_builders=1

Comment 7

[Justin Wood (:Callek)]

yep was done this week. was holding off on resolving until things look(ed) stable enough to leave it be. It is certainly that way now.