676055 - Test buildbot master migration during next downtime

Status: RESOLVED WONTFIX

(Release Engineering :: General, defect, P3)

Description

[Chris Cooper [:coop] (he/him)]

One of the things we discussed at the releng work week was building enough extra master capacity that we could migrate slaves from a busy master to an empty master. We could then stop the original master, update the code, and start it back up. This would allow us to stop doing reconfigs, thereby avoid some of the problems/slowdowns associated with reconfigs.

We talked about doubling the number of masters we currently have. After discussing with Dustin, I think the easiest way to do this would be to create a duplicate master on every master host. However, we're unsure whether migrating slaves between 2 loaded masters running on the same hosts is feasible, i.e. it is untested: load may spike, etc. The first step should be to create a duplicate master on one already-loaded test master, and then attempt a trial migration process during the next downtime window to see if/how it works.

If successful, we could do the same for all our masters, and even think about writing tools to automate switching masters on the same host.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to comment #0)
> We talked about doubling the number of masters we currently have. After
> discussing with Dustin, I think the easiest way to do this would be to
> create a duplicate master on every master host.

I recall we were worried about having enough memory on the test masters in this scenario. Was that part of your discussion ?

Comment 2

[Chris Cooper [:coop] (he/him)]

(In reply to comment #1) 
> I recall we were worried about having enough memory on the test masters in
> this scenario. Was that part of your discussion ?

Yes, that's the crux of it, actually. 

The only way to figure out whether this will work is to try it on a single heavily-loaded master. Also why we want to try this during a downtime.

Comment 3

[Armen [:armenzg]]

Marking it so we see it for next downtime.

Comment 4

[John Ford [:jhford] CET/CEST Berlin Time]

(In reply to Armen Zambrano G. [:armenzg] - Release Engineer from comment #3)
> Marking it so we see it for next downtime.

this work does not mesh with the downtime for today.  I am going to leave the bug in its current state so that the next downtime picks it up.

Comment 5

[Armen [:armenzg]]

I re-read this bug and does not seem that we are ready to do anything.
I probably got confused with the kvm auto-balancing that we wanted with ServerOps.

Comment 6

[Chris Cooper [:coop] (he/him)]

Reconfigs are less painful now, and we have plans to move away from buildbot anyways.