419071 - update buildbot masters to use build slaves randomly

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[alice nodelman [:alice] [:anode]]

Currently, builds are distributed to talos boxes sequentially (box 1 is always preferred, if it isn't available then box 2 is selected, etc).  This can mean that box 3, 4, etc are never touched for tests that run quicker than builds are generated.  We can also end up in bad states where box 1 is failing out quickly but then immediately served another build - consuming all available builds without managing to test anything.

I'd like to more equitably distribute builds to test boxes using this patch:

http://buildbot.net/trac/ticket/36

Comment 1

[Rob Campbell [:rc] (:robcee)]

note that this will affect all other instances of buildbot on qm-rhel02, namely the unittest master and talos staging master (for now). This shouldn't be a problem.

Are we going to apply this to the production branch?

Comment 2

[alice nodelman [:alice] [:anode]]

We should go with the patch that made it into buildbot:

http://buildbot.net/trac/changeset?format=diff&new=349&old=310&new_path=buildbot%2Fprocess%2Fbuilder.py&old_path=buildbot%2Fprocess%2Fbuilder.py

Comment 3

[Rob Campbell [:rc] (:robcee)]

I'm taking this.

I need to run this on the unittest staging environment for a day or so to make sure nothing breaks there.

Comment 4

[Rob Campbell [:rc] (:robcee)]

somehow I missed installing this on qm-buildbot01. Below is the console output from a successful install.

[root@qm-buildbot01 buildbot]# patch -p0 <random.patch 
patching file buildbot/process/builder.py
Hunk #1 succeeded at 1 with fuzz 2.
Hunk #2 succeeded at 227 (offset 2 lines).
[root@qm-buildbot01 buildbot]# python setup.py build
running build
running build_py
copying buildbot/process/builder.py -> build/lib/buildbot/process
running build_scripts
[root@qm-buildbot01 buildbot]# python setup.py install
running install
running build
running build_py
running build_scripts
running install_lib
copying build/lib/buildbot/process/builder.py -> /tools/python/lib/python2.5/site-packages/buildbot/process
byte-compiling /tools/python/lib/python2.5/site-packages/buildbot/process/builder.py to builder.pyc
/tools/python/lib/python2.5/site-packages/buildbot/process/builder.py:386: Warning: 'with' will become a reserved keyword in Python 2.6
/tools/python/lib/python2.5/site-packages/buildbot/process/builder.py:387: Warning: 'with' will become a reserved keyword in Python 2.6
/tools/python/lib/python2.5/site-packages/buildbot/process/builder.py:388: Warning: 'with' will become a reserved keyword in Python 2.6
/tools/python/lib/python2.5/site-packages/buildbot/process/builder.py:392: Warning: 'with' will become a reserved keyword in Python 2.6
running install_scripts
changing mode of /tools/python/bin/buildbot to 775
running install_data
running install_egg_info
Removing /tools/python/lib/python2.5/site-packages/buildbot-0.7.5-py2.5.egg-info
Writing /tools/python/lib/python2.5/site-packages/buildbot-0.7.5-py2.5.egg-info
[root@qm-buildbot01 buildbot]# 


restarting the buildbot master. Should see builds coming up on random machines now. Hopefully.

Can we get this installed into BUILDBOT_MOZILLA_PRODUCTION?

Comment 5

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

This seems like it would be useful for all the masters used by build also. Can we also get this installed on staging-1.8-master, staging-1.9-master, production-1.8-master, production-1.9-master? 

Comment 6

[[e-mail] (:bhearsum)]

(In reply to comment #5)
> This seems like it would be useful for all the masters used by build also. Can
> we also get this installed on staging-1.8-master, staging-1.9-master,
> production-1.8-master, production-1.9-master? 
> 

AFAIK all of these only have one slave right now. Even with two, dep builds would still only alternate between the builders (because they build continuously). It's a bit of a PITA to upgrade Buildbot right now so we may as well wait until we upgrade Buildbot on these to pick this up.