576339 - Setup buildbot master on test-master02.build.mozilla.org

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

This new buildbot master is to reduce load on test-master01.b.m.o. 

Once the VM is created, we'll need to:

- install and configure master instance
- move some slaves from talos-master02.b.m.o over to this new
test-master02.b.m.o
- setup some platforms on test-master02, and disable those on talos-master02
- setup wait times monitoring 
- setup puppet configs? 
- anything else?


(bug#571573 was used to setup test-master01.b.m.o, and has lots of extra post-setup installs, some still in progress. Check there for details of what else to install here.)

Comment 1

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: add test-master02 to configs and to talos-r3

Comment 2

[[e-mail] (:bhearsum)]

Comment on attachment 455759 [details] [diff] [review]
add test-master02 to configs and to talos-r3

I don't think the ACTIVE_PLATFORMS lists are right here. We only want each platform on one master. I'd recommend the following scenario:
master1.cfg: win32 only
master2.cfg: win64, linux64, mac64
master3.cfg: linux, mac

This will reduce load on both tm01 and tm02, keep the mac machines (which cause the most load) on separate masters, and avoid us moving around any Windows machines.
This way the mac machines, which cause the most load, aren't on the same master, and we we won't have to move around any Windows machines.

Comment 3

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: add test-master02 to configs and to talos-r3

New patch alters ACTIVE_PLATFORMS so load is better distributed as per bhearsum's suggestions and so that no windows slaves need to move.

Here's a comparison of the builds being done on each master:
old:
master 1 builds: linux, linux64, win64, macosx64
master 2 builds: win32, macosx

new:
master 1 builds: win64, linux64, macosx64
master 2 builds: win32
master 3 builds: macosx, linux

So far I have:
* Set-up /tools with Twisted 8.0.1, buildbot (BUILDBOT_PRODUCTION), setuptools, easy_install, simplejson, and buildbotcustom (default branch)
* Installed cltbld's profile
* Set-up /builds/buildbot with a configs checkout and talos-r3 directory
* Copied cltbld's 'bin' from test-master01, updated the scripts

Still to do:
* Set-up wait times cronjobs
* Land config updates 
* Nagios monitoring

Comment 4

[[e-mail] (:bhearsum)]

Comment on attachment 455794 [details] [diff] [review]
add test-master02 to configs and to talos-r3

This still doesn't look right to me. You've got:
master1.cfg / talos-master02: linux, macosx, win32
master2.cfg / test-master01: linux, linux64, macosx, macosx64, win64
master3.cfg / test-master02: macosx64, linux64, win32, win64.

Comment 5

[Lukas Blakk [:lsblakk] use ?needinfo]

(In reply to comment #4)
> Comment on attachment 455794 [details] [diff] [review]
> add test-master02 to configs and to talos-r3
> 
> This still doesn't look right to me. You've got:
> master1.cfg / talos-master02: linux, macosx, win32
> master2.cfg / test-master01: linux, linux64, macosx, macosx64, win64
> master3.cfg / test-master02: macosx64, linux64, win32, win64.

wait a second - the platforms with None are *not* active on that master, correct? That's how it appears to me so this is why I have the patch set this way.

I'm trying to block linux, macosx, and win32 from talos-master02 so that only the 64 bit builders are on there. Same goes for the others.

Comment 6

[[e-mail] (:bhearsum)]

Look a bit closer, we list the active platforms in ACTIVE_PLATFORMS, with a value of None, and then fill in the details below:
ACTIVE_PLATFORMS = {'win32': None, 'macosx': None}
for p in ACTIVE_PLATFORMS.keys():
    ACTIVE_PLATFORMS[p] = deepcopy(PLATFORMS[p])

None is just a placeholder.

Comment 7

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: add test-master02 to configs and to talos-r3

amazing what you can convince yourself of, and not even *see*.

this time I have the appropriate platforms in the right master.cfg - leaving the win slaves where they are so we don't have to move them to different masters.

Comment 8

[[e-mail] (:bhearsum)]

Comment on attachment 457565 [details] [diff] [review]
add test-master02 to configs and to talos-r3

Looks good this time.

I only remembered about it now, but we need to update the tac generation script in build/tools, and update the following ref platforms:
* talos-r3-leopard-ref
* talos-r3-fed-ref

Comment 9

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: updates to buildbot-tac.py for test-master02

Comment 10

[[e-mail] (:bhearsum)]

I fixed the Buildbot install with:
su -
cd /tools
rm -rf buildbot
hg clone http://hg.mozilla.org/build/buildbot
cd buildbot
hg up -r 1a62 # get buildbot 0.7.10p1 instead of 0.8.0
export PATH=/opt/local/bin:/tools/buildbot/bin:/tools/twisted/bin:/tools/twisted-core/bin:/tools/python/bin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/home/cltbld/bin
export PYTHONPATH=/tools/buildbotcustom:/tools/buildbot/lib/python2.5/site-packages:/tools/twisted/lib/python2.5/site-packages:/tools/twisted-core/lib/python2.5/site-packages:/tools/zope-interface/lib/python2.5/site-packages/
mkdir -p     /tools/buildbot/lib/python2.5/site-packages/
python setup.py install buildbot

Comment 11

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 457565 [details] [diff] [review]
add test-master02 to configs and to talos-r3

We also need to make the same type of change as attachment 452723 [details] [diff] [review] to keep releases unit tests.

Comment 12

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: add test-master02 to mozilla2/configs for release unittests

Comment 13

[Lukas Blakk [:lsblakk] use ?needinfo]

Comment on attachment 457565 [details] [diff] [review]
add test-master02 to configs and to talos-r3

http://hg.mozilla.org/build/buildbot-configs/rev/723582c57bf9

Comment 14

[Lukas Blakk [:lsblakk] use ?needinfo]

Comment on attachment 458019 [details] [diff] [review]
updates to buildbot-tac.py for test-master02

http://hg.mozilla.org/build/tools/rev/ca1cffa8fe30

Comment 15

[Lukas Blakk [:lsblakk] use ?needinfo]

test-master02 has been started up, fed and leopard slaves attached. test-master01/talos-master02 reconfig'd so that load is more balanced.

Comment 16

[[e-mail] (:bhearsum)]

Comment on attachment 458447 [details] [diff] [review]
add test-master02 to mozilla2/configs for release unittests

changeset:   2716:71384333fe4b

Comment 17

[Nick Thomas [:nthomas] (UTC+12)]

I've updated the pending graph, http://build.mozilla.org/builds/pending/, let me know if you see any strangeness. test-master01 is confused about some stuck pending jobs, needs a restart at a quiet moment.

Comment 18

[[e-mail] (:bhearsum)]

Lukas, could you please also update the crontab today, and manually run the wait times report?

Comment 19

[Lukas Blakk [:lsblakk] use ?needinfo]

(In reply to comment #18)
> Lukas, could you please also update the crontab today, and manually run the
> wait times report?

Done.

Comment 20

[Nick Thomas [:nthomas] (UTC+12)]

update_from_files is not working because it is missing the config file, and a bunch of python sql modules. Working on fixing that.

Comment 21

[Nick Thomas [:nthomas] (UTC+12)]

Steps to install sql goo for dumping history into the status db:

# we get 5.0.77-4.el5_5.3 from updates
yum install mysql mysql-devel

source ~cltbld/.bash_profile 

# this is actually a buildbot 0.8.0 dep
wget http://pypi.python.org/packages/source/s/simplejson/simplejson-2.1.1.tar.gz#md5=0bbe3a2e5e4cac040013733aca159d89
tar -zvxf simplejson-2.1.1.tar.gz 
cd simplejson-2.1.1
python setup.py build
python setup.py install

cd ~
wget http://downloads.sourceforge.net/project/sqlalchemy/sqlalchemy/0.5.5/SQLAlchemy-0.5.5.tar.gz?use_mirror=cdnetworks-us-1
tar -zvxf SQLAlchemy-0.5.5.tar.gz 
cd SQLAlchemy-0.5.5
python setup.py build
python setup.py install

cd ~
wget http://downloads.sourceforge.net/project/mysql-python/mysql-python/1.2.3/MySQL-python-1.2.3.tar.gz?use_mirror=cdnetworks-us-2
tar -zvxf MySQL-python-1.2.3.tar.gz 
cd MySQL-python-1.2.3
python setup.py build
python setup.py install

Now I'm hitting an auth issue.

Comment 22

[Nick Thomas [:nthomas] (UTC+12)]

Auth now set to match other hosts (bug 579567). ~/.my.cnf removed and ~/.bbcustom_update_from_files-rev3.conf added.

Manual initial import running ... enabling crontab pending completion of that.

Comment 23

[Nick Thomas [:nthomas] (UTC+12)]

History dump now complete, and the cronjob enabled.