532676 - Upgrade buildbot, buildbotcustom and buildbot-configs for SeaMonkey builders

Status: RESOLVED FIXED

(SeaMonkey :: Release Engineering, defect)

Description

[Robert Kaiser]

We need to upgrade buildbot installations, buildbotcustom pulls, and buildbot-configs to current states on SeaMonkey builder machines (buildmaster and slaves).

The current state is from somewhere around August 2009, from there on we tried to stay as stable as possible while going for a final 2.0 release.

The new minis (including the re-imaged cb-sea-miniosx01, see bug 526206) come with buildbot 0.7.10p1, which we should be using everywhere now, buildbotcustom has seen a number of changes, including stuff for nightly L10n updates, which we also want to adopt, and we also might want to shift more of the code from config into buildbotcustom, like it was done for the main Mozilla masters.

Comment 1

[Robert Kaiser]

For Linux slaves, did the following (buildbotcustom shouldn't matter on slaves, just did it for completeness - and linux64-01 doesn't even have it):

cd /tools/buildbot
sudo hg pull -u
sudo python setup.py install
cd /tools/buildbotcustom/buildbotcustom/
sudo hg pull -u
rm /builds/slave/twistd.log
rm -rf /builds/slave/comm-*/*

Done on cb-sea-linux-tbox, cb-seamonkey-linuxdebug-01, cn-sea-qm-centos5-01, cb-seamonkey-linux64-01. (cb-seamonkey-linux-01 is the master, -02 is down completely because of parallels restrictions.)

Comment 2

[Robert Kaiser]

On Windows slaves, did the following:

cd /d/dist/buildbot/
hg pull -u
python setup.py install
rm /e/builds/slave/twistd.log*
rm -rf /e/builds/slave/comm-*/*

Done on cb-sea-win32-tbox, cb-seamonkey-win32-01, cb-seamonkey-win32-02, cn-sea-qm-win2k3-01

Comment 3

[Robert Kaiser]

On Mac slaves, did the following:

cd /tools/buildbot/
sudo hg pull -u
sudo python setup.py install
rm /builds/slave/twistd.log*
rm -rf /builds/slave/comm-*/*

Done on cb-seamonkey-osx-01, cb-seamonkey-osx-02, cb-seamonkey-osx-03 (-04 is down for parallels reasons, cb-sea-miniosx01 should already have current code).

Comment 4

[Robert Kaiser]

On the master (cb-seamonkey-linux-01), did the following:

buildbot stop /builds/master
cd /tools/buildbot
sudo hg pull -u
sudo python setup.py install
sudo python setup.py install --prefix=/tools/buildbot
cd /tools/buildbotcustom/buildbotcustom/
sudo hg revert --all
sudo hg import --no-commit https://bugzilla.mozilla.org/attachment.cgi?id=417254
sudo hg import --no-commit -f https://bugzilla.mozilla.org/attachment.cgi?id=417272
rm -rf /builds/slave/comm-*/*
cd ~/buildbot-configs/
hg pull -u
cd /builds/master && buildbot checkconfig
buildbot start .

For buildbotcustom, we need to be careful, as we had local patches applied. We know that we need to apply two of them again, so applied them manually after upgrading (one will go upstream soon, I hope, the other should get obsoleted by a better solution).
We still had slave directories around from the times when a slave was run on the same VM, so also cleaned those up.

Comment 5

[Robert Kaiser]

Hrm, I just realized that the Linux and Mac slaves aren't suing the new buildbot yet - just like the master, they need to have it installed in /tools/buildbot

So, going into all the Linux and Mac slaves again and doing the following:
buildbot stop /builds/slave
cd /tools/buildbot
sudo python setup.py install --prefix=/tools/buildbot
sudo reboot

(The reboot should make buildbot start up automatically again.)


Done on cn-sea-qm-centos5-01, cb-sea-linux-tbox, cb-seamonkey-linuxdebug-01, cb-seamonkey-linux64-01, cb-seamonkey-osx-01, cb-seamonkey-osx-02, cb-seamonkey-osx-03

Comment 6

[Robert Kaiser]

For whatever reason MSYS on the Windows slaves didn't set MACHTYPE automatically like it should and that made the compile processes fail, so I added

export MACHTYPE=i686-pc-msys

to seabld's .bash_profile there, which fixed that for the moment.

Comment 7

[Robert Kaiser]

One more local path needed for the master:

https://bugzilla.mozilla.org/attachment.cgi?id=417299

Comment 8

[Robert Kaiser]

OK, all trees are back to their previous green/orange states, master ist running current buildbot and buildbotcustom (with above-mentioned patches), all slaves are running current buildbot, this mission has been a success!

Comment 9

[Robert Kaiser]

I found a better solution for the problem in comment #6 and am deploying it right now, removing the hack from there:

echo "export HOSTTYPE MACHTYPE OSTYPE SHELL" >> /etc/profile

This is actually a bug in MozillaBuild 1.4, and releng fixed it with the OPSI package in http://hg.mozilla.org/build/opsi-package-sources/file/55fbdeba037f/profilevars/CLIENT_DATA/profilevars.ins which does exactly what the one-liner above achieves.