567035 - Upgrade SeaMonkey buildbots to 0.8

Status: RESOLVED FIXED

(SeaMonkey :: Release Engineering, defect)

Description

[Robert Kaiser]

buildbot 0.8 should be better performance-wise, at least on the master, but it will need upgrades of all slaves as well, and probably even of twisted.
I'm following progress on this on the Firefox side closely, there's currently a blocker on Windows, but once things run well for them, we should apply it on our side as well.

Comment 1

[Robert Kaiser]

The .bash_profile changes in attachment 443360 [details] [diff] [review] are something we should look into and apply as possible. For Mac, we'll need attachment 445356 [details] [diff] [review].

For Windows, we'll need attachment 449271 [details] [diff] [review] in addition to the usual things.

Comment 2

[Robert Kaiser]

Those are the commands for Linux64:
--------------------------------------------------------------------------------
cd /tools/dist/
scp -i /home/seabld/.ssh/seabld_dsa [e-mail]:/tmp/centos5-x86_64/build/RPMs/buildbot-0.8.0-0moz2.x86_64.rpm .
scp -i /home/seabld/.ssh/seabld_dsa [e-mail]:/tmp/centos5-x86_64/build/RPMs/python26-2.6.5-0moz1.x86_64.rpm .
rpm -ihv --nodeps python26-2.6.5-0moz1.x86_64.rpm buildbot-0.8.0-0moz2.x86_64.rpm
rm buildbot-0.8.0-0moz2.x86_64.rpm python26-2.6.5-0moz1.x86_64.rpm
cd /tools/
rm -rf buildbot twisted* zope-interface*
ln -s buildbot-0.8.0 buildbot
--------------------------------------------------------------------------------

Note that the buildbot RPM contains twisted and zope.interface, but wrongly old versions of those are marked as dependencies, so installing with --nodeps is required and the old versions can be removed from /tools.

The Linux64 slave should be done and running on 0.8 now.

Comment 3

[Robert Kaiser]

Commands for 32bit Linux:
--------------------------------------------------------------------------------
cd /tools/dist/
scp -i /home/seabld/.ssh/seabld_dsa [e-mail]:/tmp/centos5-i686/build/RPMs/buildbot-0.8.0-0moz2.i686.rpm .
scp -i /home/seabld/.ssh/seabld_dsa [e-mail]:/tmp/centos5-i686/build/RPMs/python26-2.6.5-0moz1.i686.rpm .
rpm -ihv --nodeps python26-2.6.5-0moz1.i686.rpm buildbot-0.8.0-0moz2.i686.rpm
rm -f buildbot-0.8.0-0moz2.i686.rpm python26-2.6.5-0moz1.i686.rpm
cd /tools/
rm -rf buildbot twisted* zope-interface*
ln -s buildbot-0.8.0 buildbot
--------------------------------------------------------------------------------

Comment 4

[Robert Kaiser]

The upgrade should be done on all the Linux slaves now.

Comment 5

[Robert Kaiser]

Note to self: Mac DMG is at /mnt/puppet-files/production/darwin9-i386/build/DMGs/buildbot-0.8.0.dmg and installation should be similar to how we installed YASM in bug 571007 - I possible will try that this weekend.

Comment 6

[Robert Kaiser]

Commands for Mac, including some cleanup:

--------------------------------------------------------------------------------
cd /tools/dist/
sudo rm -rf ../Twisted* ../twisted* ../zope* ../Python-2.5.2* ../python ../mercurial* ../buildbot-* ../buildbot mercurial-1.2.1*
sudo chown seabld .
scp stage-old.mozilla.org:/mnt/puppet-files/production/darwin9-i386/build/DMGs/Python-2.5.2.dmg,mercurial-1.2.1.dmg,build-tools-0b149c0ad18d.dmg,buildbot-0.8.0.dmg} .
hdiutil attach Python-2.5.2.dmg
sudo installer -pkg /Volumes/Mozilla\ Python-2.5.2/Python-2.5.2.pkg -target /
hdiutil detach /Volumes/Mozilla\ Python-2.5.2
cd ..
sudo ln -s Python-2.5.2/ python
cd dist/
hdiutil attach mercurial-1.2.1.dmg
sudo installer -pkg /Volumes/Mozilla\ mercurial-1.2.1/mercurial-1.2.1.pkg -target /
hdiutil detach /Volumes/Mozilla\ mercurial-1.2.1
hdiutil attach build-tools-0b149c0ad18d.dmg
sudo installer -pkg /Volumes/Mozilla\ build-tools-0b149c0ad18d/build-tools-0b149c0ad18d.pkg -target /
hdiutil detach /Volumes/Mozilla\ build-tools-0b149c0ad18d
cd ..
sudo ln -s build-tools-0b149c0ad18d/ build-tools 
cd dist/
hdiutil attach buildbot-0.8.0.dmg
sudo installer -pkg /Volumes/Mozilla\ buildbot-0.8.0/buildbot-0.8.0.pkg -target /
hdiutil detach /Volumes/Mozilla\ buildbot-0.8.0
sudo rm *.dmg
--------------------------------------------------------------------------------

Also, attachment 445356 [details] [diff] [review] changes need to be applied via
sudo nano /Library/LaunchAgents/buildbot.start.slave.plist
nano ~/.profile

Comment 7

[Robert Kaiser]

Sorry, small correction to Mac Leopard changes:
--------------------------------------------------------------------------------
cd /tools/dist/
sudo rm -rf ../Twisted* ../twisted* ../zope* ../Python-2.5.2* ../python ../mercurial* ../buildbot-* ../buildbot mercurial-1.2.1*
sudo chown seabld .
scp stage-old.mozilla.org:/mnt/puppet-files/production/darwin9-i386/build/DMGs/{Python-2.5.2.dmg,mercurial-1.2.1.dmg,build-tools-0b149c0ad18d.dmg,buildbot-0.8.0.dmg} .
hdiutil attach Python-2.5.2.dmg
sudo installer -pkg /Volumes/Mozilla\ Python-2.5.2/Python-2.5.2.pkg -target /
hdiutil detach /Volumes/Mozilla\ Python-2.5.2
cd ..
sudo ln -s Python-2.5.2/ python
cd dist/
hdiutil attach mercurial-1.2.1.dmg
sudo installer -pkg /Volumes/Mozilla\ mercurial-1.2.1/mercurial-1.2.1.pkg -target /
hdiutil detach /Volumes/Mozilla\ mercurial-1.2.1
hdiutil attach build-tools-0b149c0ad18d.dmg
sudo installer -pkg /Volumes/Mozilla\ build-tools-0b149c0ad18d/build-tools-0b149c0ad18d.pkg -target /
hdiutil detach /Volumes/Mozilla\ build-tools-0b149c0ad18d
cd ..
sudo ln -s build-tools-0b149c0ad18d/ build-tools 
cd dist/
hdiutil attach buildbot-0.8.0.dmg
sudo installer -pkg /Volumes/Mozilla\ buildbot-0.8.0/buildbot-0.8.0.pkg -target /
hdiutil detach /Volumes/Mozilla\ buildbot-0.8.0
cd ..
sudo ln -s buildbot-0.8.0/ buildbot
cd dist/
rm *.dmg
--------------------------------------------------------------------------------
# apply attachment 445356 [details] [diff] [review] changes
sudo nano /Library/LaunchAgents/buildbot.start.slave.plist
nano ~/.profile
--------------------------------------------------------------------------------

And should be done for miniosx01-05, i.e. all our current minis!

Comment 8

[Robert Kaiser]

Er, but the bug isn't fixed, Windows and the master still need to be done *redears*

Comment 9

[Robert Kaiser]

Attachment: bash script for installing buildbot on Windows

Here's a bash script that should work on Windows slaves for installing buildbot 0.8, I'll try it on one shortly.

Comment 10

[Robert Kaiser]

Attachment: bash script for installing buildbot on Windows

Gah, no idea why I had that " in there, here's the correct script.

Comment 11

[Robert Kaiser]

Attachment: bash script for installing buildbot on Windows

Erm, and the last command is on d, not on c.

With that, installed successfully on both cn-sea-qm-win2k3-01 and cb-sea-win32-tbox, will do the others tomorrow.

Comment 12

[Robert Kaiser]

cb-seamonkey-win32-01 through -03 should be done now as well, so all slaves are upgraded, only the master is left!

Comment 13

[Robert Kaiser]

I'm starting to work on the master now, hope to get it up and running on 0.8 today to close this out finally.

Comment 14

[Robert Kaiser]

Commands for the master, including some cleanup of installed packages:

--------------------------------------------------------------------------------
cd /tools
sudo rm -rf buildbot* gcc* jdk* python* twisted* zope-interface*
sudo scp -i /home/seabld/.ssh/seabld_dsa [e-mail]:/mnt/puppet-files/production/centos5-i686/build/RPMs/{python25-2.5.1-0moz1,python26-2.6.5-0moz1,buildbot-0.8.0-0moz2,build-tools-ac05929dc0b1-0moz1,mercurial-1.6.3-0moz1}.i686.rpm /tools/dist/
sudo rpm --nodeps -Uhv /tools/dist/{python25-2.5.1-0moz1,python26-2.6.5-0moz1,buildbot-0.8.0-0moz2,build-tools-ac05929dc0b1-0moz1,mercurial-1.6.3-0moz1}.i686.rpm
sudo rm -f /tools/dist/{python25-2.5.1-0moz1,python26-2.6.5-0moz1,buildbot-0.8.0-0moz2,build-tools-ac05929dc0b1-0moz1,mercurial-1.6.3-0moz1}.i686.rpm
sudo ln -s python-2.5.1/ python
cd ~
hg clone http://hg.mozilla.org/build/buildbot/
cd buildbot/master
sudo /tools/buildbot-0.8.0/bin/python setup.py install
cd /tools
sudo ln -s buildbot-0.8.0/ buildbot
cd /tools/buildbotcustom/buildbotcustom/
sudo hg pull -r tip
sudo hg up -r default
--------------------------------------------------------------------------------

After this, both buildbot and buildbotcustom are updated. What is still to do is updating buildbot-configs and getting the build master up again.

Comment 15

[Robert Kaiser]

Here's steps for the config upgrade:

--------------------------------------------------------------------------------
cd /builds/master
buildbot upgrade-master .
--------------------------------------------------------------------------------

I needed a number of patches against buildbotcustom and buildbot-configs to make the upgrade-master succeed.

Comment 16

[Robert Kaiser]

I required the following patches:
http://hg.mozilla.org/build/buildbotcustom/rev/eb9ad4db68d1
The bug 614937 patch
http://hg.mozilla.org/build/buildbot-configs/rev/5e7e293634df
http://hg.mozilla.org/build/buildbot-configs/rev/a947bd3718d9
http://hg.mozilla.org/build/buildbot-configs/rev/080fe605ee90
http://hg.mozilla.org/build/buildbot-configs/rev/fb345052e1fa
http://hg.mozilla.org/build/buildbot-configs/rev/aee50cd4cc6a

...but it seems to be working now - let's see how well it does.