491077 - Move nightly thunderbird2.0.0.x builds to release VMs (eg bm-xserve02 to bm-xserve05)

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Please move old thunderbird2.0.0.x processes from bm-xserve05 over to the existing thunderbird 2.0.0.x build machine bm-xserve02.

This would allow us to recycle bm-xserve05 for running unittests on active development lines. (bm-xserve05 is a hard-to-find PPC xserve, hence the juggling.)

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

Morphing a bit. 02 is the nightly box, doing Tb en-US and locales with tinderbox; 05 is the release box, doing both with buildbot calling tinderbox. It's easier to move to the release box for setup verification. We could just run tinderbox on bm-xserve05 but probably better to switch to buildbot all round. I'll work on this next week.

To do
* bootstrap config files for nightly t'bird builds (x2)
* new en-US & l10n builders on tb production master for nightly, appropriate schedulers
* decide whether to move linux and win32 over to buildbot too. Potentially turn off karma, cerberus-vm, crazyhorse, & patrocles and do all nightly and release builds on production-crazyhorse, production-cerberus-vm, bm-xserve05

Comment 2

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Sounds totally great, whichever way is easier/safer is totally fine by me. 

When filing this bug, I should have really asked to "consolidate TB2 processes to run on one machine, so freeing up one rare PPC xserve".

Comment 3

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

once this is done, can we have IT make a new image of this machine onto bm-xserve01? 

We did this already once before, so that if bm-xserve02 (production machine) died, we could quickly power up bm-xserve01 (the hot backup).... want to make sure we re-do this after all changes are done.

Comment 4

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

gentle ping... jhford is starting work on bm-xserve03,04 in bug#491889. 

Once bm-xserve05 is freed up, lets use that for firefox3.0? Also, has this been backed up (does it need to be backed up?)?

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

P3ing this for now, as a gaggle of PPCs were rescued during the office move and freeing up bm-xserve02 is less urgent.

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

bm-xserve02 is not responding and may well have a corrupted file system, so this got more urgent again (so that we have tb2.0.0.x nightlies).

I'm probably going to regret this, but lets move all the en-US and l10n builds to the buildbot master rather than just mac. It also allows us to delete four VMs (patrocles, crazyhorse, cerberus-vm, karma) - they're not very busy so it'll be a disk-space win mainly.

The control sequence is Buildbot -> Bootstrap -> Tinderbox, much like we do for releases. We'll need these changes
* add schedulers and builders to tb-master.cfg
* modify the tinder-config.pl's so that we can easily adjust some parameters using the Bootstrap config
* setup tinderbox on production slaves

There's no staging environment for the 1.8 branch so we're gonna have to do this hot. Patches upcoming for an untested first cut and then we can do followups.

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: buildbot master changes

This restores the factory we used for Fx2.0.0.x dev builds when they were driven by buildbot, using a 2 hourly periodic scheduler. It adds the equivalent factory for l10n repacks, which I don't think we've run like this before, and does that once a day. TinderConfig updates both en-US and l10n configs for tinderbox in one go so l10nNightlyFactory can be short. Passes checkconfig.

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: New bootstrap config

This is what Bootstrap uses when buildbot calls into it. It's a copy of tb-moz18-bootstrap.cfg, with 
* milestone & version set to nightly
* assorted buildDir getting s/Release/Nightly/
* append ".nightly" to logDir
* send mail to build-announce  (any blackhole will do ...)
* send results to Mozilla1.8 tree for en-US
* set sshUser to tbirdbld to continue to upload as that user. We have cltbld for the cvs operations because we need to be able to write tbox config changes back to the repo

Comment 9

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: CONFIGify tinderbox configs

This makes adjusting build parameters easier by putting in the markers that TinderConfig looks for. Values for the substitution come out of the bootstrap config. The variables that get the treatment are moz_cvsroot, BuildTree for en-US (somewhat pointlessly!), sshUser, sshKey, sshServer (all three for build upload). l10n config also makes some subs in %WGetFiles and $BuildLocalesArgs, where l10n_buildDir will be changing. Also updates a headers for the machine-in-use. 

In my local checkout I have MOZILLA_1_8_BRANCH in moz18_/, and MOZILLA_1_8_BRANCH_l10n in moz18_l10n.

Comment 10

[[e-mail] (:bhearsum)]

(In reply to comment #6)
> bm-xserve02 is not responding and may well have a corrupted file system, so
> this got more urgent again (so that we have tb2.0.0.x nightlies).
> 
> I'm probably going to regret this, but lets move all the en-US and l10n builds
> to the buildbot master rather than just mac.

I'm a little concerned about this being a rathole, but if you're going to chase things down I'm happy to do the reviews.
> There's no staging environment for the 1.8 branch so we're gonna have to do
> this hot. Patches upcoming for an untested first cut and then we can do
> followups.

There should be a staging-1.8-automation master on staging-master - dunno what state it is in.

Comment 11

[[e-mail] (:bhearsum)]

Comment on attachment 384571 [details] [diff] [review]
buildbot master changes

This seems fine...but keep in mind that we've never run nightly repacks through Bootstrap before (to my knowledge, at least). Boostrap::Repack::Execute is pretty straightforward though, so this _should_ be okay.

Comment 12

[[e-mail] (:bhearsum)]

Comment on attachment 384572 [details] [diff] [review]
New bootstrap config

Oops, meant to comment on this one:

I don't see the buildDir's on the slaves yet, but I assume that's because you haven't gotten that far yet. This config seems fine/valid though.

Comment 13

[[e-mail] (:bhearsum)]

Comment on attachment 384573 [details] [diff] [review]
CONFIGify tinderbox configs

Looks fine.

Best of luck with this!

Comment 14

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 384571 [details] [diff] [review]
buildbot master changes

>+ builderNames=[
>+  'tb-linux_dep_build', 
>+  'tb-win32_dep_build', 
>+  'tb-macosx_dep_build',
>+ ]
...
>+ builderNames=[
>+  'tb-linux_l10n_nightly', 
>+  'tb-macosx_l10n_nightly',
>+  'tb-win32_l10n_nightly', 
>+ ],
>+)

I've landed this with linux and windows commented out for now.

Comment 15

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 384571 [details] [diff] [review]
buildbot master changes

Updated production-1.8-master:/tools/buildbotcustom/buildbotcustom from rev 3a369fb4e64c (Jan 27 2009) to 0bff92bdd6fc (current tip). I've checked that there are no functional changes to BootstrapFactory, which is the only usage of buildbotcustom AFAICT. 

Reconfig'd the master with this attachment.

Comment 16

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Round 2 of fixes

* fixes the log directory in the bootstrap config (this already landed)
* then en-US mac build worked fine but l10n had a problem pulling cvs.m.o:/l10n code using ffxbld, just due to a key mismatch. CONFIGify the l10n mozconfig to fix it.

Comment 17

[Nick Thomas [:nthomas] (UTC+12)]

Mac l10n successfully built a locale, so that platform should be fixed. Hooked up linux to the master too.

Comment 18

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Fix nagios monitor for Mozilla1.8 tree

This patch accounts for the change in machine names.

The Windows build is moved over to production-patrocles. You have to symlink 
  build-seamonkey.pl --> build-thunderbird.pl -->
   ../mozilla/tools/tinderbox/build-thunderbird.pl
on windows, as we use
  . $topsrcdir/mail/config/mozconfig
in the mozconfig there. en-US is running now, l10n to follow. Nearly done here.

Comment 19

[Nick Thomas [:nthomas] (UTC+12)]

This was fixed a few days ago but I only just had a chance to do some checks.