427769 - Test Thunderbird 2.0.0.x automation on staging setup

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Nick Thomas [:nthomas] (UTC+12)]

Rob worked up this list of steps, to try building Tb2.0.0.x on the 1.8 staging environment:

I think that the following is needed first in CVS. Is there a bug  
filed on this yet? Should probably do that so there's a place to  
review this stuff:

1) create new mozilla/tools/release/configs/tb-moz18-staging-
bootstrap.cfg (based on fx-moz18-staging-bootstrap.cfg is probably  
easiest)

2) create new master.cfg for staging-1.8-master, based on mozilla/
tools/buildbot-configs/automation/staging/master.cfg

3) annotate mozilla/tools/tinderbox-config/thunderbird configs with "#  
CONFIG" statements. Could look at the firefox configs for examples),  
branches are MOZILLA_1_8_BRANCH_release and  
MOZILLA_1_8_BRANCH_l10n_release)

Nick, does this look right? I don't see any firefox specific dirs in  
the Makefile (it knows how to read the bootstrap.cfg for that stuff  
now), the only place on the master that needs any changes should be  
the master.cfg, the bootstrap.cfg comes from the checkout on the slave  
now and the filename is configurable in the BootstrapFactory now.

If I'm not missing anything, then when the above lands, the only  
things needed on staging-1.8-master I think should be:

1) refresh CVS mirror, to pick up the above changes
cd /home/cltbld/real/mozilla/tools/release && make cvsmirror_main

Note that staging-1.8-master is almost out of disk space on the /cvs  
partition, this should probably be fixed before refreshing the mirror.

2) update master.cfg and restart buildbot
cd /builds/buildbot/TestBot/buildbot-configs && cvs up
# The /builds/buildbot/TestBot/master.cfg symlink should point at the  
new Thunderbird master.cfg
buildbot restart /builds/buildbot/TestBot/

Comment 1

[Rick Tessner]

Attachment: [checked in] Thunderbird bootstrap config for staging

This will be added as mozilla/tools/release/configs/tb-moz18-staging-bootstrap.cfg to the trunk

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 314355 [details] [diff] [review]
[checked in] Thunderbird bootstrap config for staging

r+ and checked in

Checking in tb-moz18-staging-bootstrap.cfg;
/cvsroot/mozilla/tools/release/configs/tb-moz18-staging-bootstrap.cfg,v  <--  tb-moz18-staging-bootstrap.cfg
initial revision: 1.1
done

Comment 3

[Rick Tessner]

Attachment: Buildbot config for tbird build automation (release build only)

Just commented out any configuration that had to do with nightly builds since tbird will be using tinderbox for a while yet.

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 314511 [details] [diff] [review]
Buildbot config for tbird build automation (release build only)

<snipping>

># This is the staging buildmaster config file for Mozilla Firefox 1.8 (aka Firefox 2.0.0.x). 

s/Firefox/Thunderbird/

>####### DEPENDENT SCHEDULERS
>prestage_depscheduler = Dependent(
> name="prestage_depscheduler",
> upstream=slave_prestage_scheduler,
> builderNames=["prestage"],
>)

Why did cvs_mirror_depscheduler get removed from here ? (and the c['schedulers'].append calls below).

>slavePrestageFactory = BootstrapFactory(
> cvsroot=cvsroot, 
> cvsmodule=cvsmodule, 
> automation_tag=automation_tag,
> logdir='/builds/logs.nightly', 
> bootstrap_config='configs/tb-moz18-staging-bootstrap.cfg',
>)

logDir should stay as /builds/logs

>prestageFactory = BootstrapFactory(
> cvsroot=cvsroot, 
> cvsmodule=cvsmodule, 
> automation_tag=automation_tag,
> logdir='/builds/logs.nightly', 
> bootstrap_config='configs/tb-moz18-staging-bootstrap.cfg',
>)

Same again for the logs.

>prestageFactory.addStep(ShellCommand, 
> description="update bootstrap config",
> workdir="/builds/buildbot/TestBot/bootstrap-configs",
> command=['cvs', 'up', '-CPd'],
>)

This isn't used anymore. Could you have started from a slightly stale copy of staging/master.cfg ? There was a refactoring checkin recently, so that revision 1.34 is the tip. Your changes for bootstrap_config, s/firefox/thunderbird/, etc look fine, but please recreate the patch against the tip.

>####### PROJECT IDENTITY
>c['projectName'] = "1.8 Staging Master"

Something like "1.8 Staging Master (Thunderbird)" would be good here.

Comment 5

[Rick Tessner]

Attachment: [checked in] 2nd attempt : Buildbot config for tbird build automation (release build only)

A second attempt at putting together a buildbot config for thunderbird automation.  The previous attempt was based on a stale copy of the master.cfg used for firefox.

Comment 6

[Rick Tessner]

I've gone thru the mozilla/tools/tinderbox-config/thunderbird/*/tinder-config.pl on both the MOZILLA_1_8_BRANCH_release and MOZILLA_1_8_BRANCH_l10n_release branches and it looks like all of them have been annotated with the "# CONFIG: " markers in the same manner that the firefox tinder-config.pl had been done.

No patch appears to be necessary for these configs.

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 314705 [details] [diff] [review]
[checked in] 2nd attempt : Buildbot config for tbird build automation (release build only)

r+ 

Checking in tb-master.cfg;
/cvsroot/mozilla/tools/buildbot-configs/automation/staging/tb-master.cfg,v  <--  tb-master.cfg
initial revision: 1.1
done

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to comment #6)
> I've gone thru the
> mozilla/tools/tinderbox-config/thunderbird/*/tinder-config.pl on both the
> MOZILLA_1_8_BRANCH_release and MOZILLA_1_8_BRANCH_l10n_release branches and it
> looks like all of them have been annotated with the "# CONFIG: " markers in the
> same manner that the firefox tinder-config.pl had been done.
> 
> No patch appears to be necessary for these configs.

Looks good to me too. I'll look into refreshing the CVS mirror.

Comment 9

[Nick Thomas [:nthomas] (UTC+12)]

I used this to update just the Thunderbird mozconfigs:

rsync -av --delete-after --exclude=CVSROOT/config --exclude=CVSROOT/loginfo cvs-mirror.mozilla.org::mozilla/mozilla/tools/tinderbox-configs/thunderbird/ /builds/cvsmirror.clean/cvsroot/mozilla/tools/tinderbox-configs/thunderbird

Comment 10

[Nick Thomas [:nthomas] (UTC+12)]

No thunderbird group on staging-prometheus-vm, did

groupadd thunderbird
usermod -a -G thunderbird cltbld

Comment 11

[Rick Tessner]

The buildbot slave on staging-prometheus-vm, which runs under the cltbld userid, had to be restarted so that it was aware of the new group permissions it had.  That was done with a

buildbot restart ~/linux-1.8-slave1

Comment 12

[Nick Thomas [:nthomas] (UTC+12)]

Hit some bustage in the TinderConfig part of Build, as I'd forgotten to remove the four THUNDERBIRD_{RELEASE,RC1}{,_l10n} tags from the tinderbox-configs after the rsync in comment 10. Did that with 8 commands like
[cltbld@staging-build-console]$ cvs -d /builds/cvsmirror.clean/cvsroot rtag -d THUNDERBIRD_2_0_0_13_RELEASE mozilla/tools/tinderbox-configs/thunderbird/
on both cvsmiror.clean and cvsmirror.

Then logged into the three slaves that builds and removed /builds/config/thunderbird-2.0.0.13rc1, so that TinderConfig would not die early in a new attempt. And also modified the tb-master.cfg to remove all .addStep's between the factory initialization and appending the builder for linux_prestage thru to source, and did a reconfig.

On the sendchange, discovered that we'd completely forgotten to set up tinderbox for Tb on these boxes. That's our next step.

Comment 13

[Nick Thomas [:nthomas] (UTC+12)]

Setting up the tinderboxes:

# connect to slave and make sure there is ~10 GB free space
cd /builds/tinderbox  # use /cygdrive/c/builds/tinderbox on windows
mkdir Tb-Mozilla1.8-Release Tb-Mozilla1.8-l10n-Release # see buildDir in cfg

cd Tb-Mozilla1.8-Release/
../mozilla/tools/tinderbox/install-links
rm tinderbox
cvs -d :ext:[e-mail]:/builds/cvsmirror/cvsroot co -r MOZILLA_1_8_BRANCH_release -d tinderbox-configs mozilla/tools/tinderbox-configs/thunderbird/linux # or macosx or win32
ln -s tinderbox-configs/mozconfig
ln -s tinderbox-configs/tinder-config.pl
# on windows for en-US only, also do
rm build-seamonkey.pl
ln -s ../mozilla/tools/tinderbox/build-thunderbird.pl
ln -s build-thunderbird.pl build-seamonkey.pl

cd ../Tb-Mozilla1.8-l10n-Release/
# repeat the previous block from install-links, but use 
# MOZILLA_1_8_BRANCH_l10n_release when checking out the config

Before restarting automation, removed the tags (so TinderConfig doesn't fail) with
for tag in THUNDERBIRD_2_0_0_13_{RC1,RELEASE}{,_l10n}; do echo $tag; cvs -d /builds/cvsmirror/cvsroot rtag -d $tag mozilla/tools/tinderbox-configs/thunderbird/; done

Comment 14

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [checked in] Merge changes from 1.9 staging

This patch merges the locks from staging-1.9 so that the linux slave doesn't try to run more than one task at once (now that it has more to do). It also brings over the cleanups to /builds/updates and /builds/config, which mean you don't have to reach into each slave if triggering manual builds from the waterfall. On the more trival side, it removes a "cat bootstrap.cfg" from a preStage (which the factory does now), and fixes a formatting difference.

Comment 15

[[e-mail] (:bhearsum)]

Comment on attachment 315523 [details] [diff] [review]
[checked in] Merge changes from 1.9 staging

Looks good.

Comment 16

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 315523 [details] [diff] [review]
[checked in] Merge changes from 1.9 staging

Checked in and reconfig'd on staging-1.8-master.

Checking in production/master.cfg;
/cvsroot/mozilla/tools/buildbot-configs/automation/production/master.cfg,v  <--  master.cfg
new revision: 1.26; previous revision: 1.25
done
Checking in staging/master.cfg;
/cvsroot/mozilla/tools/buildbot-configs/automation/staging/master.cfg,v  <--  master.cfg
new revision: 1.35; previous revision: 1.34
done
Checking in staging/tb-master.cfg;
/cvsroot/mozilla/tools/buildbot-configs/automation/staging/tb-master.cfg,v  <--  tb-master.cfg
new revision: 1.2; previous revision: 1.1
done
Checking in staging-1.9/master.cfg;
/cvsroot/mozilla/tools/buildbot-configs/automation/staging-1.9/master.cfg,v  <--  master.cfg
new revision: 1.36; previous revision: 1.35
done

Comment 17

[Nick Thomas [:nthomas] (UTC+12)]

Builds failed out in the packaging code as the pre-built packages for Talkback weren't on the slaves. Copied those on from the nightly tinderboxes, picked up the changes from attachment 315523 [details] [diff] [review], removed tags on the tinderbox configs (again!) and sent a sendchange.

Comment 18

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [checked in] Use CONFIG for $UsePrebuiltTalkback

This is the win32 tinder-config for Thunderbird releases, and a bit of hack until we get buildRoot from bug 397554 landed. Need this because patrocles uses /cygdrive/e/ and staging-pacifica-vm is on /cygdrive/c/.

Comment 19

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 315568 [details] [diff] [review]
[checked in] Use CONFIG for $UsePrebuiltTalkback

cvs.m.o:

Checking in win32/tinder-config.pl;
/cvsroot/mozilla/tools/tinderbox-configs/thunderbird/win32/tinder-config.pl,v  <--  tinder-config.pl
new revision: 1.6.4.1.2.16; previous revision: 1.6.4.1.2.15
done

Also checked into staging-1.8-master:/builds/cvsmirror.clean/cvsroot & /builds/cvsmirror/cvsroot, and fixed up the upload server for Talkback symbols.

Comment 20

[Nick Thomas [:nthomas] (UTC+12)]

After restarting builds with a sendchange, they got as far as waiting on the Sign step. Turns out staging-prometheus-vm was configured to only show /pub/mozilla.org/firefox, so I fixed the symlinks in /var/www/html. 

That revealed that no locales were produced, because the tinderbox was trying to wget them from staging-prometheus-vm and got a 404 (this doesn't cause the build to die). 

Commented out the addStep's in buildFactory before the three Repack calls, then sent another sendchange.

Comment 21

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Remove gu-IN (all platforms) and he (mac only) from update verify for 2.0b2

The run got right to the end this time, but failed out in each update verify when it couldn't find an update from 2.0b2/"gu-IN" and 2.0b2/"he"/mac. Turns out we didn't ship these for the next release (2.0.0.0) or since then, so there's none to be found and they can be removed from the config.

This also adds the locales after "he" for mac, which was mysteriously missing. FWIW, shipped-locales on the THUNDERBIRD_2_b2_RELEASE tag is bogus (eg "ko" "mk" didn't ship for all platforms for 2.0b2), so I checked each locale list against what is on the ftp site.

Comment 22

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 315736 [details] [diff] [review]
Remove gu-IN (all platforms) and he (mac only) from update verify for 2.0b2

Checked this in to the cvsmirrors for testing, and manually kicked off another set verify runs.

In other news, Stage failed because the disk filled up. Deleted all the builds from the Fx staging run to make space and kicked off a run (waits on the lock from linux update verify).

Comment 23

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [as checked in] Remove gu-IN (all platforms) and he (mac only) from update verify for 2.0b2

Checking in moz18-thunderbird-linux.cfg;
/cvsroot/mozilla/testing/release/updates/moz18-thunderbird-linux.cfg,v  <--  moz18-thunderbird-linux.cfg
new revision: 1.13; previous revision: 1.12
done
Checking in moz18-thunderbird-mac.cfg;
/cvsroot/mozilla/testing/release/updates/moz18-thunderbird-mac.cfg,v  <--  moz18-thunderbird-mac.cfg
new revision: 1.15; previous revision: 1.14
done
Checking in moz18-thunderbird-win32.cfg;
/cvsroot/mozilla/testing/release/updates/moz18-thunderbird-win32.cfg,v  <--  moz18-thunderbird-win32.cfg
new revision: 1.12; previous revision: 1.11
done

This version doesn't add the mac locales from hu on, for some reason they're excluded as exceptions in the patcher config. *confused*   Update the cvs mirrors too.

Comment 24

[Nick Thomas [:nthomas] (UTC+12)]

I checked over the logs and bits from this and can't see any problems. Put some  validation results up at 
 http://spreadsheets.google.com/ccc?key=pZm1Hv_8CTFa4rPmlBS8ZUQ&hl=en

Looks like everything got checked into cvs.m.o, so we should get this building Firefox 2.0.0.x again.

Comment 25

[Nick Thomas [:nthomas] (UTC+12)]

Google docs link that works:
http://spreadsheets.google.com/pub?key=pZm1Hv_8CTFa4rPmlBS8ZUQ

Comment 26

[Nick Thomas [:nthomas] (UTC+12)]

We got Fx running again in bug 432986.