Closed Bug 427769 Opened 12 years ago Closed 12 years ago

Test Thunderbird 2.0.0.x automation on staging setup

Categories

(Release Engineering :: General, defect, P2)

defect

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: nthomas, Assigned: rick.tessner)

Details

Attachments

(6 files, 1 obsolete file)

Rob worked up this list of steps, to try building Tb2.0.0.x on the 1.8 staging environment:

I think that the following is needed first in CVS. Is there a bug  
filed on this yet? Should probably do that so there's a place to  
review this stuff:

1) create new mozilla/tools/release/configs/tb-moz18-staging-
bootstrap.cfg (based on fx-moz18-staging-bootstrap.cfg is probably  
easiest)

2) create new master.cfg for staging-1.8-master, based on mozilla/
tools/buildbot-configs/automation/staging/master.cfg

3) annotate mozilla/tools/tinderbox-config/thunderbird configs with "#  
CONFIG" statements. Could look at the firefox configs for examples),  
branches are MOZILLA_1_8_BRANCH_release and  
MOZILLA_1_8_BRANCH_l10n_release)

Nick, does this look right? I don't see any firefox specific dirs in  
the Makefile (it knows how to read the bootstrap.cfg for that stuff  
now), the only place on the master that needs any changes should be  
the master.cfg, the bootstrap.cfg comes from the checkout on the slave  
now and the filename is configurable in the BootstrapFactory now.

If I'm not missing anything, then when the above lands, the only  
things needed on staging-1.8-master I think should be:

1) refresh CVS mirror, to pick up the above changes
cd /home/cltbld/real/mozilla/tools/release && make cvsmirror_main

Note that staging-1.8-master is almost out of disk space on the /cvs  
partition, this should probably be fixed before refreshing the mirror.

2) update master.cfg and restart buildbot
cd /builds/buildbot/TestBot/buildbot-configs && cvs up
# The /builds/buildbot/TestBot/master.cfg symlink should point at the  
new Thunderbird master.cfg
buildbot restart /builds/buildbot/TestBot/
Summary: Test Thunderbird 2.0.0.x automation staging → Test Thunderbird 2.0.0.x automation on staging setup
This will be added as mozilla/tools/release/configs/tb-moz18-staging-bootstrap.cfg to the trunk
Attachment #314355 - Flags: review?(nrthomas)
Comment on attachment 314355 [details] [diff] [review]
[checked in] Thunderbird bootstrap config for staging

r+ and checked in

Checking in tb-moz18-staging-bootstrap.cfg;
/cvsroot/mozilla/tools/release/configs/tb-moz18-staging-bootstrap.cfg,v  <--  tb-moz18-staging-bootstrap.cfg
initial revision: 1.1
done
Attachment #314355 - Attachment description: Thunderbird bootstrap config for staging → [checked in] Thunderbird bootstrap config for staging
Attachment #314355 - Flags: review?(nrthomas) → review+
Just commented out any configuration that had to do with nightly builds since tbird will be using tinderbox for a while yet.
Attachment #314511 - Flags: review?(nrthomas)
Attachment #314511 - Flags: review?(nrthomas)
Comment on attachment 314511 [details] [diff] [review]
Buildbot config for tbird build automation (release build only)

<snipping>

># This is the staging buildmaster config file for Mozilla Firefox 1.8 (aka Firefox 2.0.0.x). 

s/Firefox/Thunderbird/

>####### DEPENDENT SCHEDULERS
>prestage_depscheduler = Dependent(
> name="prestage_depscheduler",
> upstream=slave_prestage_scheduler,
> builderNames=["prestage"],
>)

Why did cvs_mirror_depscheduler get removed from here ? (and the c['schedulers'].append calls below).

>slavePrestageFactory = BootstrapFactory(
> cvsroot=cvsroot, 
> cvsmodule=cvsmodule, 
> automation_tag=automation_tag,
> logdir='/builds/logs.nightly', 
> bootstrap_config='configs/tb-moz18-staging-bootstrap.cfg',
>)

logDir should stay as /builds/logs

>prestageFactory = BootstrapFactory(
> cvsroot=cvsroot, 
> cvsmodule=cvsmodule, 
> automation_tag=automation_tag,
> logdir='/builds/logs.nightly', 
> bootstrap_config='configs/tb-moz18-staging-bootstrap.cfg',
>)

Same again for the logs.

>prestageFactory.addStep(ShellCommand, 
> description="update bootstrap config",
> workdir="/builds/buildbot/TestBot/bootstrap-configs",
> command=['cvs', 'up', '-CPd'],
>)

This isn't used anymore. Could you have started from a slightly stale copy of staging/master.cfg ? There was a refactoring checkin recently, so that revision 1.34 is the tip. Your changes for bootstrap_config, s/firefox/thunderbird/, etc look fine, but please recreate the patch against the tip.

>####### PROJECT IDENTITY
>c['projectName'] = "1.8 Staging Master"

Something like "1.8 Staging Master (Thunderbird)" would be good here.
Attachment #314511 - Flags: review?(nrthomas) → review-
Priority: -- → P2
A second attempt at putting together a buildbot config for thunderbird automation.  The previous attempt was based on a stale copy of the master.cfg used for firefox.
Attachment #314705 - Flags: review?(nrthomas)
I've gone thru the mozilla/tools/tinderbox-config/thunderbird/*/tinder-config.pl on both the MOZILLA_1_8_BRANCH_release and MOZILLA_1_8_BRANCH_l10n_release branches and it looks like all of them have been annotated with the "# CONFIG: " markers in the same manner that the firefox tinder-config.pl had been done.

No patch appears to be necessary for these configs.
Comment on attachment 314705 [details] [diff] [review]
[checked in] 2nd attempt : Buildbot config for tbird build automation (release build only)

r+ 

Checking in tb-master.cfg;
/cvsroot/mozilla/tools/buildbot-configs/automation/staging/tb-master.cfg,v  <--  tb-master.cfg
initial revision: 1.1
done
Attachment #314705 - Attachment description: 2nd attempt : Buildbot config for tbird build automation (release build only) → [checked in] 2nd attempt : Buildbot config for tbird build automation (release build only)
Attachment #314705 - Flags: review?(nrthomas)
Attachment #314705 - Flags: review+
(In reply to comment #6)
> I've gone thru the
> mozilla/tools/tinderbox-config/thunderbird/*/tinder-config.pl on both the
> MOZILLA_1_8_BRANCH_release and MOZILLA_1_8_BRANCH_l10n_release branches and it
> looks like all of them have been annotated with the "# CONFIG: " markers in the
> same manner that the firefox tinder-config.pl had been done.
> 
> No patch appears to be necessary for these configs.

Looks good to me too. I'll look into refreshing the CVS mirror.

I used this to update just the Thunderbird mozconfigs:

rsync -av --delete-after --exclude=CVSROOT/config --exclude=CVSROOT/loginfo cvs-mirror.mozilla.org::mozilla/mozilla/tools/tinderbox-configs/thunderbird/ /builds/cvsmirror.clean/cvsroot/mozilla/tools/tinderbox-configs/thunderbird
No thunderbird group on staging-prometheus-vm, did

groupadd thunderbird
usermod -a -G thunderbird cltbld
The buildbot slave on staging-prometheus-vm, which runs under the cltbld userid, had to be restarted so that it was aware of the new group permissions it had.  That was done with a

buildbot restart ~/linux-1.8-slave1
Hit some bustage in the TinderConfig part of Build, as I'd forgotten to remove the four THUNDERBIRD_{RELEASE,RC1}{,_l10n} tags from the tinderbox-configs after the rsync in comment 10. Did that with 8 commands like
[cltbld@staging-build-console]$ cvs -d /builds/cvsmirror.clean/cvsroot rtag -d THUNDERBIRD_2_0_0_13_RELEASE mozilla/tools/tinderbox-configs/thunderbird/
on both cvsmiror.clean and cvsmirror.

Then logged into the three slaves that builds and removed /builds/config/thunderbird-2.0.0.13rc1, so that TinderConfig would not die early in a new attempt. And also modified the tb-master.cfg to remove all .addStep's between the factory initialization and appending the builder for linux_prestage thru to source, and did a reconfig.

On the sendchange, discovered that we'd completely forgotten to set up tinderbox for Tb on these boxes. That's our next step.

Setting up the tinderboxes:

# connect to slave and make sure there is ~10 GB free space
cd /builds/tinderbox  # use /cygdrive/c/builds/tinderbox on windows
mkdir Tb-Mozilla1.8-Release Tb-Mozilla1.8-l10n-Release # see buildDir in cfg

cd Tb-Mozilla1.8-Release/
../mozilla/tools/tinderbox/install-links
rm tinderbox
cvs -d :ext:cltbld@staging-build-console.build.mozilla.org:/builds/cvsmirror/cvsroot co -r MOZILLA_1_8_BRANCH_release -d tinderbox-configs mozilla/tools/tinderbox-configs/thunderbird/linux # or macosx or win32
ln -s tinderbox-configs/mozconfig
ln -s tinderbox-configs/tinder-config.pl
# on windows for en-US only, also do
rm build-seamonkey.pl
ln -s ../mozilla/tools/tinderbox/build-thunderbird.pl
ln -s build-thunderbird.pl build-seamonkey.pl

cd ../Tb-Mozilla1.8-l10n-Release/
# repeat the previous block from install-links, but use 
# MOZILLA_1_8_BRANCH_l10n_release when checking out the config

Before restarting automation, removed the tags (so TinderConfig doesn't fail) with
for tag in THUNDERBIRD_2_0_0_13_{RC1,RELEASE}{,_l10n}; do echo $tag; cvs -d /builds/cvsmirror/cvsroot rtag -d $tag mozilla/tools/tinderbox-configs/thunderbird/; done
This patch merges the locks from staging-1.9 so that the linux slave doesn't try to run more than one task at once (now that it has more to do). It also brings over the cleanups to /builds/updates and /builds/config, which mean you don't have to reach into each slave if triggering manual builds from the waterfall. On the more trival side, it removes a "cat bootstrap.cfg" from a preStage (which the factory does now), and fixes a formatting difference.
Attachment #315523 - Flags: review?(bhearsum)
Comment on attachment 315523 [details] [diff] [review]
[checked in] Merge changes from 1.9 staging

Looks good.
Attachment #315523 - Flags: review?(bhearsum) → review+
Comment on attachment 315523 [details] [diff] [review]
[checked in] Merge changes from 1.9 staging

Checked in and reconfig'd on staging-1.8-master.

Checking in production/master.cfg;
/cvsroot/mozilla/tools/buildbot-configs/automation/production/master.cfg,v  <--  master.cfg
new revision: 1.26; previous revision: 1.25
done
Checking in staging/master.cfg;
/cvsroot/mozilla/tools/buildbot-configs/automation/staging/master.cfg,v  <--  master.cfg
new revision: 1.35; previous revision: 1.34
done
Checking in staging/tb-master.cfg;
/cvsroot/mozilla/tools/buildbot-configs/automation/staging/tb-master.cfg,v  <--  tb-master.cfg
new revision: 1.2; previous revision: 1.1
done
Checking in staging-1.9/master.cfg;
/cvsroot/mozilla/tools/buildbot-configs/automation/staging-1.9/master.cfg,v  <--  master.cfg
new revision: 1.36; previous revision: 1.35
done
Attachment #315523 - Attachment description: Merge changes from 1.9 staging → [checked in] Merge changes from 1.9 staging
Builds failed out in the packaging code as the pre-built packages for Talkback weren't on the slaves. Copied those on from the nightly tinderboxes, picked up the changes from attachment 315523 [details] [diff] [review], removed tags on the tinderbox configs (again!) and sent a sendchange.
This is the win32 tinder-config for Thunderbird releases, and a bit of hack until we get buildRoot from bug 397554 landed. Need this because patrocles uses /cygdrive/e/ and staging-pacifica-vm is on /cygdrive/c/.
Attachment #315568 - Flags: review?(bhearsum)
Attachment #315568 - Flags: review?(bhearsum) → review+
Comment on attachment 315568 [details] [diff] [review]
[checked in] Use CONFIG for $UsePrebuiltTalkback

cvs.m.o:

Checking in win32/tinder-config.pl;
/cvsroot/mozilla/tools/tinderbox-configs/thunderbird/win32/tinder-config.pl,v  <--  tinder-config.pl
new revision: 1.6.4.1.2.16; previous revision: 1.6.4.1.2.15
done

Also checked into staging-1.8-master:/builds/cvsmirror.clean/cvsroot & /builds/cvsmirror/cvsroot, and fixed up the upload server for Talkback symbols.
Attachment #315568 - Attachment description: Use CONFIG for $UsePrebuiltTalkback → [checked in] Use CONFIG for $UsePrebuiltTalkback
After restarting builds with a sendchange, they got as far as waiting on the Sign step. Turns out staging-prometheus-vm was configured to only show /pub/mozilla.org/firefox, so I fixed the symlinks in /var/www/html. 

That revealed that no locales were produced, because the tinderbox was trying to wget them from staging-prometheus-vm and got a 404 (this doesn't cause the build to die). 

Commented out the addStep's in buildFactory before the three Repack calls, then sent another sendchange.

The run got right to the end this time, but failed out in each update verify when it couldn't find an update from 2.0b2/"gu-IN" and 2.0b2/"he"/mac. Turns out we didn't ship these for the next release (2.0.0.0) or since then, so there's none to be found and they can be removed from the config.

This also adds the locales after "he" for mac, which was mysteriously missing. FWIW, shipped-locales on the THUNDERBIRD_2_b2_RELEASE tag is bogus (eg "ko" "mk" didn't ship for all platforms for 2.0b2), so I checked each locale list against what is on the ftp site.
Attachment #315736 - Flags: review?(bhearsum)
Comment on attachment 315736 [details] [diff] [review]
Remove gu-IN (all platforms) and he (mac only) from update verify for 2.0b2

Checked this in to the cvsmirrors for testing, and manually kicked off another set verify runs.

In other news, Stage failed because the disk filled up. Deleted all the builds from the Fx staging run to make space and kicked off a run (waits on the lock from linux update verify).
Attachment #315736 - Flags: review?(bhearsum) → review+
Checking in moz18-thunderbird-linux.cfg;
/cvsroot/mozilla/testing/release/updates/moz18-thunderbird-linux.cfg,v  <--  moz18-thunderbird-linux.cfg
new revision: 1.13; previous revision: 1.12
done
Checking in moz18-thunderbird-mac.cfg;
/cvsroot/mozilla/testing/release/updates/moz18-thunderbird-mac.cfg,v  <--  moz18-thunderbird-mac.cfg
new revision: 1.15; previous revision: 1.14
done
Checking in moz18-thunderbird-win32.cfg;
/cvsroot/mozilla/testing/release/updates/moz18-thunderbird-win32.cfg,v  <--  moz18-thunderbird-win32.cfg
new revision: 1.12; previous revision: 1.11
done

This version doesn't add the mac locales from hu on, for some reason they're excluded as exceptions in the patcher config. *confused*   Update the cvs mirrors too.
Attachment #315736 - Attachment is obsolete: true
I checked over the logs and bits from this and can't see any problems. Put some  validation results up at 
 http://spreadsheets.google.com/ccc?key=pZm1Hv_8CTFa4rPmlBS8ZUQ&hl=en

Looks like everything got checked into cvs.m.o, so we should get this building Firefox 2.0.0.x again.

We got Fx running again in bug 432986.
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.