428063 - [Bootstrap] Support major releases & quit using rc in overloaded ways

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [superceded] Use buildN rather than rcN

This patch does two major things:

1, files live in the likes of ../nightly/2.0.0.18-candidates/build2 instead of ../rc2  (ie s/rc/build/)
2, tags are the likes of FIREFOX_2_0_0_18_BUILD1, FIREFOX_2_0_0_18_BUILD2, FIREFOX_2_0_0_18_RELEASE   (ie s/RC/BUILD/)

It might be one way of resolving the 3.0 RC2 rc3 confusion (the third candidate build for the 2nd public RC).

Also tweaks the config for the master on 1.9 staging, which I'm hoping to try this out on.

Comment 1

[Robert Helmer [:rhelmer]]

Comment on attachment 314657 [details] [diff] [review]
[superceded] Use buildN rather than rcN

Looks good to me. Only place that "rc" still exists is PatcherConfig, this means that "beta" maintenance releases will still get "RCn" in the update dialog, right?

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

Good call. If this works out OK for the automation, then I'll look into fixing up patcher too.

Commandeering 1.9 staging to test this.

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

Bumps along the way:

* make stage
** needed to be adjust to pull from .../3.0b4-candidates/rc1/ rather than .../build1/, a transition glitch. We could create symlinks on ftp.m.o to work around this
** substituted stage.m.o for relases.m.o to it speed up, will get this reviewed elsewhere
* l10n configs use rc$rc in WGetFiles, so TinderConfig failed. Needed to s/rc/build/ and checkin to staging mirrors (will follow up here)
* Sign step failed as master.cfg part of attachment 314657 [details] [diff] [review] hadn't been applied

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

* Update generation failed, as it was trying to use UPDATE_PACKAGING_R1 in fast mode
** bumped the bootstrap config to ..._R3 to match production, landed in cvs.m.o and the two staging mirrors
** then _R3 doesn't exist in cvs mirror. Verified that _R3 runs in fast mode on slave by checking out from cvs.m.o
** Updated the clean cvs mirror from cvs.m.o, checked in the Bootstrap and l10n tinderconfig changes, undid the commenting in master and started a new run
** partial update generation failed due to running out of space in /tmp, not related to this patch and filed bug 428650 

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

** checked in attachment 315276 [details] [diff] [review] from bug 428650 into /builds/cvsmirror/clean, moved the _R3 tag, and started update gen again (NB: will fail again on a run that recreates the cvs mirror)

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

** had to setup up backupsnip on staging-1.9-master, ended up setting up a bunch of machines, see bug 409347 for details

** linux & mac verification went ok, win32 failed because oldBuild was still 1 in the bootstrap config. We had rc2 info in the patcher config (from prod), but the 1 value meant the update verify config was bumped wrong and we end up querying with the rc1 buildID and not finding anything (shock!). Checked in a bump from rc1 to rc2 in the clean cvs mirror.

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Patcher changes

Makes the update dialog say "2.0.0.15 build 2" or "3.0 RC2 build 1" on the beta & betatest channels, rather than "2.0.0.15rc2" and "3.0 RC2rc1".

Tested on the staging setup for 1.9, doing a recursive diff between snippets generated without and with this patch. Only the appv lines of beta and betatest snippets changed.

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

We're going to go ahead with this before the Fx3.0 release candidates come along. Morphing this to include other "major release" support.

Comment 9

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [checked in] Fix up l10n_release configs for staging

Using CONFIG on ssh_key breaks trunk staging for l10n builds, because we don't have a ~/.ssh/cltbld.dsa key there. Turns out l10n_release is the odd one out - we don't specify ssh_key for the en-US configs on the release branch - and we should just pick up the default key.

Comment 10

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 317244 [details] [diff] [review]
[checked in] Fix up l10n_release configs for staging

Checked in:

mozilla/tools/tinderbox-configs/firefox/linux/tinder-config.pl 	1.1.2.9.2.14
mozilla/tools/tinderbox-configs/firefox/macosx/tinder-config.pl 	1.8.2.12.2.13
mozilla/tools/tinderbox-configs/firefox/win32/tinder-config.pl 	1.2.4.8.2.13

Comment 11

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [checked in] Patcher changes, v2

betatest gets "Firefox 3.0 RC 1 (build 1)"/"Firefox 2.0.0.15 (build 2)";
release gets  "Firefox 3.0 RC 1"/"Firefox 2.0.0.15"

The output is a bit wonky still, eg
  Partial patch info - 399 to create
        [  1/399] 3.0b5-3.0build1rc1/linux-i686/en-US/betatest... done
        [  2/399] 3.0b5-3.0rc1/linux-i686/en-US/beta... done
I was aiming for 3.0b5-3.0rc1build1.

Comment 12

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to comment #11)
> Created an attachment (id=317269) [details]
> Patcher changes, v2

The aim would be to combine this and bug 428650, and create an UPDATE_PACKAGING_R4 tag.

Comment 13

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Use buildN rather than rcN, v2

With changes to tinderbox config for l10n. Needs attachment 317269 [details] [diff] [review] to work.

Comment 14

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Support major versions

The main point of this one is to allow us to have a release-to-the-public of "3.0 RC n" that has an application version of 3.0. The optional bootstrap variable of appVersion is new, and is used to bump the application version, and for referring to the files that come off the tinderbox (installers/dmg/tarball/complete mars). The changes in fx-moz19-staging-bootstrap.cfg illustrate how the config is done.

Assorted sundry fixes:
* pull previous release from stage.m.o instead of releases.m.o, so that a slow mirror doesn't slow down staging setup
* Make using the beta channel prior to a release optional, via useBetaChannel
* support turning a milestone of 1.9 into a relbranch tag of GECKO19_2008....
* Explicitly set the versions in the partial update file name (parity with completes)
* Update verify-locales.pl for files with rc or RC in their name
* Modify the l10n configs to use appVersion. We'll either need to keep appVersion in the config (and equal to version), or revert this after 3.0 gets into maintenance mode

This doesn't resolve the problem of Tag/Bump.pm wanting a particular set of input for the first build of each RC, but that may not be an issue if we branch from the tip each time we start a new RC.

Missing are changes to production configs (useBetaChannel).

Next up is merging these two patches and attempting an end-to-end run of 3.0 RC1 build 1 with a refreshed CVS mirror.

Comment 15

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Merged patch

This combines attachment 317273 [details] [diff] [review] and attachment 317299 [details] [diff] [review]. I'm going to set it up on staging and see how it goes.

Comment 16

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [checked in] Merged patch, v1.1

Fixes a merge typo in PatcherConfig.pm.

Comment 17

[Nick Thomas [:nthomas] (UTC+12)]

This patch has almost completed a full run for 3.0 RC1 build1. I hit a few bumps along the way, mostly due to the staging setup:

* prestage failure trying to pull .../firefox/nightly/3.0b5-candidates/build1 on stage, --> added symlink from rc1 to build1 (also for 2.0.0.14) and restarted
run 

* Tag failure in "do valid l10n checkout"
 "cvs [checkout aborted]: *PANIC* administration files missing". 
Couldn't reproduce this error on the command line. --> Commented out prestage,stage,cvsmirror and tag factory addSteps and restarted run

* Build failure when trying to checkout source using wrong tag (all platforms). This is fallout from refreshing the cvs mirror, so that cvs up on the tinderbox-configs dir doesn't pick up new changes (thinks it already has the latest revision) --> removed files in both build dirs on each slave, and cvs up to get them back, removed FIREFOX_3_0RC1_{BUILD1,RELEASE}{,_l10n} tags in "dirty" mirrorm, commented out Source addStep and restarted run

* Update verify failures - a few issues here
** firstly, accumulated old snippets in staging-1.9-master:/opt/aus2/incoming/3 meant that an update was offered when it shouldn't have been --> removed all snippets and rsync'd back in 20080424-Firefox-3.0rc1-test, rerunning 
[for the long term, we should make sure we clean this up in the prestage builder]
** secondly, some locales got dropped along the way, or will be for RC1, but we didn't change the update verify configs. In particular
*** All: ga-IE, not shipped for b5 (but was for b4 and will be for RC1)
*** Mac only: he & gu-IN not shipping for rc1; ar stopped shipping for b5
** On Win32 only, the two chk files give spurious FAILs

Comment 18

[Robert Helmer [:rhelmer]]

Comment on attachment 317269 [details] [diff] [review]
[checked in] Patcher changes, v2

>Index: patcher2.pl
>===================================================================
>RCS file: /cvsroot/mozilla/tools/patcher/patcher2.pl,v
>retrieving revision 1.31
>diff -u -p -r1.31 patcher2.pl
>--- patcher2.pl	14 Mar 2008 20:19:44 -0000	1.31
>+++ patcher2.pl	23 Apr 2008 14:25:12 -0000
>@@ -1246,7 +1246,7 @@ sub CreatePartialPatchinfo {
>                     # appv's.
>                     my $progressVersion = $u;
>                     if ($snippetToAppVersion ne $to->{'appv'}) {
>-                        $progressVersion =~ s/$to->{'appv'}/$snippetToAppVersion/;
>+                        $progressVersion =~ s/\-$to->{'appv'}/-$snippetToAppVersion/;

The rest looks ok to me, why is substituting "-$to" needed? Is this just to make the progress meter look correct?

Comment 19

[Robert Helmer [:rhelmer]]

Comment on attachment 317559 [details] [diff] [review]
[checked in] Merged patch, v1.1

r=rhelmer :)

Since the s/rc/build/ is mostly a mechanical change, it'd be easier on the reviewer to do that kinda of thing first as a separate patch.

The new features like supporting RC and Beta explicitly sounds great though.

Comment 20

[Robert Helmer [:rhelmer]]

(In reply to comment #17)
> * Tag failure in "do valid l10n checkout"
>  "cvs [checkout aborted]: *PANIC* administration files missing". 
> Couldn't reproduce this error on the command line. --> Commented out
> prestage,stage,cvsmirror and tag factory addSteps and restarted run

This sounds like something went wrong syncing the mirror.. did you use the Makefile targets to do this or do it by hand? I tend to screw it up when running by hand.

The only one needed now that Talkback is binary-only is the cvsmirror_main target I believe.

Comment 21

[[e-mail] (:bhearsum)]

Comment on attachment 317559 [details] [diff] [review]
[checked in] Merged patch, v1.1

>Index: release-merged/bin/verify-locales.pl
>===================================================================
>RCS file: /cvsroot/mozilla/tools/release/bin/verify-locales.pl,v
>retrieving revision 1.1
>diff -u -r1.1 verify-locales.pl
>--- release-merged/bin/verify-locales.pl	12 Apr 2008 15:32:22 -0000	1.1
>+++ release-merged/bin/verify-locales.pl	23 Apr 2008 22:28:49 -0000
>@@ -79,7 +79,7 @@
>       return 0;
>    }
> 
>-   my $fileRegex = '^' . $product . '\-[\d\.]+(b\d+)?\.tar\.' . $extension .
>+   my $fileRegex = '^' . $product . '\-[\d\.]+((b|rc)\d+)?\.tar\.' . $extension .
>     '(\.asc)?$';

Side note: We should make verify-locales.pl support alphas at some point. I filed bug 431069 on this.


Other than that, everything looks fine to me. This was a much simpler patch than I imagined. Nonetheless, thanks for taking this on!

Comment 22

[[e-mail] (:bhearsum)]

Comment on attachment 314657 [details] [diff] [review]
[superceded] Use buildN rather than rcN

This patch is obsoleted by the merged one, right?

Comment 23

[[e-mail] (:bhearsum)]

Comment on attachment 317269 [details] [diff] [review]
[checked in] Patcher changes, v2

Other than what Rob already mentioned this looks fine.

Comment 24

[[e-mail] (:bhearsum)]

I'll deal with checkins/tagging/etc. tomorrow.

Comment 25

[[e-mail] (:bhearsum)]

Taking this to checkin/tag/do a new staging run.

Comment 26

[[e-mail] (:bhearsum)]

On the l10n_release branch:
Checking in firefox/linux/tinder-config.pl;
/cvsroot/mozilla/tools/tinderbox-configs/firefox/linux/tinder-config.pl,v  <--  tinder-config.pl
new revision: 1.1.2.9.2.15; previous revision: 1.1.2.9.2.14
done
Checking in firefox/macosx/tinder-config.pl;
/cvsroot/mozilla/tools/tinderbox-configs/firefox/macosx/tinder-config.pl,v  <--  tinder-config.pl
new revision: 1.8.2.12.2.14; previous revision: 1.8.2.12.2.13
done
Checking in firefox/win32/tinder-config.pl;
/cvsroot/mozilla/tools/tinderbox-configs/firefox/win32/tinder-config.pl,v  <--  tinder-config.pl
new revision: 1.2.4.8.2.14; previous revision: 1.2.4.8.2.13
done


On trunk:
/cvsroot/mozilla/tools/release/Makefile,v  <--  Makefile
new revision: 1.30; previous revision: 1.29
done
Checking in Bootstrap/Config.pm;
/cvsroot/mozilla/tools/release/Bootstrap/Config.pm,v  <--  Config.pm
new revision: 1.18; previous revision: 1.17
done
Checking in Bootstrap/Step.pm;
/cvsroot/mozilla/tools/release/Bootstrap/Step.pm,v  <--  Step.pm
new revision: 1.18; previous revision: 1.17
done
Checking in Bootstrap/Step/Build.pm;
/cvsroot/mozilla/tools/release/Bootstrap/Step/Build.pm,v  <--  Build.pm
new revision: 1.29; previous revision: 1.28
done
Checking in Bootstrap/Step/PatcherConfig.pm;
/cvsroot/mozilla/tools/release/Bootstrap/Step/PatcherConfig.pm,v  <--  PatcherConfig.pm
new revision: 1.21; previous revision: 1.20
done
Checking in Bootstrap/Step/Repack.pm;
/cvsroot/mozilla/tools/release/Bootstrap/Step/Repack.pm,v  <--  Repack.pm
new revision: 1.26; previous revision: 1.25
done
Checking in Bootstrap/Step/Sign.pm;
/cvsroot/mozilla/tools/release/Bootstrap/Step/Sign.pm,v  <--  Sign.pm
new revision: 1.14; previous revision: 1.13
done
Checking in Bootstrap/Step/Source.pm;
/cvsroot/mozilla/tools/release/Bootstrap/Step/Source.pm,v  <--  Source.pm
new revision: 1.16; previous revision: 1.15
done
Checking in Bootstrap/Step/Stage.pm;
/cvsroot/mozilla/tools/release/Bootstrap/Step/Stage.pm,v  <--  Stage.pm
new revision: 1.38; previous revision: 1.37
done
Checking in Bootstrap/Step/Tag.pm;
/cvsroot/mozilla/tools/release/Bootstrap/Step/Tag.pm,v  <--  Tag.pm
new revision: 1.24; previous revision: 1.23
done
Checking in Bootstrap/Step/TinderConfig.pm;
/cvsroot/mozilla/tools/release/Bootstrap/Step/TinderConfig.pm,v  <--  TinderConfig.pm
new revision: 1.10; previous revision: 1.9
done
Checking in Bootstrap/Step/Updates.pm;
/cvsroot/mozilla/tools/release/Bootstrap/Step/Updates.pm,v  <--  Updates.pm
new revision: 1.43; previous revision: 1.42
done
Checking in Bootstrap/Step/Tag/Bump.pm;
/cvsroot/mozilla/tools/release/Bootstrap/Step/Tag/Bump.pm,v  <--  Bump.pm
new revision: 1.13; previous revision: 1.12
done
Checking in Bootstrap/Step/Tag/Mozilla.pm;
/cvsroot/mozilla/tools/release/Bootstrap/Step/Tag/Mozilla.pm,v  <--  Mozilla.pm
new revision: 1.6; previous revision: 1.5
done
Checking in Bootstrap/Step/Tag/Talkback.pm;
/cvsroot/mozilla/tools/release/Bootstrap/Step/Tag/Talkback.pm,v  <--  Talkback.pm
new revision: 1.7; previous revision: 1.6
done
Checking in Bootstrap/Step/Tag/l10n.pm;
/cvsroot/mozilla/tools/release/Bootstrap/Step/Tag/l10n.pm,v  <--  l10n.pm
new revision: 1.10; previous revision: 1.9
done
Checking in bin/verify-locales.pl;
/cvsroot/mozilla/tools/release/bin/verify-locales.pl,v  <--  verify-locales.pl
new revision: 1.2; previous revision: 1.1
done
Checking in configs/fx-moz18-bootstrap.cfg;
/cvsroot/mozilla/tools/release/configs/fx-moz18-bootstrap.cfg,v  <--  fx-moz18-bootstrap.cfg
new revision: 1.43; previous revision: 1.42
done
Checking in configs/fx-moz18-nightly-bootstrap.cfg;
/cvsroot/mozilla/tools/release/configs/fx-moz18-nightly-bootstrap.cfg,v  <--  fx-moz18-nightly-bootstrap.cfg
new revision: 1.14; previous revision: 1.13
done
Checking in configs/fx-moz18-nightly-staging-bootstrap.cfg;
/cvsroot/mozilla/tools/release/configs/fx-moz18-nightly-staging-bootstrap.cfg,v  <--  fx-moz18-nightly-staging-bootstrap.cfg
new revision: 1.5; previous revision: 1.4
done
Checking in configs/fx-moz18-staging-bootstrap.cfg;
/cvsroot/mozilla/tools/release/configs/fx-moz18-staging-bootstrap.cfg,v  <--  fx-moz18-staging-bootstrap.cfg
new revision: 1.25; previous revision: 1.24
done
Checking in configs/fx-moz180-bootstrap.cfg;
/cvsroot/mozilla/tools/release/configs/fx-moz180-bootstrap.cfg,v  <--  fx-moz180-bootstrap.cfg
new revision: 1.9; previous revision: 1.8
done
Checking in configs/fx-moz19-bootstrap.cfg;
/cvsroot/mozilla/tools/release/configs/fx-moz19-bootstrap.cfg,v  <--  fx-moz19-bootstrap.cfg
new revision: 1.25; previous revision: 1.24
done
Checking in configs/fx-moz19-nightly-bootstrap.cfg;
/cvsroot/mozilla/tools/release/configs/fx-moz19-nightly-bootstrap.cfg,v  <--  fx-moz19-nightly-bootstrap.cfg
new revision: 1.15; previous revision: 1.14
done
Checking in configs/fx-moz19-nightly-staging-bootstrap.cfg;
/cvsroot/mozilla/tools/release/configs/fx-moz19-nightly-staging-bootstrap.cfg,v  <--  fx-moz19-nightly-staging-bootstrap.cfg
new revision: 1.4; previous revision: 1.3
done
Checking in configs/fx-moz19-staging-bootstrap.cfg;
/cvsroot/mozilla/tools/release/configs/fx-moz19-staging-bootstrap.cfg,v  <--  fx-moz19-staging-bootstrap.cfg
new revision: 1.23; previous revision: 1.22
done
Checking in configs/tb-moz18-bootstrap.cfg;
/cvsroot/mozilla/tools/release/configs/tb-moz18-bootstrap.cfg,v  <--  tb-moz18-bootstrap.cfg
new revision: 1.22; previous revision: 1.21
done
Checking in configs/tb-moz18-staging-bootstrap.cfg;
/cvsroot/mozilla/tools/release/configs/tb-moz18-staging-bootstrap.cfg,v  <--  tb-moz18-staging-bootstrap.cfg
new revision: 1.2; previous revision: 1.1
done
Checking in configs/tb-moz180-bootstrap.cfg;
/cvsroot/mozilla/tools/release/configs/tb-moz180-bootstrap.cfg,v  <--  tb-moz180-bootstrap.cfg
new revision: 1.12; previous revision: 1.11
done
Checking in configs/xr-moz19-bootstrap.cfg;
/cvsroot/mozilla/tools/release/configs/xr-moz19-bootstrap.cfg,v  <--  xr-moz19-bootstrap.cfg
new revision: 1.4; previous revision: 1.3
done
Checking in configs/xr-moz19-staging-bootstrap.cfg;
/cvsroot/mozilla/tools/release/configs/xr-moz19-staging-bootstrap.cfg,v  <--  xr-moz19-staging-bootstrap.cfg
new revision: 1.6; previous revision: 1.5
done
Checking in staging-1.9/master.cfg;
/cvsroot/mozilla/tools/buildbot-configs/automation/staging-1.9/master.cfg,v  <--  master.cfg
new revision: 1.37; previous revision: 1.36
done

Comment 27

[[e-mail] (:bhearsum)]

Tagged as RELEASE_AUTOMATION_M9.

Comment 28

[[e-mail] (:bhearsum)]

I've re-tagged tools/update-packaging, tools/release, and tools/patcher with UPDATE_PACKAGING_R4:

cvs co -r UPDATE_PACKAGING_R3 mozilla/client.mk
cd mozilla
make -f client.mk MOZ_CO_PROJECT=all checkout
cvs up -A tools/update-packaging
cvs up -AdP tools/release
cvs up -AdP tools/patcher
cvs -q tag UPDATE_PACKAGING_R4 2>&1 | tee ../tag.log
cvs -v '^T' ../tag.log
cvs tag -b UPDATE_PACKAGING_R4_minibranch client.mk
cvs up -r UPDATE_PACKAGING_R4_minibranch client.mk

edited client.mk, bumped to UPDATE_PACKAGING_R4
commited client.mk

Checking in client.mk;
/cvsroot/mozilla/client.mk,v  <--  client.mk
new revision: 1.314.2.1.2.1.2.1.2.1.2.1; previous revision: 1.314.2.1.2.1.2.1.2.1
done

re-tagged client.mk:
cvs -q tag UPDATE_PACKAGING_R4 client.mk
W client.mk : UPDATE_PACKAGING_R4 already exists on version 1.314.2.1.2.1.2.1.2.1 : NOT MOVING tag to version 1.314.2.1.2.1.2.1.2.1.2.1
cvs -q tag -F UPDATE_PACKAGING_R4 client.mk
T client.mk

Comment 29

[[e-mail] (:bhearsum)]

Attachment: [checked in] bump 1.9 staging to new UPDATE_PACKAGING_R4 tag

Kicking off a staging run with this.

Comment 30

[[e-mail] (:bhearsum)]

Checking in fx-moz19-staging-bootstrap.cfg;
/cvsroot/mozilla/tools/release/configs/fx-moz19-staging-bootstrap.cfg,v  <--  fx-moz19-staging-bootstrap.cfg
new revision: 1.24; previous revision: 1.23
done

Comment 31

[[e-mail] (:bhearsum)]

I've got a run going on 1.9 staging. The Linux slave is extremely slow (Tag took 4 hours) but it's going fine. As of right now Mac has finished build/repack, linux is repacking without problem, and windows is finishing building. updates/l10nverify/update_verify are the big question though.

Comment 32

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to comment #18)
> The rest looks ok to me, why is substituting "-$to" needed? Is this just to
> make the progress meter look correct?

Yep, it's just the progress meter, which was screwing up in the case of 
  $u = 3.0b5-3.0rc1, appv = 3.0 
and the regexp taking the leftmost match. 

The progress meter still has problems with the patch I posted, I got output like this:

Partial patch info - 396 to create
        [  1/396] 3.0b5-3.0build1rc1/linux-i686/af/betatest... done
        [  2/396] 3.0b5-3.0rc1/linux-i686/af/beta... done
...
Complete patch info - 396 to create
        [  1/396] 3.0b5-3.0build1rc1/linux-i686/af/betatest... done
        [  2/396] 3.0b5-3.0rc1/linux-i686/af/beta... done
...
Past release patch info - 2364 to create
        [   1/2364] 3.0a1-3.0build1/Linux_x86-gcc3/en-US/betatest/complete... done
        [   2/2364] 3.0a1-3.0build1/Linux_x86-gcc3/en-US/betatest/partial... done
        [   3/2364] 3.0a1-3.0/Linux_x86-gcc3/en-US/beta/complete... done
        [   4/2364] 3.0a1-3.0/Linux_x86-gcc3/en-US/beta/partial... done
        [   5/2364] 3.0a1-3.0/Linux_x86-gcc3/en-US/releasetest/complete... done
        [   6/2364] 3.0a1-3.0/Linux_x86-gcc3/en-US/releasetest/partial... done

Hoping to follow up on this.

Comment 33

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to comment #20)
> (In reply to comment #17)
> > * Tag failure in "do valid l10n checkout"
> >  "cvs [checkout aborted]: *PANIC* administration files missing". 
> > Couldn't reproduce this error on the command line. --> Commented out
> > prestage,stage,cvsmirror and tag factory addSteps and restarted run
> 
> This sounds like something went wrong syncing the mirror.. did you use the
> Makefile targets to do this or do it by hand? I tend to screw it up when
> running by hand.
> 
> The only one needed now that Talkback is binary-only is the cvsmirror_main
> target I believe.

IIRC I ran the cvsmirror_main target, which timed out on the first attempt. I interrupted it the second time, since I didn't want the FIREFOX_3_0b5_RELEASE tag removed from shipped-locales (or to wait for ages), and that may have been when the chmod or chgrp was running. So quite possibly self-induced.

Thanks for picking up and running with this guys.

Comment 34

[[e-mail] (:bhearsum)]

Updates failed out because I didn't sync the mirror first. Nick, what changes did you have to make to shipped-locales?

Comment 35

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 317269 [details] [diff] [review]
[checked in] Patcher changes, v2

Checking in patcher2.pl;
/cvsroot/mozilla/tools/patcher/patcher2.pl,v  <--  patcher2.pl
new revision: 1.32; previous revision: 1.31
done

and moved the UPDATE_PACKAGING_R4 tag to rev 1.32

Comment 36

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to comment #34)
> Updates failed out because I didn't sync the mirror first. Nick, what changes
> did you have to make to shipped-locales?

(From IRC) I didn't make any changes to shipped-locales, but Axel did in bug 428567 (the second patch) during the last week.

BTW, not sure why I was worrying about tags on shipped-locales in comment #33. There's nothing in the Makefile target that affects the tags on the old version.

Comment 37

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Fix up l10n configs for the 1.8 branch

Pulls builds from ...dates/build%build%/ rather than rc%rc% to unbreak staging on the 1.8 branch. Safe since we shipped Fx & Tb 2.0.0.14 and will use the code on this bug for the next release.

Comment 38

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [checked in] Fix up l10n configs for the 1.8 branch

Sorry, attached the wrong patch.

Comment 39

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 318997 [details] [diff] [review]
[checked in] Fix up l10n configs for the 1.8 branch

/cvsroot/mozilla/tools/tinderbox-configs/firefox/linux/tinder-config.pl,v  <--  tinder-config.pl
new revision: 1.1.18.36; previous revision: 1.1.18.35
done
Checking in moz18_l10n_release/firefox/macosx/tinder-config.pl;
/cvsroot/mozilla/tools/tinderbox-configs/firefox/macosx/tinder-config.pl,v  <--  tinder-config.pl
new revision: 1.8.10.36; previous revision: 1.8.10.35
done
Checking in moz18_l10n_release/firefox/win32/tinder-config.pl;
/cvsroot/mozilla/tools/tinderbox-configs/firefox/win32/tinder-config.pl,v  <--  tinder-config.pl
new revision: 1.2.14.37; previous revision: 1.2.14.36
done
Checking in moz18_l10n_release/thunderbird/linux/tinder-config.pl;
/cvsroot/mozilla/tools/tinderbox-configs/thunderbird/linux/tinder-config.pl,v  <--  tinder-config.pl
new revision: 1.1.14.1.2.17; previous revision: 1.1.14.1.2.16
done
Checking in moz18_l10n_release/thunderbird/macosx/tinder-config.pl;
/cvsroot/mozilla/tools/tinderbox-configs/thunderbird/macosx/tinder-config.pl,v  <--  tinder-config.pl
new revision: 1.4.6.1.2.18; previous revision: 1.4.6.1.2.17
done
Checking in moz18_l10n_release/thunderbird/win32/tinder-config.pl;
/cvsroot/mozilla/tools/tinderbox-configs/thunderbird/win32/tinder-config.pl,v  <--  tinder-config.pl
new revision: 1.6.6.1.2.20; previous revision: 1.6.6.1.2.19
done

Comment 40

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [checked in] Fix up tag replacement in client.mk

For 3.0b5, client.mk has
  MOZ_CO_TAG           = FIREFOX_3_0b5_RELEASE
  NSPR_CO_TAG          = FIREFOX_3_0b5_RELEASE
  NSS_CO_TAG           = FIREFOX_3_0b5_RELEASE
  LDAPCSDK_CO_TAG      = LDAPCSDK_6_0_3_CLIENT_BRANCH
  LOCALES_CO_TAG       =
after it's been bumped during tagging. 

For comparison
  MOZ_CO_TAG           = FIREFOX_2_0_0_14_RELEASE
  NSPR_CO_TAG          = FIREFOX_2_0_0_14_RELEASE
  NSS_CO_TAG           = FIREFOX_2_0_0_14_RELEASE
  LDAPCSDK_CO_TAG      = FIREFOX_2_0_0_14_RELEASE
  LOCALES_CO_TAG       = FIREFOX_2_0_0_14_RELEASE
shows how it should be.

This happens because our assumptions about the existing values for LDAP and LOCALES tags work are based on 1.8 branch, but things work differently on the trunk (specifically, MOZILLA_1_8_BRANCH is used for both on branch, while the trunk values are as shown in the 3.0b5 example). If we weren't setting LOCALES_CO_TAG in the mozconfigs for the l10n builds then this would have been a big problem. This patch fixes it up for people building locales from source/CVS, and keeps the LDAP in sync with all the others.

Comment 41

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 319554 [details] [diff] [review]
[checked in] Fix up tag replacement in client.mk

Checking in Bootstrap/Step/Tag/Bump.pm;
/cvsroot/mozilla/tools/release/Bootstrap/Step/Tag/Bump.pm,v  <--  Bump.pm
new revision: 1.14; previous revision: 1.13
done

and moved RELEASE_AUTOMATION_M9 onto the new revision.

Comment 42

[Nick Thomas [:nthomas] (UTC+12)]

Note to self - this doesn't remove the existing text:

>+             '^LDAPCSDK_CO_TAG\s+=\s+' =>

Need a .*$ in there

Comment 43

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [checked in] Fix regresssion from tag replacement

This fixes up the regression introduced in attachment 319554 [details] [diff] [review], where 
  LDAPCSDK_CO_TAG      = LDAPCSDK_6_0_3_CLIENT_BRANCH
got replaced with
  LDAPCSDK_CO_TAG      = THUNDERBIRD_3_0a1_RELEASELDAPCSDK_6_0_3_CLIENT_BRANCH
when tagging Tb3.0a1. This is tested against fx & tb on 1.8 branch and trunk.

Comment 44

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 319970 [details] [diff] [review]
[checked in] Fix regresssion from tag replacement

Checking in Bootstrap/Step/Tag/Bump.pm;
/cvsroot/mozilla/tools/release/Bootstrap/Step/Tag/Bump.pm,v  <--  Bump.pm
new revision: 1.15; previous revision: 1.14
done

I'm thinking of making a RELEASE_AUTOMATION_M9_1 for this fix and another in the works - RELEASE_AUTOMATION_M9 was used for Tb3.0a1

Comment 45

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [checked in] Fix up update verify for 3.0rcN

When testing 3.0rc1 -> 3.0rc2 update, had a problem where Updates::BumpVerifyConfig() was setting 
  release="3.0rc1" product="Firefox" platform....
at the top of the configs. That release value goes into URL for the update request so it should be 3.0 to match the application version. This fix makes use of Config::GetOldAppVersion() that was added earlier (knew I needed it for something!).

Comment 46

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 319983 [details] [diff] [review]
[checked in] Fix up update verify for 3.0rcN 

Checking in Bootstrap/Step/Updates.pm;
/cvsroot/mozilla/tools/release/Bootstrap/Step/Updates.pm,v  <--  Updates.pm
new revision: 1.44; previous revision: 1.43
done

Comment 47

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [checked in] Fix up long filenames for previous release in update verify

Config::GetOldVersion() needs the same treatment Config::GetVersion() had in attachment 317559 [details] [diff] [review] - picking up support for RC filenames. We need this for doing an update verify _from_ an RC build on win32/mac.

Comment 48

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 319990 [details] [diff] [review]
[checked in] Fix up long filenames for previous release in update verify

Checking in Bootstrap/Config.pm;
/cvsroot/mozilla/tools/release/Bootstrap/Config.pm,v  <--  Config.pm
new revision: 1.19; previous revision: 1.18
done

Comment 49

[Nick Thomas [:nthomas] (UTC+12)]

The build of 3.0rc2 build1 was successful with these update verify fixes. The only issue I see is the long-standing patcher problem bringing forward locales that were dropped for a release (ie on mac - ar 3.0b3 & 3.0b4; he 3.0b2, 3.0b3 & 3.0b4).

Restarting staging at 3.0 rc1 build1 with an updated cvs mirror, and using productTag of FIREFOX_3_0rc1 to match historical precedence (ie not ..3_0RC1). Also tagging release automation and tinderbox with RELEASE_AUTOMATION_M9_1. This is the final dress rehearsal.

Comment 50

[Nick Thomas [:nthomas] (UTC+12)]

The 3.0rc2 build1 run ran without issue. Verifications at 
  http://spreadsheets.google.com/pub?key=pZm1Hv_8CTFa4rPmlBS8ZUQ

Since we also didn't regress Fx 2.0.0.x (bug 432986) this bug is pretty much done. We just need to fix up comment #32, see bug 433479.