449209 - write a BuildFactory containing all the necessary steps to perform branching and tagging for a mercurial-based release

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[[e-mail] (:bhearsum)]

Comment 1

[[e-mail] (:bhearsum)]

Attachment: script to grab l10n repository revisions

djc pointed out that 'hg identify' works on remote repositories. I went this route for simplicity.

I'm open to better names for the script and output file.

Where do you think this should live? hg.m.o/build/tools might be the right place.

Comment 2

[[e-mail] (:bhearsum)]

Attachment: output

When run like this: bash get-l10n-release-revs af ar be ca cs da de el en-GB es-AR eu fi fr gu-IN ja it id mk mn nl pl ro si sk sl sq sr tr uk fake

Comment 3

[Axel Hecht [:Pike]]

Comment on attachment 333613 [details] [diff] [review]
script to grab l10n repository revisions

Could you silence the warning of the mv line if there is no existing l10n-changesets?

With that, r=me. And yes, I think this should live somewhere in the build repos.

As a disclaimer, we might at some point get changeset ids with a different length, so code reading this file and doing stuff based upon it should make sure that it works at least for the 12 and 40(?) length IDs.

Comment 4

[[e-mail] (:bhearsum)]

Attachment: send mv errors to /dev/null

Comment 5

[Mike Shaver (:shaver emeritus)]

That silences _all_ mv errors, not just no-existing-file.  I think you want the diagnostic for things like permission errors or source-file-name-somehow-wrong, and would recommend pre-testing, comme ca:

[ -f $DUMP_FILE ] && mv $DUMP_FILE $DUMP_FILE.old

Also, you have an unbalanced quote in your diagnostic indicating that you're running identify.

Also also, you probably want to spit the result line out to stdout, because everyone who ever runs this script is immediately going to want to be checking for problems by opening the file, and this way you have the results interleaved with the commands.  Perhaps something like:

echo "$locale $id" | tee -a $DUMP_FILE

?

Otherwise, the bike shed looks _great_.

Comment 6

[[e-mail] (:bhearsum)]

Attachment: [checked in] address shaver's comments

Comment 7

[[e-mail] (:bhearsum)]

Attachment: tagging factory.

Here's my attempt at a Buildbot factory for tagging Mercurial based releases. There's lots of comments inline, but here's an overview:
* 'repositories' is a data structure that contains repository names and for each repository: signed-off revision numbers, where applicable, a list of files that need to be version bumped, and when applicable, a 'release branch' name. The other parameters are necessary mostly for version bumping, and a little bit of build 1 vs. respin logic.
* For l10n, we are going to rely on an external file that contains locales + revisions. The master.cfg will be responsible for interpreting this into the 'repositories' data structure. later on, it might be good to have helper classes and/or functions to organize this a bit better. for now, i think this is OK.
* The Factory has logic to determine when to create a branch vs. update to one, how to name release branches + tags, etc.

This process is almost identical to the way that Bootstrap works. appName, version, etc. will be defined somewhere in the Buildbot configuration and passed on.

For 'build 1's, a typical call would like this:

repositories = {
  'http://hg.mozilla.org/mozilla-central': {
    'revision': 'b6be543f6734',
    'bumpFiles': ['mozilla/client.mk', 'config/milestone.txt', 'browser/app/module.ver', 'browser/config/version.txt'],
    'relbranchOverride': ''
  }
}

<add in l10n stuff>

tag_factory = ReleaseTaggingFactory(
  repositories=repositories,
  buildToolsRepo='https://hg.mozilla.org/build/tools',
  appName='browser',
  appVersion='3.1b1',
  milestone='1.9.1b1',
  baseTag='FIREFOX_3_1b1',
  buildNumber=1
)

For respins, it would be like this:
repositories = {
  'http://hg.mozilla.org/mozilla-central': {
    'revision': 'b6be543f6734',
    'bumpFiles': []
    'relbranchOverride': 'GECKO191_20080827_RELBRANCH'
  }
}

<add in l10n stuff>

tag_factory = ReleaseTaggingFactory(
  repositories=repositories,
  buildToolsRepo='https://hg.mozilla.org/build/tools',
  appName='browser',
  appVersion='3.1b1',
  milestone='1.9.1b1',
  baseTag='FIREFOX_3_1b1',
  buildNumber=2
)

Comment 8

[Axel Hecht [:Pike]]

Minor note, sorted() is python 2.4 and greater, not sure what our requirements are.

I'm wondering if something like buildNumber > 1 and not set relbranchOverride should error.

The os.path.basename to get the repoName sounds wrong, and that's brittle with http://hg.mozilla.org/mozilla-central/ vs http://hg.mozilla.org/mozilla-central. In particular, hg locally prefers the trailing slash, at which point the repo name turns out empty.

I don't get the version bump check-in message :-?

2 cts.

Comment 9

[[e-mail] (:bhearsum)]

Comment on attachment 335746 [details] [diff] [review]
tagging factory.

(In reply to comment #8)
> Minor note, sorted() is python 2.4 and greater, not sure what our requirements
> are.
> 

I think 2.4 or greater is an OK bar for us to set. Honestly though, I think all of our HG build machinery are Python 2.5.

> I'm wondering if something like buildNumber > 1 and not set relbranchOverride
> should error.
> 

Yeah, that sounds good.

> The os.path.basename to get the repoName sounds wrong, and that's brittle with
> http://hg.mozilla.org/mozilla-central/ vs
> http://hg.mozilla.org/mozilla-central. In particular, hg locally prefers the
> trailing slash, at which point the repo name turns out empty.
> 

That's true. I could've sworn I switched this code to use urlparse - I guess not!



I'm going to fix up the things mentioned above in a new patch.

Comment 10

[Chris Cooper [:coop] (he/him)]

Comment on attachment 335746 [details] [diff] [review]
tagging factory.

Nits are all in the comments.

>+            relbranchOverride is typically used in two situations:
>+             1) During a respin (buildNumber > 2) when the "release" branch has

That should be buildNumber >= 2, no?

+        # we don't need to do any version bumping if this is a respin

Good comment, just not sure it's in the right place. Down by where version-bump.pl is actually called would be better.

>+            # generally, this is specifide for Firefox respins and non-Firefox

typo in specified

+            # if buildNumber > 2 we need to switch to it with 'hg up -C'

buildNumber >= 2 (again)

Comment 11

[[e-mail] (:bhearsum)]

Attachment: [checked in] address axel's and coop's comments

Changes:
* Fix typos
* Correctly use buildNumber > 1 (instead of buildNumber > 2)
* Use productName in check-in message instead of repoName.
* Some simple validation in __init__()
* Support http, https, ssh, and local repositories (use really simple string manipulation to grab the repository name and pushRepo)

Comment 12

[[e-mail] (:bhearsum)]

Comment on attachment 336493 [details] [diff] [review]
[checked in] address axel's and coop's comments

Checking in factory.py;
/cvsroot/mozilla/tools/buildbotcustom/process/factory.py,v  <--  factory.py
new revision: 1.11; previous revision: 1.10
done

Comment 13

[[e-mail] (:bhearsum)]

Comment on attachment 333766 [details]
[checked in] address shaver's comments

changeset:   5:82400503b9f0

Comment 14

[[e-mail] (:bhearsum)]

(comment #13 was me landing the l10n script in hg.m.o/build/tools/release/get-l10n-revs

Comment 15

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [checked in] Fix up URL for l10n repos

I took get-l10n-revs for a spin and got
  $ bash get-l10n-revs af
  Running 'hg identify http://hg.mozilla.org/l10n-central/index.cgi/af'
    af real URL is http://hg.mozilla.org/l10n-central/af
  8f83d4d24ad4
instead of 
  $ bash get-l10n-revs af
  Running 'hg identify http://hg.mozilla.org/l10n-central/af'
    af 8f83d4d24ad4
The l10n-changesets file also has the message instead of the rev. 

Ben, perhaps you have -q set as a global option for hg ? I'd rather not set that when we can trim '/index.cgi' off L10N_CENTRAL. Also sets the executable bit. Look OK ?

Comment 16

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 337816 [details] [diff] [review]
[checked in] Fix up URL for l10n repos

changeset:   7:f45852093b61
tag:         tip
user:        Nick Thomas <[e-mail]>
date:        Thu Sep 11 08:12:37 2008 +1200
files:       release/get-l10n-revs
description:
Bug 449209, tweak url to l10n-central, r=bhearsum