711534 - Configure Thunderbird release builders

Status: RESOLVED FIXED

(Release Engineering :: Release Automation, defect)

Description

[John Hopkins (:jhopkins)]

Let's get Thunderbird release builds running on dev-master01 + Firefox build slaves.
Development buildbot-configs are here:

 https://hg.mozilla.org/users/john.hopkins_mozillamessaging.com/buildbot-configs-stage/

Look in the thunderbird2/ subdirectory.

Comment 1

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

found in triage.

Comment 2

[John Hopkins (:jhopkins)]

The actual comm-release builds are looking good, aside from get_codesize_log failing on Linux and OS X.

Comment 3

[[e-mail] (:bhearsum)]

John, I think you'll be the one looking at this, right?

Comment 4

[John Hopkins (:jhopkins)]

Attachment: initial patch passes checkconfig

Comment 5

[[e-mail] (:bhearsum)]

Comment on attachment 618254 [details] [diff] [review]
initial patch passes checkconfig

Review of attachment 618254 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozilla/staging_release-thunderbird-comm-beta.py
@@ +34,5 @@
> +releaseConfig['userRepoRoot'] = 'users/stage-ffxbld'
> +releaseConfig['sourceRepositories']  = {
> +    'mozilla': {
> +        'name': 'comm-beta',
> +        'clonePath': 'releases/comm-beta',

You should be using mozilla-beta here, as the 'mozilla' repository I think....comm-beta should go in as another entry in the dict. sourceRepoKey may need adjusting, we create some names and make some decisions based on the repo named by 'sourceRepoInfo', have a look at process/release.py and it should be a bit clearer.

@@ +50,5 @@
> +#  L10n repositories
> +releaseConfig['l10nRelbranch']       = None
> +releaseConfig['l10nRepoClonePath']   = 'releases/l10n/mozilla-beta'
> +releaseConfig['l10nRepoPath']        = 'users/stage-ffxbld'
> +releaseConfig['l10nRevisionFile']    = 'l10n-thunderbird-changesets-beta'

Naming nit: we call these files "l10n-changesets" normally, so I'd prefer to see them named l10n-changesets_thunderbird-$branch

@@ +80,5 @@
> +releaseConfig['hgSshKey']            = '~cltbld/.ssh/ffxbld_dsa'
> +
> +# Update-specific configuration
> +releaseConfig['cvsroot']             = ':ext:[e-mail]:/cvsroot'
> +releaseConfig['patcherConfig']       = 'mozBeta-branch-patcher2.cfg'

Need to use a different patcher config for Thunderbird.

@@ +104,5 @@
> +    'linux64': 'mail/config/mozconfigs/linux64/release',
> +    'macosx64': 'mail/config/mozconfigs/macosx-universal/release',
> +    'win32': 'mail/config/mozconfigs/win32/release',
> +}
> +releaseConfig['xulrunner_mozconfigs']          = {}

You can just omit this entirely.

::: mozilla/universal_master_sqlite.cfg
@@ -37,1 @@
>  c['status'].append(QueuedCommandHandler(

builder_master.cfg will need similar changes as this file.

@@ +109,5 @@
>      from buildbotcustom.process.release import generateReleaseBranchObjects
>      if RESERVED_SLAVES:
>          setReservedFileName(RESERVED_SLAVES)
> +    for product, sourceRepoKey, productBranches, releaseBranch in \
> +            [('thunderbird', 'mozilla', THUNDERBIRD_BRANCHES, b)

You probably want to use 'comm' or something as the sourceRepoKey rather than 'mozilla'...see more in the release configs.

Comment 6

[John Hopkins (:jhopkins)]

Attachment: ability to pass extra environment vars to builds

Will pass revision details to Thunderbird mozconfigs via environment variables.  This patch provides that feature.

Comment 7

[John Hopkins (:jhopkins)]

Attachment: thunderbird release configs

Addresses feedback except for l10n changeset file names (will address once i've copied those files to a new name).

passes checkconfig and 'setup-master.py -t' tests.

Comment 8

[[e-mail] (:bhearsum)]

Comment on attachment 618713 [details] [diff] [review]
thunderbird release configs

Review of attachment 618713 [details] [diff] [review]:
-----------------------------------------------------------------

Everything looks ok except the extraBuildEnv stuff.

::: mozilla/builder_master.cfg
@@ +98,5 @@
> +        if x.branch == cs.branch:
> +            cnt = cnt + 1
> +    if cnt > 1:
> +        buildObjects['change_source'].remove(cs)
> +

Huh, this code doesn't already exist in builder_master.cfg?

@@ +113,5 @@
>      from buildbotcustom.process.release import generateReleaseBranchObjects
>      if RESERVED_SLAVES:
>          setReservedFileName(RESERVED_SLAVES)
> +    for product, sourceRepoKey, productBranches, releaseBranch in \
> +            [('thunderbird', 'mozilla', THUNDERBIRD_BRANCHES, b)

I think you want 'comm' here, not 'mozilla'.

::: mozilla/staging_release-thunderbird-comm-beta.py
@@ +70,5 @@
> +}
> +releaseConfig['extraBuildEnv'] = {
> +    'COMM_REV':      releaseConfig['sourceRepositories']['comm']['revision'],
> +    'MOZILLA_REV':   releaseConfig['sourceRepositories']['mozilla']['revision'],
> +}

I don't think you need these at all. ReleaseBuildFactory already has buildRevision, which is the _RELEASE tag. You can have it set these two variables rather than defining them in every release config. If you really think you need to set them here, please merge them in with 'platform_env' in release.py -- we definitely don't want to have 'env' and 'extraBuildEnv' as args to the same factory - very confusing.

Comment 9

[[e-mail] (:bhearsum)]

Comment on attachment 618712 [details] [diff] [review]
ability to pass extra environment vars to builds

Review of attachment 618712 [details] [diff] [review]:
-----------------------------------------------------------------

I don't think this is necessary, per my previous comment.

Comment 10

[John Hopkins (:jhopkins)]

Attachment: copy Thunderbird l10n files under mozilla/ directory

Note that l10n changeset files are also renamed to conform to Firefox's naming convention.

Comment 11

[John Hopkins (:jhopkins)]

Attachment: buildbotcustom patch to set COMM_REV and MOZILLA_REV in platform_env[]

Comment 12

[John Hopkins (:jhopkins)]

Attachment: release configs

Comment 13

[Mark Banner (:standard8)]

Comment on attachment 618903 [details] [diff] [review]
copy Thunderbird l10n files under mozilla/ directory

Hang on a mo, you need to update thunderbird_config.py for the all-locales.* changes.

Comment 14

[John Hopkins (:jhopkins)]

Attachment: buildbot-configs patch to reflect new all-locales locations

standard8: if you're happy with this patch, can you please review the 'copy Thunderbird l10n files under mozilla/ directory' patch as well?

Comment 15

[John Hopkins (:jhopkins)]

Comment on attachment 618904 [details] [diff] [review]
buildbotcustom patch to set COMM_REV and MOZILLA_REV in platform_env[]

Review of attachment 618904 [details] [diff] [review]:
-----------------------------------------------------------------

Landed in https://hg.mozilla.org/build/buildbotcustom/rev/28f4d9562676

Comment 16

[John Hopkins (:jhopkins)]

Comment on attachment 618908 [details] [diff] [review]
release configs

Review of attachment 618908 [details] [diff] [review]:
-----------------------------------------------------------------

Landed in https://hg.mozilla.org/build/buildbot-configs/rev/f335696d8e8b

Comment 18

[Rail Aliiev [:rail]]

Merged to production today

Comment 19

[Mark Banner (:standard8)]

Comment on attachment 619015 [details] [diff] [review]
buildbot-configs patch to reflect new all-locales locations

r=me if you correct the line that says 'browser/locales/all-locales' for BRANCHES[comm-esr10] as pointed out over irc.

Comment 20

[Mark Banner (:standard8)]

Comment on attachment 618903 [details] [diff] [review]
copy Thunderbird l10n files under mozilla/ directory

r+ though we'll probably want to copy the esr file from comm-esr10 as well.

Comment 21

[John Hopkins (:jhopkins)]

Comment on attachment 618903 [details] [diff] [review]
copy Thunderbird l10n files under mozilla/ directory

Landed in https://hg.mozilla.org/build/buildbot-configs/rev/d344e17405c7

Comment 22

[John Hopkins (:jhopkins)]

Comment on attachment 619015 [details] [diff] [review]
buildbot-configs patch to reflect new all-locales locations

Review of attachment 619015 [details] [diff] [review]:
-----------------------------------------------------------------

Landed in https://hg.mozilla.org/build/buildbot-configs/rev/753ffadb0b1b

Comment 23

[John Hopkins (:jhopkins)]

Release builders are deployed to production masters. Any issues will be tracked in separate bugs.