502876 - mercurial release automation needs to easily support multiple branches on the same master instance at the same time

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P4)

Description

[[e-mail] (:bhearsum)]

Currently, the configuration side of the Mercurial release automation consists of two files: release_master.py and release_config.py. Because of this assumption of only one configuration file it will be ugly and difficult to run two branches within the same Buildbot. We need to fix this in some way that allows us to have multiple configuration files, and easily switch between them.

We should be able to do this in such a way that release_master.py can be branch independent, and the configuration files can be swapped in and out.

We should do this before we ship the first Alpha of Firefox.next.

Comment 1

[Chris AtLee [:catlee]]

Can we also do it so that the master doesn't need to be reconfigured to update the parameters for the release factories?

Comment 2

[[e-mail] (:bhearsum)]

(In reply to comment #1)
> Can we also do it so that the master doesn't need to be reconfigured to update
> the parameters for the release factories?

I'm not convinced this is of great benefit, but either way it's a separate thing IMHO.

Comment 3

[[e-mail] (:bhearsum)]

Attachment: stopgap solution

This is by no means a long term solution to the problem, but it will enable 3.5.2/3.6a1 to happen this week. The -mozilla-1.9.1 file is a straight copy of the existing release_config.py, and the -mozilla-central one is a copy with obvious changes made. We'll also need temporary changes to release_master.py for 3.6a1, those are being tracked in bug 506401.

Comment 4

[Chris AtLee [:catlee]]

Comment on attachment 390814 [details] [diff] [review]
stopgap solution

>diff -r 471850cdabae mozilla2/release-firefox-mozilla-central.py
>--- /dev/null	Thu Jan 01 00:00:00 1970 +0000
>+++ b/mozilla2/release-firefox-mozilla-central.py	Mon Jul 27 10:30:14 2009 -0400
>@@ -0,0 +1,36 @@

<snip>

>+patcherConfig      = 'moz191-branch-patcher2.cfg'
>+patcherToolsTag    = 'UPDATE_PACKAGING_R9'
>+ftpServer          = 'ftp.mozilla.org'
>+stagingServer      = 'stage-old.mozilla.org'
>+bouncerServer      = 'download.mozilla.org'
>+ausServerUrl       = 'https://aus2.mozilla.org'
>+useBetaChannel     = 1
>+verifyConfigs      = {'linux':  'moz191-firefox-linux.cfg',
>+                      'macosx': 'moz191-firefox-mac.cfg',
>+                      'win32':  'moz191-firefox-win32.cfg'}

Let's get these setup for the mozilla-central / mozilla-1.9.2 configs.

Comment 5

[[e-mail] (:bhearsum)]

Attachment: comments addressed

It doesn't make sense to create the patcher or update_verify configs yet - as they would be empty. I've left TODOs in the file to get them created next time, though. I also flipped useBetaChannel which, when we generate updates for a2, will prevent release snippets from being generated.

Comment 6

[Chris AtLee [:catlee]]

Comment on attachment 391375 [details] [diff] [review]
comments addressed

small mismatch between staging and production for the mozilla-central configs, but the production ones look good.

Comment 7

[[e-mail] (:bhearsum)]

Comment on attachment 391375 [details] [diff] [review]
comments addressed

changeset:   1379:79806e4aa97d

Comment 8

[Axel Hecht [:Pike]]

Per IRC, we'll need this for l10n-changesets on 1.9.1 and 1.9.2, too.

Comment 9

[[e-mail] (:bhearsum)]

So, this bug was originally about being able to do multiple releases on the same master at the same time. We've worked around this largely by having more masters. This is still a valid bug, but less pressing now.

Comment 10

[Chris Cooper [:coop] (he/him)]

Mass move of bugs from Release Engineering:Future -> Release Engineering. See
http://coop.deadsquid.com/2010/02/kiss-the-future-goodbye/ for more details.

Comment 11

[Syed Albiz [:salbiz]]

Attachment: concurrent multibranch releases

This consolidates the releaseConfigs for all branches into one file and then generates releaseObjects for every branch. Which should be fine now that all release-builders have unique names between branches

I have only tested that the releaseObjects are generated correctly, so there could still be problems encountered during the builder steps if two branches run releases concurrently.

Comment 12

[Chris AtLee [:catlee]]

Comment on attachment 487343 [details] [diff] [review]
concurrent multibranch releases

Having the staging and production configs in the same file is a bit weird.

Also, we need a way to enable/disable branches per master.

What about something like this...

release_config1.py:

ACTIVE_RELEASE_BRANCHES = ['mozilla-central', 'mozilla-1.9.2']

releaseConfig = {}
for b in ACTIVE_RELEASE_BRANCHES:
    releaseConfig[b] = loadConfig(b)

Comment 13

[[e-mail] (:bhearsum)]

I have similar thoughts. I think I'd rather see the configs remain in separate files, too....it's much easier for reading.

Nick, any thoughts here?

Comment 14

[Syed Albiz [:salbiz]]

Attachment: split release_configs into staging_release_config.py, production_release_config.py

For comparison, here is the version of the patch that splits them up into staging, and production releases. Also adds the ACTIVE_RELEASE_BRANCHES list to the release_configs themselves.

Comment 15

[[e-mail] (:bhearsum)]

From IRC, I prefer to have the configs split by branch, too. I think it helps readability and discourages factoring out common things -- which is anti-feature here IMHO.

Comment 16

[Chris AtLee [:catlee]]

Comment on attachment 487653 [details] [diff] [review]
split release_configs into staging_release_config.py, production_release_config.py

Maybe check somehow that we don't already have a PBChangeSource in c['change_source']

The rest of the code in the main master config looks good.

But yeah, I think we need to split the configs by branch as well...how about something like:

for releaseBranch in ACTIVE_RELEASE_BRANCHES:
   releaseBranchConfig = {}
   execfile('release-%s.cfg' % releaseBranch, releaseBranchConfig, releaseBranchConfig)
   ...

Comment 17

[Armen [:armenzg]]

(In reply to comment #16)
> for releaseBranch in ACTIVE_RELEASE_BRANCHES:
>    releaseBranchConfig = {}
>    execfile('release-%s.cfg' % releaseBranch, releaseBranchConfig,
> releaseBranchConfig)
>    ...
I vote for this approach as well.

Comment 18

[Syed Albiz [:salbiz]]

Attachment: use execfile to keep existing release_configs structure

Done. I've kept the ACTIVE_RELEASE_BRANCHES in the master_localconfigs so that it will be configurable per master.

Comment 19

[Chris AtLee [:catlee]]

Comment on attachment 488307 [details] [diff] [review]
use execfile to keep existing release_configs structure

This is looking better.

I think you need to move releaseBranchConfig = {} to inside the for loop.  It's also missing from the universal config.

Can you add a check that we're not adding more than one PBChangeSource to the masters?

Comment 20

[[e-mail] (:bhearsum)]

Comment on attachment 488307 [details] [diff] [review]
use execfile to keep existing release_configs structure

>diff --git a/mozilla/production_builder_master_bm01_localconfig.py b/mozilla/production_builder_master_bm01_localconfig.py
>--- a/mozilla/production_builder_master_bm01_localconfig.py
>+++ b/mozilla/production_builder_master_bm01_localconfig.py
>diff --git a/mozilla/production_builder_master_bm02_localconfig.py b/mozilla/production_builder_master_bm02_localconfig.py
>--- a/mozilla/production_builder_master_bm02_localconfig.py
>+++ b/mozilla/production_builder_master_bm02_localconfig.py

We're going to have to leave release disabled on these two masters, for now at least, because they don't have any Mac build machines.


> releaseConfig['hgUsername']          = 'ffxbld'
> releaseConfig['hgSshKey']            = '~cltbld/.ssh/ffxbld_dsa'
>-releaseConfig['sourceRepoName']      = 'mozilla-central'
>+releaseConfig['sourceRepoName']      = 'mozilla-2.0'

mozilla-central is actually correct, for now. Please leave this unchanged, and in the staging config.

Looks fine other than those two things.

Comment 21

[Syed Albiz [:salbiz]]

Attachment: check for PBChangeSource(), shuffle localconfigs

added a rough check to see if c['changes'] has a PBChangeSource already in the list, fixed the execfile stanza in both configs and moved around some configs based on bhearsum's comment. One issue that came up was that I had to disable mozilla-central as the sourceRepoName collides with mozilla-2.0 causing duplicate buildernames.

Comment 22

[Syed Albiz [:salbiz]]

Comment on attachment 488866 [details] [diff] [review]
check for PBChangeSource(), shuffle localconfigs

An issue arose during testing, where the repo_setup step clobbers all the l10n builders on staging, which causes other branch builds using those repos to fail, or use the wrong repos as the l10n repos that are recreated are not for the correct branch. This can be partially worked around by disabling repo_setup on all but one of the release branches.

Comment 23

[[e-mail] (:bhearsum)]

Attachment: make share() capable of updating from a repo

This is going to be a pretty common thing, so the share() library may as well handle it.

Comment 24

[[e-mail] (:bhearsum)]

Attachment: continue settings got_revision

Comment 25

[[e-mail] (:bhearsum)]

Attachment: integrate share even more

So..I think my previous patch was a bit hasty. When I went to integrate it with my l10n work I realized that I had to use share() everywhere -- including library functions. I don't think that's the right thing to do, especially library functions, so I wrote this. This should have no behaviour change unless HG_SHARE_BASE_DIR is set in the environment, or a consumer overrides util.hg.SHARE_BASE_DIR.

With this, we can set HG_SHARE_BASE_DIR per platform in the buildbot configs and should end up using share in anything using the mercurial() method, including hgtool.py uses.

Comment 26

[[e-mail] (:bhearsum)]

I...don't know why I posted the share patches here.

Comment 27

[[e-mail] (:bhearsum)]

Comment on attachment 490091 [details] [diff] [review]
integrate share even more

Moving this to the correct bug...

Comment 28

[[e-mail] (:bhearsum)]

Comment on attachment 488866 [details] [diff] [review]
check for PBChangeSource(), shuffle localconfigs

changeset:   3334:2d630f4871d0

Comment 29

[[e-mail] (:bhearsum)]

All done here!