Closed Bug 502876 Opened 12 years ago Closed 11 years ago

mercurial release automation needs to easily support multiple branches on the same master instance at the same time

Categories

(Release Engineering :: General, defect, P4)

x86
macOS
defect

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: bhearsum, Assigned: salbiz)

References

Details

(Whiteboard: [automation])

Attachments

(3 files, 6 obsolete files)

Currently, the configuration side of the Mercurial release automation consists of two files: release_master.py and release_config.py. Because of this assumption of only one configuration file it will be ugly and difficult to run two branches within the same Buildbot. We need to fix this in some way that allows us to have multiple configuration files, and easily switch between them.

We should be able to do this in such a way that release_master.py can be branch independent, and the configuration files can be swapped in and out.

We should do this before we ship the first Alpha of Firefox.next.
Priority: -- → P3
Can we also do it so that the master doesn't need to be reconfigured to update the parameters for the release factories?
(In reply to comment #1)
> Can we also do it so that the master doesn't need to be reconfigured to update
> the parameters for the release factories?

I'm not convinced this is of great benefit, but either way it's a separate thing IMHO.
Attached patch stopgap solutionSplinter Review
This is by no means a long term solution to the problem, but it will enable 3.5.2/3.6a1 to happen this week. The -mozilla-1.9.1 file is a straight copy of the existing release_config.py, and the -mozilla-central one is a copy with obvious changes made. We'll also need temporary changes to release_master.py for 3.6a1, those are being tracked in bug 506401.
Attachment #390814 - Flags: review?(catlee)
Comment on attachment 390814 [details] [diff] [review]
stopgap solution

>diff -r 471850cdabae mozilla2/release-firefox-mozilla-central.py
>--- /dev/null	Thu Jan 01 00:00:00 1970 +0000
>+++ b/mozilla2/release-firefox-mozilla-central.py	Mon Jul 27 10:30:14 2009 -0400
>@@ -0,0 +1,36 @@

<snip>

>+patcherConfig      = 'moz191-branch-patcher2.cfg'
>+patcherToolsTag    = 'UPDATE_PACKAGING_R9'
>+ftpServer          = 'ftp.mozilla.org'
>+stagingServer      = 'stage-old.mozilla.org'
>+bouncerServer      = 'download.mozilla.org'
>+ausServerUrl       = 'https://aus2.mozilla.org'
>+useBetaChannel     = 1
>+verifyConfigs      = {'linux':  'moz191-firefox-linux.cfg',
>+                      'macosx': 'moz191-firefox-mac.cfg',
>+                      'win32':  'moz191-firefox-win32.cfg'}

Let's get these setup for the mozilla-central / mozilla-1.9.2 configs.
Attachment #390814 - Flags: review?(catlee) → review-
It doesn't make sense to create the patcher or update_verify configs yet - as they would be empty. I've left TODOs in the file to get them created next time, though. I also flipped useBetaChannel which, when we generate updates for a2, will prevent release snippets from being generated.
Attachment #391375 - Flags: review?(catlee)
Comment on attachment 391375 [details] [diff] [review]
comments addressed

small mismatch between staging and production for the mozilla-central configs, but the production ones look good.
Attachment #391375 - Flags: review?(catlee) → review+
Comment on attachment 391375 [details] [diff] [review]
comments addressed

changeset:   1379:79806e4aa97d
Attachment #391375 - Flags: checked-in+
Per IRC, we'll need this for l10n-changesets on 1.9.1 and 1.9.2, too.
So, this bug was originally about being able to do multiple releases on the same master at the same time. We've worked around this largely by having more masters. This is still a valid bug, but less pressing now.
Summary: mercurial release automation needs to easily support multiple branches at the same time → mercurial release automation needs to easily support multiple branches on the same master instance at the same time
Whiteboard: [automation]
Mass move of bugs from Release Engineering:Future -> Release Engineering. See
http://coop.deadsquid.com/2010/02/kiss-the-future-goodbye/ for more details.
Component: Release Engineering: Future → Release Engineering
Assignee: nobody → armenzg
Status: NEW → ASSIGNED
Priority: P3 → P4
Attached patch concurrent multibranch releases (obsolete) — Splinter Review
This consolidates the releaseConfigs for all branches into one file and then generates releaseObjects for every branch. Which should be fine now that all release-builders have unique names between branches

I have only tested that the releaseObjects are generated correctly, so there could still be problems encountered during the builder steps if two branches run releases concurrently.
Assignee: armenzg → salbiz
Attachment #487343 - Flags: feedback?(catlee)
Attachment #487343 - Flags: feedback?(armenzg)
Comment on attachment 487343 [details] [diff] [review]
concurrent multibranch releases

Having the staging and production configs in the same file is a bit weird.

Also, we need a way to enable/disable branches per master.

What about something like this...

release_config1.py:

ACTIVE_RELEASE_BRANCHES = ['mozilla-central', 'mozilla-1.9.2']

releaseConfig = {}
for b in ACTIVE_RELEASE_BRANCHES:
    releaseConfig[b] = loadConfig(b)
Attachment #487343 - Flags: feedback?(catlee) → feedback-
I have similar thoughts. I think I'd rather see the configs remain in separate files, too....it's much easier for reading.

Nick, any thoughts here?
For comparison, here is the version of the patch that splits them up into staging, and production releases. Also adds the ACTIVE_RELEASE_BRANCHES list to the release_configs themselves.
Attachment #487343 - Attachment is obsolete: true
Attachment #487653 - Flags: feedback?(catlee)
Attachment #487653 - Flags: feedback?(bhearsum)
Attachment #487343 - Flags: feedback?(armenzg)
From IRC, I prefer to have the configs split by branch, too. I think it helps readability and discourages factoring out common things -- which is anti-feature here IMHO.
Attachment #487653 - Flags: feedback?(bhearsum) → feedback-
Comment on attachment 487653 [details] [diff] [review]
split release_configs into staging_release_config.py, production_release_config.py

Maybe check somehow that we don't already have a PBChangeSource in c['change_source']

The rest of the code in the main master config looks good.

But yeah, I think we need to split the configs by branch as well...how about something like:

for releaseBranch in ACTIVE_RELEASE_BRANCHES:
   releaseBranchConfig = {}
   execfile('release-%s.cfg' % releaseBranch, releaseBranchConfig, releaseBranchConfig)
   ...
Attachment #487653 - Flags: feedback?(catlee) → feedback-
(In reply to comment #16)
> for releaseBranch in ACTIVE_RELEASE_BRANCHES:
>    releaseBranchConfig = {}
>    execfile('release-%s.cfg' % releaseBranch, releaseBranchConfig,
> releaseBranchConfig)
>    ...
I vote for this approach as well.
Done. I've kept the ACTIVE_RELEASE_BRANCHES in the master_localconfigs so that it will be configurable per master.
Attachment #487653 - Attachment is obsolete: true
Attachment #488307 - Flags: feedback?(catlee)
Attachment #488307 - Flags: feedback?(bhearsum)
Comment on attachment 488307 [details] [diff] [review]
use execfile to keep existing release_configs structure

This is looking better.

I think you need to move releaseBranchConfig = {} to inside the for loop.  It's also missing from the universal config.

Can you add a check that we're not adding more than one PBChangeSource to the masters?
Attachment #488307 - Flags: feedback?(catlee) → feedback+
Comment on attachment 488307 [details] [diff] [review]
use execfile to keep existing release_configs structure

>diff --git a/mozilla/production_builder_master_bm01_localconfig.py b/mozilla/production_builder_master_bm01_localconfig.py
>--- a/mozilla/production_builder_master_bm01_localconfig.py
>+++ b/mozilla/production_builder_master_bm01_localconfig.py
>diff --git a/mozilla/production_builder_master_bm02_localconfig.py b/mozilla/production_builder_master_bm02_localconfig.py
>--- a/mozilla/production_builder_master_bm02_localconfig.py
>+++ b/mozilla/production_builder_master_bm02_localconfig.py

We're going to have to leave release disabled on these two masters, for now at least, because they don't have any Mac build machines.


> releaseConfig['hgUsername']          = 'ffxbld'
> releaseConfig['hgSshKey']            = '~cltbld/.ssh/ffxbld_dsa'
>-releaseConfig['sourceRepoName']      = 'mozilla-central'
>+releaseConfig['sourceRepoName']      = 'mozilla-2.0'

mozilla-central is actually correct, for now. Please leave this unchanged, and in the staging config.

Looks fine other than those two things.
Attachment #488307 - Flags: feedback?(bhearsum) → feedback+
added a rough check to see if c['changes'] has a PBChangeSource already in the list, fixed the execfile stanza in both configs and moved around some configs based on bhearsum's comment. One issue that came up was that I had to disable mozilla-central as the sourceRepoName collides with mozilla-2.0 causing duplicate buildernames.
Attachment #488307 - Attachment is obsolete: true
Attachment #488866 - Flags: feedback?(catlee)
Attachment #488866 - Flags: feedback?(bhearsum)
Attachment #488866 - Flags: feedback?(bhearsum) → feedback+
Attachment #488866 - Flags: feedback?(catlee) → feedback+
Comment on attachment 488866 [details] [diff] [review]
check for PBChangeSource(), shuffle localconfigs

An issue arose during testing, where the repo_setup step clobbers all the l10n builders on staging, which causes other branch builds using those repos to fail, or use the wrong repos as the l10n repos that are recreated are not for the correct branch. This can be partially worked around by disabling repo_setup on all but one of the release branches.
This is going to be a pretty common thing, so the share() library may as well handle it.
Attachment #489894 - Flags: review?(catlee)
Attachment #489894 - Flags: review?(salbiz)
Attached patch continue settings got_revision (obsolete) — Splinter Review
Attachment #489916 - Flags: review?(salbiz)
Attachment #489916 - Flags: review?(catlee)
Attachment #489894 - Attachment is obsolete: true
Attachment #489894 - Flags: review?(salbiz)
Attachment #489894 - Flags: review?(catlee)
Attachment #489916 - Flags: review?(catlee) → review+
Attachment #489916 - Flags: review?(salbiz) → review+
Attached patch integrate share even more (obsolete) — Splinter Review
So..I think my previous patch was a bit hasty. When I went to integrate it with my l10n work I realized that I had to use share() everywhere -- including library functions. I don't think that's the right thing to do, especially library functions, so I wrote this. This should have no behaviour change unless HG_SHARE_BASE_DIR is set in the environment, or a consumer overrides util.hg.SHARE_BASE_DIR.

With this, we can set HG_SHARE_BASE_DIR per platform in the buildbot configs and should end up using share in anything using the mercurial() method, including hgtool.py uses.
Attachment #490091 - Flags: review?(salbiz)
Attachment #490091 - Flags: review?(catlee)
I...don't know why I posted the share patches here.
Comment on attachment 490091 [details] [diff] [review]
integrate share even more

Moving this to the correct bug...
Attachment #490091 - Attachment is obsolete: true
Attachment #490091 - Flags: review?(salbiz)
Attachment #490091 - Flags: review?(catlee)
Attachment #489916 - Attachment is obsolete: true
Depends on: 553300
Flags: needs-reconfig?
Comment on attachment 488866 [details] [diff] [review]
check for PBChangeSource(), shuffle localconfigs

changeset:   3334:2d630f4871d0
Attachment #488866 - Flags: review+
Attachment #488866 - Flags: checked-in+
All done here!
Status: ASSIGNED → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
Flags: needs-reconfig?
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.