498110 - Abstract buildbotcustom factories for comm-central release automation

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Robert Kaiser]

For making comm-central release automation work, we need to abstract release factories as well as CC* factories somewhat, I'm working on this and will post a patch once I can test everything reasonably.

Comment 1

[Robert Kaiser]

Attachment: WIP1: passes tag, source, *_build, *_repack

FYI, here's the current patch that makes my config in http://hg.mozilla.org/users/kairo_kairo.at/bbconf-sm2/ (including prototyped CC* classes in ccfactory.py) pass the tag, source, *_build, and *_repack steps.
updates and the various verification steps might need more work based on this, and I still need to find out what to do about SeaMonkey probably not wanting prettynames.

Comment 2

[Robert Kaiser]

Attachment: main patch, v1: pass all worker steps

With this patch, I get all the worker steps (tag, source, *_build, *_repack, updates) to pass. I didn't test the verify steps yet (L10n verify doesn't make sense without having locales for the previous release), but I think we should drive this part in. A major part of this is actually in CC*Factory classes, so asking both Ben and gozer for reviews.

Comment 3

[Robert Kaiser]

Attachment: main patch, v1.1: unbitrot for recent changes

Sorry to post a new patch so fast but I didn't have it based on newest code last night. This is the same patch as the previous, but unbitrotted to apply cleanly to current buildbotcustom (bug 496509 caused conflicts).

While I'm at it, here's a bit of explanation what the patch does.

common release factories:
* Allow a brand name to be passed to release repack ("SeaMonkey" just like e.g. "GranParadiso" are not just the capitalized project name, but used in file names to be pulled from FTP)
* Switch more assuptions of the original source being in a self.brandName directory to self.origSrcDir (see work in bug 494577)
* Allow a different prefix than "GECKO" for the relbranch in tagging (SeaMonkey/Thunderbird use "COMM" there)
* Don't re-set the relbranchName when an override is used, as else the override will be used for all repos processed after the current one
* Use the same "generic" (in-PATH) post_upload.py in the source factory as in all other factories
* Allow a mozRepoPath to be passed to the updates factory in case our main repository is no Mozilla one (patcher needs a Mozilla repo as HGROOT to build the tools)

CC*Factory:
* Make it possible to pass explicit locations for all repositories client.py can pull from
* Also include a flag that makes it skip any repository that isn't passed (so Thunderbird can get rid of checking out or shipping sources for inspector/venkman/chatzilla)
* Make it possible to update all sources to a specific tag, passed as buildRevision (same named tag must exist on all checked out repositories)

Comment 4

[Robert Kaiser]

Oh, one more note: You need to have the patches for bug 494577 and bug 496368 applied to make this patch apply as well.

Comment 5

[Robert Kaiser]

Attachment: main patch, v2: add brandName to updates factory

I realized that in the end we actually do need the correct brandName in the updates after all, so I needed to add it to the factory and the tools it calls (need bug 499594 patches applied to use it).

So, this one change to the list above in in the new patch:

common release factories:
* Support a brandName param and hand it to tools that need to care about it.

Comment 6

[Robert Kaiser]

As a note, the configurations I used for SeaMonkey release testing, including the CC* factories prototyped in ccfactory.py, are now moved to the production repo in http://hg.mozilla.org/build/buildbot-configs/file/tip/seamonkey/

Comment 7

[Robert Kaiser]

Attachment: main patch, v2.1: unbitrot patch context

unbitrot the v2 patch, cleanly applies to buildbotcustom rev. 2c8a4ab0c551 on top
of latest bug 494577 and bug 496368 patches (only cited context in the patch was bitrotted, actually).

Comment 8

[[e-mail] (:bhearsum)]

I did a mock release with this patch today and it went great. I'll do an actual review tomorrow.

Comment 9

[[e-mail] (:bhearsum)]

Comment on attachment 387043 [details] [diff] [review]
main patch, v2.1: unbitrot patch context

> class ReleaseTaggingFactory(ReleaseFactory):
>     def __init__(self, repositories, productName, appName, version, appVersion,
>                  milestone, baseTag, buildNumber, hgUsername, hgSshKey=None,
>-                 buildSpace=1.5, **kwargs):
>+                 relbranchPrefix=None, buildSpace=1.5, **kwargs):

I'd prefer this to default to 'GECKO' and avoid setting a default inside this method.

>-        relbranchName = 'GECKO%s_%s_RELBRANCH' % (
>-          milestone.replace('.', ''), datetime.now().strftime('%Y%m%d'))
>+        if not relbranchPrefix:
>+            relbranchPrefix = 'GECKO'
>+        relbranchName = '%s%s_%s_RELBRANCH' % (
>+          relbranchPrefix, milestone.replace('.', ''),
>+          datetime.now().strftime('%Y%m%d'))
> 


>+            # use repo-specific variable so that a changed name doesn't
>+            # propagate to later repos without an override
>             relbranchOverride = False
>             if repositories[repoPath]['relbranchOverride']:
>                 relbranchOverride = True
>-                relbranchName = repositories[repoPath]['relbranchOverride']
>+                repoRelbranchName = repositories[repoPath]['relbranchOverride']
>+            else:
>+                repoRelbranchName = relbranchName
>

I'm not sure I understand the need for this. Are you using different relbranch names for different repositories?

>+           mozRepoPath: The path for the Mozilla repo to hand patcher as the
>+                        HGROOT (if omitted, the default repoPath is used).
>+                        Apps not rooted in the Mozilla repo need this.

Why? Patcher only needs the update packaging tools, which are in the mozilla repository...

Everything else in this patch looks fine, holding off on +/- until I hear back, though.

Comment 10

[Robert Kaiser]

(In reply to comment #9)
> (From update of attachment 387043 [details] [diff] [review])
> > class ReleaseTaggingFactory(ReleaseFactory):
> >     def __init__(self, repositories, productName, appName, version, appVersion,
> >                  milestone, baseTag, buildNumber, hgUsername, hgSshKey=None,
> >-                 buildSpace=1.5, **kwargs):
> >+                 relbranchPrefix=None, buildSpace=1.5, **kwargs):
> 
> I'd prefer this to default to 'GECKO' and avoid setting a default inside this
> method.

OK, no problem.

> >+            # use repo-specific variable so that a changed name doesn't
> >+            # propagate to later repos without an override
> >             relbranchOverride = False
> >             if repositories[repoPath]['relbranchOverride']:
> >                 relbranchOverride = True
> >-                relbranchName = repositories[repoPath]['relbranchOverride']
> >+                repoRelbranchName = repositories[repoPath]['relbranchOverride']
> >+            else:
> >+                repoRelbranchName = relbranchName
> >
> 
> I'm not sure I understand the need for this. Are you using different relbranch
> names for different repositories?

Yes, we do, as we want to ride on a given GECKO_* relbranch for mozilla but use a COMM_* relbranch for comm-central. But actually, the factory before this patch has the problem that when any repo specifies a relbranchOverride, all repos being processed after it happen to use the same override as well, even though none was specified for them.

> >+           mozRepoPath: The path for the Mozilla repo to hand patcher as the
> >+                        HGROOT (if omitted, the default repoPath is used).
> >+                        Apps not rooted in the Mozilla repo need this.
> 
> Why? Patcher only needs the update packaging tools, which are in the mozilla
> repository...

Exactly. And without the patch we're handing it the main repo we build from (repoPath), which happens to be the comm-central one for SeaMonkey and Thunderbird - but as you state correctly, we need to hand it the mozilla repo instead.

Comment 11

[[e-mail] (:bhearsum)]

Comment on attachment 387043 [details] [diff] [review]
main patch, v2.1: unbitrot patch context

r=bhearsum, given KaiRo's last comment.

Comment 12

[Philippe M. Chiasson (:gozer)]

Comment on attachment 387043 [details] [diff] [review]
main patch, v2.1: unbitrot patch context

Nothing more to add to the existing comments. I've applied this to my testing setup, and it came up without problems. Looks good to me.

And, KaiRo, you've been running the SeaMonkey buildbot master with these patch applied locally already, correct?

Comment 13

[[e-mail] (:bhearsum)]

Comment on attachment 387043 [details] [diff] [review]
main patch, v2.1: unbitrot patch context

changeset:   365:b22d7105b0a5

Comment 14

[Robert Kaiser]

(In reply to comment #12)
> And, KaiRo, you've been running the SeaMonkey buildbot master with these patch
> applied locally already, correct?

Right, I've been running it for a while now with that stack of patches that landed today, and with the configs (and additional CC* release factories) linked to in comment #6.

Comment 15

[Robert Kaiser]

Thanks for checking in, marking fixed as the patch seems to stick.