Abstract buildbotcustom factories for comm-central release automation

RESOLVED FIXED

Status

Release Engineering
General
RESOLVED FIXED
9 years ago
5 years ago

People

(Reporter: Robert Kaiser, Assigned: Robert Kaiser)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment, 4 obsolete attachments)

(Assignee)

Description

9 years ago
For making comm-central release automation work, we need to abstract release factories as well as CC* factories somewhat, I'm working on this and will post a patch once I can test everything reasonably.
(Assignee)

Updated

9 years ago
Blocks: 493451
(Assignee)

Comment 1

9 years ago
Created attachment 383450 [details] [diff] [review]
WIP1: passes tag, source, *_build, *_repack

FYI, here's the current patch that makes my config in http://hg.mozilla.org/users/kairo_kairo.at/bbconf-sm2/ (including prototyped CC* classes in ccfactory.py) pass the tag, source, *_build, and *_repack steps.
updates and the various verification steps might need more work based on this, and I still need to find out what to do about SeaMonkey probably not wanting prettynames.
(Assignee)

Comment 2

9 years ago
Created attachment 383840 [details] [diff] [review]
main patch, v1: pass all worker steps

With this patch, I get all the worker steps (tag, source, *_build, *_repack, updates) to pass. I didn't test the verify steps yet (L10n verify doesn't make sense without having locales for the previous release), but I think we should drive this part in. A major part of this is actually in CC*Factory classes, so asking both Ben and gozer for reviews.
Attachment #383450 - Attachment is obsolete: true
Attachment #383840 - Flags: review?(bhearsum)
(Assignee)

Updated

9 years ago
Attachment #383840 - Flags: review?(gozer)
(Assignee)

Comment 3

9 years ago
Created attachment 383993 [details] [diff] [review]
main patch, v1.1: unbitrot for recent changes

Sorry to post a new patch so fast but I didn't have it based on newest code last night. This is the same patch as the previous, but unbitrotted to apply cleanly to current buildbotcustom (bug 496509 caused conflicts).

While I'm at it, here's a bit of explanation what the patch does.

common release factories:
* Allow a brand name to be passed to release repack ("SeaMonkey" just like e.g. "GranParadiso" are not just the capitalized project name, but used in file names to be pulled from FTP)
* Switch more assuptions of the original source being in a self.brandName directory to self.origSrcDir (see work in bug 494577)
* Allow a different prefix than "GECKO" for the relbranch in tagging (SeaMonkey/Thunderbird use "COMM" there)
* Don't re-set the relbranchName when an override is used, as else the override will be used for all repos processed after the current one
* Use the same "generic" (in-PATH) post_upload.py in the source factory as in all other factories
* Allow a mozRepoPath to be passed to the updates factory in case our main repository is no Mozilla one (patcher needs a Mozilla repo as HGROOT to build the tools)

CC*Factory:
* Make it possible to pass explicit locations for all repositories client.py can pull from
* Also include a flag that makes it skip any repository that isn't passed (so Thunderbird can get rid of checking out or shipping sources for inspector/venkman/chatzilla)
* Make it possible to update all sources to a specific tag, passed as buildRevision (same named tag must exist on all checked out repositories)
Attachment #383840 - Attachment is obsolete: true
Attachment #383993 - Flags: review?(gozer)
Attachment #383993 - Flags: review?(bhearsum)
Attachment #383840 - Flags: review?(gozer)
Attachment #383840 - Flags: review?(bhearsum)
(Assignee)

Comment 4

9 years ago
Oh, one more note: You need to have the patches for bug 494577 and bug 496368 applied to make this patch apply as well.
(Assignee)

Updated

9 years ago
Depends on: 494577, 496368
(Assignee)

Updated

9 years ago
Depends on: 499594
(Assignee)

Comment 5

9 years ago
Created attachment 384321 [details] [diff] [review]
main patch, v2: add brandName to updates factory

I realized that in the end we actually do need the correct brandName in the updates after all, so I needed to add it to the factory and the tools it calls (need bug 499594 patches applied to use it).

So, this one change to the list above in in the new patch:

common release factories:
* Support a brandName param and hand it to tools that need to care about it.
Attachment #383993 - Attachment is obsolete: true
Attachment #384321 - Flags: review?(gozer)
Attachment #384321 - Flags: review?(bhearsum)
Attachment #383993 - Flags: review?(gozer)
Attachment #383993 - Flags: review?(bhearsum)
(Assignee)

Comment 6

9 years ago
As a note, the configurations I used for SeaMonkey release testing, including the CC* factories prototyped in ccfactory.py, are now moved to the production repo in http://hg.mozilla.org/build/buildbot-configs/file/tip/seamonkey/
(Assignee)

Comment 7

9 years ago
Created attachment 387043 [details] [diff] [review]
main patch, v2.1: unbitrot patch context

unbitrot the v2 patch, cleanly applies to buildbotcustom rev. 2c8a4ab0c551 on top
of latest bug 494577 and bug 496368 patches (only cited context in the patch was bitrotted, actually).
Attachment #384321 - Attachment is obsolete: true
Attachment #387043 - Flags: review?(gozer)
Attachment #387043 - Flags: review?(bhearsum)
Attachment #384321 - Flags: review?(gozer)
Attachment #384321 - Flags: review?(bhearsum)
I did a mock release with this patch today and it went great. I'll do an actual review tomorrow.
Comment on attachment 387043 [details] [diff] [review]
main patch, v2.1: unbitrot patch context

> class ReleaseTaggingFactory(ReleaseFactory):
>     def __init__(self, repositories, productName, appName, version, appVersion,
>                  milestone, baseTag, buildNumber, hgUsername, hgSshKey=None,
>-                 buildSpace=1.5, **kwargs):
>+                 relbranchPrefix=None, buildSpace=1.5, **kwargs):

I'd prefer this to default to 'GECKO' and avoid setting a default inside this method.

>-        relbranchName = 'GECKO%s_%s_RELBRANCH' % (
>-          milestone.replace('.', ''), datetime.now().strftime('%Y%m%d'))
>+        if not relbranchPrefix:
>+            relbranchPrefix = 'GECKO'
>+        relbranchName = '%s%s_%s_RELBRANCH' % (
>+          relbranchPrefix, milestone.replace('.', ''),
>+          datetime.now().strftime('%Y%m%d'))
> 


>+            # use repo-specific variable so that a changed name doesn't
>+            # propagate to later repos without an override
>             relbranchOverride = False
>             if repositories[repoPath]['relbranchOverride']:
>                 relbranchOverride = True
>-                relbranchName = repositories[repoPath]['relbranchOverride']
>+                repoRelbranchName = repositories[repoPath]['relbranchOverride']
>+            else:
>+                repoRelbranchName = relbranchName
>

I'm not sure I understand the need for this. Are you using different relbranch names for different repositories?

>+           mozRepoPath: The path for the Mozilla repo to hand patcher as the
>+                        HGROOT (if omitted, the default repoPath is used).
>+                        Apps not rooted in the Mozilla repo need this.

Why? Patcher only needs the update packaging tools, which are in the mozilla repository...

Everything else in this patch looks fine, holding off on +/- until I hear back, though.
(Assignee)

Comment 10

9 years ago
(In reply to comment #9)
> (From update of attachment 387043 [details] [diff] [review])
> > class ReleaseTaggingFactory(ReleaseFactory):
> >     def __init__(self, repositories, productName, appName, version, appVersion,
> >                  milestone, baseTag, buildNumber, hgUsername, hgSshKey=None,
> >-                 buildSpace=1.5, **kwargs):
> >+                 relbranchPrefix=None, buildSpace=1.5, **kwargs):
> 
> I'd prefer this to default to 'GECKO' and avoid setting a default inside this
> method.

OK, no problem.

> >+            # use repo-specific variable so that a changed name doesn't
> >+            # propagate to later repos without an override
> >             relbranchOverride = False
> >             if repositories[repoPath]['relbranchOverride']:
> >                 relbranchOverride = True
> >-                relbranchName = repositories[repoPath]['relbranchOverride']
> >+                repoRelbranchName = repositories[repoPath]['relbranchOverride']
> >+            else:
> >+                repoRelbranchName = relbranchName
> >
> 
> I'm not sure I understand the need for this. Are you using different relbranch
> names for different repositories?

Yes, we do, as we want to ride on a given GECKO_* relbranch for mozilla but use a COMM_* relbranch for comm-central. But actually, the factory before this patch has the problem that when any repo specifies a relbranchOverride, all repos being processed after it happen to use the same override as well, even though none was specified for them.

> >+           mozRepoPath: The path for the Mozilla repo to hand patcher as the
> >+                        HGROOT (if omitted, the default repoPath is used).
> >+                        Apps not rooted in the Mozilla repo need this.
> 
> Why? Patcher only needs the update packaging tools, which are in the mozilla
> repository...

Exactly. And without the patch we're handing it the main repo we build from (repoPath), which happens to be the comm-central one for SeaMonkey and Thunderbird - but as you state correctly, we need to hand it the mozilla repo instead.
Attachment #387043 - Flags: review?(bhearsum) → review+
Comment on attachment 387043 [details] [diff] [review]
main patch, v2.1: unbitrot patch context

r=bhearsum, given KaiRo's last comment.
Comment on attachment 387043 [details] [diff] [review]
main patch, v2.1: unbitrot patch context

Nothing more to add to the existing comments. I've applied this to my testing setup, and it came up without problems. Looks good to me.

And, KaiRo, you've been running the SeaMonkey buildbot master with these patch applied locally already, correct?
Attachment #387043 - Flags: review?(gozer) → review+
Attachment #387043 - Flags: checked‑in+
Comment on attachment 387043 [details] [diff] [review]
main patch, v2.1: unbitrot patch context

changeset:   365:b22d7105b0a5
(Assignee)

Comment 14

9 years ago
(In reply to comment #12)
> And, KaiRo, you've been running the SeaMonkey buildbot master with these patch
> applied locally already, correct?

Right, I've been running it for a while now with that stack of patches that landed today, and with the configs (and additional CC* release factories) linked to in comment #6.
(Assignee)

Comment 15

9 years ago
Thanks for checking in, marking fixed as the patch seems to stick.
Status: ASSIGNED → RESOLVED
Last Resolved: 9 years ago
Resolution: --- → FIXED
(Assignee)

Updated

9 years ago
Blocks: 508243
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.