497156 - Add partner repacks to release automation for 3.5

Status: RESOLVED FIXED

(Release Engineering :: Release Requests, defect, P2)

Description

[Chris Cooper [:coop] (he/him)]

Similar to bug 483233, we need to add some partner repack build steps to the release automation process for 3.5.

Comment 1

[Chris Cooper [:coop] (he/him)]

Resurrecting my patch into staging today.

Comment 2

[Chris Cooper [:coop] (he/him)]

My automation run crapped out pretty early in the process last night, but for reasons completely unrelated to partner builds. I'll be trying again tonight.

Comment 3

[Chris Cooper [:coop] (he/him)]

We're on the verge of releasing 3.6 so this bug may soon morph into a 3.6 bug if partners make that switch quickly enough. The build steps should remain the same regardless.

In terms of where to upload the automated repacks, after a brief discussion on IRC today, I'm going to upload the partner repacks under the same dir as the other builds for the same release, i.e. firefox/nightly/3.x.x-candidates/buildx/partner-repacks. Keeping all the related builds together makes sense to me (and bhearsum).

My trial runs in staging show that generating the full suite of partner builds now takes slightly longer than one hour, so I don't think we want to block regular signing. As such, we will need to be careful to exclude the partner repacks from the core signing download and SUM creation. In most cases, partner repacks should be done by the time the core signing is done, so repack signing could begin immediately thereafter. We already have instructions for doing partner repack signing separately, so this is maintaining the status quo.

We'll also need to ensure that the partner repacks never get rsync-ed to the releases/ dir and pushed out to mirrors.

Comment 4

[Chris Cooper [:coop] (he/him)]

I tried a bunch of different ways to try to get partner repacks to start as soon as the l10n repacks are all complete. Unfortunately, it's not straightforward.

I can't key directly off the DependentL10n schedulers otherwise I end up with as many partner schedulers as we have locales. The l10n repacks can also happen in any order, i.e. assuming zh-TW is even in the list of locales it may not be the last one created, so I can't use a simple FtpPoller to check that the all the expected localized builds are present.

In the end, I decided to add the partner repack scheduler to the post_signing branch. This means that partner repacks won't start until after signing is done for the core builds. While not ideal, at least they'll be automated. :/

If it's an issue, we can write a custom poller to check for all the expected locales before triggering the partner repacks.

Comment 5

[Chris Cooper [:coop] (he/him)]

Attachment: [checked in] Add command-line switches for staging server and pkg-dmg location

Just a few changes to make running this via automation (pkg-dmg) and in staging (staging server) easier.

Comment 6

[Chris Cooper [:coop] (he/him)]

Attachment: Add PartnerRepackFactory and RepackPartners step

Factory is pretty straightforward. I've opted to upload *all* the partner repacks under the unsigned/ directory.

I created a custom build step to allow us to force single partner repacks via the Force Build form.

Tested successfully in staging.

Comment 7

[Chris Cooper [:coop] (he/him)]

Attachment: [checked in] Add partner repacks to release automation, enabled only for 1.9.1

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 422250 [details] [diff] [review]
[checked in] Add command-line switches for staging server and pkg-dmg location

>diff --git a/scripts/partner-repacks.py b/scripts/partner-repacks.py

>+ 
>+        if not which(options.pkg_dmg):
>             print "Error: couldn't find the pkg-dmg executable in PATH."
>-            error = True
>+            error = True            

Please remove the stray whitespace in the line before the if block and after True.

Comment 9

[Chris Cooper [:coop] (he/him)]

Comment on attachment 422250 [details] [diff] [review]
[checked in] Add command-line switches for staging server and pkg-dmg location

http://hg.mozilla.org/build/partner-repacks/rev/6fd9a91acc5a

Comment 10

[[e-mail] (:bhearsum)]

Comment on attachment 422252 [details] [diff] [review]
Add PartnerRepackFactory and RepackPartners step

>diff --git a/process/factory.py b/process/factory.py
>+        self.addStep(ShellCommand,
>+            name='clone_partners_repo',
>+            command=['bash', '-c',
>+                     'if [ ! -d %s ]; then hg clone http://%s/%s %s;fi' % \
>+                       (self.partnersRepackDir,
>+                        self.hgHost,
>+                        self.partnersRepoPath,
>+                        self.partnersRepackDir)
>+                    ],

I don't think we should be using an existing clone here. For the first-attempt-of-a-release, if the directory already exists, that means it wasn't clobbered properly, and we definitely don't want to proceed.

In the respin-a-single-partner-case, this works fine for now, since the revision we're using is 'default'. If we ever switch to tags there we could easily run into cases where we're updating to the wrong revision after moving a tag.

I'm torn on this, since the repository is of non-trivial size. What do you think, coop?


>+        self.addStep(ShellCommand,
>+            name='download_pkg-dmg',
>+            command=['bash', '-c',
>+                     'wget http://hg.mozilla.org/%s/raw-file/default/build/package/mac_osx/pkg-dmg' % self.repoPath],
>+            description=['download', 'pkg-dmg'],
>+            workdir='%s/scripts' % self.partnersRepackDir,
>+            haltOnFailure=True            
>+        )

Definitely want to use the _RELEASE tag here, for later reproducibility.


>+class RepackPartners(ShellCommand):
>+    '''This step allows a regular ShellCommand to be optionally extended
>+       based on provided properties. This is useful for tweaking the command
>+       to be run based on, e.g., properties supplied by the user in the 
>+       force builds web interface.
>+    '''

I'm not a huge fan of overriding the command at runtime, but within the constraints of the partner repack script this seems like the only way to make Force Build work.


This is basically ok - need to change the pkg-dmg wget, and I'm curious what you think about my comments on cloning.

Comment 11

[[e-mail] (:bhearsum)]

Comment on attachment 422253 [details] [diff] [review]
[checked in] Add partner repacks to release automation, enabled only for 1.9.1

Looks fine to me.

We need to be careful when we push to mirrors that we do the right thing and --exclude=partner-repacks in that rsync.

Comment 12

[Chris Cooper [:coop] (he/him)]

(In reply to comment #11)
> We need to be careful when we push to mirrors that we do the right thing and
> --exclude=partner-repacks in that rsync.

Yeah, I'll be updating the build notes *before* landing the patch to be sure.

bhearsum: anything wrong with the factory patch still pending review (attachment 422252 [details] [diff] [review])?

Comment 13

[[e-mail] (:bhearsum)]

(In reply to comment #12)
> (In reply to comment #11)
> > We need to be careful when we push to mirrors that we do the right thing and
> > --exclude=partner-repacks in that rsync.
> 
> Yeah, I'll be updating the build notes *before* landing the patch to be sure.
> 
> bhearsum: anything wrong with the factory patch still pending review
> (attachment 422252 [details] [diff] [review])?

Whoops, forgot mark it. It's mostly fine, needs at least one small fix though. See comment #10 for details

Comment 14

[Chris Cooper [:coop] (he/him)]

(In reply to comment #10)
> I'm torn on this, since the repository is of non-trivial size. What do you
> think, coop?

We're not blocking any other parts of the release process on this, so I say blow it away and re-clone every time. I'll fix that up.
 
> Definitely want to use the _RELEASE tag here, for later reproducibility.

Roger that. New patch coming up.

Comment 15

[Chris Cooper [:coop] (he/him)]

Attachment: [checked in] Add PartnerRepackFactory and RepackPartners step, v2

* remove and clone partner-repacks fresh each time
* adds a getReleaseTag function to be used when downloading pkg-dmg. Should we push this function up into the parent class?

Comment 16

[Chris Cooper [:coop] (he/him)]

Comment on attachment 422541 [details] [diff] [review]
[checked in] Add PartnerRepackFactory and RepackPartners step, v2

http://hg.mozilla.org/build/buildbotcustom/rev/7a6775d7c0d0

Comment 17

[Chris Cooper [:coop] (he/him)]

Comment on attachment 422253 [details] [diff] [review]
[checked in] Add partner repacks to release automation, enabled only for 1.9.1

http://hg.mozilla.org/build/buildbot-configs/rev/01c5460116be

Comment 18

[Chris Cooper [:coop] (he/him)]

Ben pushed the changes live this morning.

I'll file a follow-up bug about switching partner repacks to 3.6 rather than 3.5.