1118798 - Downstream job to create partner repacks

Status: RESOLVED FIXED

(Release Engineering :: Release Automation: Other, defect)

Description

[Chris AtLee [:catlee]]

Port the existing partner repack script to the new system. Check the existing inputs to script and minimize, e.g. pass en-US build location in, and determine version if required. (aka remove dependency on information from release config).

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [partner-repacks] Adjust the repacking script

The key points are
* add a new mode triggered by --use-tinderbox-builds, needs arg --revision set
* query taskcluster for the en-US builds using the revision, and download from there since we have no candidates dir to use
* ignore l10n for now, that'll get done later
* handle different naming styles for nightly/release
* some minor decrufting

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [buildbot-configs] Moar variables

Carries over a couple of partner variables from the release config, and bumps the gecko version for the recent merge.

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [buildbotcustom] Schedule partner builds, flay the factory

* schedule builds, based on the code in process/release.py but not passing a version or buildNumber; new releasePromotion parameter
* ride through PartnerRepackFactory with the four horsemen, dropping if/else around the place
* diff -w should help see the new code vs indentation

The uploads end up in platform specific directories, eg date-linux-partner_repacks, but it may be better to upload into one. They're also overwriting on each run, but I'm deferring something better until we have a better idea on where we're uploading in S3.

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [buildbotcustom] Schedule partner builds, flay the factory, v2

Same as comment #3 plus changing the branch for the sendchange to be foo-release-promotion.

Comment 5

[[e-mail] (:bhearsum)]

(In reply to Nick Thomas [:nthomas] from comment #3)
> The uploads end up in platform specific directories, eg
> date-linux-partner_repacks, but it may be better to upload into one. They're
> also overwriting on each run, but I'm deferring something better until we
> have a better idea on where we're uploading in S3.

Sounds fine to me. Private partner repacks will probably affect this too (which may be the same or at least lumped in with the s3 transition).

Comment 6

[[e-mail] (:bhearsum)]

Comment on attachment 8586130 [details] [diff] [review]
[partner-repacks] Adjust the repacking script

Review of attachment 8586130 [details] [diff] [review]:
-----------------------------------------------------------------

Other than the few comments on this patch, everything here looks great! Much easier than I imagined it might be.

::: scripts/partner-repacks.py
@@ +24,5 @@
>  HGROOT = 'https://hg.mozilla.org'
>  REPO = 'releases/mozilla-release'
>  DEFAULT_OUTPUT_DIR = 'partner-repacks/%(partner)s/%(platform)s/%(locale)s'
> +TASKCLUSTER_INDEX='https://index.taskcluster.net/v1/task/buildbot.revisions.%(revision)s.%(base_repo)s.%(tc_platform)s'
> +TASKCLUSTER_ARTIFACT='https://queue.taskcluster.net/v1/task/%(taskId)s/artifacts/public/build/%(filename)s'

Are we going to need netflows for these, or are they already open?

@@ +261,5 @@
> +
> +def getTaskId(platform):
> +    tc_platform = getTaskClusterPlatform(platform)
> +    revision = options.revision
> +    base_repo = path.basename(options.repo)

Not the end of the world, but it's probably better to pass these in rather than relying on options being in the global scope.

@@ +681,5 @@
>      )
>      parser.add_option(
> +        "--use-tinderbox-builds", action="store_true", dest="use_tinderbox_builds",
> +        default=False,
> +        help="Use tinderbox builds (ie release promotion)"

I wonder if --use-ci-builds might be a better name. We're not actually pulling from the "tinderbox-builds" directory, so this name is a bit weird to me...

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [partner-repacks] Adjust the repacking script w/ review comments

(In reply to Ben Hearsum [:bhearsum] from comment #6)
> > +TASKCLUSTER_ARTIFACT='https://queue.taskcluster.net/v1/task/%(taskId)s/artifacts/public/build/%(filename)s'
> Are we going to need netflows for these, or are they already open?> 

They're open already thankfully, and no auth needed - I tested with a mac slave attached to my staging master while developing this.

I've incorporated the other feedback here, carrying r+. Will land after the chemspill is done.

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8586131 [details] [diff] [review]
[buildbot-configs] Moar variables

Landed with partners_repo_path set to production repo:
https://hg.mozilla.org/build/buildbot-configs/rev/04ab7b9bb920

Comment 9

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8586364 [details] [diff] [review]
[buildbotcustom] Schedule partner builds, flay the factory, v2

https://hg.mozilla.org/build/buildbotcustom/rev/a4247fb4c834

Comment 10

[Release Engineering SlaveAPI Service]

In production: https://hg.mozilla.org/build/buildbot-configs/rev/04ab7b9bb920

Comment 11

[Release Engineering SlaveAPI Service]

In production: https://hg.mozilla.org/build/buildbotcustom/rev/a4247fb4c834

Comment 12

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8587115 [details] [diff] [review]
[partner-repacks] Adjust the repacking script w/ review comments

I found one more tinderbox-builds to fix up, so the landing is slightly different from what's attached here.

https://hg.mozilla.org/build/partner-repacks/rev/c72118fff0e6