1236954 - Add standalone builder for partner repacks

Status: RESOLVED FIXED

(Release Engineering :: Release Requests, defect)

Description

[Chris Cooper [:coop] (he/him)]

One regular use-case that isn't covered by current automation is generating one-off partner repacks for a particular partner. This happens frequently enough that we should have a standalone builder.

The new mh script in bug 1236951 can already handle this, we just need the corresponding builder created.

Comment 1

[Chris Cooper [:coop] (he/him)]

Attachment: MozReview Request: Bug 1236954 - add standalone partner repack factory; r?jlund

Review commit: https://reviewboard.mozilla.org/r/32327/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/32327/

Comment 2

[Chris Cooper [:coop] (he/him)]

Attachment: MozReview Request: Bug 1236954 - partner repack script changes to read properties from buildprops.json; r?jlund

Adds --require-buildprops option used by the standalone builder. When this option is
specified, a partner property must be present in the buildprops.json file. This limits
the repack operation to that single partner.

Review commit: https://reviewboard.mozilla.org/r/32339/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/32339/

Comment 3

[Jordan Lund (:jlund)]

Comment on attachment 8711829 [details]
MozReview Request: Bug 1236954 - partner repack script changes to read properties from buildprops.json; r?jlund

https://reviewboard.mozilla.org/r/32339/#review29111

lgtm :)

Comment 4

[Jordan Lund (:jlund)]

Comment on attachment 8711823 [details]
MozReview Request: Bug 1236954 - add standalone partner repack factory; r?jlund

https://reviewboard.mozilla.org/r/32327/#review29113

looks great. just a couple questions about platform.

::: process/release.py:910
(Diff revision 1)
>                          "--platform", platform,

note: in your non standalone repack builders you pass '--platform' in mh script arg and you pass "platform" in buildbot properties. in standalone you only pass via buildbot props.

::: process/release.py:944
(Diff revision 1)
> +        platform = "macosx64"

hmm, was this for debugging or did you only want standalone for osx?

Comment 5

[Chris Cooper [:coop] (he/him)]

https://reviewboard.mozilla.org/r/32327/#review29113

> hmm, was this for debugging or did you only want standalone for osx?

Regardless of target platform, repacks only work on osx, so that's the only platform we need.

Comment 6

[Chris Cooper [:coop] (he/him)]

https://reviewboard.mozilla.org/r/32327/#review29113

> note: in your non standalone repack builders you pass '--platform' in mh script arg and you pass "platform" in buildbot properties. in standalone you only pass via buildbot props.

The standard use-case for this is respinning a single partner because of a late change in their config. In this case, you want all platforms and just provide the partner name in the properties.

Comment 7

[Chris Cooper [:coop] (he/him)]

Comment on attachment 8711823 [details]
MozReview Request: Bug 1236954 - add standalone partner repack factory; r?jlund

https://reviewboard.mozilla.org/r/32327/#review29767

Comment 8

[Chris Cooper [:coop] (he/him)]

https://hg.mozilla.org/build/buildbotcustom/rev/a7bd1793a7abadad4a672fffc384cdba334a7790
Bug 1236954 - add standalone partner repack factory - r=jlund

Comment 9

[Chris Cooper [:coop] (he/him)]

https://reviewboard.mozilla.org/r/32327/#review31763

Comment 10

[Chris Cooper [:coop] (he/him)]

Comment on attachment 8711823 [details]
MozReview Request: Bug 1236954 - add standalone partner repack factory; r?jlund

https://hg.mozilla.org/build/buildbotcustom/rev/a7bd1793a7ab

Comment 11

[Chris Cooper [:coop] (he/him)]

https://hg.mozilla.org/integration/mozilla-inbound/rev/499c7f2126a0aefc1d8509864e0be4523fef7fc1
Bug 1236954 - partner repack script changes to read properties from buildprops.json (NPOTB, DONTBUILD); r=jlund

Comment 12

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/499c7f2126a0

Comment 13

[Release Engineering SlaveAPI Service]

In production: https://hg.mozilla.org/build/buildbotcustom/rev/a7bd1793a7ab

Comment 14

[Jordan Lund (:jlund)]

Comment on attachment 8711823 [details]
MozReview Request: Bug 1236954 - add standalone partner repack factory; r?jlund

backed out for now till we diagnose:

10:22:01 <Callek> https://irccloud.mozilla.com/pastebin/pek5LJz7
10:23:10 <Callek> ahhh its more likely jlund and coop: https://hg.mozilla.org/build/buildbotcustom/rev/a7bd1793a7ab
10:25:35 <jlund> so we are missing config_file somewhere for partnerRepackConfig
10:26:38 <•coop> do we need uplift?
10:27:08 <jlund> oh, maybe config_file is too nested
10:28:21 <jlund> coop: https://dxr.mozilla.org/build-central/source/buildbot-configs/mozilla/release-firefox-mozilla-beta.py#164 vs https://dxr.mozilla.org/build-central/source/buildbot-configs/mozilla/release-fennec-mozilla-release.py#99


backout and merge to default:

remote:   https://hg.mozilla.org/build/buildbotcustom/rev/708d6f514771
remote:   https://hg.mozilla.org/build/buildbotcustom/rev/dd991a2a11b2

merged to production:
remote:   https://hg.mozilla.org/build/buildbotcustom/rev/90c1128b4ce9

Comment 15

[Chris Cooper [:coop] (he/him)]

Backed out due to fennec config errors. New patch incoming.

Comment 16

[Chris Cooper [:coop] (he/him)]

Attachment: [buildbotcustom] Add standalone partner repack factory, v1

The interdiff is adding the following conditional:

+        if releaseConfig['productName'] == 'firefox':

...and then an extra level of indentation for the rest of the patch.

AFAIK, we still don't even do mobile repacks, so we shouldn't need to worry about this for fennec.

Comment 17

[Chris Cooper [:coop] (he/him)]

I uplifted the script change today:

https://hg.mozilla.org/releases/mozilla-aurora/rev/b71f6d0688ea
https://hg.mozilla.org/releases/mozilla-beta/rev/22333de3d1c8
https://hg.mozilla.org/releases/mozilla-release/rev/b0856b132647

Comment 18

[Jordan Lund (:jlund)]

Comment on attachment 8720416 [details] [diff] [review]
[buildbotcustom] Add standalone partner repack factory, v1

we discussed and a further patch is coming

Comment 19

[Chris Cooper [:coop] (he/him)]

Attachment: [buildbotcustom] Add standalone partner repack factory, v2

(In reply to Jordan Lund (:jlund) from comment #14)
> 10:22:01 <Callek> https://irccloud.mozilla.com/pastebin/pek5LJz7
> 10:23:10 <Callek> ahhh its more likely jlund and coop:
> https://hg.mozilla.org/build/buildbotcustom/rev/a7bd1793a7ab
> 10:25:35 <jlund> so we are missing config_file somewhere for
> partnerRepackConfig
> 10:26:38 <•coop> do we need uplift?
> 10:27:08 <jlund> oh, maybe config_file is too nested
> 10:28:21 <jlund> coop:
> https://dxr.mozilla.org/build-central/source/buildbot-configs/mozilla/
> release-firefox-mozilla-beta.py#164 vs
> https://dxr.mozilla.org/build-central/source/buildbot-configs/mozilla/
> release-fennec-mozilla-release.py#99

Same patch as before, except now I check that the product != "fennec" at the start of the block. We setup release fennec partner repacks, but have never actually had any.

Comment 20

[Jordan Lund (:jlund)]

Comment on attachment 8751503 [details] [diff] [review]
[buildbotcustom] Add standalone partner repack factory, v2

Review of attachment 8751503 [details] [diff] [review]:
-----------------------------------------------------------------

i'm probably missing something but I still feel like we should be driving this condition from configs: e.g. https://dxr.mozilla.org/build-central/source/buildbot-configs/mozilla/release-fennec-mozilla-release.py#91

regardless though, everything we add to buildbot I suppose is a stopgap so keying off a restricted product list defined at buildbotcustom factory level, I don't really mind :)

Comment 21

[Chris Cooper [:coop] (he/him)]

(In reply to Jordan Lund (:jlund) from comment #20)
> Comment on attachment 8751503 [details] [diff] [review]
> [buildbotcustom] Add standalone partner repack factory, v2
> 
> Review of attachment 8751503 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> i'm probably missing something but I still feel like we should be driving
> this condition from configs: e.g.
> https://dxr.mozilla.org/build-central/source/buildbot-configs/mozilla/
> release-fennec-mozilla-release.py#91

Oh, we do. All the partner factories (including this one) are wrapped in a doPartnerRepacks() check at the outset. They aren't even created if that's set to False.

Comment 22

[Chris Cooper [:coop] (he/him)]

https://hg.mozilla.org/build/buildbotcustom/rev/fd4a361e9d388db5f4d0881e4d92fc9e9a7d235c
Bug 1236954 - Add standalone partner repack factory - r=jlund

Comment 23

[Chris AtLee [:catlee]]

This is breaking travis:
https://travis-ci.org/mozilla/build-buildbot-configs/builds/130016034

Comment 24

[Chris AtLee [:catlee]]

I backed this out:
https://hg.mozilla.org/build/buildbotcustom/rev/31fcb09c4488
https://hg.mozilla.org/build/buildbotcustom/rev/21165e565d41

Comment 25

[Chris Cooper [:coop] (he/him)]

(In reply to Chris AtLee [:catlee] from comment #23)
> This is breaking travis:
> https://travis-ci.org/mozilla/build-buildbot-configs/builds/130016034

I'm confused. That link shows a passing build.

Comment 26

[Chris AtLee [:catlee]]

gah, I re-ran them in travis after I backed the code out, and now can't find the previous results.

it was generating this error:
https://bugzilla.mozilla.org/show_bug.cgi?id=1272467#c2

Comment 27

[Chris Cooper [:coop] (he/him)]

https://hg.mozilla.org/build/buildbotcustom/rev/546df4a987231618e8ac3a09af8d75fbbe1eb59e
Bug 1236954 - Add standalone partner repack factory - r=jlund

Comment 28

[Release Engineering SlaveAPI Service]

In production: https://hg.mozilla.org/build/buildbotcustom/rev/546df4a98723