748800 - replace TuxedoEntrySubmitterFactory with a mozharness script

Status: RESOLVED FIXED

(Release Engineering :: Applications: MozharnessCore, defect, P3)

Description

[[e-mail] (:bhearsum)]

Needed for reconfigless releases.

Comment 1

[Rail Aliiev [:rail]]

I'm going to look at this this week since I'm on release duty

Comment 2

[Rail Aliiev [:rail]]

Attachment: tuxedo-mozharness.diff

* Once the blocker is fixed I can test it in staging. The generated URLs look OK to me. 
* SSL-only product API call added.
* WCPGW? :)

Comment 3

[Rail Aliiev [:rail]]

Attachment: tuxedo-buildbotcustom.diff

Comment 4

[Rail Aliiev [:rail]]

Attachment: tuxedo-buildbot-configs.diff

Comment 5

[Rail Aliiev [:rail]]

Attachment: tuxedo-mozharness-1.diff

* removed a TODO comment which is not a TODO anymore :)

Comment 6

[Rail Aliiev [:rail]]

Attachment: tuxedo-buildbot-configs-1.diff

* no solaris for betas
* no euballot for betas

Comment 7

[Rail Aliiev [:rail]]

Attachment: tuxedo-buildbotcustom-1.diff

Use bouncer_add_euballot to avoid submitting euballot for betas

Comment 8

[Rail Aliiev [:rail]]

Attachment: tuxedo-buildbotcustom-2.diff

oopsie, syntax fix

Comment 9

[Rail Aliiev [:rail]]

Attachment: tuxedo-buildbot-configs-2.diff

Updated *.py files as well to make test-masters.sh happier.

Comment 10

[Rail Aliiev [:rail]]

Attachment: tuxedo-buildbotcustom-3.diff

forgot to update the second entry of bouncer_add_euballot

Comment 11

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8380708 [details] [diff] [review]
tuxedo-mozharness-1.diff

Review of attachment 8380708 [details] [diff] [review]:
-----------------------------------------------------------------

Looks good, r+ with some tweaks. I didn't check the configs in Super Pedantic Man mode, so I suggest we compare locations against a prior release (either in staging before this lands, or in prod after) to make sure no little details changed in the transfer.

::: scripts/bouncer_submitter.py
@@ +47,5 @@
> +            "help": "Specify buildbot platforms",
> +        }],
> +        [["--bouncer-api-prefix"], {
> +            "dest": "bouncer-api-prefix",
> +            "help": "Specify Bouncer API prefix",

Some of these help messages just repeat the argument rather than adding any extra information. Could you make them more human friendly ? eg 'URL of bouncer admin host"

@@ +72,5 @@
> +                                'download-shipped-locales',
> +                                'add-product',
> +                                'add-ssl-only-product',
> +                                'add-complete-upadtes',
> +                                'add-partial-upadtes',

s/upadtes/updates/g.

@@ +113,5 @@
> +        if not self.config.get("shipped-locales-url"):
> +            self.info("Not downloading shipped-locales")
> +            return
> +        if self.config["no-locales"]:
> +            self.info("No need downloading shipped-locales")

nit, s/downloading/to download/

@@ +198,5 @@
> +        }
> +        for p in platforms:
> +            c = self.config["platform-config"][p]
> +            bouncer_platform = c["bouncer-platform"]
> +            path = c[path_type] % replace_dict

Might get KeyError's here with incomplete configs.

@@ +209,5 @@
> +        self.add_locations(product_name, "installer")
> +
> +    def add_ssl_only_product(self):
> +        if not self.config.get("add-ssl-only-product"):
> +            self.info("SSL-only product disaled. Skipping...")

nit, typo in disabled

Comment 12

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8380951 [details] [diff] [review]
tuxedo-buildbot-configs-2.diff

Review of attachment 8380951 [details] [diff] [review]:
-----------------------------------------------------------------

r+ with tweaks on landing.

::: mozilla/release-thunderbird-comm-beta.py
@@ +136,5 @@
>  releaseConfig['tuxedoServerUrl']     = 'https://bounceradmin.mozilla.com/api/'
>  releaseConfig['extraBouncerPlatforms'] = ('solaris-sparc', 'solaris-i386',
>                                            'opensolaris-sparc',
>                                            'opensolaris-i386')
> +releaseConfig['bouncer_submitter_config'] = 'releases/bouncer_firefox.py'

Wrong config file - s/firefox/thunderbird/. There are several Thunderbird configs that need this fix.

::: mozilla/release-thunderbird-comm-beta.py.template
@@ -135,5 @@
> -releaseConfig['tuxedoConfig']        = 'firefox-tuxedo.ini'
> -releaseConfig['tuxedoServerUrl']     = 'https://bounceradmin.mozilla.com/api/'
> -releaseConfig['extraBouncerPlatforms'] = ('solaris-sparc', 'solaris-i386',
> -                                          'opensolaris-sparc',
> -                                          'opensolaris-i386')

Apparently the Solaris elves out there are still doing Thunderbird betas, just not Firefox ones, so should add extraBouncerPlatforms back for them.

::: mozilla/staging_release-firefox-mozilla-beta.py.template
@@ +141,3 @@
>  releaseConfig['extraBouncerPlatforms'] = ('solaris-sparc', 'solaris-i386',
>                                            'opensolaris-sparc',
>                                            'opensolaris-i386')

Please remove extraBouncerPlatforms so the staging config is closer to prod.

::: mozilla/staging_release-firefox-mozilla-release.py.template
@@ +142,4 @@
>  releaseConfig['extraBouncerPlatforms'] = ('solaris-sparc', 'solaris-i386',
>                                            'opensolaris-sparc',
>                                            'opensolaris-i386')
> +releaseConfig['bouncer_submitter_config'] = 'releases/bouncer_firefox.py'

Missing the new EU ballot pref here and in non-template file.

Comment 13

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8380959 [details] [diff] [review]
tuxedo-buildbotcustom-3.diff

Review of attachment 8380959 [details] [diff] [review]:
-----------------------------------------------------------------

r+ with tweak.

::: process/release.py
@@ +1610,5 @@
> +                          releaseConfig['tuxedoServerUrl'],
> +                          "--platform", "win32-EUBallot",
> +                          "--no-add-ssl-only-product",
> +                          "--no-add-complete-upadtes",
> +                          "--no-add-partial-upadtes",

s/upadtes/updates/g to match mozharness fix.

Comment 14

[Rail Aliiev [:rail]]

Attachment: tuxedo-mozharness-2.diff

* addressed comments
* the KeyError issue is made more obvious by checking and calling self.fatal(). I didn't want to make the initial verification code (_pre_config_lock) too complex.
* carry on r=nthomas

I don't feel like landing this tonight. If I won't be able to test this in staging I'll land this before the next beta.

Comment 15

[Rail Aliiev [:rail]]

Attachment: tuxedo-buildbotcustom-4.diff

Comment 16

[Rail Aliiev [:rail]]

Attachment: tuxedo-buildbot-configs-3.diff

Comment 17

[Rail Aliiev [:rail]]

Comment on attachment 8381113 [details] [diff] [review]
tuxedo-mozharness-2.diff

https://hg.mozilla.org/build/mozharness/rev/ab3e42e05a4f

with one change (s/browser/mail/) in the Thunderbird config

Comment 18

[Rail Aliiev [:rail]]

Comment on attachment 8381115 [details] [diff] [review]
tuxedo-buildbotcustom-4.diff

https://hg.mozilla.org/build/buildbotcustom/rev/6bc616bed2a2

Comment 19

[Rail Aliiev [:rail]]

Comment on attachment 8381116 [details] [diff] [review]
tuxedo-buildbot-configs-3.diff

https://hg.mozilla.org/build/buildbot-configs/rev/c5fdf825ac97

Comment 20

[Rail Aliiev [:rail]]

I tested the patches in staging with disabled self.fatal(). Everything looked fine to me (except 401s). I'm going to watch this on Thursday.

Comment 21

[Rail Aliiev [:rail]]

:Callek, you may want to create a SeaMonkey config, so we can retire tuxedo-add.py soon.

Comment 22

[John Hopkins (:jhopkins)]

In production.

Comment 23

[Rail Aliiev [:rail]]

Attachment: [custom] fix --revision

Comment 24

[Rail Aliiev [:rail]]

Comment on attachment 8383433 [details] [diff] [review]
[custom] fix --revision

remote:   https://hg.mozilla.org/build/buildbotcustom/rev/3077ae98d429
remote:   https://hg.mozilla.org/build/buildbotcustom/rev/08f73658eb00

Comment 25

[Rail Aliiev [:rail]]

Attachment: [mozharness] fail early

A small tweak to make the script exit if it fails to download shipped-locales and avoid running the next steps.

Comment 26

[Rail Aliiev [:rail]]

Comment on attachment 8383447 [details] [diff] [review]
[mozharness] fail early

remote:   https://hg.mozilla.org/build/mozharness/rev/440e5eaa8837
remote:   https://hg.mozilla.org/build/mozharness/rev/80b65eebf449

Comment 27

[Jordan Lund (:jlund)]

buildbotcustom patch is merged and live in production! :)

Comment 28

[Jordan Lund (:jlund)]

this might have been already in production as a hot fix. please ignore if that is the case