1210539 - Add "updates" builder to release promotion task

Status: RESOLVED FIXED

(Release Engineering :: Release Automation, defect)

Description

[Rail Aliiev [:rail]]

Probably using BBB for now, because we need credentials to push to HG.

Comment 1

[Rail Aliiev [:rail]]

hijack!

Comment 2

[Rail Aliiev [:rail]]

Attachment: updates_release_promotion-gecko-dev.diff

Just wanted to make sure that I'm not in a wrong direction. :)

The idea is to use the same (or almost) tools to bump configs, tag (still need this for update verification) and push the top blob info to balrog. Also avoid 2 builders for RC that sometimes give us troubles racing each other.

Instead of using rule IDs I use aliases (I hope it works without any changes).

Comment 3

[Rail Aliiev [:rail]]

Attachment: updates_release_promotion-tools.diff

* ignore deletes (esr31)
* to simplify things I imported Config::General under lib/perl. I hope it dies some day. :)
* added 2 scripts (*-rp.py). They are called from the MH script and don't require any release configs
* haven't tested the balrog script yet

Comment 4

[Rail Aliiev [:rail]]

Attachment: update_release_promotion-buildbotcustom.diff

adds a BBB builder

Comment 5

[Rail Aliiev [:rail]]

Attachment: update_release_promotion-buildbot-configs.diff

Comment 6

[Justin Wood (:Callek)]

Comment on attachment 8717498 [details] [diff] [review]
updates_release_promotion-gecko-dev.diff

Review of attachment 8717498 [details] [diff] [review]:
-----------------------------------------------------------------

::: testing/mozharness/configs/releases/updates_date.py
@@ +32,5 @@
> +                "linux":  "mozDate-firefox-linux.cfg",
> +                "linux64":  "mozDate-firefox-linux64.cfg",
> +                "macosx64": "mozDate-firefox-mac64.cfg",
> +                "win32":  "mozDate-firefox-win32.cfg",
> +                "win64":  "mozDate-firefox-win64.cfg",

nope, wrong format.

date-firefox-... and note that mac64 is wrong its macosx64 (this is now programtically deterministic)

Comment 7

[Rail Aliiev [:rail]]

(In reply to Justin Wood (:Callek) from comment #6)
> nope, wrong format.
> 
> date-firefox-... and note that mac64 is wrong its macosx64 (this is now
> programtically deterministic)

Ah, thanks for the comment. I'll fix them and remove verify_configs from configs completely.

Comment 8

[[e-mail] (:bhearsum)]

Comment on attachment 8717498 [details] [diff] [review]
updates_release_promotion-gecko-dev.diff

Review of attachment 8717498 [details] [diff] [review]:
-----------------------------------------------------------------

(In reply to Rail Aliiev [:rail] from comment #2)
> Instead of using rule IDs I use aliases (I hope it works without any
> changes).

It should, but you probably need to add alias' to more rules still. There's only one alias active in production!

I'm a bit unsure about putting config bumping and balrog submission into the same script...they feel like very different things to me. I know patcher configs are going away at some point, but update verify will remain. I'm not going to r- for this, but I'm not going to be shocked if we have separate these things out later.

::: testing/mozharness/configs/releases/updates_date.py
@@ +20,5 @@
> +    "archive_prefix": "https://s3-us-west-2.amazonaws.com/mozilla-releng-beet-mover-dev",
> +    "previous_archive_prefix": "https://archive.mozilla.org",
> +    "download_domain": "download.mozilla.org",
> +    "balrog_url": "https://aus5-dev.allizom.org",
> +    "balrog_username": "ffxbld",

This should be stage-ffxbld for dev. You may also want to use https://github.com/mozilla/gecko-dev/blob/master/testing/mozharness/configs/balrog/staging.py as an additional config than duplicating here.

@@ +36,5 @@
> +                "win64":  "mozDate-firefox-win64.cfg",
> +            },
> +            "update_verify_channel": "date-localtest",
> +            "mar_channel_ids": [
> +                # "firefox-date", "firefox-mozilla-beta",

Any particular reason this is commented out here?

@@ +39,5 @@
> +            "mar_channel_ids": [
> +                # "firefox-date", "firefox-mozilla-beta",
> +            ],
> +            # TODO: find better naming
> +            "blob_channels": ["date", "date-localtest", "date-cdntest"],

How about "channel_names" for this?

@@ +40,5 @@
> +                # "firefox-date", "firefox-mozilla-beta",
> +            ],
> +            # TODO: find better naming
> +            "blob_channels": ["date", "date-localtest", "date-cdntest"],
> +            "test_rules": ["firefox-date-cdntest", "firefox-date-localtest"],

And "rules_to_update" or "test_channel_rules" for this?

::: testing/mozharness/mozharness/mozilla/merge.py
@@ +111,5 @@
>                  ignore_no_changes=self.config.get("ignore_no_changes", False)
>              )
>          self.info("Now verify |hg out| and |hg out --patch| if you're paranoid, and --push")
>  
> +    def hg_tag(self, cwd, tags, user=None, message=None, revision=None,

It's really weird for the updates script to inherit from GeckoMigrationMixin. Can you move this to a more independent location?

::: testing/mozharness/scripts/release/updates.py
@@ +1,1 @@
> +#!/usr/bin/env python

Please find a more descriptive name for this script.

@@ +36,5 @@
> +        # dot. Compose a structure that will be sorted by StrictVersion and
> +        # return untouched version
> +        composed = sorted([(v, StrictVersion(v)) for v in partial_versions if
> +                           v != version], key=lambda x: x[1], reverse=True)
> +        return composed[0][0]

This looks like something that should live in a library, and have some tests (it's a pure function after all!).

Comment 9

[[e-mail] (:bhearsum)]

Comment on attachment 8717499 [details] [diff] [review]
updates_release_promotion-tools.diff

Review of attachment 8717499 [details] [diff] [review]:
-----------------------------------------------------------------

I'm not a huge fan of duplicating the two scripts, but I can live with it if you find a better name for them. These will end up being the only ones that exist after we fully switch to release promotion, so please choose a name that makes sense for that world.

::: release/update-verify-bump.pl
@@ -1,1 @@
> -#!/usr/bin/perl -w

Yay, script death!

::: scripts/updates/balrog-release-pusher-rp.py
@@ +25,5 @@
> +    parser.add_argument(
> +        "--username", required=True, help="Username in the credentials file")
> +    parser.add_argument(
> +        "--channel", dest="channels", action="append", required=True,
> +        help="Release blob channels")

This is not a very useful help message. How about something like "Channels to set-up fileUrls for"?

@@ +28,5 @@
> +        "--channel", dest="channels", action="append", required=True,
> +        help="Release blob channels")
> +    parser.add_argument(
> +        "--test-rule", dest="test_rules", action="append", required=True,
> +        help="Test channels that will be automatically enabled")

Please change the name of this to match whatever is used in the mozharness config.

Comment 10

[Rail Aliiev [:rail]]

Comment on attachment 8717514 [details] [diff] [review]
update_release_promotion-buildbot-configs.diff

https://hg.mozilla.org/build/buildbot-configs/rev/af83ded3aee2

Comment 11

[Rail Aliiev [:rail]]

Comment on attachment 8717513 [details] [diff] [review]
update_release_promotion-buildbotcustom.diff

https://hg.mozilla.org/build/buildbotcustom/rev/48181923d5ae

Comment 12

[Kim Moir [:kmoir] ET]

in production

Comment 13

[Rail Aliiev [:rail]]

Attachment: updates_release_promotion-tools.diff

Almost the same thing (the same approach), but working :)
http://buildbot-master72.bb.releng.usw2.mozilla.com:8001/builders/release-date-firefox_updates/builds/1

Comment 14

[Rail Aliiev [:rail]]

Attachment: updates_release_promotion-gecko-dev-3.diff

works on date!

Comment 15

[Rail Aliiev [:rail]]

Attachment: releasetasks patch

Comment 16

[[e-mail] (:bhearsum)]

Comment on attachment 8721083 [details] [diff] [review]
updates_release_promotion-tools.diff

Review of attachment 8721083 [details] [diff] [review]:
-----------------------------------------------------------------

Much better names/locations, thank you!

Comment 17

[Rail Aliiev [:rail]]

Comment on attachment 8721083 [details] [diff] [review]
updates_release_promotion-tools.diff

https://hg.mozilla.org/build/tools/rev/8c70fceff603

Comment 18

[Jordan Lund (:jlund)]

Comment on attachment 8721085 [details] [diff] [review]
updates_release_promotion-gecko-dev-3.diff

Review of attachment 8721085 [details] [diff] [review]:
-----------------------------------------------------------------

::: testing/mozharness/scripts/release/updates.py
@@ +65,5 @@
> +                'tag',
> +                'push',
> +                'submit-to-balrog',
> +            ],
> +            default_actions=[

fyi - if default_actions == all_actions, it's not needed. mozharness will assume that if default_actions is not explicitly defined.

@@ +154,5 @@
> +            repos=self.query_repos())
> +
> +    def download_shipped_locales(self):
> +        dirs = self.query_abs_dirs()
> +        self.mkdir_p(dirs["abs_work_dir"])

is this needed?

Comment 19

[Rail Aliiev [:rail]]

(In reply to Jordan Lund (:jlund) from comment #18) 
> @@ +154,5 @@
> > +            repos=self.query_repos())
> > +
> > +    def download_shipped_locales(self):
> > +        dirs = self.query_abs_dirs()
> > +        self.mkdir_p(dirs["abs_work_dir"])
> 
> is this needed?

I hit some issues without that, probably when I ran it with a lot of --no-something args.

Comment 20

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/93654b236653

Comment 21

[Rail Aliiev [:rail]]

Comment on attachment 8721086 [details] [review]
releasetasks patch

r+ed in the PR

Comment 22

[Phil Ringnalda (:philor)]

https://hg.mozilla.org/mozilla-central/rev/93654b236653

Comment 23

[Rail Aliiev [:rail]]

Attachment: typos.diff

Typos... Pushed to date: https://hg.mozilla.org/projects/date/rev/d5ea8d4fd43e

Comment 24

[Rail Aliiev [:rail]]

Attachment: typos2.diff

moar fixes

Comment 25

[Jordan Lund (:jlund)]

Comment on attachment 8724501 [details] [diff] [review]
typos.diff

Review of attachment 8724501 [details] [diff] [review]:
-----------------------------------------------------------------

this patch is "moBeta" :)

Comment 26

[Rail Aliiev [:rail]]

Comment on attachment 8724501 [details] [diff] [review]
typos.diff

https://hg.mozilla.org/integration/mozilla-inbound/rev/fc271aac7545

Comment 27

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/8f6efc17373b

Comment 28

[Rail Aliiev [:rail]]

Comment on attachment 8724582 [details] [diff] [review]
typos2.diff

https://hg.mozilla.org/integration/mozilla-inbound/rev/8f6efc17373b

Comment 29

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/fc271aac7545
https://hg.mozilla.org/mozilla-central/rev/8f6efc17373b