Add "updates" builder to release promotion task

RESOLVED FIXED

Status

Release Engineering
Release Automation: Other
RESOLVED FIXED
3 years ago
2 years ago

People

(Reporter: rail, Assigned: rail)

Tracking

unspecified
Dependency tree / graph

Firefox Tracking Flags

(firefox47 fixed)

Details

Attachments

(7 attachments, 2 obsolete attachments)

(Assignee)

Description

3 years ago
Probably using BBB for now, because we need credentials to push to HG.

Updated

3 years ago
Assignee: nobody → jlund
(Assignee)

Comment 1

2 years ago
hijack!
Assignee: jlund → rail
(Assignee)

Comment 2

2 years ago
Created attachment 8717498 [details] [diff] [review]
updates_release_promotion-gecko-dev.diff

Just wanted to make sure that I'm not in a wrong direction. :)

The idea is to use the same (or almost) tools to bump configs, tag (still need this for update verification) and push the top blob info to balrog. Also avoid 2 builders for RC that sometimes give us troubles racing each other.

Instead of using rule IDs I use aliases (I hope it works without any changes).
Attachment #8717498 - Flags: feedback?(bhearsum)
(Assignee)

Comment 3

2 years ago
Created attachment 8717499 [details] [diff] [review]
updates_release_promotion-tools.diff

* ignore deletes (esr31)
* to simplify things I imported Config::General under lib/perl. I hope it dies some day. :)
* added 2 scripts (*-rp.py). They are called from the MH script and don't require any release configs
* haven't tested the balrog script yet
Attachment #8717499 - Flags: feedback?(bhearsum)
(Assignee)

Comment 4

2 years ago
Created attachment 8717513 [details] [diff] [review]
update_release_promotion-buildbotcustom.diff

adds a BBB builder
Attachment #8717513 - Flags: review?(bugspam.Callek)
(Assignee)

Comment 5

2 years ago
Created attachment 8717514 [details] [diff] [review]
update_release_promotion-buildbot-configs.diff
Attachment #8717514 - Flags: review?(bugspam.Callek)
Comment on attachment 8717498 [details] [diff] [review]
updates_release_promotion-gecko-dev.diff

Review of attachment 8717498 [details] [diff] [review]:
-----------------------------------------------------------------

::: testing/mozharness/configs/releases/updates_date.py
@@ +32,5 @@
> +                "linux":  "mozDate-firefox-linux.cfg",
> +                "linux64":  "mozDate-firefox-linux64.cfg",
> +                "macosx64": "mozDate-firefox-mac64.cfg",
> +                "win32":  "mozDate-firefox-win32.cfg",
> +                "win64":  "mozDate-firefox-win64.cfg",

nope, wrong format.

date-firefox-... and note that mac64 is wrong its macosx64 (this is now programtically deterministic)
Attachment #8717498 - Flags: feedback-
(Assignee)

Comment 7

2 years ago
(In reply to Justin Wood (:Callek) from comment #6)
> nope, wrong format.
> 
> date-firefox-... and note that mac64 is wrong its macosx64 (this is now
> programtically deterministic)

Ah, thanks for the comment. I'll fix them and remove verify_configs from configs completely.
Comment on attachment 8717498 [details] [diff] [review]
updates_release_promotion-gecko-dev.diff

Review of attachment 8717498 [details] [diff] [review]:
-----------------------------------------------------------------

(In reply to Rail Aliiev [:rail] from comment #2)
> Instead of using rule IDs I use aliases (I hope it works without any
> changes).

It should, but you probably need to add alias' to more rules still. There's only one alias active in production!

I'm a bit unsure about putting config bumping and balrog submission into the same script...they feel like very different things to me. I know patcher configs are going away at some point, but update verify will remain. I'm not going to r- for this, but I'm not going to be shocked if we have separate these things out later.

::: testing/mozharness/configs/releases/updates_date.py
@@ +20,5 @@
> +    "archive_prefix": "https://s3-us-west-2.amazonaws.com/mozilla-releng-beet-mover-dev",
> +    "previous_archive_prefix": "https://archive.mozilla.org",
> +    "download_domain": "download.mozilla.org",
> +    "balrog_url": "https://aus5-dev.allizom.org",
> +    "balrog_username": "ffxbld",

This should be stage-ffxbld for dev. You may also want to use https://github.com/mozilla/gecko-dev/blob/master/testing/mozharness/configs/balrog/staging.py as an additional config than duplicating here.

@@ +36,5 @@
> +                "win64":  "mozDate-firefox-win64.cfg",
> +            },
> +            "update_verify_channel": "date-localtest",
> +            "mar_channel_ids": [
> +                # "firefox-date", "firefox-mozilla-beta",

Any particular reason this is commented out here?

@@ +39,5 @@
> +            "mar_channel_ids": [
> +                # "firefox-date", "firefox-mozilla-beta",
> +            ],
> +            # TODO: find better naming
> +            "blob_channels": ["date", "date-localtest", "date-cdntest"],

How about "channel_names" for this?

@@ +40,5 @@
> +                # "firefox-date", "firefox-mozilla-beta",
> +            ],
> +            # TODO: find better naming
> +            "blob_channels": ["date", "date-localtest", "date-cdntest"],
> +            "test_rules": ["firefox-date-cdntest", "firefox-date-localtest"],

And "rules_to_update" or "test_channel_rules" for this?

::: testing/mozharness/mozharness/mozilla/merge.py
@@ +111,5 @@
>                  ignore_no_changes=self.config.get("ignore_no_changes", False)
>              )
>          self.info("Now verify |hg out| and |hg out --patch| if you're paranoid, and --push")
>  
> +    def hg_tag(self, cwd, tags, user=None, message=None, revision=None,

It's really weird for the updates script to inherit from GeckoMigrationMixin. Can you move this to a more independent location?

::: testing/mozharness/scripts/release/updates.py
@@ +1,1 @@
> +#!/usr/bin/env python

Please find a more descriptive name for this script.

@@ +36,5 @@
> +        # dot. Compose a structure that will be sorted by StrictVersion and
> +        # return untouched version
> +        composed = sorted([(v, StrictVersion(v)) for v in partial_versions if
> +                           v != version], key=lambda x: x[1], reverse=True)
> +        return composed[0][0]

This looks like something that should live in a library, and have some tests (it's a pure function after all!).
Attachment #8717498 - Flags: feedback?(bhearsum) → feedback+
Comment on attachment 8717499 [details] [diff] [review]
updates_release_promotion-tools.diff

Review of attachment 8717499 [details] [diff] [review]:
-----------------------------------------------------------------

I'm not a huge fan of duplicating the two scripts, but I can live with it if you find a better name for them. These will end up being the only ones that exist after we fully switch to release promotion, so please choose a name that makes sense for that world.

::: release/update-verify-bump.pl
@@ -1,1 @@
> -#!/usr/bin/perl -w

Yay, script death!

::: scripts/updates/balrog-release-pusher-rp.py
@@ +25,5 @@
> +    parser.add_argument(
> +        "--username", required=True, help="Username in the credentials file")
> +    parser.add_argument(
> +        "--channel", dest="channels", action="append", required=True,
> +        help="Release blob channels")

This is not a very useful help message. How about something like "Channels to set-up fileUrls for"?

@@ +28,5 @@
> +        "--channel", dest="channels", action="append", required=True,
> +        help="Release blob channels")
> +    parser.add_argument(
> +        "--test-rule", dest="test_rules", action="append", required=True,
> +        help="Test channels that will be automatically enabled")

Please change the name of this to match whatever is used in the mozharness config.
Attachment #8717499 - Flags: feedback?(bhearsum) → feedback+

Updated

2 years ago
Attachment #8717513 - Flags: review?(bugspam.Callek) → review+

Updated

2 years ago
Attachment #8717514 - Flags: review?(bugspam.Callek) → review+
(Assignee)

Comment 10

2 years ago
Comment on attachment 8717514 [details] [diff] [review]
update_release_promotion-buildbot-configs.diff

https://hg.mozilla.org/build/buildbot-configs/rev/af83ded3aee2
Attachment #8717514 - Flags: checked-in+
(Assignee)

Comment 11

2 years ago
Comment on attachment 8717513 [details] [diff] [review]
update_release_promotion-buildbotcustom.diff

https://hg.mozilla.org/build/buildbotcustom/rev/48181923d5ae
Attachment #8717513 - Flags: checked-in+
in production
(Assignee)

Updated

2 years ago
Depends on: 1248883
(Assignee)

Updated

2 years ago
Depends on: 1249070
(Assignee)

Comment 13

2 years ago
Created attachment 8721083 [details] [diff] [review]
updates_release_promotion-tools.diff

Almost the same thing (the same approach), but working :)
http://buildbot-master72.bb.releng.usw2.mozilla.com:8001/builders/release-date-firefox_updates/builds/1
Attachment #8717499 - Attachment is obsolete: true
Attachment #8721083 - Flags: review?(bhearsum)
(Assignee)

Comment 14

2 years ago
Created attachment 8721085 [details] [diff] [review]
updates_release_promotion-gecko-dev-3.diff

works on date!
Attachment #8717498 - Attachment is obsolete: true
Attachment #8721085 - Flags: review?(jlund)
(Assignee)

Comment 15

2 years ago
Created attachment 8721086 [details] [review]
releasetasks patch
Attachment #8721086 - Flags: review?(jlund)
Comment on attachment 8721083 [details] [diff] [review]
updates_release_promotion-tools.diff

Review of attachment 8721083 [details] [diff] [review]:
-----------------------------------------------------------------

Much better names/locations, thank you!
Attachment #8721083 - Flags: review?(bhearsum) → review+
(Assignee)

Comment 17

2 years ago
Comment on attachment 8721083 [details] [diff] [review]
updates_release_promotion-tools.diff

https://hg.mozilla.org/build/tools/rev/8c70fceff603
Attachment #8721083 - Flags: checked-in+
Comment on attachment 8721085 [details] [diff] [review]
updates_release_promotion-gecko-dev-3.diff

Review of attachment 8721085 [details] [diff] [review]:
-----------------------------------------------------------------

::: testing/mozharness/scripts/release/updates.py
@@ +65,5 @@
> +                'tag',
> +                'push',
> +                'submit-to-balrog',
> +            ],
> +            default_actions=[

fyi - if default_actions == all_actions, it's not needed. mozharness will assume that if default_actions is not explicitly defined.

@@ +154,5 @@
> +            repos=self.query_repos())
> +
> +    def download_shipped_locales(self):
> +        dirs = self.query_abs_dirs()
> +        self.mkdir_p(dirs["abs_work_dir"])

is this needed?
Attachment #8721085 - Flags: review?(jlund) → review+
(Assignee)

Comment 19

2 years ago
(In reply to Jordan Lund (:jlund) from comment #18) 
> @@ +154,5 @@
> > +            repos=self.query_repos())
> > +
> > +    def download_shipped_locales(self):
> > +        dirs = self.query_abs_dirs()
> > +        self.mkdir_p(dirs["abs_work_dir"])
> 
> is this needed?

I hit some issues without that, probably when I ran it with a lot of --no-something args.
(Assignee)

Updated

2 years ago
Attachment #8721085 - Flags: checked-in+
(Assignee)

Comment 21

2 years ago
Comment on attachment 8721086 [details] [review]
releasetasks patch

r+ed in the PR
Attachment #8721086 - Flags: review?(jlund)
Attachment #8721086 - Flags: review+
Attachment #8721086 - Flags: checked-in+

Comment 22

2 years ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/93654b236653
Status: NEW → RESOLVED
Last Resolved: 2 years ago
status-firefox47: --- → fixed
Resolution: --- → FIXED
(Assignee)

Comment 23

2 years ago
Created attachment 8724501 [details] [diff] [review]
typos.diff

Typos... Pushed to date: https://hg.mozilla.org/projects/date/rev/d5ea8d4fd43e
Attachment #8724501 - Flags: review?(jlund)
(Assignee)

Updated

2 years ago
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
(Assignee)

Comment 24

2 years ago
Created attachment 8724582 [details] [diff] [review]
typos2.diff

moar fixes
Attachment #8724582 - Flags: review?(jlund)
Comment on attachment 8724501 [details] [diff] [review]
typos.diff

Review of attachment 8724501 [details] [diff] [review]:
-----------------------------------------------------------------

this patch is "moBeta" :)
Attachment #8724501 - Flags: review?(jlund) → review+

Updated

2 years ago
Attachment #8724582 - Flags: review?(jlund) → review+

Comment 29

2 years ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/fc271aac7545
https://hg.mozilla.org/mozilla-central/rev/8f6efc17373b
Status: REOPENED → RESOLVED
Last Resolved: 2 years ago2 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.