multilocale repo configs should be override-friendly

RESOLVED FIXED

Status

Release Engineering
General
P5
normal
RESOLVED FIXED
7 years ago
5 years ago

People

(Reporter: aki, Assigned: aki)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [releases][mozharness])

Attachments

(3 attachments, 2 obsolete attachments)

(Assignee)

Description

7 years ago
We should easily be able to pass in

a) release tag
b) base user repo location

at the very least, and have them apply to all repos.
(Assignee)

Updated

7 years ago
Priority: -- → P5
(Assignee)

Comment 1

7 years ago
This would have helped during the chemspills of the week of August 29.
(Assignee)

Comment 2

7 years ago
(In reply to Aki Sasaki [:aki] from comment #0)
> b) base user repo location

I think I'm going to pass on this.
We can easily change what config file that mozharness uses ( http://hg.mozilla.org/build/buildbot-configs/file/b1174cccecbf/mozilla/release-fennec-mozilla-beta.py#l112) and I've been thinking we should have mozharness/configs/users/USERNAME directories where users can check in their own config files for their own use without review.

However, I think getting rid of the multi_locale config edits for releases, by passing in the release tag via commandline, would be very helpful.
(Assignee)

Comment 3

7 years ago
Created attachment 559348 [details] [diff] [review]
multi tag override (mozharness)

This patch:

* removes version-specific release tags from configs/multi_locale/[staging_]release_*

* adds tag_override to required_config_vars in configs/multi_locale/[staging_]release_*

* adds a check in BaseScript.__init__() to verify all required_config_vars are set in self.config; else self.fatal()

* adds an option for tag_override in MercurialMixin.scm_checkout_repos()

* adds a commandline option for --tag-override in MultiLocaleBuild

* adds tag_override to both self.scm_checkout_repos() calls in MultiLocaleBuild

* makes hg_l10n_tag optional in MultiLocaleBuild.pull_locale_source()

I verified this with

mozharness/scripts/multil10n.py --config-file multi_locale/release_mozilla-beta_linux-android.json
# Failed due to missing --tag-override

mozharness/scripts/multil10n.py --config-file multi_locale/release_mozilla-beta_linux-android.json --tag-override FENNEC_7_0b3_RELEASE --only-pull-locale-source
grep 'update -C' upload_dir/logs/multilocale_info.log 
# Used tag FENNEC_7_0b3_RELEASE for all locales

mozharness/scripts/multil10n.py --config-file multi_locale/release_mozilla-beta_linux-android.json --tag-override FENNEC_7_0b3_RELEASE --only-pull-build-source
grep 'update -C' upload_dir/logs/multilocale_info.log 
# Used tag FENNEC_7_0b3_RELEASE for all non-locale repos

mozharness/scripts/multil10n.py --config-file multi_locale/mozilla-central_linux-android.json --only-pull-build-source
grep 'update -C' upload_dir/logs/multilocale_info.log
# Used standard nightly tags for all non-locale repos

mozharness/scripts/multil10n.py --config-file multi_locale/mozilla-central_linux-android.json --only-pull-locale-source
grep 'update -C' upload_dir/logs/multilocale_info.log
# Used default for all locales


Buildbotcustom patch coming; I'll also test an Android release build because I don't want to break 7.0b6 / 7.0 final.
(Assignee)

Comment 4

7 years ago
Created attachment 559351 [details] [diff] [review]
[untested] multi tag override (buildbotcustom)

add '--tag-override=%s' % kwargs['mozharnessTag'] to ReleaseBuildFactory's mozharnessMultiOptions.  Nested this inside an |if kwargs.get('multiLocale')| since mozharnessTag isn't set for non-Android non-Maemo release factories.

As mentioned above, I want to test this in staging.
(Assignee)

Comment 5

7 years ago
(In reply to Aki Sasaki [:aki] from comment #2)
> (In reply to Aki Sasaki [:aki] from comment #0)
> > b) base user repo location
> 
> I think I'm going to pass on this.
> We can easily change what config file that mozharness uses (
> http://hg.mozilla.org/build/buildbot-configs/file/b1174cccecbf/mozilla/
> release-fennec-mozilla-beta.py#l112) and I've been thinking we should have
> mozharness/configs/users/USERNAME directories where users can check in their
> own config files for their own use without review.
> 
> However, I think getting rid of the multi_locale config edits for releases,
> by passing in the release tag via commandline, would be very helpful.

I just got bit by this in my staging run, so it's back on. New patches tomorrow.
(Assignee)

Comment 6

7 years ago
Created attachment 559578 [details] [diff] [review]
multi tag/user repo override (mozharness)

Same as above:

* removes version-specific release tags from configs/multi_locale/[staging_]release_*

* adds tag_override to required_config_vars in configs/multi_locale/[staging_]release_*

* adds a check in BaseScript.__init__() to verify all required_config_vars are set in self.config; else self.fatal()

* adds an option for tag_override in MercurialMixin.scm_checkout_repos()

* adds a commandline option for --tag-override in MultiLocaleBuild

* adds tag_override to both self.scm_checkout_repos() calls in MultiLocaleBuild

* makes hg_l10n_tag optional in MultiLocaleBuild.pull_locale_source()

Also:

* adds %(user_repo_override)s to all configs/multi_locale/staging_release*.json
* adds user_repo_override to the list of required config vars
* adds --user-repo-override to MultiLocaleBuild's options
* replaces %(user_repo_override)s in both pull_build_source() and pull_locale_source()
Attachment #559348 - Attachment is obsolete: true
(Assignee)

Comment 7

7 years ago
Created attachment 559583 [details] [diff] [review]
multi tag/user repo override (buildbot-configs)

* added multiLocaleOptions to the releaseConfig['mozharness_config'] dictionary

* --tag-override comes from releaseConfig['baseTag']

* in staging release configs, added --user-repo-override=users/stage-ffxbld (we can change this here before doing a staging release)
(Assignee)

Comment 8

7 years ago
Created attachment 559584 [details] [diff] [review]
multi tag/user repo override (buildbotcustom)

* stopped setting mozharnessMultiOptions inside ReleaseBuildFactory

* set them in generateReleaseBranchObjects()
Attachment #559351 - Attachment is obsolete: true
(Assignee)

Comment 9

7 years ago
Testing:

* Ran a staging release android mozilla-beta build.

** I didn't have to edit any tags in the mozharness configs; that was passed down through staging_release_fennec-mozilla-beta.py's --tag-override option.

** I didn't have to edit any user repo locations in the mozharness configs; that was passed down through staging_release_fennec-mozilla-beta.py's --user-repo-override option.

** This does assume that all repos for a staging release are under one user repo root.
However, if that isn't the case, we still have the option of landing a new user mozharness config file and specifying that inside staging_release_fennec-mozilla-beta.py.

* Ran a mozilla-central android nightly.

Both runs were good.  Asking for r?
(Assignee)

Updated

7 years ago
Attachment #559578 - Flags: review?(lsblakk)
(Assignee)

Updated

7 years ago
Attachment #559583 - Flags: review?(lsblakk)
(Assignee)

Updated

7 years ago
Attachment #559584 - Flags: review?(lsblakk)
Attachment #559578 - Flags: review?(lsblakk) → review+
Attachment #559583 - Flags: review?(lsblakk) → review+
Attachment #559584 - Flags: review?(lsblakk) → review+
(Assignee)

Updated

7 years ago
Flags: needs-reconfig?
(Assignee)

Comment 10

7 years ago
Comment on attachment 559578 [details] [diff] [review]
multi tag/user repo override (mozharness)

http://hg.mozilla.org/build/mozharness/rev/86ed3a78d76b
Attachment #559578 - Flags: checked-in+
(Assignee)

Comment 11

7 years ago
Comment on attachment 559583 [details] [diff] [review]
multi tag/user repo override (buildbot-configs)

http://hg.mozilla.org/build/buildbot-configs/rev/24c37d7d04be
Attachment #559583 - Flags: checked-in+
(Assignee)

Comment 12

7 years ago
Comment on attachment 559584 [details] [diff] [review]
multi tag/user repo override (buildbotcustom)

http://hg.mozilla.org/build/buildbotcustom/rev/855428250f00
Attachment #559584 - Flags: checked-in+
(Assignee)

Updated

7 years ago
Status: NEW → RESOLVED
Last Resolved: 7 years ago
Flags: needs-reconfig?
Resolution: --- → FIXED
(Assignee)

Updated

7 years ago
Duplicate of this bug: 676488
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.