Closed Bug 679153 Opened 9 years ago Closed 9 years ago

multilocale repo configs should be override-friendly

Categories

(Release Engineering :: General, defect, P5)

defect

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: aki, Assigned: aki)

References

Details

(Whiteboard: [releases][mozharness])

Attachments

(3 files, 2 obsolete files)

We should easily be able to pass in

a) release tag
b) base user repo location

at the very least, and have them apply to all repos.
Priority: -- → P5
This would have helped during the chemspills of the week of August 29.
(In reply to Aki Sasaki [:aki] from comment #0)
> b) base user repo location

I think I'm going to pass on this.
We can easily change what config file that mozharness uses ( http://hg.mozilla.org/build/buildbot-configs/file/b1174cccecbf/mozilla/release-fennec-mozilla-beta.py#l112) and I've been thinking we should have mozharness/configs/users/USERNAME directories where users can check in their own config files for their own use without review.

However, I think getting rid of the multi_locale config edits for releases, by passing in the release tag via commandline, would be very helpful.
Attached patch multi tag override (mozharness) (obsolete) — Splinter Review
This patch:

* removes version-specific release tags from configs/multi_locale/[staging_]release_*

* adds tag_override to required_config_vars in configs/multi_locale/[staging_]release_*

* adds a check in BaseScript.__init__() to verify all required_config_vars are set in self.config; else self.fatal()

* adds an option for tag_override in MercurialMixin.scm_checkout_repos()

* adds a commandline option for --tag-override in MultiLocaleBuild

* adds tag_override to both self.scm_checkout_repos() calls in MultiLocaleBuild

* makes hg_l10n_tag optional in MultiLocaleBuild.pull_locale_source()

I verified this with

mozharness/scripts/multil10n.py --config-file multi_locale/release_mozilla-beta_linux-android.json
# Failed due to missing --tag-override

mozharness/scripts/multil10n.py --config-file multi_locale/release_mozilla-beta_linux-android.json --tag-override FENNEC_7_0b3_RELEASE --only-pull-locale-source
grep 'update -C' upload_dir/logs/multilocale_info.log 
# Used tag FENNEC_7_0b3_RELEASE for all locales

mozharness/scripts/multil10n.py --config-file multi_locale/release_mozilla-beta_linux-android.json --tag-override FENNEC_7_0b3_RELEASE --only-pull-build-source
grep 'update -C' upload_dir/logs/multilocale_info.log 
# Used tag FENNEC_7_0b3_RELEASE for all non-locale repos

mozharness/scripts/multil10n.py --config-file multi_locale/mozilla-central_linux-android.json --only-pull-build-source
grep 'update -C' upload_dir/logs/multilocale_info.log
# Used standard nightly tags for all non-locale repos

mozharness/scripts/multil10n.py --config-file multi_locale/mozilla-central_linux-android.json --only-pull-locale-source
grep 'update -C' upload_dir/logs/multilocale_info.log
# Used default for all locales


Buildbotcustom patch coming; I'll also test an Android release build because I don't want to break 7.0b6 / 7.0 final.
add '--tag-override=%s' % kwargs['mozharnessTag'] to ReleaseBuildFactory's mozharnessMultiOptions.  Nested this inside an |if kwargs.get('multiLocale')| since mozharnessTag isn't set for non-Android non-Maemo release factories.

As mentioned above, I want to test this in staging.
(In reply to Aki Sasaki [:aki] from comment #2)
> (In reply to Aki Sasaki [:aki] from comment #0)
> > b) base user repo location
> 
> I think I'm going to pass on this.
> We can easily change what config file that mozharness uses (
> http://hg.mozilla.org/build/buildbot-configs/file/b1174cccecbf/mozilla/
> release-fennec-mozilla-beta.py#l112) and I've been thinking we should have
> mozharness/configs/users/USERNAME directories where users can check in their
> own config files for their own use without review.
> 
> However, I think getting rid of the multi_locale config edits for releases,
> by passing in the release tag via commandline, would be very helpful.

I just got bit by this in my staging run, so it's back on. New patches tomorrow.
Same as above:

* removes version-specific release tags from configs/multi_locale/[staging_]release_*

* adds tag_override to required_config_vars in configs/multi_locale/[staging_]release_*

* adds a check in BaseScript.__init__() to verify all required_config_vars are set in self.config; else self.fatal()

* adds an option for tag_override in MercurialMixin.scm_checkout_repos()

* adds a commandline option for --tag-override in MultiLocaleBuild

* adds tag_override to both self.scm_checkout_repos() calls in MultiLocaleBuild

* makes hg_l10n_tag optional in MultiLocaleBuild.pull_locale_source()

Also:

* adds %(user_repo_override)s to all configs/multi_locale/staging_release*.json
* adds user_repo_override to the list of required config vars
* adds --user-repo-override to MultiLocaleBuild's options
* replaces %(user_repo_override)s in both pull_build_source() and pull_locale_source()
Attachment #559348 - Attachment is obsolete: true
* added multiLocaleOptions to the releaseConfig['mozharness_config'] dictionary

* --tag-override comes from releaseConfig['baseTag']

* in staging release configs, added --user-repo-override=users/stage-ffxbld (we can change this here before doing a staging release)
* stopped setting mozharnessMultiOptions inside ReleaseBuildFactory

* set them in generateReleaseBranchObjects()
Attachment #559351 - Attachment is obsolete: true
Testing:

* Ran a staging release android mozilla-beta build.

** I didn't have to edit any tags in the mozharness configs; that was passed down through staging_release_fennec-mozilla-beta.py's --tag-override option.

** I didn't have to edit any user repo locations in the mozharness configs; that was passed down through staging_release_fennec-mozilla-beta.py's --user-repo-override option.

** This does assume that all repos for a staging release are under one user repo root.
However, if that isn't the case, we still have the option of landing a new user mozharness config file and specifying that inside staging_release_fennec-mozilla-beta.py.

* Ran a mozilla-central android nightly.

Both runs were good.  Asking for r?
Attachment #559578 - Flags: review?(lsblakk)
Attachment #559583 - Flags: review?(lsblakk)
Attachment #559584 - Flags: review?(lsblakk)
Attachment #559578 - Flags: review?(lsblakk) → review+
Attachment #559583 - Flags: review?(lsblakk) → review+
Attachment #559584 - Flags: review?(lsblakk) → review+
Flags: needs-reconfig?
Comment on attachment 559578 [details] [diff] [review]
multi tag/user repo override (mozharness)

http://hg.mozilla.org/build/mozharness/rev/86ed3a78d76b
Attachment #559578 - Flags: checked-in+
Comment on attachment 559583 [details] [diff] [review]
multi tag/user repo override (buildbot-configs)

http://hg.mozilla.org/build/buildbot-configs/rev/24c37d7d04be
Attachment #559583 - Flags: checked-in+
Comment on attachment 559584 [details] [diff] [review]
multi tag/user repo override (buildbotcustom)

http://hg.mozilla.org/build/buildbotcustom/rev/855428250f00
Attachment #559584 - Flags: checked-in+
Status: NEW → RESOLVED
Closed: 9 years ago
Flags: needs-reconfig?
Resolution: --- → FIXED
Duplicate of this bug: 676488
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.