679153 - multilocale repo configs should be override-friendly

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P5)

Description

[Aki Sasaki (not active)]

We should easily be able to pass in

a) release tag
b) base user repo location

at the very least, and have them apply to all repos.

Comment 1

[Aki Sasaki (not active)]

This would have helped during the chemspills of the week of August 29.

Comment 2

[Aki Sasaki (not active)]

(In reply to Aki Sasaki [:aki] from comment #0)
> b) base user repo location

I think I'm going to pass on this.
We can easily change what config file that mozharness uses ( http://hg.mozilla.org/build/buildbot-configs/file/b1174cccecbf/mozilla/release-fennec-mozilla-beta.py#l112) and I've been thinking we should have mozharness/configs/users/USERNAME directories where users can check in their own config files for their own use without review.

However, I think getting rid of the multi_locale config edits for releases, by passing in the release tag via commandline, would be very helpful.

Comment 3

[Aki Sasaki (not active)]

Attachment: multi tag override (mozharness)

This patch:

* removes version-specific release tags from configs/multi_locale/[staging_]release_*

* adds tag_override to required_config_vars in configs/multi_locale/[staging_]release_*

* adds a check in BaseScript.__init__() to verify all required_config_vars are set in self.config; else self.fatal()

* adds an option for tag_override in MercurialMixin.scm_checkout_repos()

* adds a commandline option for --tag-override in MultiLocaleBuild

* adds tag_override to both self.scm_checkout_repos() calls in MultiLocaleBuild

* makes hg_l10n_tag optional in MultiLocaleBuild.pull_locale_source()

I verified this with

mozharness/scripts/multil10n.py --config-file multi_locale/release_mozilla-beta_linux-android.json
# Failed due to missing --tag-override

mozharness/scripts/multil10n.py --config-file multi_locale/release_mozilla-beta_linux-android.json --tag-override FENNEC_7_0b3_RELEASE --only-pull-locale-source
grep 'update -C' upload_dir/logs/multilocale_info.log 
# Used tag FENNEC_7_0b3_RELEASE for all locales

mozharness/scripts/multil10n.py --config-file multi_locale/release_mozilla-beta_linux-android.json --tag-override FENNEC_7_0b3_RELEASE --only-pull-build-source
grep 'update -C' upload_dir/logs/multilocale_info.log 
# Used tag FENNEC_7_0b3_RELEASE for all non-locale repos

mozharness/scripts/multil10n.py --config-file multi_locale/mozilla-central_linux-android.json --only-pull-build-source
grep 'update -C' upload_dir/logs/multilocale_info.log
# Used standard nightly tags for all non-locale repos

mozharness/scripts/multil10n.py --config-file multi_locale/mozilla-central_linux-android.json --only-pull-locale-source
grep 'update -C' upload_dir/logs/multilocale_info.log
# Used default for all locales


Buildbotcustom patch coming; I'll also test an Android release build because I don't want to break 7.0b6 / 7.0 final.

Comment 4

[Aki Sasaki (not active)]

Attachment: [untested] multi tag override (buildbotcustom)

add '--tag-override=%s' % kwargs['mozharnessTag'] to ReleaseBuildFactory's mozharnessMultiOptions.  Nested this inside an |if kwargs.get('multiLocale')| since mozharnessTag isn't set for non-Android non-Maemo release factories.

As mentioned above, I want to test this in staging.

Comment 5

[Aki Sasaki (not active)]

(In reply to Aki Sasaki [:aki] from comment #2)
> (In reply to Aki Sasaki [:aki] from comment #0)
> > b) base user repo location
> 
> I think I'm going to pass on this.
> We can easily change what config file that mozharness uses (
> http://hg.mozilla.org/build/buildbot-configs/file/b1174cccecbf/mozilla/
> release-fennec-mozilla-beta.py#l112) and I've been thinking we should have
> mozharness/configs/users/USERNAME directories where users can check in their
> own config files for their own use without review.
> 
> However, I think getting rid of the multi_locale config edits for releases,
> by passing in the release tag via commandline, would be very helpful.

I just got bit by this in my staging run, so it's back on. New patches tomorrow.

Comment 6

[Aki Sasaki (not active)]

Attachment: multi tag/user repo override (mozharness)

Same as above:

* removes version-specific release tags from configs/multi_locale/[staging_]release_*

* adds tag_override to required_config_vars in configs/multi_locale/[staging_]release_*

* adds a check in BaseScript.__init__() to verify all required_config_vars are set in self.config; else self.fatal()

* adds an option for tag_override in MercurialMixin.scm_checkout_repos()

* adds a commandline option for --tag-override in MultiLocaleBuild

* adds tag_override to both self.scm_checkout_repos() calls in MultiLocaleBuild

* makes hg_l10n_tag optional in MultiLocaleBuild.pull_locale_source()

Also:

* adds %(user_repo_override)s to all configs/multi_locale/staging_release*.json
* adds user_repo_override to the list of required config vars
* adds --user-repo-override to MultiLocaleBuild's options
* replaces %(user_repo_override)s in both pull_build_source() and pull_locale_source()

Comment 7

[Aki Sasaki (not active)]

Attachment: multi tag/user repo override (buildbot-configs)

* added multiLocaleOptions to the releaseConfig['mozharness_config'] dictionary

* --tag-override comes from releaseConfig['baseTag']

* in staging release configs, added --user-repo-override=users/stage-ffxbld (we can change this here before doing a staging release)

Comment 8

[Aki Sasaki (not active)]

Attachment: multi tag/user repo override (buildbotcustom)

* stopped setting mozharnessMultiOptions inside ReleaseBuildFactory

* set them in generateReleaseBranchObjects()

Comment 9

[Aki Sasaki (not active)]

Testing:

* Ran a staging release android mozilla-beta build.

** I didn't have to edit any tags in the mozharness configs; that was passed down through staging_release_fennec-mozilla-beta.py's --tag-override option.

** I didn't have to edit any user repo locations in the mozharness configs; that was passed down through staging_release_fennec-mozilla-beta.py's --user-repo-override option.

** This does assume that all repos for a staging release are under one user repo root.
However, if that isn't the case, we still have the option of landing a new user mozharness config file and specifying that inside staging_release_fennec-mozilla-beta.py.

* Ran a mozilla-central android nightly.

Both runs were good.  Asking for r?

Comment 10

[Aki Sasaki (not active)]

Comment on attachment 559578 [details] [diff] [review]
multi tag/user repo override (mozharness)

http://hg.mozilla.org/build/mozharness/rev/86ed3a78d76b

Comment 11

[Aki Sasaki (not active)]

Comment on attachment 559583 [details] [diff] [review]
multi tag/user repo override (buildbot-configs)

http://hg.mozilla.org/build/buildbot-configs/rev/24c37d7d04be

Comment 12

[Aki Sasaki (not active)]

Comment on attachment 559584 [details] [diff] [review]
multi tag/user repo override (buildbotcustom)

http://hg.mozilla.org/build/buildbotcustom/rev/855428250f00