627308 - Develop a release config manipulation tool

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Rail Aliiev [:rail]]

As a part of automatic pre-production releases we need a tool which:

* Generates pre-production release config using already existing production config. It should be able to change some variables, such as FTP server, ssh username, repositories, etc.

* Bumps already existing configs. It should be able (at least) to bump version, build number.

It would be great, if the output of the bumper would be human readable (like the current configs).

Comment 1

[Rail Aliiev [:rail]]

Attachment: release_config_bumper.py

This version works fine for release configs. 

By default it bumps buildNumber and the corresponding variables.

Run with -v to bump version.

At the moment support for pre-production is minimal. It assumes that we don't land any changes on default branch in pre-production repository clone and uses tip as revision and branch for builds>1.

TODO: full support for pre-production by merging already existing pre-production release configs (to be added manually) and production release configs.

Comment 2

[Chris AtLee [:catlee]]

Comment on attachment 506680 [details] [diff] [review]
release_config_bumper.py

Looks good!

Comment 3

[Rail Aliiev [:rail]]

Attachment: tools

This tool is supposed to be used in pre-production to clone production repos to preproduction account, bump the version (atm version+1 build1) + override some configs, tag tools and configs repos.

* cleanOutgoingRevs moved to util.hg and reused in some places
* release_config_bumper.py processes only dynamically changing variables, static variables are in the override files (the next patch).
* mozilla-central is not used in clone_config.py because the config is not supposed to be used w/o manual modifications.

Worked fine on local computer using user repo.

Comment 4

[Rail Aliiev [:rail]]

Attachment: pre-production release overrides

Comment 5

[[e-mail] (:bhearsum)]

Comment on attachment 521984 [details] [diff] [review]
tools

Overall this looks pretty good, there's a few things I'd like to see addressed:
- The naming of some things made it hard to understand some of this. Specifically:
-- Using the word "clone" when more than just cloning is happening. I'd suggest re-using "repo_setup" or coming up with something new that encompasses what's happening
-- "bumpBranchConfigs" would be better named as "overrides"
- In bumpReleaseConfig(), can we get rid the conflation of build1BumpVars and build1ProcessVars? Specifically, not adding the former to the latter. That way you can iterate over build1ProcessVars instead of explicitly referring to old* again later.
- I'm not sure I see the need for the "preproduction" option to the release config bumper. It seems like you could drop it completely and have preproduction always passing "tip" for both revision and relbranch. 
Can you drop the "preproduction" option and replace it with --preserve-{revision,branch} or similar?
- Please use "relbranch" instead of "branch" as an option name to the release config bumper. Branch is already overloaded enough around here ;).

Apologies for the big delay on getting this reviewed, you're doing good work here!

Comment 6

[[e-mail] (:bhearsum)]

Comment on attachment 521985 [details] [diff] [review]
pre-production release overrides

Mostly fine, a few thoughts:
- I don't think staging-stage.b.m.o is the correct ausServerUrl for preproduction
- You've got got enable_repo_setup listed twice
- Until we've got more slaves in preproduction, I think it'd be good to lower the number of l10n chunks. Maybe to 2, to make sure we're still testing the chunking code properly?

I really like the idea of these overrides, hopefully we can do something similar for staging!

Comment 7

[Rail Aliiev [:rail]]

Attachment: tools

(In reply to comment #5)
> - I'm not sure I see the need for the "preproduction" option to the release
> config bumper. It seems like you could drop it completely and have
> preproduction always passing "tip" for both revision and relbranch. 
> Can you drop the "preproduction" option and replace it with

Yeah. This is simpler.

> --preserve-{revision,branch} or similar?

--preserve-relbranch won't touch relbranch (useful for chemspills).

Comment 8

[Rail Aliiev [:rail]]

Attachment: pre-production release overrides

(In reply to comment #6)
> Mostly fine, a few thoughts:
> - I don't think staging-stage.b.m.o is the correct ausServerUrl for
> preproduction
> - You've got got enable_repo_setup listed twice

Ooops.

> - Until we've got more slaves in preproduction, I think it'd be good to lower
> the number of l10n chunks. Maybe to 2, to make sure we're still testing the
> chunking code properly?

OK

> I really like the idea of these overrides, hopefully we can do something
> similar for staging!

Yeah, I hate the fact that we need to adjust configs every time in staging. Maybe it will be a different approach, because we still need release configs as monolithic files to execfile them on slaves.

Comment 9

[Rail Aliiev [:rail]]

Comment on attachment 523434 [details] [diff] [review]
tools

http://hg.mozilla.org/build/tools/rev/5a3b9828c905

Comment 10

[Rail Aliiev [:rail]]

Comment on attachment 523437 [details] [diff] [review]
pre-production release overrides

http://hg.mozilla.org/build/buildbot-configs/rev/6ae7ca9494ab

Comment 11

[Rail Aliiev [:rail]]

All done here.