Closed Bug 1166464 Opened 10 years ago Closed 9 years ago

Bump version after shipping release builds

Categories

(Release Engineering :: General, defect)

defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: RyanVM, Assigned: rail)

References

Details

Attachments

(9 files, 5 obsolete files)

17.89 KB, patch
jlund
: review+
Details | Diff | Splinter Review
3.28 KB, patch
jlund
: review+
Details | Diff | Splinter Review
31.43 KB, patch
jlund
: feedback+
Details | Diff | Splinter Review
7.08 KB, patch
jlund
: review+
Details | Diff | Splinter Review
4.29 KB, patch
jlund
: review+
Details | Diff | Splinter Review
3.17 KB, patch
nthomas
: review+
Details | Diff | Splinter Review
47 bytes, text/x-github-pull-request
jlund
: review+
Details | Review
1.18 KB, patch
Callek
: review+
Details | Diff | Splinter Review
32.31 KB, patch
jlund
: review+
Details | Diff | Splinter Review
Since we haven't shipped any official Fennec 38.0.5 beta builds yet, dep builds are currently picking up a 38.0.1 version number from mobile/android/confvars.sh. This has been a source of confusion for people on the Fennec team trying to verify feature work landing on the branch prior to official betas going on. Furthermore, it occurs to me that this could become a bigger issue with release promotion. Right now, we bump the version number automatically when an official build is requested via Ship It. What happens when we're using a pre-existing build instead? I don't have a great proposal for what to do here, but we should be aware of this possible problem at least.
So you're proposing http://hg.mozilla.org/releases/mozilla-release/file/06bdddc6463d/mobile/android/confvars.sh#l8 be changed to '38.0.5' ? There's a beta coming on Thursday which will have the same effect, but we can change it now if that's too long. With release promotion, we'd have to be more proactive about setting the version to have it right when the on-push build runs. Merging would take some of that, and then after a chemspill ships we could bump again. Thanks for raising it.
Yeah, I thought about just bumping it myself with some uplifts I did today but decided to let it ride until Thursday since it's not really *that* big of a deal as long as we know about it. I mainly just filed this to get it on the release promotion radar.
For release promotion, we should bump the version to the next point version after shipping to the release channel. Can't do that before because we might respin, and one of the nice things about promoting is not having release branches any more.
Summary: Confusing version number for Fennec builds on mozilla-release → Bump version after shipping release builds
Bug 1145175 changes this a bit - we'll need to bump after each beta.
Assignee: nobody → rail
Attached patch bump_version.diff (obsolete) — Splinter Review
* Refactor config to include optional version suffix (for beta) * add `bump_second_digit` to handle esr * refactor get_fx_version() * refactor some methods to deal with missing "from" repo * ran on most repos, worked fine
Attachment #8709485 - Flags: review?(jlund)
Comment on attachment 8709485 [details] [diff] [review] bump_version.diff Review of attachment 8709485 [details] [diff] [review]: ----------------------------------------------------------------- ::: testing/mozharness/scripts/merge_day/gecko_migration.py @@ +127,5 @@ > self.abs_dirs['abs_tools_lib_dir'] = os.path.join( > dirs['abs_work_dir'], 'tools', 'lib', 'python' > ) > for k in ('from', 'to'): > + url = self.config.get("%s_repo_url" % k) oh, right. because no from_* for bump_esr @@ +333,5 @@ > curr_weave_version = str(int(curr_version) + 2) > next_weave_version = str(int(curr_weave_version) + 1) > for f in self.config["version_files"]: > + from_ = "%s.0%s" % (curr_version, curr_suffix) > + to = "%s.0%s" % (next_version, next_suffix) hm, maybe we want file in the config's version_files to have explicit f["suffix"]? Off first read it's a bit confusing sometimes having a value for next_suffix param of bump_version() and sometimes having "suffix" defined in the config but ensuring both are not defined. @@ +486,5 @@ > + # bump the second digit > + next_version[1] = str(int(next_version[1]) + 1) > + # in case we have third digit, reset it to 0 > + if len(next_version) > 2: > + next_version[2] = '0' right now this equals '2esrpre'[1]. Do we ever need to preserve the 'esrpre' bit or something similar? [1] https://hg.mozilla.org/releases/mozilla-esr38/file/aaf922ae6796/browser/config/version.txt @@ +491,5 @@ > + next_version = ".".join(next_version) > + for f in self.config["version_files"]: > + to = next_version > + if f.get("suffix"): > + to += f["suffix"] I notice in the bump_esr config you don't define 'suffix' anywhere. do you predict we will eventually or are you just putting it in here for completeness?
Attachment #8709485 - Flags: review?(jlund) → review+
(In reply to Jordan Lund (:jlund) from comment #6) > hm, maybe we want file in the config's version_files to have explicit > f["suffix"]? Off first read it's a bit confusing sometimes having a value > for next_suffix param of bump_version() and sometimes having "suffix" > defined in the config but ensuring both are not defined. Yeah. I can explicitly set them. > right now this equals '2esrpre'[1]. Do we ever need to preserve the 'esrpre' > bit or something similar? "esrpre" will go away, because those would be unshipable. > I notice in the bump_esr config you don't define 'suffix' anywhere. do you > predict we will eventually or are you just putting it in here for > completeness? See comment above, no suffix for esr. I'll update the patch with explicit suffixes.
(In reply to Jordan Lund (:jlund) from comment #6) > I notice in the bump_esr config you don't define 'suffix' anywhere. do you > predict we will eventually or are you just putting it in here for > completeness? Misread this, and found a bug in this block. :) to is never used.
Interdiff :https://gist.github.com/rail/32c15ae7a9c3d2ec11af * use suffix explicitly * don't bump b2g/confvars.sh - we stopped building b2g off of train branches
Attachment #8709485 - Attachment is obsolete: true
Attachment #8709626 - Flags: review?(jlund)
Attached patch version_bump_postrelease.diff (obsolete) — Splinter Review
This one is to bump version as a part of postrelease: * get_version() skips comments and blank lines now * I need to add a buildbotcustom and releasetasks patches to schedule it * need to add commit and push actions, probably by explicitly passing them, to prevent accidental pushes * may need to explicitly specify ssh user/key
Attachment #8709633 - Flags: feedback?(jlund)
Comment on attachment 8709626 [details] [diff] [review] version_bump-gecko-dev.diff Review of attachment 8709626 [details] [diff] [review]: ----------------------------------------------------------------- ::: testing/mozharness/scripts/merge_day/gecko_migration.py @@ +333,5 @@ > curr_weave_version = str(int(curr_version) + 2) > next_weave_version = str(int(curr_weave_version) + 1) > for f in self.config["version_files"]: > + from_ = "%s.0%s" % (curr_version, curr_suffix) > + to = "%s.0%s%s" % (next_version, next_suffix, f["suffix"]) I was actually thinking of replacing `next_suffix` with `f["suffix"]` altogether. However I guess that wouldn't be so straight forward for certain migrations like central_to_aurora because that would require something like: self.config['version_files]['central'][files] and self.config['version_files]['aurora'][files] as we make two bump_version calls with central_to_aurora.py. Might make it just as messy :/
Attachment #8709626 - Flags: review?(jlund) → review+
Keywords: leave-open
Comment on attachment 8709633 [details] [diff] [review] version_bump_postrelease.diff Review of attachment 8709633 [details] [diff] [review]: ----------------------------------------------------------------- looks great! the configs don't leave room for fennec. do we need per product configs? i.e. s/postrelease_beta.py/desktop_fx_postrelease_beta.py/ ? ::: testing/mozharness/configs/merge_day/postrelease_beta.py @@ +1,3 @@ > +config = { > + "log_name": "bump_esr", > + "version_files": [ now that 'suffix' is assumed to exist (.get('suffix') instead of ['suffix']) for migration and bump_second_digit actions, I wonder if that will add confusion or cause errors in the future. Granted these post release configs will only be used with the 'bump_postrelease' actions and that method does not assume 'suffix' is present right now probably not an issue for now, just pointing that out @@ +2,5 @@ > + "log_name": "bump_esr", > + "version_files": [ > + { > + "file": "browser/config/version_display.txt", > + "split_by": "b", # split using "b" character this kind of makes me nervous because we would end up with something like ['45.0', '1']. I guess that's okay as long as we forever ignore the first index because it is not a true split in that it combines major and minor versions around the '.' ::: testing/mozharness/configs/merge_day/postrelease_esr.py @@ +19,5 @@ > + }, > + ], > + "tools_repo_url": "https://hg.mozilla.org/build/tools", > + "tools_repo_revision": "default", > + "to_repo_url": "ssh://hg.mozilla.org/releases/mozilla-esr38", should we add another TODO here when we have mozilla-esr45? ::: testing/mozharness/scripts/merge_day/gecko_migration.py @@ +187,3 @@ > contents = self.read_from_file(version_path, error_level=FATAL) > + for line in contents.splitlines(): > + # skip comments, empty lines ++ @@ +507,5 @@ > + version = self.get_version(dirs['abs_to_dir'], f["file"], > + f["split_by"]) > + curr_version = f["split_by"].join(version) > + next_version = list(version) > + # bump required digit the only one I'm confused about is with esr. Because esr we sometimes bump the first digit, the second digit (index 1) and sometimes we bump the third (index 2). e.g. '38.5.2 to 38.6.0' vs '38.5.2 to 38.5.3' vs 'introduction of 45' so to sanity check I we expect the same thing: iiuc, after we do a major release, e.g. 38.5.2 to 38.6.0, and then run this post release step, version.txt will have 38.6.1 as contents correct? where as if we do a dot release, 38.5.2 to 38.5.3, after post release, version.txt will have contents 38.5.4 correct?
Attachment #8709633 - Flags: feedback?(jlund) → feedback+
* Move the "enable_release_promotion" logic to configs * Explicitly pass product name to generateReleasePromotionBuilders(), similar to generateReleaseBuilders(). In the future it will simplify Fennec work. Also, no need to use pf["product_name"] for platform independent builders. * add postrelease_version_bump_* vars
Attachment #8711061 - Flags: review?(jlund)
* renames: config -> branch_config, name -> branch_name * explicitly take product name as a parameter * get rid of early return if enable_release_promotion is not set * use product instead of pf["product_name"] * be explicit when use pf for platform independent builders * get rid of hard coded properties - they should be set by bbb, leave only required by log_uploader.py * reformat some blocks * add version_bump_builder
Attachment #8711063 - Flags: review?(jlund)
I decided to simplify this and get rid of action bumping. Instead, we'll be using explicit next_version set as property, passed vial releasetasks. I thought that in ship-it we can generate a suggested next version programmatically and give a chance to override it for unusual cases. This also solves the issue when you try to rerun the same builder and get a newer version every run. Also, will solve the issue when we have Fennec and Firefox releases running in parallel. Whichever bumps first wins the race, the other one won't do anything. I'm going to implement next_version in releaserunner for now, but it should be done in ship-it. Maybe I need to add a guard to prevent reducing the number, but I'm not quite sure if we want this yet...
Attachment #8709626 - Attachment is obsolete: true
Attachment #8711128 - Flags: review?(jlund)
Attachment #8709626 - Attachment is obsolete: false
Attachment #8709633 - Attachment is obsolete: true
This is a temp solution for version bump
Attachment #8711136 - Flags: review?(jlund)
Comment on attachment 8711063 [details] [diff] [review] version_bump-buildbotcustom-1.diff Review of attachment 8711063 [details] [diff] [review]: ----------------------------------------------------------------- can the version bump job be a TC task instead? We just need commit privileges correct? Can we use some TC secret _something_? setting to r- until we discuss the explicit 'product' comment. feel free to push back :) ::: process/release.py @@ +1764,3 @@ > l10n_buildername = "release-{branch}_{product}_{platform}_l10n_repack".format( > + branch=branch_name, > + product=product, so in https://bugzilla.mozilla.org/page.cgi?id=splinter.html&bug=1166464&attachment=8711061 you are forcing the product to be 'firefox' but what if later down the line branch_config["platforms"][platform]["product_name"] points to something that is not "firefox"? This won't be caught with an error and might cause a footgun later down the line. e.g. we forget to update builder_master.cfg with more release_promotion products as l10n_release_platforms list gets extended with say fennec platforms @@ +1818,5 @@ > bouncer_buildername = "release-{branch}_{product}_bncr_sub".format( > + branch=branch_name, product=product) > + # Explicitly define pf using the slave platform (linux64 in this case) > + bouncer_submitter_factory = makeMHFactory( > + config=branch_config, pf=branch_config["platforms"]['linux64'], whoa, I'm confused. was this a bug before? were we just taking the last `pf` value in the l10n_release_platforms loop? @@ +1840,5 @@ > } > builders.append(bouncer_builder) > > + version_bump_mh_cfg = { > + "script_name": "scripts/merge_day/gecko_migration.py", so, iiuc, we will be using the release branch's mozharness for this gecko_migration.py call. This might cause confusion because up till now we have been using m-c tip for all things gecko_migration.py...
Attachment #8711063 - Flags: review?(jlund) → review-
Comment on attachment 8711063 [details] [diff] [review] version_bump-buildbotcustom-1.diff Review of attachment 8711063 [details] [diff] [review]: ----------------------------------------------------------------- ::: process/release.py @@ +1842,5 @@ > > + version_bump_mh_cfg = { > + "script_name": "scripts/merge_day/gecko_migration.py", > + "extra_args": [ > + "-c", branch_config['postrelease_version_bump_config'], note: this means that we must have a postrelease_version_bump_config defined even if it's not enabled or on a branch that supports this.
Comment on attachment 8711061 [details] [diff] [review] version_bump-buildbot-configs-1.diff Review of attachment 8711061 [details] [diff] [review]: ----------------------------------------------------------------- r+ with comment nit ::: mozilla/builder_master.cfg @@ +166,5 @@ > # In other words, if a branch is 'active' and has 'enable_release_promotion', the > # build master will have both CI and release promotion builders for that branch. > for branch in ACTIVE_BRANCHES: > + if BRANCHES[branch].get('enable_release_promotion'): > + # Only Firefox is supporten for now nit: s/supporten/supported/ ::: mozilla/universal_master_sqlite.cfg @@ +155,5 @@ > # In other words, if a branch is 'active' and has 'enable_release_promotion', the > # build master will have both CI and release promotion builders for that branch. > for branch in ACTIVE_BRANCHES: > + if BRANCHES[branch].get('enable_release_promotion'): > + # Only Firefox is supporten for now nit: s/supporten/supported/
Attachment #8711061 - Flags: review?(jlund) → review+
Comment on attachment 8711136 [details] [diff] [review] version_bump-tools-3.diff Review of attachment 8711136 [details] [diff] [review]: ----------------------------------------------------------------- ::: buildfarm/release/release-runner.py @@ +287,5 @@ > > kwargs = { > "public_key": docker_worker_key, > "version": release["version"], > + "next_version": bump_version(release["version"]), I guess this will make more sense when you put up the releasetasks patch but I am assuming this will be a bbb property?
Attachment #8711136 - Flags: review?(jlund) → review+
Comment on attachment 8711128 [details] [diff] [review] version_bump_postrelease-gecko-dev-1.diff Review of attachment 8711128 [details] [diff] [review]: ----------------------------------------------------------------- r+ with comments but no blockers ::: testing/mozharness/scripts/merge_day/gecko_migration.py @@ +109,5 @@ > self.run_sanity_check() > > + def _pre_config_lock(self, rw_config): > + super(GeckoMigration, self)._pre_config_lock(rw_config) > + # override properties from buildbot properties here as defined by ++ @@ +521,5 @@ > self.touch_clobber_file(dirs['abs_to_dir']) > > + def bump_postrelease(self, *args, **kwargs): > + """Bump version""" > + if not self.config["next_version"]: maybe put this in _pre_config_lock so we fail before checking out gecko? @@ +526,5 @@ > + self.fatal("Next version has to be set. Use --next-version or " > + "pass `next_version' via buildbot properties.") > + dirs = self.query_abs_dirs() > + for f in self.config["version_files"]: > + curr_version = ".".join( oh right, I guess this works in the postrelease_beta.py case even though next_version is derived by splitting off 'b' and curr_version is from a split with '.' because we are only interested in the full 'join' of each at this stage. @@ +651,5 @@ > self.run_command(hg + ["diff"], cwd=cwd) > self.hg_commit( > cwd, user=self.config['hg_user'], > + message="Update configs. IGNORE BROKEN CHANGESETS CLOSED TREE NO BUG a=release ba=release", > + ignore_no_changes=self.config.get("ignore_no_changes", False) do we want to know when postrelease bump doesn't make changes?
Attachment #8711128 - Flags: review?(jlund) → review+
(In reply to Jordan Lund (:jlund) from comment #19) > can the version bump job be a TC task instead? We just need commit > privileges correct? Can we use some TC secret _something_? I've chosen the way with less resistance here. :) We can work on replacing it with a TC task once we have a good way to store secrets (ssh key to push to hg.m.o in this case). I'm a bit nervous to make it happen in one step. > so in > https://bugzilla.mozilla.org/page.cgi?id=splinter. > html&bug=1166464&attachment=8711061 you are forcing the product to be > 'firefox' but what if later down the line > branch_config["platforms"][platform]["product_name"] points to something > that is not "firefox"? I'd say ignore the platform value and use the global one for consistency. At least for releases. I scanned config.py, thunderbird_config.py and b2g_config.py and they have the same value for "product_name" regardless of platform. I can move it to GLOBAL_VAR, actually. > This won't be caught with an error and might cause a > footgun later down the line. e.g. we forget to update builder_master.cfg > with more release_promotion products as l10n_release_platforms list gets > extended with say fennec platforms Meh. We don't add/remove products too often. I'd rather have something missing instead of something popping up automagically. > @@ +1818,5 @@ > > bouncer_buildername = "release-{branch}_{product}_bncr_sub".format( > > + branch=branch_name, product=product) > > + # Explicitly define pf using the slave platform (linux64 in this case) > > + bouncer_submitter_factory = makeMHFactory( > > + config=branch_config, pf=branch_config["platforms"]['linux64'], > > whoa, I'm confused. was this a bug before? were we just taking the last `pf` > value in the l10n_release_platforms loop? yeah... makeMHFactory requires pf, but I'm not sure if I want to refactor it now. > > @@ +1840,5 @@ > > } > > builders.append(bouncer_builder) > > > > + version_bump_mh_cfg = { > > + "script_name": "scripts/merge_day/gecko_migration.py", > > so, iiuc, we will be using the release branch's mozharness for this > gecko_migration.py call. This might cause confusion because up till now we > have been using m-c tip for all things gecko_migration.py... Probably I should split this script and move the shared methods to mozharness/something.
(In reply to Jordan Lund (:jlund) from comment #22) > I guess this will make more sense when you put up the releasetasks patch but > I am assuming this will be a bbb property? Yes, it'll be used as a property. The releasetasks patch needs some tests before I publish it.
> > > This won't be caught with an error and might cause a > > footgun later down the line. e.g. we forget to update builder_master.cfg > > with more release_promotion products as l10n_release_platforms list gets > > extended with say fennec platforms > > Meh. We don't add/remove products too often. I'd rather have something > missing instead of something popping up automagically. sure, I guess I meant, in this scenario, it won't be missing. It would still work but pf["product_name"] and `product` could differ and that be bad. I guess it just seemed wrong to use the literal value "firefox" twice: 1) your patch https://bug1166464.bmoattachments.org/attachment.cgi?id=8711061 + for product in ("firefox",): 2) e.g. https://dxr.mozilla.org/build-central/source/buildbot-configs/mozilla/config.py#172 'product_name': 'firefox', maybe not a big deal. I'm okay if you think we can land this as is.
* split gecko migration and postrelase bump * move things around * move some things into "query_*" methods * rename configs * make the configs clearer Jordan, can you take a look at this? No need for in-depth review - I'm going to test this on date first once I land the configs/custom patches. Any better names for the class names?
Attachment #8711128 - Attachment is obsolete: true
Attachment #8711822 - Flags: feedback?(jlund)
(In reply to Rail Aliiev [:rail] from comment #25) > (In reply to Jordan Lund (:jlund) from comment #22) > > I guess this will make more sense when you put up the releasetasks patch but > > I am assuming this will be a bbb property? > > Yes, it'll be used as a property. The releasetasks patch needs some tests > before I publish it. I pushed the current state (no explicit tests yet): https://github.com/mozilla/releasetasks/compare/master...rail:version_bump?expand=1
Just a one line change (the scrip name). I'd probably stick with the initial approach - don't want to add to many levels of indirection. We will refactor this function in any case when we add Fennec support.
Attachment #8711063 - Attachment is obsolete: true
Attachment #8711827 - Flags: review?(jlund)
Nothing new, just typo fixes and config name change.
Attachment #8711061 - Attachment is obsolete: true
Attachment #8711828 - Flags: review?(jlund)
Comment on attachment 8711822 [details] [diff] [review] version_bump_postrelease-gecko-dev.diff Review of attachment 8711822 [details] [diff] [review]: ----------------------------------------------------------------- this looks great and means we can probably use any repo's postrelease_version_bump.py now :) ::: testing/mozharness/mozharness/mozilla/merge.py @@ +4,5 @@ > +from mozharness.base.log import FATAL, INFO > +from mozharness.base.vcs.mercurial import MercurialVCS > + > + > +class GeckoMigrationMixin(object): name seems fine :) ::: testing/mozharness/scripts/release/postrelease_version_bump.py @@ +19,5 @@ > +from mozharness.mozilla.merge import GeckoMigrationMixin > + > + > +# GeckoMigration {{{1 > +class PostReleaseVersionBump(MercurialScript, BuildbotMixin, GeckoMigrationMixin): I wonder if this will be the home script for all things post-release or if we will stay with tools script version..
Attachment #8711822 - Flags: feedback?(jlund) → feedback+
Attachment #8711828 - Flags: review?(jlund) → review+
Attachment #8711827 - Flags: review?(jlund) → review+
Fix for busted reconfigs: https://hg.mozilla.org/build/buildbotcustom/rev/a4bc25f76f2d https://hg.mozilla.org/build/buildbotcustom/rev/a212ce1d8529 I thought we had some checks to make sure category was defined on all builders, could you verify if that is true and it we're somehow not checking release promotion builders ?
Attached patch category.diffSplinter Review
Unify category usage. "release-" is something that makes buildbot prioritize the builders properly.
Attachment #8712673 - Flags: review?(nthomas)
Attached file releasetasks patch
Attachment #8712769 - Flags: review?(jlund)
See Also: → 1243477
Attachment #8712673 - Flags: review?(nthomas) → review+
Attachment #8712769 - Flags: review?(jlund) → review+
Attachment #8712769 - Flags: checked-in+
Attachment #8713209 - Flags: review?(bugspam.Callek) → review+
Attachment #8713209 - Flags: checked-in+
Comment on attachment 8713209 [details] [diff] [review] buildbotcustom-fix-buildername.diff merged to production: https://hg.mozilla.org/build/buildbotcustom/rev/0d2e8be233b4
Finally. \o/ * refactored gecko_migration.py and moved shared code to a mixin * postrelase bump now pulls from https:// and pushes to ssh:// * example TC task: https://tools.taskcluster.net/task-inspector/#Wat4QkIXS_eOjj4M0dO8Lg/ * the corresponding buildbot job: http://buildbot-master94.bb.releng.use1.mozilla.com:8001/builders/release-date-firefox_version_bump/builds/1 * the corresponding bump https://hg.mozilla.org/projects/date/rev/6ee98fbb7ada
Attachment #8714117 - Flags: review?(jlund)
Comment on attachment 8714117 [details] [diff] [review] version_bump_postrelease_v2-gecko-dev.diff interdiff against the f+ed patch: https://gist.github.com/rail/da89c8b1338346f81222
Comment on attachment 8714117 [details] [diff] [review] version_bump_postrelease_v2-gecko-dev.diff Review of attachment 8714117 [details] [diff] [review]: ----------------------------------------------------------------- this looks awesome. ::: testing/mozharness/scripts/merge_day/gecko_migration.py @@ +174,5 @@ > + def query_push_dirs(self): > + dirs = self.query_abs_dirs() > + return dirs.get('abs_from_dir'), dirs.get('abs_to_dir') > + > + def query_push_args(self, cwd): just thinking. should we be explicit and push with '-r' and get in habit? I think hgmo has a magic hook to default to '.' iirc so this might not be so important now.
Attachment #8714117 - Flags: review?(jlund) → review+
Attachment #8714117 - Flags: checked-in+
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
Blocks: 1252085
Removing leave-open keyword from resolved bugs, per :sylvestre.
Keywords: leave-open
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: