464014 - Mercurial Release Automation needs to properly support version/appVersion differences

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[[e-mail] (:bhearsum)]

From bug 462145:
> In Bootstrap we must have a version, which is the RelEng friendly name of the
> release we're working on, and can have an appVersion if the version used inside
> the app is different. There are matching old vars, eg
>  version       = 3.0rc3
>  appVersion    = 3.0
>  oldVersion    = 3.0rc2
>  oldAppVersion = 3.0
>
> The vars in this class seem to match up with that, but the config uses
> appVersion as version. I'd like to resolve that confusion before the rc's
> come along.

So, the configuration files definitely need to be fixed up. We should also take a pass at things that care about them (patcher, updates) and make sure everything is using them properly.

Comment 1

[[e-mail] (:bhearsum)]

Not sure how this got into Custom Builds.

Comment 2

[[e-mail] (:bhearsum)]

Okay, I'm going to post a patch for this in a few minutes. I did a full test run (3.1b2 -> 3.1rc1) to give this a shot. Obviously this isn't a 100% complete test (oldAppVersion is not tested at all) - but it should catch most things. I imagine we'll do a fake 3.1rc2 before the real thing, and same thing goes for 3.1.1 - if there's follow up issues we can catch those there. I don't think it's worthwhile to do testruns of those right now. I fear they will be too different from the real thing to do useful.

With that said, these are pretty simple patches as patcher2.pl and the patcher/update verify config bump scripts already support this for the most part. There's a key difference in this implementation that should be noted:
For Firefox 3 RCs MARs did *not* contain the rcN in the filename. I couldn't really figure out why, but suspect it's just because Tinderbox didn't do it by default. I took the liberty of changing this in for 3.1 releases for the sake of consistency.

As with the rest of the final deliverables we'll want to rename these when do the final 3.1 release.

That's pretty much it for things I wanted to mention up front. I'll have some more specific notes about the patches & test run when I post them.

Comment 3

[[e-mail] (:bhearsum)]

Attachment: support version/appVersion - buildbotcustom

Okay, so this patch gets our BuildFactorys accepting the appropriate version/appVersion options. Here's a summary:
* ReleaseBuildFactory/ReleaseRepackFactory/SingleSourceFactory uses version as MOZ_PKG_VERSION, to make sure filenames and the candidates dir have the 'rc' part on them
* ReleaseTaggingFactory uses version for the commit message, and appVersion for the actual version bumping (eg, browser/config/version.txt)
* ReleaseUpdatesFactory passes version, appVersion, and oldVersion onto the patcher config bump script (who's code was tested in 3.0 RCs and again here)
* L10nVerifyFactory uses version to pull builds
* UpdateVerify didn't need any buildbotcustom side changes, but we started passing the correct variables (no more version=appVersion!)

I did a full test run, which ended up passing. There's still some cruft in the logs if you have a look on staging-master, but it's all explainable stuff:
* update_verify fails all over the place because I did a test run with only ar, de, en-US, and zh-TW. Because of how we bump that config file we just have to deal with this. (We should probably fix it, but I'm not taking that on as part of this.)
* Lots of spurious failure for update_verify/l10nverify on Mac - this is because Buildbot had been restarted over ssh, causing hdiutil to barf.

Overall, I'm confident this works as intended.

Comment 4

[[e-mail] (:bhearsum)]

Attachment: support version/appVersion - master side

This one is dead simple. Just adds the new parameters and passes them appropriately. I'll be updating the docs for this shortly.

Comment 5

[[e-mail] (:bhearsum)]

Attachment: support version/appVersion - fix bump scripts

Minor, minor bugfixes here. patcher-config-bump.pl had a typo which caused appVersion not to be interpreted properly. update-verify-bump.pl needed an update to get linux builds to use version (rather than appVersion) in candidate directories, when pretty names are on.

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: support version/appVersion - buildbotcustom (unrotted)

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: support version/appVersion - master side (unrotted)

This looks good for 3.1 rc1. Before rc2 we'll need to pass in oldAppVersion to UpdateVerifyFactory rather than falling back to oldVersion - I think it'll try to query AUS with a version of 3.1rc1 otherwise.

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

Re the unrotting (penance for the tardiness reviewing the patches), it's mostly a merge of the changes with the tip and fixed one typo in a comment. Do take a look for do double check though. The collisions were at the top of ReleaseBuildFactory and ReleaseUpdatesFactory in process/factory.py.

When do you think we should land this relative to 3.1b3 ?

Comment 9

[[e-mail] (:bhearsum)]

The interdiffs look fine to me. Since you're on the hook for 3.1b3 I'll leave it up to you as to when to land. For the sake of closing this bug it would be nice to do it before b3, but if you'd rather wait until afterwardness for paranoia's sake, that's cool.

Comment 10

[Nick Thomas [:nthomas] (UTC+12)]

I'll handle landing this after 3.1b3, because I'm a wuss.

Comment 12

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: support version/appVersion - buildbotcustom (unrotted again)

Updated patch to account for other changes in buildbotcustom. Also fixes this regression from bug 478229.
-            self.env['ZIP_IN'] = WithProperties('%(srcdir)s/' + self.project)
+            self.env['ZIP_IN'] = WithProperties('%(srcdir)s/' + filename)

Going to take this patch, and the one that follows for buildbot-configs, for a spin on mozilla-staging as part of bug 481469.

Comment 13

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: support version/appVersion - master side (unrotted again)

Updated patch for other changes. Also does s/appVersion/version/ in the FtpPoller change source for the signing log (bug 473415).

Comment 14

[Nick Thomas [:nthomas] (UTC+12)]

Hit bug 486076 with a staging run with these changes.

Comment 15

[Nick Thomas [:nthomas] (UTC+12)]

Attachment 370121 [details] [diff] and attachment 370128 [details] [diff] [review] worked fine in a mock 3.5rc1 release (some of the context is slightly out of date now). It didn't run right through because of staging setup issues with the preceeding mock 3.5b4 release. And I also ran into bug 486444, which mucked up win32 update verify runs.

Attachment 357833 [details] [diff] I forgot to test but I think that's OK. The patcher-config-bump.pl change is only the command line option, while update-verify-bump.pl is only exercised for rc2 type releases. We need to do a mock run in this case anyway.

Comment 16

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 370128 [details] [diff] [review]
support version/appVersion - master side (unrotted again)

Carrying my original review over. 

committed changeset 1075:f769b81ee738

Comment 17

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 370121 [details] [diff] [review]
support version/appVersion - buildbotcustom (unrotted again)

Carrying my original review over.

committed changeset 241:ea76a9d5a770

Comment 18

[Nick Thomas [:nthomas] (UTC+12)]

production-master:moz2-master reconfig'd with this, at long last.