542142 - Figure out how to version Lorentz nightlies and milestone builds

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Nick Thomas [:nthomas] (UTC+12)]

Lorentz is a clone of mozilla-1.9.2 for the backport of out-of-process-plugins (OOPP) to the 1.9.2 branch. We need to support providing nightly builds and updates, and weekly betas/milestones with updates, until the work is complete and merged back to m-1.9.2.

Bug 536188 comment #9 has a suggestion for how we might do that. Subject to testing our systems to make sure it'll work, here's a full plan:
* very shortly the version in firefox-lorentz would be 3.6plugin1pre/1.9.2.1pre - I don't think we should track the 3.6.N changes in 1.9.2 as it doesn't add anything
* the first beta/milestone will be 3.6plugin1/1.9.2.Npre where N is whatever we get from the last merge from m-1.9.2. Don't see any value in setting 1.9.2plugin1 for the milestone. They should be on the beta update channel so that they join our existing beta cohort when merged back to 1.9.2, and we don't need to explicitly prepare updates for oopp testers forever more
* rinse and repeat

Outstanding questions
- what branding do we want to use for the beta/milestone builds ? Not the official branding presumabley, but should they be identified as 'Firefox Lorentz Beta N' in the titlebar and elsewhere ? What about the nightly builds ? Are filenames like 'Firefox Setup 3.6plugin1', 'Firefox 3.6plugin1.dmg', 'firefox-3.6plugin1.tar.bz2' ok ?
- how do we prevent branding/versioning changes from landing back in m-1.9.2, bsmedberg says this is easy for the person merging, I'd love to hear more details. It may be possible to write a hook to prevent this too.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

Related to the branding question, should we be naming the project-branch nightly builds so it's obvious what the build is ? Eg tracemonkey, electrolysis etc all look identical to mozilla-central unless you go to about:buildconfig

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

Do we want to update our existing 3.6 beta users onto OOPP betas ?

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

Successfully updated from a fake 3.6plugin1pre nightly to another with small changes to AUS and buildbot configs. Extensions marked compatible with 3.6.* or 3.7a1pre can still be installed.

Comment 4

[Benjamin Smedberg]

> * the first beta/milestone will be 3.6plugin1/1.9.2.Npre where N is whatever we
> get from the last merge from m-1.9.2. Don't see any value in setting
> 1.9.2plugin1 for the milestone. They should be on the beta update channel so
> that they join our existing beta cohort when merged back to 1.9.2, and we don't
> need to explicitly prepare updates for oopp testers forever more

There is one issue with this. 3.6plugin1 is a "smaller" version than 3.6, which is the current version on the beta channel. In order for automatic update to work correctly, I think we should use 3.6.1plugin1 -> 3.6.1pluginN

We haven't decided (yet) to update the beta users to the first Lorentz release:

* How long can we delay that decision?
* Is it harder to update users to 3.6.1plugin3 or whichever milestone we decide is stable enough?
* How hard is it to present the first milestone as a prompted (major) offer instead of a minor offer?

> - how do we prevent branding/versioning changes from landing back in m-1.9.2,
> bsmedberg says this is easy for the person merging, I'd love to hear more
> details. It may be possible to write a hook to prevent this too.

At the final merge point we simply un-do the branding changes before merge, or during the merge process.

Comment 5

[tymerkaev]

Maybe 3.6 Revision 2 (3.6r2pre) at an early stage of development -> 3.6.x pre at final stage of development? ;)

Comment 6

[Chris AtLee [:catlee]]

How much of the branding / versioning could we do in the mozconfig?

Comment 7

[Benjamin Smedberg]

I think version bumps require code changes to milestone.txt at least. The branding changes probably require us to set up the browser/branding/unofficial and browser/branding/nightly code once. robstrong knows more about this than I.

Comment 8

[Robert Strong (they/them - no direct email)]

Setting MOZ_APP_DISPLAYNAME in the appropriate configure.sh should be sufficient.

The value for MOZ_APP_DISPLAYNAME is also in the update string so I don't see much if any value in having plugin or any other string besides what is used currently in the version string and it *might* prevent us from updating to the release / different branch.

The main issue I see with updating users from one branch to another is if files were added on one that need to be removed which would require these files being added to removed-files.in and could easily be overlooked.

Comment 9

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to comment #4)
* We can update 3.6.x beta users to Lorentz milestones at any time, as long as we stick with your scheme of using 3.6.MpluginN for a current 3.6.M release
* No difference that I can think of for doing it to plugin3 vs earlier/later
* Major updates are a simple change on our side, but if you want/need localization of the billboard then pascalc and the localizers need some lead-time (on the order of weeks probably)

This would be in addition to keeping anyone who's on a plugin build updated to the latest milestone.


(In reply to comment #6)
> How much of the branding / versioning could we do in the mozconfig?

I had a thought about branding. We could duplicate browser/branding/unofficial to ..../lorentz and tweak names and so on there. That change would land in m-1.9.2 and merge to firefox-lorentz (f-l), enabling us to say 
  ac_add_options --with-branding=browser/branding/lorentz
in the mozconfig. Don't think we can do anything about the versioning though.


(In reply to comment #8)
> The value for MOZ_APP_DISPLAYNAME is also in the update string 

How does it factor in ? For the 3.6 alphas (with the unofficial Namoroka branding) we still request https://aus2.mozilla.org/update/3/Firefox/3.6a1/...

Not sure how forgetting about removed-files.in will be an issue. Are you thinking of users that might update to a lorentz build but then decide to re-install a 3.6.x ?

Comment 10

[Robert Strong (they/them - no direct email)]

(In reply to comment #9)
> (In reply to comment #8)
> > The value for MOZ_APP_DISPLAYNAME is also in the update string 
> 
> How does it factor in ? For the 3.6 alphas (with the unofficial Namoroka
> branding) we still request https://aus2.mozilla.org/update/3/Firefox/3.6a1/...
damn... I thought it used the codename off the top of my head but you are correct. It would be trivial to override the value via the pref but not sure it is worth it.

> Not sure how forgetting about removed-files.in will be an issue. Are you
> thinking of users that might update to a lorentz build but then decide to
> re-install a 3.6.x ?
I was referring to project branches in general moving back to a main / different branch.

Comment 11

[tymerkaev]

Attachment: Simple Patch for update unofficial branding and versions (correct?) for firefox-lorentz repository (before beta 1)

Maybe this?

Comment 12

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [buildbot-configs] Enable creation of complete mars and stuff snippets into partial generator

This will create the complete mar files, and stuff the snippets describing them into a directory separate from the other branches. The partial update generator picks the latter up automatically, and will promote any en-US lorentz build ahead of l10n (as per the other branches). Patch applies on top of attachment 425482 [details] [diff] [review].

Landing this is blocked on a version bump in the lorentz repo, to 3.6.1plugin1pre or 3.6.3plugin1pre.

Comment 13

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [AUS] Direct nightly updates to correct directory

Mike, welcome to the versioning hilarity that is Lorentz. This will pick off 3.6.MpluginNpre requests (lorentz) before we hit the broader '3.6*' match (mozilla-1.9.2) on the next line.

We may be able to push this live along with bug 544458.

Comment 14

[Nick Thomas [:nthomas] (UTC+12)]

Comments 12 and 13 are for nightly updates. 

I've also done mock 3.6.2plugin1 and 3.6.3plugin2 releases (yeah the versions aren't quite right), which mostly worked fine. We end up with files named like this:
 win32/en-US/Firefox Setup 3.6.3plugin2.exe
 mac/en-US/Firefox 3.6.3plugin2.dmg
 linux-i686/en-US/firefox-3.6.3plugin2.tar.bz2
 update/linux-i686/en-US/firefox-3.6.3plugin2.complete.mar
 update/linux-i686/en-US/firefox-3.6.2plugin1-3.6.3plugin2.partial.mar
 source/firefox-3.6.3plugin2.bundle
 source/firefox-3.6.3plugin2.source.tar.bz2
Obviously I left the official branding on and will defer the actual branding choice to bsmedberg and beltzner.

Issues to fix
* tagging: version bumping when 'plugin' is present in the version (have a patch)
* l10n_verification doesn't run, http://pastebin.mozilla.org/701628, which is probably just something we need to fix in the staging setup (later python version?)
* signing is untested, expecting it'll fail near http://hg.mozilla.org/build/tools/file/tip/release/signing/signing.py#l60 (need to test + patch)
* bouncer setup is untested but it should be fine

Comment 15

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 425742 [details] [diff] [review]
[buildbot-configs] Enable creation of complete mars and stuff snippets into partial generator

http://hg.mozilla.org/build/buildbot-configs/rev/87d504e69427

pm02 reconfig'd.

Comment 16

[Michael Morgan [:morgamic]]

Comment on attachment 425743 [details] [diff] [review]
[AUS] Direct nightly updates to correct directory

This works.  * is analagous to .* and the pattern will match correctly.

Comment 17

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 425743 [details] [diff] [review]
[AUS] Direct nightly updates to correct directory

Checking in config-dist.php;
/cvsroot/mozilla/webtools/aus/xml/inc/config-dist.php,v  <--  config-dist.php
new revision: 1.68; previous revision: 1.67
done

Moved AUS2_PRODUCTION to 1.68 of config-dist.php (also picks up attachment 432606 [details] [diff] [review]), AUS2_RTM_201003181531 applied to all of mozilla/webtools/aus/.

Comment 18

[Nick Thomas [:nthomas] (UTC+12)]

Nightly updates will work once we get a version bump in the repo, which if IIRC should be 3.6.3plugin1pre to be able to update from 3.6.2. Tested that by overriding the update url with yesterdays nightly.

Comment 19

[Benjamin Smedberg]

Do you need me to do anything here, or will you take care of the version bump?

Comment 20

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to comment #19)
> Do you need me to do anything here, or will you take care of the version bump?

Probably easier if you do it, I'll just have to get a review from you. Just browser/config/version.txt and (maybe?) config/milestone.txt and js/src/config/milestone.txt.

Comment 21

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [tools] Support using words when bumping release versions

This worked for me in a test release with version 3.6.3plugin2. The full command was 
perl tools/release/version-bump.pl -w firefox-lorentz -t FIREFOX_3_6_3plugin2_RELEASE -a browser -v 3.6.3plugin2 -m 1.9.2.3pre config/milestone.txt js/src/config/milestone.txt browser/config/version.txt

I wasn't trying to modify the milestone on the assumption that a OOOP build wouldn't get a gecko revision, but it's the same regexp and works if we do want to.

Comment 22

[Benjamin Smedberg]

Bumped lorentz nightly version to 3.6.3plugin1pre

http://hg.mozilla.org/projects/firefox-lorentz/rev/52eee8ac4f91

I didn't change the platform version at all, just the Firefox version.

Comment 23

[tymerkaev]

Maybe you should update branding (Lorentz instead Namoroka)? Any ideas about Lorentz Start Page?

P.S. I think platform should be upgraded because it also backported to 1.9.2.x.

Comment 24

[Benjamin Smedberg]

If you'd like to provide a branding-change patch, I'll review it. I'm not that concerned about it.

What exactly do you think the platform version should be upgraded to? It's currently 1.9.2.3pre, and I don't see any reason to make it 1.9.2.3plugin1pre, since that version number doesn't have any effect on updates.

Comment 25

[tymerkaev]

Attachment: Lorentz branding (unofficial updated)

Comment 26

[Benjamin Smedberg]

Comment on attachment 434515 [details] [diff] [review]
Lorentz branding (unofficial updated)

Gavin, can you approve this? Looks fine to me, but it's been a long time.

I'll set up the web pages if this is acceptable.

Comment 27

[[e-mail] (:bhearsum)]

Comment on attachment 434458 [details] [diff] [review]
[tools] Support using words when bumping release versions

This seems fine. Do we care about the pretty version in the update snippets or anything like that?

Comment 28

[:Gavin Sharp [email: [e-mail]]]

Comment on attachment 434515 [details] [diff] [review]
Lorentz branding (unofficial updated)

Looks fine to me, though what happened to the plan in comment 9?

(Are there any plans for putting pages under http://www.mozilla.org/projects/lorentz/ , or should those links point somewhere else? It's probably also OK for them to just be 404, I guess.)

Comment 29

[Benjamin Smedberg]

sounds complicated and involves buildbot config changes which I'd prefer to avoid for now. And I'll do something simple with the webpages.

Comment 30

[tymerkaev]

(In reply to comment #28)

> (Are there any plans for putting pages under
> http://www.mozilla.org/projects/lorentz/ , or should those links point
> somewhere else? It's probably also OK for them to just be 404, I guess.)

I think this page should be similar to http://www.mozilla.org/projects/devpreview/

Comment 31

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 434458 [details] [diff] [review]
[tools] Support using words when bumping release versions

http://hg.mozilla.org/build/tools/rev/fc5631aaa10a

Comment 32

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [tools] Match on 3.6.XpluginY when signing

This adds a few more tests and tweaks the regular expressions for matching on mar and exe names. 'python tests.py' passes, and signing a 3.6.3plugin2 test release also succeeded (en-US + 2 locales).

Comment 33

[Benjamin Smedberg]

Branding: http://hg.mozilla.org/projects/firefox-lorentz/rev/8cc55bc13291

Comment 34

[tymerkaev]

(In reply to comment #33)
> Branding: http://hg.mozilla.org/projects/firefox-lorentz/rev/8cc55bc13291

Maybe after Lorentz (Fx 3.6.3) will be released, this patch should be applied for 3.6.4pre nightly builds (in mozilla-1.9.2)?

Comment 35

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 434828 [details] [diff] [review]
[tools] Match on 3.6.XpluginY when signing

http://hg.mozilla.org/build/tools/rev/47ad24744db6

Comment 36

[Nick Thomas [:nthomas] (UTC+12)]

Verified nightly updates are working, as is the branding from attachment 434515 [details] [diff] [review]. FWIW, it can't find 
  http://www.mozilla.org/projects/lorentz/
on startup for a new profile. The useragent uses 'Lorentz/3.6.3plugin2pre' which might cause problems with sites that sniff for Firefox.

We also have support in the release automation. --> FIXED.

Comment 37

[tymerkaev]

Attachment: Update link to startup homepage + version bump to 3.6.4plugin2pre

Comment 38

[Benjamin Smedberg]

Azat, the Lorentz branch has merged with 1.9.2 and is closed.

Comment 39

[tymerkaev]

sorry, np