Closed Bug 908134 Opened 11 years ago Closed 9 years ago

Figure out how to offer release build to beta users

Categories

(Release Engineering :: Release Automation: Other, defect)

x86
All
defect
Not set
major

Tracking

(firefox24-, firefox25-, firefox26- wontfix, firefox27+ affected, firefox28 affected, firefox29 fixed)

RESOLVED FIXED
Tracking Status
firefox24 - ---
firefox25 - ---
firefox26 - wontfix
firefox27 + affected
firefox28 --- affected
firefox29 --- fixed

People

(Reporter: nthomas, Assigned: nthomas)

References

Details

(Whiteboard: [leave open])

Attachments

(11 files, 7 obsolete files)

7.31 KB, text/x-python-script
Details
3.34 KB, text/plain
Details
77.47 KB, application/x-bzip2
Details
14.22 KB, text/x-python-script
Details
614.17 KB, application/x-bzip2
Details
2.30 KB, patch
bhearsum
: review+
Details | Diff | Splinter Review
14.62 KB, text/x-python-script
Details
441.62 KB, patch
Details | Diff | Splinter Review
10.07 KB, patch
catlee
: review+
Details | Diff | Splinter Review
33.19 KB, patch
Details | Diff | Splinter Review
39.09 KB, patch
Details | Diff | Splinter Review
The post-mortem for Firefox 23.0 discussed how we didn't detect the Radeon crash (bug 902349) prior to shipping, due to the small audience testing it. We want to try updating the beta audience the release build a few days before the release day, giving us a lot more users to submit crash reports. If needed we would spin another build.

I've done some quick testing on Windows 8:
* Updating from 23.0b10 to 23.0.1 works fine
* Updating onwards to 24.0b4, using the service, fails
 AUS:SVC readStatusFile - status: failed: 22, path: C:\Users\mozilla\AppData\Local\Mozilla\Firefox\Mozilla Firefox Beta Test\updates\0\update.status
which toolkit/mozapps/update/common/errors.h says is MAR_CHANNEL_MISMATCH_ERROR.

Looks like this happens because the complete.mar for 23.0.1 changes a line in update-settings.ini from
  ACCEPTED_MAR_CHANNEL_IDS=firefox-mozilla-beta,firefox-mozilla-release
to
  ACCEPTED_MAR_CHANNEL_IDS=firefox-mozilla-release
and the 24.0b4 mar is tagged with firefox-mozilla-beta.

So perhaps we should be excluding update-settings.ini from mar files, like we do for channel-prefs.js. bbondy/rstrong, what do you think of that ? Would it have any impact on the security we built in around the maintenance service ?
Supposing we solve that problem, RelEng will also need to modify the release automation to generate snippets for the beta channel.
The signing can be removed, the product information can be refreshed, and the mar can be resigned using the mar binary.
http://mxr.mozilla.org/mozilla-central/source/modules/libmar/tool/mar.c#36

The update-settings.ini file should be fine as is since it lists both release and beta.

The channel is preprocessed into nsUpdateService.js as a fallback so we won't orphan users in case the channel pref from the channel-prefs.js file isn't read in for any reason. This will break applying partials on beta unless it is updated as part of a new repackaging process.

Since the channel-prefs.js file has gotten screwed up in the past (bug 756325) bug 900251 will add a new instruction to re-add it back if it is missing. The worst I think could happen in this case is a beta user would be moved to release if they are missing the channel-prefs.js file. It should be possible to update this as part of a new repackaging process.
caveat emptor, the above might not include all other issues.
It would be ideal to have this tested and in place in time for our last FF24 Desktop Beta - I'll set up a meeting and loop in security so we can evaluate the feasibility of doing this and getting it tested thoroughly by QA as well.
Curtis - are you the best person to attend for security review/assessment of this change or do you have another person to recommend?
Flags: needinfo?(curtisk)
(In reply to Robert Strong [:rstrong] (do not email) from comment #2)
> The signing can be removed, the product information can be refreshed, and
> the mar can be resigned using the mar binary.
> http://mxr.mozilla.org/mozilla-central/source/modules/libmar/tool/mar.c#36
> 
> The update-settings.ini file should be fine as is since it lists both
> release and beta.
Actually not true, I had only looked at beta and while with beta it has both release and beta release only lists release. We might be able to get away with listing both as is done in beta.
(In reply to lsblakk@mozilla.com [:lsblakk] from comment #5)
> Curtis - are you the best person to attend for security review/assessment of
> this change or do you have another person to recommend?

That depends really, are you all looking for someone to review the code involved or are we just talking through the process? Or do we need a larger review of the architecture?
Flags: needinfo?(curtisk)
Flags: needinfo?(lsblakk)
(In reply to Robert Strong [:rstrong] (do not email) from comment #6)
> Actually not true, I had only looked at beta and while with beta it
> has both release and beta release only lists release. We might be
> able to get away with listing both as is done in beta.

No we can't, that's an important factor in our defense against downgrade attacks. Let's say we're in week 4 of the cycle--we've already patched a bunch of security bugs and anyone watching the Beta channel repo can see what we've patched. We don't want attackers to be able to "upgrade" Release users to the first 24 beta build that is still vulnerable to those bugs.

I like nthomas's suggestion of letting people keep the list of acceptable mar channels they initially had. If people were originally on beta they should still be on the beta channel and get beta updates even if they happen to have a release build installed.
Flags: sec-review?(dveditz)
Flags: needinfo?(lsblakk)
(In reply to Daniel Veditz [:dveditz] from comment #8)
> (In reply to Robert Strong [:rstrong] (do not email) from comment #6)
> > Actually not true, I had only looked at beta and while with beta it
> > has both release and beta release only lists release. We might be
> > able to get away with listing both as is done in beta.
> 
> No we can't, that's an important factor in our defense against downgrade
> attacks. Let's say we're in week 4 of the cycle--we've already patched a
> bunch of security bugs and anyone watching the Beta channel repo can see
> what we've patched. We don't want attackers to be able to "upgrade" Release
> users to the first 24 beta build that is still vulnerable to those bugs.
For clarity, it would be an upgrade attack since it would be going from release (N) to beta (N+1). The downgrade attack would be going from beta (N) to release (N-1) which was ok per the security review discussion.

> I like nthomas's suggestion of letting people keep the list of acceptable
> mar channels they initially had. If people were originally on beta they
> should still be on the beta channel and get beta updates even if they happen
> to have a release build installed.
So just to verify, are you still ok with beta having both beta and release? The reason this was implemented this way and accepted during the security review is specifically due to needing to be able to apply a release build to beta. We *might* be able to come up with a different process but there would definitely need to be additional work when creating the mar files which might not be needed anyways.
Flags: needinfo?(dveditz)
(In reply to Robert Strong [:rstrong] (do not email) from comment #9)
> (In reply to Daniel Veditz [:dveditz] from comment #8)
> > I like nthomas's suggestion of letting people keep the list of acceptable
> > mar channels they initially had. If people were originally on beta they
> > should still be on the beta channel and get beta updates even if they happen
> > to have a release build installed.
> So just to verify, are you still ok with beta having both beta and release?
> The reason this was implemented this way and accepted during the security
> review is specifically due to needing to be able to apply a release build to
> beta. We *might* be able to come up with a different process but there would
> definitely need to be additional work when creating the mar files which
> might not be needed anyways.
s/might not be/might be/ needed anyways
(In reply to Daniel Veditz [:dveditz] from comment #8)
> I like nthomas's suggestion of letting people keep the list of acceptable
> mar channels they initially had. If people were originally on beta they
> should still be on the beta channel and get beta updates even if they happen
> to have a release build installed.

We just need to have those installations still send the "beta" channel in their crashes (and probably also their update requests), just like they did back when we had this working in the past. We need to get the info in crashes (and I guess also ADI) that people are using this build on the beta channel to be able to diagnose stability correctly.
Confirmed this works (Win 8)
* Install 23.0b10 en-US
* set app.update.url.override to http://people.mozilla.com/~nthomas/update-test/23.0.xml
* Help > About to initiate update to 23.0 --> successful

* reset app.update.url.override, check for update --> we're still on beta:
AUS:SVC Checker:checkForUpdates - sending request to: https://aus3.mozilla.org/update/3/Firefox/23.0/20130730113002/WINNT_x86-msvc/en-US/beta/Windows_NT%206.2.0.0%20(x64)/default/default/update.xml?force=1
* install crashme extension (https://code.google.com/p/crashme/downloads/list), generate crash --> it has 'Release Channel: beta'
https://crash-stats.mozilla.com/report/index/d036ee27-1f8f-4ecf-a364-1aea52130822

* change <install dir>\update-setting.ini (eg by moving to desktop, edit, move back), so it has this again
    ACCEPTED_MAR_CHANNEL_IDS=firefox-mozilla-beta,firefox-mozilla-release
* set app.update.url.override to http://people.mozilla.com/~nthomas/update-test/24.0b4.xml
* Help > About to initiate update to 24.0b4 --> successful

* reset app.update.url.override, check for update --> we're still on beta:
[10:50:03.937] AUS:SVC Checker:checkForUpdates - sending request to: https://aus3.mozilla.org/update/3/Firefox/24.0/20130819170952/WINNT_x86-msvc/en-US/beta/Windows_NT%206.2.0.0%20(x64)/default/default/update.xml?force=1
* initiate a crash --> it has 'Release Channel: beta'
https://crash-stats.mozilla.com/report/index/421bfac8-dbef-4bf2-a790-272b22130822
Nick, that's awesome.  I see Dan is flagged twice, it looks like we still need to do:

* Security review (dveditz)
* full OS testing on all OSes - adding QA wanted

ni? on nthomas to see if the steps in comment 12 could be run on other OS if custom testing snippets exist and if not can you generate some and put them up for QA to test against?
Flags: needinfo?(nthomas)
Keywords: qawanted
I've set up snippets for the other platforms. QA please follow comment #12, with the change that app.update.url.override should be set to the appropriate snippet in http://people.mozilla.com/~nthomas/update-test/. 

eg for mac use 
 http://people.mozilla.com/~nthomas/update-test/23.0-mac.xml
 http://people.mozilla.com/~nthomas/update-test/24.0b4-mac.xml

Please note that the windows snippets have changed url, they end -win.xml now.

We think there might be problems with mac & app signing, where signing is going to be invalidated by changing update-settings.ini. Please check for that. 

We'd probably have to exclude MacOS/update-setting.ini from signing in http://hg.mozilla.org/mozilla-central/file/default/browser/app/macbuild/Contents/_CodeSignature/CodeResources to work around that. Something else for the security folks to consider.
Flags: needinfo?(nthomas)
Windows XP SP2
--------------
1. Install Firefox 23.0b10 20130729175331 en-US
2. Set app.update.url.override to http://people.mozilla.com/~nthomas/update-test/23.0-win.xml
3. Check for updates via the About dialog
> Update downloaded and installed
4. Restart Firefox and check the app version in about:support
> Firefox 23.0 20130730113002 en-US
5. Reset app.update.url.override and set app.update.log to true
6. Check for updates and verify the update channel in the AUS URL
> https://aus3.mozilla.org/update/3/Firefox/23.0/20130730113002/WINNT_x86-msvc/en-US/beta/Windows_NT%205.1.3.0%20(x86)/default/default/update.xml
7. Install the Crashme extension, trigger a crash, and restart Firefox
8. Load the crash report from about:crashes and check the Release Channel in the report
> Release Channel: beta
> https://crash-stats.mozilla.com/report/index/dc562890-f713-442d-8955-46ed22130828
9. Edit <install_dir>/update-settings.ini to have:
ACCEPTED_MAR_CHANNEL_IDS=firefox-mozilla-beta,firefox-mozilla-release
10. Start Firefox and set app.update.url.override to http://people.mozilla.com/~nthomas/update-test/24.0b4-win.xml
11. Check for and install any found updates
> Update found and installed
12. Check about:support for the app version
> Firefox 24.0 20130819170952
13. Reset app.update.url.override and check for updates, verifying the AUS update channel
> Update channel is beta
> https://aus3.mozilla.org/update/3/Firefox/24.0/20130819170952/WINNT_x86-msvc/en-US/beta/Windows_NT%205.1.3.0%20(x86)/default/default/update.xml?force=1
Note: partial update to 24.0b6 is served at this point
14. Trigger a crash using the Crashme extension
15. Load the crash report from about:crashes and verify the Release Channel
> Release Channel: beta
> https://crash-stats.mozilla.com/report/index/fabebec3-dfe7-45cc-b791-e20732130828

Note: partial update to Firefox 24.0b6 was found and installed successfully during step 13. I think this is expected but can you please confirm?

I repeated this test using Firefox 23.0b10 de and it went the same except that during step 3 I was updated to Firefox 23.0 en-US. I would anticipate we want to keep users on their locale, no?

I will wait for feedback on this before I move on to testing Linux and Mac OSX.
QA Contact: bhearsum → anthony.s.hughes
(In reply to Anthony Hughes, Mozilla QA (:ashughes) from comment #15)
> I repeated this test using Firefox 23.0b10 de and it went the same except
> that during step 3 I was updated to Firefox 23.0 en-US. I would anticipate
> we want to keep users on their locale, no?

Definitely. But going to en-US might just have been because of how this testing is set up, i.e. that http://people.mozilla.com/~nthomas/update-test/23.0-win.xml points to an en-US complete update.
(In reply to Robert Kaiser (:kairo@mozilla.com) from comment #16)
> http://people.mozilla.com/~nthomas/update-test/23.0-win.xml points to an
> en-US complete update.

Yeah, I think that's fair. I will continue testing Mac OSX and Linux keeping that in mind. We'll want to make sure that locales are being kept on update once we get closer to the real thing.
That's right, I forgot to call out that this was en-US only for testing.
Linux 32-bit and Linux 64-bit matched the results of comment 15.
Now testing Mac OSX.
Mac OSX 10.7 matched the results of comment 15. I did not encounter any "signing" issues with this test. Is there something I need to do differently to make sure signing is okay?
Keywords: qawanted
I looked into mac signing a little bit. Modifying Contents/MacOS/update-settings.ini does invalidate the signing:
 $ codesign -v Firefox.app/
 Firefox.app/: a sealed resource is missing or invalid
 In architecture: i386
but the app still launches fine. Dunno why that is the case, any idea Ben ?

In other news, the updater only checks the channel-id on the mar file on windows at the moment, so technically we don't have to worry about update-settings.ini on other platforms. We should anyway for consistency, and all platforms will check in the future (bug 903135).


Given QA's testing, I suspect the changes we'd need in mozilla-central & co would be
* not change update-settings.ini in an update - add an exclude for update-settings.ini to list_files() in tools/update-packaging/common.sh (plus appropriate other changes in this dir)
* avoid issues with signing of mac builds - add an exclusion for update-settings.ini to browser/app/macbuild/Contents/_CodeSignature/CodeResources

Then RelEng would have figure out how to add automation support for this, which might be non-trivial given we're talking about mixing together things that normally stay quite separate. Plus lots and lots of testing.
Flags: needinfo?(bhearsum)
(In reply to Robert Strong [:rstrong] (do not email) from comment #9)
> (In reply to Daniel Veditz [:dveditz] from comment #8)
> > We don't want attackers to be able to "upgrade" Release users
> > to the first 24 beta build that is still vulnerable
> For clarity, it would be an upgrade attack since it would be
> going from release (N) to beta (N+1).

I stand by my scare quotes. It's numerically higher so it looks like an upgrade (and is, in terms of features) but in security terms it's a chronological step backwards. It doesn't matter what we call it, we don't want it to happen.

> The downgrade attack would be going from beta (N) to
> release (N-1) which was ok per the security review discussion.

No, we agreed to allow Beta N to Release N. Version numbers must always be the same or higher. The code in the method ArchiveReader::VerifyProductInformation appears to match what I think ought to happen.
https://mxr.mozilla.org/mozilla-central/source/toolkit/mozapps/update/updater/archivereader.cpp#191

> So just to verify, are you still ok with beta having both beta
> and release? The reason this was implemented this way and
> accepted during the security review is specifically due to
> needing to be able to apply a release build to beta.

Yes, this is why beta is an exception and is allowed to have two channel ids. Because of the numeric constraints beta users are only allowed to be upgraded to the Release build with the same number, that is, a future Release, not the current N-1 Release available during the Beta period.
Flags: needinfo?(dveditz)
(In reply to Nick Thomas [:nthomas] from comment #21)
> I looked into mac signing a little bit. Modifying
> Contents/MacOS/update-settings.ini does invalidate the signing:

I'd hope so!

> but the app still launches fine.

That doesn't give me much confidence in Mac signing. Windows executables run fine without being signed though, so maybe broken just equals unsigned.

> In other news, the updater only checks the channel-id on the mar
> file on windows at the moment, 

Yes, the channel and version checks are triggered as part of the mar signature verification. If the mar isn't signed you have no confidence it's not lying about the version and channel, and it could be a trojanized hacked copy of Firefox anyway. Checking doesn't buy you anything.

sec-review+ on the plan described in this bug to allow Firefox Beta users to be upgraded to the final Release of the same version without breaking future updates or getting the user stuck on Release.
Flags: sec-review?(dveditz) → sec-review+
(In reply to Nick Thomas [:nthomas] from comment #21)
> I looked into mac signing a little bit. Modifying
> Contents/MacOS/update-settings.ini does invalidate the signing:
>  $ codesign -v Firefox.app/
>  Firefox.app/: a sealed resource is missing or invalid
>  In architecture: i386
> but the app still launches fine. Dunno why that is the case, any idea Ben ?

If you ever used the "right click+open" way of running Firefox (to bypass an invalid signature), I'm pretty sure this gets cached forever. I think it's worth retesting after resetting your state as described in https://bugzilla.mozilla.org/show_bug.cgi?id=752613#c20.
Flags: needinfo?(bhearsum)
What's left to do here?
Flags: needinfo?(nthomas)
It's on me to do create a full test using staging, using the in-tree changes outlined above.
Assignee: nobody → nthomas
Flags: needinfo?(nthomas)
Severity: normal → major
This is taking longer in staging than expected, and I can't delay my other higher priority work any longer, so I'm setting it aside for now.

Status dump:
I'll attach the patch I was going to use for mozilla-release. The goal was to rebuild the last release with that (all locales/platforms etc), and create some snippets for 
  last beta of last branch --> rebuilt release --> latest beta of current branch
for a complete update only; then ask QA to hammer on it. There are edge cases, like not serving an update to any locale which is in beta but not release. The snippet generation/patcher config/update verify config work in the release automation is unscoped.
Assignee: nthomas → nobody
Attached patch mozilla-release patch (obsolete) — Splinter Review
This has been targeted for FF26 on Beta at the earliest.
(In reply to Alex Keybl [:akeybl] from comment #29)
> This has been targeted for FF26 on Beta at the earliest.

Resetting QA Contact since this falls within Juan's milestone.
QA Contact: anthony.s.hughes → jbecerra
This back on my plate. A natural time to do this is just after we merge for 26.0b1 because all the repos are setup and modified as needed, making them easy to clone and patch as described above.
Assignee: nobody → nthomas
Attached patch mozilla-release patch, v2 (obsolete) — Splinter Review
Drops the mac signing change - if we're going to leave update-settings.ini unchanged during updates it can remain in the inputs for signing.
Attachment #806981 - Attachment is obsolete: true
Progress update - a script has been developed to generate updates for large QA test. It repacks existing complete updates to remove update-settings.ini, and creates partial updates where we don't have them (eg, 25.0b12 -> 25.0; I want to aim for this in the automation, given we want the beta population on the release build asap). I've verified the updates work for mac & windows en-US.

The script is cranking through all the combinations of platform, locale, and firefox versions. I'll publish the results during the Friday afternoon Pacific time.
This is for a one-off repack of complete mar files, plus generating extra partials, and the update snippets to point at them. We'd do something completely different to automate this.
Depends on: 937672
I have gone through the update scenarios described via email, and here are the results:

https://etherpad.mozilla.org/updates-brbtest-908134

There was a problem with offering updates to 32bit Fx installations in Mac OS X, but that was resolved server side.
Removing tracking for 26 based on the timeline and impending holidays.  We'll aim for FF27 to put this into place for the first time.  January 27th will be the release build candidate we'll want to put our beta users onto (nd then off of).
Ben, Rail, and I talked about how to modify the release automation in person last week.

Design spec:
* a week before we ship from mozilla-release we merge from mozilla-beta, but we don't do a beta build. We do a release build and put it on the beta channel. I'm anticipating QA will check updates and functional tests first, so we can get the build out on beta ASAP. We should have a partial update from the last beta to help with rapid uptake. We would serve updates out of firefox/candidates/XX.0buildY/updates/, so that we don't have to push into firefox/releases/ earlier than we want
* if we need to respin the release build, then the same applies. ie we want to get beta folks over asap
* we still push to mirrors roughly a day before release, QA checks releasetest with release builds etc
* come release day we enable updates on the release channel

Any comments/corrections on that lsblakk ?


Here's the implementation plan (subject to scheduling a meeting with reality and getting along ok). The basic idea is to modify the updates builder for the release build (but not try to use any existing builders for beta releases). We would bump the release patcher config as normal, then bump the beta patcher config to add the new release. We would change the update verify bump to take read both the beta and release patcher configs and test all the variations. We would generate snippets for the beta channel separately, so that they can be pushed live independently. 

Even lower level detail:
* we need to have the right 'from' value when bumping patcher configs, so probably need to explicitly set this on ship-it for both the last release and beta, then pass this through the release config
* we should try for names like '26.0-build1' instead of '26.0' in the patcher configs, so that we support respins automatically. This also means we can offer a partial update for anyone who's downloaded build1 from ftp.m.o/some random site, and deprecate our current snippet munging
* we still need a list of older builds to generate partials at compile time, which may be constructed as lastRelease + lastBeta + listOfOtherPartials in ship-it
* we will need to specify the name of the beta patcher config, and possibly some boolean to enable mixTheStreams mode
* to serve from firefox/candidates/NN.0-buildM-candidates/ we'll need to set up bouncer,; probably only need a releasetest channel. Since the complete updates are shared between beta and release, we either need to be able to change a location from using canididates/ to releases/, or us a different product for the beta channel updates
Flags: needinfo?(lsblakk)
(In reply to Nick Thomas [:nthomas] from comment #38)
> * we still need a list of older builds to generate partials at compile time,
> which may be constructed as lastRelease + lastBeta + listOfOtherPartials in
> ship-it

I'm not sure if we'll be able to or not, but we should try to update the suggestions logic on Ship It to suggest releases for beta releases and recent betas for releases. This may end up being more difficult than is worthwhile.

> * we will need to specify the name of the beta patcher config, and possibly
> some boolean to enable mixTheStreams mode

Rail and I talked about this a bit this morning and I think we came up with a pretty clean way of doing this. We could get rid of "patcherConfig" and "verifyConfigs" in the release configs and replace them with a single "updatesConfigs" which would have the following structure:
releaseConfig["updateConfigs"] = {
  "mozRelease-branch-patcher2.cfg": {
    "linux": "mozRelease-firefox-linux.cfg",
    ...
  },
  "mozBeta-branch-patcher2.cfg": {
    "linux": "mozBeta-firefox-linux.cfg",
    ...
  },
}

ReleaseUpdatesFactory would be modified to loop over the keys of updateConfigs and:
1) update any patcher configs found there
2) update the associated verify configs

chunked-verify.py would need to be updated to iterate over updateConfigs and chunk each one for the platform its currently running on.

Nick, this is a little bit different than what we talked about last week (where we'd pluralize patcherConfig -> patcherConfigs and put all of the betas+releases in the mozRelease update verify configs). Either way should be fine, but what Rail and I talked about today gets rid of most of the stream crossing, I think. I'm not fussy, I just wanted to put the idea out there.
Comment 38 looks good to me - anything we should know about the processs for creating the first beta from version.next?
Flags: needinfo?(lsblakk)
I think that's just making sure the release build is set as the 'previous' build, and an appropriate set of partials is selected in ship-it.

One specific exception to the plan above: we can't use it for a point release once we've shipped a beta, because the updater doesn't allow version downgrades. ie if we've shipped 27.0b1 to beta users, then giving them a 26.0.1 is not possible. Since point releases are normally urgent chemspills we probably wouldn't be a big deal.


Other things Ben + Rail + I also talked about:
* making update verify smarter (caching files, caching checks on file sizes & hashes), to speed up verification a bit and make life much easier for stage.m.o
* once we move to Balrog we'll need to look at changing the channel names, to avoid confusion on betatest/releasetest. That's because AUS2 uses point to point mapping, and we can use buildID to tell between beta and release builds, but Balrog would have a rule Product=Firefox AND Channel=betatest mapping to either a beta or a release
This preserves the accepted channel id's from install time, same as we do for the channel in channel-prefs.js.

I'd like to land this on inbound soon, then wait a couple of days on central before uplifting to aurora & beta.
Attachment #828868 - Attachment is obsolete: true
Attachment #8360189 - Flags: review?(bhearsum)
Attachment #8360189 - Flags: review?(bhearsum) → review+
Just to make sure this is written down somewhere - it came up in the channel meeting yesterday to ask about what the potential scenarios are if we should happen to need a beta11 (or more) after we create the RC from mozilla-release and push its snippets to the beta channel.  An explanation of what our options are would be great so we know what to do if the use case actually happens on a particular release.
The short answer is that there are no more betas after you do a release build, you do a build2 (or N+1) for the release build and that also goes to the beta users. I will expect the Spanish Inquisition at the next channel meeting.
Comment on attachment 8360189 [details] [diff] [review]
Exclude update-settings.ini when creating mar files

https://hg.mozilla.org/integration/mozilla-inbound/rev/4b71e71c27a5
Attachment #8360189 - Flags: checked-in+
Setting status-firefox29 to fixed for the attachment just landed, not expecting any further changes in-tree.
Whiteboard: [leave open]
still needs to land in mozilla-beta, right?
(In reply to Chris AtLee [:catlee] from comment #47)
> still needs to land in mozilla-beta, right?

Yes:

(In reply to Nick Thomas [:nthomas] from comment #42)
> I'd like to land this on inbound soon, then wait a couple of days on central
> before uplifting to aurora & beta.
Comment on attachment 8360189 [details] [diff] [review]
Exclude update-settings.ini when creating mar files

Backed out as a precautionary measure:
https://hg.mozilla.org/mozilla-central/rev/57317659a609

In complete updates, the precomplete file is being processed first. That's generated at http://hg.mozilla.org/mozilla-central/file/57317659a609/config/createprecomplete.py#l23, and it doesn't have an exclude for update-settings.ini, so the file gets removed. Fortunately this doesn't break updates because the updater carries on if the file is missing http://mxr.mozilla.org/mozilla-central/source/toolkit/mozapps/update/updater/updater.cpp#2216. I want to have a nightly without this issue for the start of the week, so I backed out.
Attachment #8360189 - Flags: checked-in+ → checked-in-
Updates are disabled for desktop on nightly channel until the backout cycles though.
Since the update-settings.ini is included with complete updates the precomplete file also includes it. If for whatever reason (we've had 'whatever' reasons in the past) the update-settings.ini file is deleted we won't be able to add it back unlike the update-channel (bug 900251).
Are you going to update the update-settings.ini for these builds?
I don't follow why we can't re-add update-settings.ini back in, could you elaborate ? With attachment 8360189 [details] [diff] [review] backed out, I think there will be ADD instruction in the complete (and also the first partial). That gets nightly users back to what they had. In testing on Windows 8 this worked:
 * Jan 16 (prelanding) to Jan 18 (post-landing) via complete (update-settings.ini now missing)
 * then on to Jan 19 via partial (no id mismatch error seen)
So I think users will be able to get an update that fixes them up.
Quite possibly but to verify I have cc'd and needinfo'd bbondy.

My concern is that I don't want to add more one-off's where these files aren't included in updates as we currently do for the channel-prefs.js file. If it is temporary sure... just not for the long term. Especially when it should be possible to migrate users using a one-off mar that only affects these users vs. the entire user base.
Flags: needinfo?(netzen)
If there is no update-settings.ini file no MAR restriction will be made. Those MAR restriction checks are made before the MAR is applied.  So as far as I can tell, you can add an update-settings.ini to an install that doesn't have one without a problem. 

If applying to an install dir without an update-settings.ini.  The MAR that contains the added update-settings.ini will not have a MAR channel check done on it.  
The next update after that will need to be served from MARs with a matching channel ID.

Disclaimer: This was never designed to be a use case to support, so please test everything thoroughly just in case.
Flags: needinfo?(netzen)
Thanks for the info. We got nightlies on the backout rev (57317659a609) so I'll test updates up to that, and fake something onwards to make sure we're ok.
Tested Windows 8 with maintenance service, Win XP without, and Mac. Updating from a Jan 19 nightly (without update-settings.ini) via partial to Jan 20 adds the ini file back. You can then reapply the Jan 20 complete update without errors. 

--> updates re-enabled for nightly channel

For posterity - on mozilla-central, up to Jan 16 and from Jan 20 nightlies included update-settings.ini; the ones for 17th through 19th didn't.
Putting aside the miscue in mozilla-central, doing this permanently is the idea. To recap - we not looking to move users in a once-off case; we want to give beta users the same bits as the release builders, but a week before the general release. 

I see what you're saying about modifying the mar files generated for the release channel, but it doesn't scale very well by the time you have 4 platforms with 85+ locales each - that's quite a lot of resources (compute/time, disk). Not including update-settings.ini in the mar file neatly side steps that (attached is the new patch I think we'll need). To preserve the security aspect, could we add a fallback to the updater utility, similar to what we have for the channel in the javascript service ?
Comment on attachment 8362729 [details] [diff] [review]
Exclude update-settings.ini when creating mar files, v2

There's a problem with this approach, which means we'll need a change the plan.

When a complete update is applied the precomplete file is processed first (this removes all the files of the existing install, including any deprecated ones). It's the precomplete already on disk rather than the one in the mar file though, so the createprecomplete.py part of this patch affects the *next* update. When I was testing this, I was ending up with precomplete removing update-settings.ini, the mar not adding it again - which means no channel id checking happening on the next update. We missed this around comment #34-36 because we didn't do a directory comparison between beta-updated-to-release and installed-release.

A couple of options come to mind, other ideas welcome:
* remove update-settings.ini from precomplete first, then remove it from the mar later. For beta, we would need to set the updates to go 
  old --> precomplete fixed -> latest
For nightly it's harder, because there's such a big smear of ADIs over old builds, so hard to test this there first.
* bite the bullet and repack the release mar files so that we put the beta version of update-settings.ini in it. This is probably the best option for 27.0.
I would very much prefer that this is repacked which would use the tried and true mechanism and wouldn't add one-off's which have caused problems in the past.
(In reply to Nick Thomas [:nthomas] from comment #60)
> A couple of options come to mind, other ideas welcome:
> * remove update-settings.ini from precomplete first, then remove it from the
> mar later. For beta, we would need to set the updates to go 
>   old --> precomplete fixed -> latest
> For nightly it's harder, because there's such a big smear of ADIs over old
> builds, so hard to test this there first.

If this is a one-off and not something we have to do regularly, I think it's worth it to take the hit. In addition to the extra resources that Nick already mentioned, it requires specialized automation that won't be run any other time to do repacks. (Our original plan for automation, in comment #38, makes use of more generic and well tested automation features.) We know from experience that this sort of automation is much more prone to breaking. I don't think it's a good trade-off to avoid this one-off if we have to live with that sort of automation.

If I'm wrong, and we're talking about one-offs on a regular basis going forward, I'd be in support of repacking.
(In reply to Ben Hearsum [:bhearsum] from comment #62)
> If this is a one-off and not something we have to do regularly

I hope you don't imply that shipping the release build to the beta channel is something we only do once. We should expect to do that for the forseeable future for every release. The AMD problem that prompted us to go down this road isn't going away, it has been with us for a long time and people will not magically install BIOS updates all around the world if they haven't done so by now (we know the BIOS can work around the issue and "fix" it). Also, I expect that doing this will be an overall quality win as we'll be testing the exact release builds with beta users.
(In reply to Robert Kaiser (:kairo@mozilla.com) from comment #63)
> (In reply to Ben Hearsum [:bhearsum] from comment #62)
> > If this is a one-off and not something we have to do regularly
> 
> I hope you don't imply that shipping the release build to the beta channel
> is something we only do once.

I'm not - I'm fully aware that we're doing this for the forseeable future. I'm talking about whether or not Nick's first bullet point from comment #60 is a one-off change or not.
I've used this to generate updates for
  26.0b10 -> 26.0 build1 -> 26.0 build2 -> 27.0b1
All four platforms; locales de, en-US, and ja/ja-JP-mac; partial and complete updates for all steps; channel is brbtest.

QA, please run this through mozmill and let me know.


Preliminary testing: Done on Windows 8 and OSX 10.9, en-US locale, checked both types of updates applied, diffed the install dir against the equivalent release/beta build. Found expected differences in 
* update-settings.ini, eg
-ACCEPTED_MAR_CHANNEL_IDS=firefox-mozilla-release
+ACCEPTED_MAR_CHANNEL_IDS=firefox-mozilla-beta,firefox-mozilla-release
* channel-prefs.js (deliberate for testing, excluded from mar files)
-pref("app.update.channel", "release");
+pref("app.update.channel", "brbtest");

On windows, the line endings on update-settings.ini change (we lose the CR/LF) but updates still work fine. install.log differs but I think this is from the installer and unrelated.

I want to run RelEng's update verify jobs to automate this comparison across the other platforms and locales.
Attachment #8365000 - Attachment description: Script to repack existing mar files (WIP) → Script to repack existing mar files, replacing update-settings.ini rather than removing (WIP)
For Releng:
I'm aiming small here. This is designed to run on a linux builder (mac may work too, untested), with calls like
 python repack.py -f 26.0b10-build1 -t 26.0-build1   -T 26.0b10-build1
 python repack.py -f 26.0-build1    -t 26.0-build2   -T 26.0b10-build1
 python repack.py -f 26.0-build2    -t 27.0b1-build1 -T 27.0b1-build1
(those would be used on successive releases)

Before starting it needs a signing token for the release certificates, and a checkout of the tools repo, in the same dir as repack.py. Beware hardcoded paths in MOZ_SIGN_CMD (TODO). Post-run actions include uploading mar files to the candidates dir on the ftp server, and snippets to aus3-staging. We'll need bouncer products to serve this (see also below re snippets).

Known issues -  It's not very fast, 20 mins for 12 platform-locale combos on a bld-centos6-hp. Parallelizing may help as a lot of it is CPU bound, either in the script or multiple slaves. Turning off partials would help too, but I'd like to keep 'em. I was hoping to integrate this with the updates factory but at the moment the inputs are not readily available, so probably we'll have to run this manually for now.

The snippet generation is focused on QA's test. Going to investigate using patcher instead, dependent on how close the existing bumper is to what we want (eg special name for complete mar to avoid collision with existing mar used by release channel, different bouncer setup).
(In reply to Nick Thomas [:nthomas] from comment #66)
> Before starting it needs a signing token for the release certificates, and a
> checkout of the tools repo, in the same dir as repack.py. 

Also 
  export PYTHONPATH=tools/lib/python; export PATH=$PATH:`pwd`

Doesn't need a mock environment.
(In reply to Nick Thomas [:nthomas] from comment #65)
> I've used this to generate updates for
>   26.0b10 -> 26.0 build1 -> 26.0 build2 -> 27.0b1
> All four platforms; locales de, en-US, and ja/ja-JP-mac; partial and
> complete updates for all steps; channel is brbtest.
> 
> QA, please run this through mozmill and let me know.

Flagging Juan given that he was working on verification the last time. Now we will be faster given that mozmill-ci supports the brbtest update channel.
Flags: needinfo?(jbecerra)
(In reply to Henrik Skupin (:whimboo) from comment #68)
> > QA, please run this through mozmill and let me know.
> 
> Flagging Juan given that he was working on verification the last time.

I'm taking care of it this time around.
Flags: needinfo?(jbecerra)
QA Contact: jbecerra → anthony.s.hughes
(In reply to Nick Thomas [:nthomas] from comment #65)
> QA, please run this through mozmill and let me know.

It's been determined that we will have to test this end-to-end manually. Be advised that I expect the testing to be complete by Monday morning at which point we can investigate any issues reported. I've set up a test plan to track the work: https://wiki.mozilla.org/QA/Desktop_Firefox/RC_Beta_Testing#Issues
Results of automated testing. Due to limitations in the test harness I was able to test
* 26.0b10 -> 26.0build1, and diff against 26.0build1 from installer
* 26.0build2 -> 27.0b1, and diff against 27.0b1 from installer
but not the 26.0 build1 -> build2 scenario, or a string of updates. QA and my earlier cover those.

There were differences but there were all expected or OK:
* channel-pref.js (not in mar file, expected to be different in installers)
* update-settings.ini (in mar file, see difference from 26.0 build installer (expected) but not 27.0b1)
* line endings of update-settings.ini changes on mac for complete update (no issues seen in my earlier testing from this)

I've included the patch to the tools repo I needed to do this testing (harness and config changes).
Status: assuming QA finds no issues with their update testing, then we're ready to put the release build(s) on the beta channel using a temporary, manually run, solution.

* Release Management:
 * Nothing needs to land in m-b or m-r before the 27.0 build starts
 * Please use ship-it as usual, no change to process or inputs 

* RelEng:
 * I'm counting myself on the hook for this part of 27.0, given the state of it right now
 * attachment 8365000 [details] [diff] [review] can be used to generate mar files for 27.0b9 -> 27.0 build1. It'd be run after all
   the builds were available (or with commenting the PLATFORMS block on separate slaves as each
   platform is done), eg
     python repack.py -f 27.0b19-build1 -t 27.0-build1 -T 27.0b9-build1 --no-snippets
 * see comment #66 and #67 for further details and setup requirements (tools checkout, 2x exports)
 * I need to do a little more work to generate production snippets, or to use patcher to generate
   snippets for a large set of older betas, which I will work on when I start ~ noon Monday (PT)
(In reply to Nick Thomas [:nthomas] from comment #72)
> Status: assuming QA finds no issues with their update testing, then we're
> ready to put the release build(s) on the beta channel using a temporary,
> manually run, solution.

Just following up, QA found no blocking issues in overnight testing as per https://wiki.mozilla.org/QA/Desktop_Firefox/RC_Beta_Testing#Issues
Latest version of repack script (handles locales properly). Running for 27.0 build1 mac/linux/linux64, in bld-centos6-hp-006:~/nthomas/bug908134/:

time python -u repack.py -f 27.0b9-build1 -t 27.0-build1 -T 27.0b9-build1 2>&1 | tee mac-linux.log
Attachment #8360189 - Attachment is obsolete: true
Attachment #8362729 - Attachment is obsolete: true
Attachment #8365000 - Attachment is obsolete: true
Linux64 ran first and is done (but not uploaded or available for testing). Adjusted the code to generate mac, win32, linux-i686, on the basis Windows repacks will be done before mac is done here.
Latest version of script, used for 27.0 build1 repacking.
Attachment #8366521 - Attachment is obsolete: true
Attached patch throttle.diffSplinter Review
Attachment #8370033 - Flags: review?(bhearsum)
Attachment #8370033 - Flags: review?(bhearsum) → review+
Comment on attachment 8370033 [details] [diff] [review]
throttle.diff

Wrong bug :(, but

cvs commit -m "Bug 908134 - Figure out how to offer release build to beta users, r=bhearsum"
Checking in config-dist.php;
/cvsroot/mozilla/webtools/aus/xml/inc/config-dist.php,v  <--  config-dist.php
new revision: 1.285; previous revision: 1.284
done

cvs tag -F AUS2_PRODUCTION 
T config-dist.php
Attachment #8370033 - Flags: checked-in+
Forked from attachment 8367171 [details], with improvements but also partial specific changes. Highpoints - handles updating checksums files to make it easier to use patcher to generate snippets, handles continuing on from an interruption.
Updated version of attachment 8370667 [details] [diff] [review] (s/build1/build2/ for 28.0b1, buildIDs)
Attachment #8370667 - Attachment is obsolete: true
No need to track this bug anymore, the process is in place and being honed but the main goal has been achieved.
Depends on: 974057
Depends on: 997499
This adds products like Firefox-29.0build1-Complete, Firefox-29.0build1-Partial-29.0b9, with locations having paths in firefox/candidates/. The buildN is there so that we support shipping more than one RC on beta without changing locations after shipping build1. This is tested and working.

Example call:
scripts/scripts/bouncer_submitter.py -c releases/bouncer_firefox_release_on_beta.py --revision FIREFOX_29_0_RELEASE --repo releases/mozilla-release --version 29.0 --build 1 --credentials-file oauth.txt --bouncer-api-prefix https://bounceradmin.mozilla.org/api --previous-version 29.0b7
where scripts/ is a mozharness checkout. To just submit the complete product leave off the --previous-version argument.
Attachment #8408160 - Flags: review?(catlee)
There's lots of staging noise in this, but see releaseConfig['updateConfigs'] in mozilla/staging_release-firefox-mozilla-release.py for the main point - bundling up all the related prefs that differ when we want to put release on beta.

Known bugs - having releaseConfig outside of the new structure, which is needed in the environment at build/repack time to plugin to
  --enable-update-channel=${MOZ_UPDATE_CHANNEL}
in the mozconfigs.
Need to add some comments about updating both beta and release mozconfigs when their duplicated data changes.
Things this does:
* adds a second bouncer submitter in the release on beta case (works with attachment 8408160 [details] [diff] [review]). Add support for running on all builds, rather than just build2, to support RC2+ on beta channel
* re-arranges the ReleaseUpdatesBuilder to attempt a loop over updateConfigs inside the builder. Did it this way instead of two builders to avoid handling push races
** Fails at the first patcher config bump, when it passes the -c argument without the tools/release/patcher-configs/ prefix. I think I'm probably doing the wrong thing with assigning updateConfig onto self, and expecting to work at runtime
** drops support for the old useBetaChannelForRelease variable, that functionality doesn't help us
** tags and pushes the patcher and update verify configs after pushing snippets, which may be a little unwise. We could tag and push twice I suppose.
** The balrog submission is probably broken
* adds more update verify builders, for the beta checks. Needs some scheduling love to run those first, and make the release ones wait. If only to not melt the ftp cluster
Status for 29.0:
* the above buildbot-configs and buildbotcustom work isn't close enough to finished & tested to use for 29.0
* backup plan:
 * we could probably generate a partial from 29.0b9 at build time, with some small hacks
   * add a releaseConfig['extraPartialUpdates'] block describing 29.0b9 to release-firefox-mozilla-release.py.template
   * at http://hg.mozilla.org/build/buildbotcustom/file/f4aa173ad7d0/process/release.py#l700 and
        http://hg.mozilla.org/build/tools/file/6d68be2cb18a/scripts/l10n/create-release-repacks.py#l336
     pass a dict which combines releaseConfig['partialUpdates'] and releaseConfig['extraPartialUpdates']
     Sending a mixture of release and beta versions worked for me in staging
 * add special products to bouncer, see comment #85
 * then we would need to generate snippets for the beta channel. There are notes at
     https://wiki.mozilla.org/Releases/Firefox_27.0/BuildNotes
   on how to bump the patcher config, generate snippets, remove betatest snippets, push live on releasetest
   (repacking and checksums not needed!)
 * update verify configs can probably be done using the usual creation script, running them is more painful. There's probably a way of faking up release-firefox-mozilla-beta.py (eg version and whatever else), and dropping the right runtime tag on tools, in order to use the existing beta builders. Alternatively you can do a cut-down test for 29.0b9 -> 29.0, with a hand crafted update config and manual execute. Some details on that on https://wiki.mozilla.org/Releases/Firefox_28.0b1
My staging master is at    (private link, sorry)
  http://dev-master1.build.mozilla.org:8710/waterfall?category=release-mozilla-release-
aka dev-master1:/builds/buildbot/nthomas/build1, for anyone wanting to avoid the setup hassle.
Attachment #8408168 - Attachment description: [buildbot-configs] WIP → [buildbotcustom] WIP
Comment on attachment 8408160 [details] [diff] [review]
[mozharness] bouncer submitter support

Review of attachment 8408160 [details] [diff] [review]:
-----------------------------------------------------------------

::: scripts/bouncer_submitter.py
@@ +31,5 @@
>              "help": "Current version",
>          }],
> +        [["--build"], {
> +            "dest": "build",
> +            "default": None,

do you really want this to default to None? it looks like this could end up in expanded paths like 
firefox/candidates/29.0.1-candidates/buildNone/update/win32/:lang/firefox-29.0-29.0.1.partial.mar
Attachment #8408160 - Flags: review?(catlee) → review+
Update verify needs to learn how to test build1 -> build2 partials, at the moment the filenames are the same so we overwrite build1 with build2. We unpack build1 (actually build2), and unsurprisingly the partial doesn't apply on top of it.
(In reply to Nick Thomas [:nthomas] from comment #91)
> Update verify needs to learn how to test build1 -> build2 partials, at the
> moment the filenames are the same so we overwrite build1 with build2. We
> unpack build1 (actually build2), and unsurprisingly the partial doesn't
> apply on top of it.

I filed this as bug 1021697
Depends on: 1021697
Blocks: 1049689
Depends on: 1055858
Looks like some of the dependencies may make progress in Q4. This is pretty old at this point but I think will still add value to our release cycle.
Is this fixed by bug 1105485 now?
Flags: needinfo?(nthomas)
Thanks to all your work on that we are very close. There's bug 1149544 for tidying up the bouncer entries, and bug 1021697 so we can do RC1 -> RC2 update verifies. We're currently skipping the latter because the patcher config only stores the last RC for each version.
Flags: needinfo?(nthomas)
And bug 1151755 to cover the case where you generate a partial from an RC early in the beta cycle.
Lets close this and deal with followups as needed.
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: