Closed Bug 657789 Opened 13 years ago Closed 11 years ago

For locales no longer developing on m-c, generate custom updates to migrate users of those locales from m-c to mozilla-aurora

Categories

(Release Engineering :: Release Requests, defect, P3)

Product:
Release Engineering
Component:
Release Requests
Platform:
x86
All
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: joduinn, Assigned: nthomas)

References

Details

(Whiteboard: [l10n][updates][x-functional])

Attachments

(2 files)

Doc for creating channel change mar
2.81 KB, text/plain
Details
[tools] WIP script to repack complete mar for channel change
7.75 KB, patch
Details | Diff | Splinter Review 
splitting out from bug#655129.

For the faster release cadence, some locales are changing from developing on m-c to developing on mozilla-aurora. For those locales, RelEng needs to generate custom updates to migrate m-c l10n nightly m-c users to mozilla-aurora l10n nightly.
This is as simple as copying some set of snippets from mozilla-aurora to mozilla-central on AUS. No actual generation required.
Whiteboard: [l10n]
Priority: -- → P3
Whiteboard: [l10n] → [l10n][updates]
(In reply to comment #1)
> This is as simple as copying some set of snippets from mozilla-aurora to
> mozilla-central on AUS. No actual generation required.

Great to know, thanks nthomas. We'll just wait for the list of locales from 	bug#655129, and then do that.
per chofmann in bug#655129:


For nightly users on mozilla-central, go ahead and do the following migrations:

1) migrate 1 locale on mozilla-central to en-US on mozilla-central.
af


2) migrate 29 locales on mozilla-central to the same locale on aurora.
ak
as
br
bn-IN
da
eu
gd
gu-IN
hy-AM
is
ka
kk
km
kn
lg
mai
mk
mn
mr
nso
oc
or
rm
si
son
sw
te
ta-LK
zu


3) at same time as these updates are published, stop generating nightly builds on mozilla-central for these 30 locales.




(once we get these rolled out, chofmann will loop back with another batch of locales to migrate.)
'oc' is a stale locale, can we migrate those users off to en-US as well, please?
(In reply to Axel Hecht [:Pike] from comment #4)
> 'oc' is a stale locale, can we migrate those users off to en-US as well,
> please?
sure, we havent started the migration work yet, so I'll update the locale lists as requested.
per chofmann in bug#655129, and axel in comment#4:


For nightly users on mozilla-central, go ahead and do the following migrations:

1) migrate 2 locale on mozilla-central to en-US on mozilla-central.
af
oc


2) migrate 28 locales on mozilla-central to the same locale on aurora.
ak
as
br
bn-IN
da
eu
gd
gu-IN
hy-AM
is
ka
kk
km
kn
lg
mai
mk
mn
mr
nso
or
rm
si
son
sw
te
ta-LK
zu


3) at same time as these updates are published, stop generating nightly builds on mozilla-central for these 30 locales.
Some of those locales are on other products, too. If they want to stop working on central, those peers should probably agree, not sure if chofmann checked that. If so, we want to have mapping actions for those products.

Here's the data:

sea_central

da
ka
si


cal_central

da
eu
gd
is
ka
mk
si
ta-LK


fx_central

af
ak
as
bn-IN
br
da
eu
gd
gu-IN
hy-AM
is
ka
kk
km
kn
lg
mai
mk
mn
mr
nso
oc
or
rm
si
son
ta-LK
te
zu


tb_central

af
br
da
eu
gd
is
ka
rm
si
ta-LK


fennec_central

da
eu
gd
Whiteboard: [l10n][updates] → [l10n][updates][x-functional]
(In reply to Nick Thomas [:nthomas] from comment #1)
> This is as simple as copying some set of snippets from mozilla-aurora to
> mozilla-central on AUS. No actual generation required.

Lies! We need to change the channel-prefs.js file so that users request updates on the aurora instead of nightly. We can do this with a custom mar file, being careful that we need one for mac and one for everything else. Bug 694873 should really be resolved first, so that we keep serving the channel change for more than a few days.
Depends on: 694873
I think I no longer need Khmer m-c builds.
(In reply to Nick Thomas [:nthomas] from comment #8)
> (In reply to Nick Thomas [:nthomas] from comment #1)
> > This is as simple as copying some set of snippets from mozilla-aurora to
> > mozilla-central on AUS. No actual generation required.
> 
> Lies! We need to change the channel-prefs.js file so that users request
> updates on the aurora instead of nightly. We can do this with a custom mar
> file, being careful that we need one for mac and one for everything else.
> Bug 694873 should really be resolved first, so that we keep serving the
> channel change for more than a few days.
There are also checks for channel name, app version, and cert that occur now per requirements from Mozilla Security. cc'ing bbondy for input regarding this.
So as I understand you're changing some users from m-c to aurora for some locales.  
After that change you want that channel to accept MARs from the aurora channel only.

You need to also:

1) Update update-settings.ini 
- ACCEPTED_MAR_CHANNEL_IDS=firefox-mozilla-central 
+ ACCEPTED_MAR_CHANNEL_IDS=firefox-mozilla-aurora
This indicates which types of MARs will be accepted moving forward.

2) If the updates are going to a user of m-c then you need to have a MARChannelName in the MAR set to firefox-mozilla-central.
You can set a specific channel name on an unsigned (not signed) MAR by using:
mar [-H MARChannelID] [-V ProductVersion] [-C workingDir] -i unsigned_archive_to_refresh.mar

3) The version which is contained inside the the MAR file needs to be higher than the version that the user will have installed when they take this update.
You can refresh a product version on a MAR that hasn't been signed yet by using:
mar [-H MARChannelID] [-V ProductVersion] [-C workingDir] -i unsigned_archive_to_refresh.mar

4) You will need to sign the MAR using the private key that is usually used for signing m-c builds. 
signmar [-C workingDir] -d NSSConfigDir -n certname -s archive.mar out_signed_archive.mar

To get a signmar in your build you need to enable add this into your .mozconfig: 
ac_add_options --enable-signmar

If you need to strip the signature off of an existing MAR you can do that with:
signmar [-C workingDir] -r signed_input_archive.mar output_archive.mar

All of this was written to the best of my knowledge, but I have never tried these steps.
It is recommended to:
i) Test the channel change itself before widely deploying
ii) Test an aurora update after the channel change.
Depends on: 759469, 759460, 756325
Do you know if the m-c builds you want to change have been getting Nightly updates? When is the last time they have been updated?
No longer depends on: 756325, 759460, 759469
Also we would need to know if they plan on getting updates from m-c anytime soon.
Looking in http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-central-l10n/, and comparing to comment #6, we have builds from today for all but the sw locale - so yes, it's likely that the (very small) number of users will have been getting nighty updates, and also we might have some users with defaults/preferences/channel-prefs.js file from installs.
(In reply to Brian R. Bondy [:bbondy] from comment #13)
> Also we would need to know if they plan on getting updates from m-c anytime
> soon.

If a valid translation of that question is 'will this bug be resolved before the next merge from central -> aurora' then no.
No I just wanted to know that if a Nightly update was not issued already for those locales, if it would be at some point soon, but as per Comment 14 one has already been issued.
Re-adding the dependencies now that they have been confirmed to affect this bug (see comment #14) due to the regression caused by bug 746156.

The following bugs will all need to be fixed first before this bug:
bug 756325
bug 759460
bug 759469
bug 760387
Depends on: 756325, 759460, 759469, 760387
Brian, what's the behaviour if the user gets an unsigned or improperly signed mar?  Is the update rejected?
I think we can probably generate a small mar that changes channel-prefs.js and update-settings.js, so that the app then queries and accepts mozilla-aurora updates.
The update would be rejected correct.
I think you want to have the mar you're offering be signed with the m-c channel inside the mar, but have the files mentioned in Comment 19 have the aurora channel string.
:catlee, per meeting today, it looks like the locales in comment#6 are ready to migrate a) from locales -> en-US or b) from m-c --> m-aurora.

This will also reduce our nightly-repack-load, so thanks for the help here.
Component: Release Engineering → Release Engineering: Custom Builds
bah - lost assignment in mid-air.
Assignee: nobody → catlee
(In reply to Brian R. Bondy [:bbondy] from comment #20)
> The update would be rejected correct.
> I think you want to have the mar you're offering be signed with the m-c
> channel inside the mar, but have the files mentioned in Comment 19 have the
> aurora channel string.

Rejected outright completely (as in the mar could never be applied), or would it prompt the user if they wanted to apply the update?
Rejected outright.
I don't remember the exact error handling but I think it gets deleted and re-downloaded. I think the user gets an error wizard to re-download as well.  But of course they'd just get another bad mar in this case.
Brian, if you have a moment could you please take a look at http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/channel-switch/switch-to-aurora.mar

I'm hitting
nthomas@HD-mandala ~/Desktop/FirefoxNightly.app/Contents/MacOS/updates $ cat last-update.log 
Performing a background update
SOURCE DIRECTORY /Users/nthomas/Desktop/FirefoxNightly.app/Contents/MacOS/updates/0
DESTINATION DIRECTORY /Users/nthomas/Desktop/FirefoxNightly.app/Updated.app
DoUpdate: error extracting manifest file
failed: 6
calling QuitProgressUI

when updating a mac nightly (m-c, de locale). Somehow I'm ending up at
 http://mxr.mozilla.org/mozilla-central/source/toolkit/mozapps/update/updater/updater.cpp#3601
It's signed with the nightly cert for firefox-mozilla-central, the code indicates it's gotten past verifying that already but I'm most paranoid about getting that right.
There's a test update at you can point at with app.update.url.override:
 https://aus3.mozilla.org/update/3/%PRODUCT%/%VERSION%/%BUILD_ID%/%BUILD_TARGET%/test/%CHANNEL%/%OS_VERSION%/%DISTRIBUTION%/%DISTRIBUTION_VERSION%/update.xml
It's locale and platform agnostic.
FTR, I don't think we should take data from comment 6 as current, that's 1.5 years old.
I checked bug 655129, but couldn't find the reasoning behind moving af nightly users to en_US. I guess not many people are affected, but I guess af can be handled just like the other languages with active translations on Aurora.
(In reply to Axel Hecht [:Pike] from comment #28)
> FTR, I don't think we should take data from comment 6 as current, that's 1.5
> years old.

I posted to newsgroups yesterday to see if any of these locales had changed their mind.
(In reply to Friedel Wolff from comment #29)
> I checked bug 655129, but couldn't find the reasoning behind moving af
> nightly users to en_US. I guess not many people are affected, but I guess af
> can be handled just like the other languages with active translations on
> Aurora.


This was from l10n-drivers, but a while ago. If you the official "af" lead, just let us know what your community wants - we are happy to move "af" users on m-c to aurora instead of merging them into "en-US" users on m-c. 

Just let us know what "af" community would prefer.
Depends on: 863050
(In reply to John O'Duinn [:joduinn] from comment #31)
> (In reply to Friedel Wolff from comment #29)
> > I checked bug 655129, but couldn't find the reasoning behind moving af
> > nightly users to en_US. I guess not many people are affected, but I guess af
> > can be handled just like the other languages with active translations on
> > Aurora.
> 
> 
> This was from l10n-drivers, but a while ago. If you the official "af" lead,
> just let us know what your community wants - we are happy to move "af" users
> on m-c to aurora instead of merging them into "en-US" users on m-c. 
> 
> Just let us know what "af" community would prefer.


Friedal: ping?
Flags: needinfo?(friedel)
We shouldn't try to resolve issues about single locales in this bug.

I've gathered some data on ADIs and activity, and l10n-drivers will put the data up for comments in .l10n, and then follow up here.
Flags: needinfo?(friedel)
Depends on: 863452
I figured out my problem at comment #26 (not precise enough manual hackery). When creating the mar file the (simplified) call is 
 mar -c output.mar <list of files>
but the <list of files> must be
 update.manifest [updatev2.manifest] <files in same order as update manifest>

Without the manifest first you get the error in comment #26; without the file ordering you get things like:
$ cat updates/last-update.log 
Performing a background update
SOURCE DIRECTORY /Users/nthomas/Desktop/FirefoxNightly.app/Contents/MacOS/updates/0
DESTINATION DIRECTORY /Users/nthomas/Desktop/FirefoxNightly.app/Updated.app
PREPARE ADD Contents/MacOS/defaults/pref/channel-prefs.js
PREPARE ADD Contents/MacOS/update-settings.ini
EXECUTE ADD Contents/MacOS/defaults/pref/channel-prefs.js
### execution failed
FINISH ADD Contents/MacOS/defaults/pref/channel-prefs.js
FINISH ADD Contents/MacOS/update-settings.ini
backup_restore: backup file doesn't exist: Contents/MacOS/update-settings.ini.moz-backup
failed: 6
calling QuitProgressUI
Status: 
* channel changing mar created (see attachment for method)
* verified it works on Windows XP (using the service) and mac

However
* when you query for an Aurora update you get
AUS:SVC UpdateService:selectUpdate - skipping update because the update's application version is less than the current application version
 and nothing is downloaded (Aurora version 22.0a2 is less than nightly 23.0a1)
* if you fake the snippets to offer 23.0a1 then the d/l will happen. Mac will apply the update with great success, but the windows service will report this in last-update.log:
  failed: 23

Per http://mxr.mozilla.org/mozilla-central/source/toolkit/mozapps/update/common/errors.h#40 this is a VERSION_DOWNGRADE_ERROR, from the 22.0a2 version encoded in the aurora mar file.

So, this at this point we have a choice:
1) Deliberately fake version 23.0a1 in Aurora snippets and when versioning the mar files, and use the channel-switching mar. Would have to do this for some undetermined time until users have upgraded.

2) bite the bullet and repack some set of aurora updates. Something like this:
* looping over platforms and locales
 * unpack mar file
 * add defaults/prefs/channel-prefs.js and update-settings.js files
 * recreate mar file with -V 23.0a1 -H firefox-mozilla-central
 * sign mar file with nightly cert, host on ftp.m.o
 * create and host snippets
This would be a one-off job for each locale, but much more time consuming.
Setting myself as QA Contact so I can test this once ready.
QA Contact: anthony.s.hughes
We can move Afrikaans (af) users to Aurora like for the other locales.
To put some context on ADI here, for 2013-04-29 on the nightly channel and version 23.0a1 we got blocklist pings for
 af (1), bg (4), ca (3), da (2), el (6), fi (3), hr (1), mk (1)
for a total of 21. The other locales removed in attachment 741623 [details] [diff] [review] didn't have any pings.

We might find slightly more if a week was considered instead (metrics interface is misbehaving so can't get that right now) but that data says there's a big reduction in the locales we need to provide paths for, if any at all.
Results are not hugely changed by considering the week ending 2013-04-29. A total of 145 blocklist pings averaging to 20.7/day. The locales listed in comment #38 drop a bit to accomadate adding en-ZA (1), eu (1), sl(1)

https://docs.google.com/a/mozilla.com/spreadsheet/ccc?key=0AiC1WGFz4NMDdFRNQ19SVm5NTzJGR2xkQVVsOWZ2TUE&usp=sharing
Component: Release Engineering: Custom Builds → Release Engineering: Releases
Assignee: catlee → nthomas
No longer depends on: 694873
Product: mozilla.org → Release Engineering
Blocks: 713159
No longer depends on: 713159
Depends on: 945179
catlee pointed out that the locales in bug 863452 were disabled long enough ago to not hit the mar versioning problem in comment #35. So I've created a switch-to-aurora.mar file per attachment 739417 [details], and put on ftp at http://ftp.mozilla.org/pub/mozilla.org/firefox/releases/custom-updates/switch-to-aurora.mar. 

Snippets have been created on aus3-staging, as the last builds were created before we switched to Balrog (so no dependency on bug 945179). They have a buildID of 99999999999999, which has been excluded from the cron job which cleans up old snippets.

Testing had to be done by taking an en-US build and hardcoding the %LOCALE% (after copying app.update.url to app.update.url.override), as we don't keep l10n builds that far back. On Windows 8, Mac 10.8, and Ubuntu 32bit, I successfully updated from the last nightly before bug 863452 took effect [1]
in two steps. The first applies switch-to-aurora.mar, which sets up Firefox to query on the aurora channel, and accept mars with channelID of firefox-mozilla-aurora; then the latest complete mar is applied.

[1] http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2013/04/2013-04-25-03-08-45-mozilla-central/
Status: NEW → RESOLVED
Closed: 11 years ago
No longer depends on: 759469, 760387, 863050, 945179
Resolution: --- → FIXED
Notes for the future, when bug 713159 is resolved, and we want to do another round of this sort of update.

If it's nearly merge time, I suggest:
* before the merge: disabling the locales on mozilla-central  (so they never get a version bump)
* after the merge: using the channel change trick to point at aurora builds (which will now have the version which was on central). Would need to set up a rule and a release in Balrog, rather than create snippets

If it's early/mid-cycle and we want to update straight away we'll to repack some aurora complete mars:
* finish this script I've attached and use it to repack some aurora locales
* known deficiencies - signing and Balrog submission

In either case, having bug 945179 would be very desirable to write a single rule in Balrog (rather than one per locale).
See Also: → 1737802
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: