Closed Bug 1366893 Opened 7 years ago Closed 6 years ago

Migrate aurora users to Beta channel for Thunderbird

Categories

(Release Engineering :: Release Requests, enhancement, P2)

Product:
Release Engineering
Component:
Release Requests
Type:
enhancement

Tracking

(Not tracked)

Status:
RESOLVED INVALID

People

(Reporter: Fallen, Unassigned)

References

Details

Attachments

(6 files, 2 obsolete files)

custom_mar_2.tar.gz
954 bytes, application/x-gzip
Details
Firefox-aurora-to-beta-switcher-watershed.json
8.55 KB, text/plain
Details
Script to generate watershed file from list of locales and template.
1.15 KB, text/x-python
Details
List of all thunderbird locales with nightly releases in 2017/2017.
239 bytes, text/plain
Details
Template balrog release for aurora to beta migration, v2
3.31 KB, text/plain
Details
Thunderbird-aurora-to-beta-switcher-watershed.json
133.63 KB, text/plain
Details 
Thunderbird has followed project dawn and removed the aurora channel. We'd like to migrate our aurora users to beta just like in bug 1353825. We have the convenience of not having a strongly branded "Developer Edition", so we can probably just migrate the users to beta.

I don't know for sure what needs to be done for this, I suspect the minimum would be to set up the auto-update rules, but looking at the dependencies on bug 1353825 there may be more needed.

I'd appreciate if someone could give us the scoop on what needs to be done.
I think we'll need some special MARs that switch signing and update channel from aurora to beta.
Priority: -- → P2
Chris, anything we can do to help create these MARs?
Flags: needinfo?(catlee)
(In reply to Aki Sasaki [:aki] from comment #3)
> Created attachment 8875383 [details]
> custom_mar.tar.gz
> 
> Edited the files in https://bugzilla.mozilla.org/attachment.cgi?id=8842539

We may have to do something to adjust signing?  We're not sure if this is a complete beta mar with channel switching, aurora-signed, or what.  I think this is going to take someone with more background than me, or for someone with said background to walk me through it.
catlee and I had a quick look at the updater code, which embeds certs inside the updater executable. However we are not sure if Thunderbird actually enables the cert check, as there are no --enable-verify-mar lines in the configs like https://dxr.mozilla.org/comm-central/source/mail/config/mozconfigs/win32/nightly and release (and we do for Firefox). So for a fairly minimal investment of time we could create a channel switching mar file, not even sign it, set up a test channel in Balrog, then see what happens.
Ah yes, I didn't ever merge bug 1189843 so they are not signed. If we can do this in the test channel it would be great!
Attached file custom_mar_2.tar.gz — 
A few small fixes (mac update channel, set channelID and version on mar file).

Uploaded a copy of the file to https://archive.mozilla.org/pub/thunderbird/candidates/aurora-beta-channel-switch/update.mar
Attachment #8875383 - Attachment is obsolete: true
Last Linux and Windows nightly was 20170620004006; last mac nightly was 20170510004022. I've removed latest-comm-aurora, latest-comm-aurora-l10n, and latest-comm-esr45 from https://archive.mozilla.org/pub/thunderbird/nightly/.

Still to do - set up switcher release in balrog, and rules to do aurora -> older beta -> latest beta on a test channel.
Assignee: nobody → nthomas
Flags: needinfo?(catlee)
Tom, this is what we're using for Firefox to switch a couple of locales. I suggest you determine what set of locales you need for Thunderbird and extend it; each locale gets exactly the same data. You could drop the WINNT_x86_64-msvc section, and the alias that refers to it, given no 64-bit windows builds on comm-aurora.

If you attach your json to the bug I'll take a look and add it to Balrog.
Flags: needinfo?(mozilla)
Reading through https://bugzilla.mozilla.org/show_bug.cgi?id=1353825 I think a lot of the dependent bugs are about having a special DevEdition version of beta builds for people migrating from Aurora (but I haven't actually checked the dependencies themselves). Since I don't think we enable different features for different releases, those changes aren't relevant for us.
https://bugzilla.mozilla.org/show_bug.cgi?id=1315083 is the issue where the mar comes from, that talks about specfics of how the update will work.
We already have a channel changing mar file, see comment #7. It just modifies update-settings.ini and channel-prefs.js, so that aurora users start querying for beta updates and are able to accept them. 

At this point we just need a Thunderbird equivalent of attachment 8887320 [details], me to set that up on a test channel, some people to test, and if all is well enable on the aurora channel.
This is the same as all the locales in the most recent aurora. I verified this by scrapping the file lists on https://ftp.mozilla.org/pub/thunderbird/nightly/
This only includes `en-US`. Use the attached script to generate the complete release.
(In reply to Tom Prince [:tomprince] from comment #13)
> Created attachment 8888027 [details]
> Script to generate watershed file from list of locales and template.

Needed a tiny fix for this, otherwise worked fine.

--- script.py.bak	2017-07-20 11:58:05.000000000 +1200
+++ script.py	2017-07-20 11:58:19.000000000 +1200
@@ -29,3 +29,3 @@
     else:
-        output_file = open(args['output'], "w")
+        output_file = open(args.output, "w")
Note that we don't currently have betas for kk, ka or af. I know we intend to ship at least kk in the future, though.
Turns out you can't have appVersion/displayVersion/platformVersion at the platform level - the schema is at https://github.com/mozilla/balrog/blob/master/auslib/blobs/schemas/apprelease-v4.yml. Instead they go either at the top-level or down in locale (chose top to avoid duplication).
Attachment #8888031 - Attachment is obsolete: true
Ah. I had consulted the schema, but it isn't the easiest to read. I think I got confused seeing the buildid at the platform level and assumed I could put the other versions there too. In any case, having the versions at the top-level makes more sense anyway.
Added this to Balrog and pointed the auroratest channel to it.

To test:
* install a comm-aurora build
* modify <install dir>/defaults/pref/channel-prefs.js to use the auroratest channel instead of aurora
* maybe make define app.update.log to true for debugging
* start up Thunderbird and check for updates
I've tested upgrading from a few different en-US aurora releases and a fr aurora release on linux x86_64, and things appear to work.

But I've had a report that the update failed on a Win10 machine. I've asked the reporter to test applying the mar manually.
I can reproduce that Win10 issue:

AUS:SVC getCanStageUpdates - able to stage updates using the service
AUS:SVC Downloader:onStopRequest - attempting to stage update: Earlybird 54.0a2
AUS:SVC readStatusFile - status: failed: 19, path: C:\Users\....

https://dxr.mozilla.org/mozilla-aurora/source/toolkit/mozapps/update/common/errors.h#38 says CERT_VERIFY_ERROR, so I'd say the maintenance service is still checking the cert. I'll need to sign the mar file, and adjust the release in Balrog.
I disabled staging (thought we had some problems there) and attempted a manual update pretty the same result
PATCH DIRECTORY C:\TBUPDATE
INSTALLATION DIRECTORY c:\Users\Joe\Downloads\thunderbird-54.0a2.en-US.win32
WORKING DIRECTORY c:\Users\Joe\Downloads\thunderbird-54.0a2.en-US.win32
failed: 19
calling QuitProgressUI 

BTW if that manual cmd line process was the only was to update, mine would be few and far between :)
Any update on getting the mar signed? Is there anything I can do to help?
Flags: needinfo?(nthomas)
https://archive.mozilla.org/pub/thunderbird/candidates/aurora-beta-channel-switch/update-signed.mar is the same mar as comment #7 but signed with the nightly cert for updates. I've also updated Thunderbird-aurora-to-beta-switcher-watershed in Balrog with new url, size, and hash.

The channel switching then works on Windows, but we get an error applying 56.0b2 on the next update request - it's CERT_VERIFY_ERROR again. This stems from the maintenance service having the cert encoded in the binary or in the registry. It's expecting a nightly-signed mar file and gets a release-signed one, and the computer says no. This is a fundamental difference from what Firefox is doing switching XP+Vista users from beta/release to esr, where both beta and release accept release-signed mar files from ESR..

AFAIK there is no way to update the maintenance service, so we're looking at significantly more work to create special repacks of a beta but with nightly signing. How many users are we talking about here ?
Flags: needinfo?(nthomas)
iirc, some ways back we had 2k-3k aurora users.  not sure what is is now, until bug 1389402 is sorted
I think the number of aurora users may be about the same as it was two months ago.

I crudely base this on the number of crashes of 54.0a2 for two different time periods 
* May-June, pre-summer, 581 [1]
* July-Aug 440 [2]
A 25% drop. But we know summer usage is significantly lower, so I'd say the number of users is roughly comparable and hasn't gone down significantly.  The number of users might be is hard to say, because the query which shows ADI no longer offers aurora versions in the query https://crash-stats.mozilla.com/crashes-per-day/?p=Thunderbird  


[1] https://crash-stats.mozilla.com/search/?release_channel=aurora&product=Thunderbird&date=%3E%3D2017-05-11T12%3A38%3A31.000Z&date=%3C2017-06-11T12%3A38%3A00.000Z&_sort=-date&_facets=signature&_facets=version&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#facet-version
[2] https://crash-stats.mozilla.com/search/?release_channel=aurora&product=Thunderbird&date=%3E%3D2017-07-11T12%3A40%3A23.000Z&date=%3C2017-08-11T12%3A40%3A23.000Z&_sort=-date&_facets=signature&_facets=version&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#facet-version
Are there any in-app messaging options which could be used to encourage people to re-install ?
(In reply to Nick Thomas [:nthomas] from comment #29)
> Are there any in-app messaging options which could be used to encourage
> people to re-install ?

We have the Thunderbird Start Page. Is it possible to put up a Thunderbird start page that is version and/or channel aware to provide a customized message to users who we want to download a specific update?

I'm not sure what else is available.  Are there other tools used by Firefox that we could adapt?
Flags: needinfo?(mkmelin+mozilla)
Flags: needinfo?(kohei.yoshino)
We can put a message on the start page, but I thought the Thunderbird content was moving to thunderbird.net (Bug 1363162). :sancus, any update on the migration?
Flags: needinfo?(kohei.yoshino) → needinfo?(sancus)
However, I don't want a pure static page, I want it to be version-aware. 

For example I don't want version 52 users to see a message saying "check to see if your version is out of date. But if a user is on version 38 I want them to see "Hey, you are out of date. here's what to do."
The start page redirects to start.thunderbird.net the versions on mozilla.org are no longer used at all.

The rest of the content is on thunderbird.net but is blocked by the other bugs listed in the tracker. 

We can make a start page that asks you to update and only redirect specific versions to it. Localization is another story, however.
Since this bug has been open for a long time, I'm not sure it's worth extensive effort anymore. Aurora users should be tech savy enough to migrate manually. I'd assume the almost all have by now. Opinions?
Flags: needinfo?(mkmelin+mozilla)
(In reply to Magnus Melin from comment #34)
> Since this bug has been open for a long time, I'm not sure it's worth
> extensive effort anymore. Aurora users should be tech savy enough to migrate
> manually. I'd assume the almost all have by now. Opinions?

I do not think we can assume that. 
1. I've encountered users on various channels who have no idea how they got there.  (I wouldn't expect it to be a high percentage, but they exist)
2. Based on your assumption, must nightly users should be on a current version even though they are not being updated. But it looks like about half are still on 56.0a1 [1]
3. At a bare minimum we are due to update our channel messages anyway.
4. I think it is fair guess is at least half are no longer on nightly (and many will have been localizers or people checking l10n).  With a quick look at the aurora channel crash rate, I estimate 1/2 - 3/4 of users may be off aurora. But that's just a quick guess.


[1] https://crash-stats.mozilla.com/search/?release_channel=nightly&product=Thunderbird&date=%3E%3D2017-09-27T21%3A23%3A53.000Z&date=%3C2017-10-04T21%3A23%3A53.000Z&_sort=-date&_facets=signature&_facets=version&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#facet-version
besides, you didn't actually answer my question :)

(but perhaps others did)
Yes we have the beta and nightly channel pages that should be easy enough to update. Those aren't critical for localization either. Those could well have some version check in place that would display a warning if you're out of date.

If there really are a lot of nightly users running two versions behind, I wonder what their motivation to run nightlies is and how they didn't notice. If you aren't actively following development and reporting new bugs you find, nightly is pretty pointless to run. And aurora was more meant for add-on authors.
> If there really are a lot of nightly users running two versions behind, I wonder what their motivation to run nightlies is and how they didn't notice.

I can only speak about macOS, but on macOS there are no nightly updates since several weeks, you have to manually download new nightly builds…
(In reply to Magnus Melin from comment #37)
> Yes we have the beta and nightly channel pages that should be easy enough to
> update. Those aren't critical for localization either. Those could well have
> some version check in place that would display a warning if you're out of
> date.

Good to hear we can target out of date users. 


> If there really are a lot of nightly users running two versions behind, 

I've rechecked, and it's definitely not "if", so many users rely upon automatic updates - which as you say is somewhat unexpected. 

And looking at the version facet, it's not just users at two versions behind.  https://crash-stats.mozilla.com/search/?release_channel=nightly&product=Thunderbird&date=%3E%3D2017-09-10T05%3A10%3A48.000Z&date=%3C2017-10-10T05%3A10%3A48.000Z&_sort=-date&_facets=signature&_facets=version&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#facet-version   

> I wonder what their motivation to run nightlies is and how they didn't notice.
> If you aren't actively following development and reporting new bugs you
> find, nightly is pretty pointless to run. And aurora was more meant for
> add-on authors.

Unfortunately, without direct contact with these users we can only guess - and that's partly our fault.  Some probably think there's a good, intentional reason they are not being updated.  Perhaps we do too good a job at silent updates?

As for reporting bugs - most users don't report bugs *regardless* of what channel they are on.  If we don't get enough bug reports from then, then that's probably, again, on us not informing and educating.  Also, reporting crashes or bugs doesn't require following development. (And we can't expect users to follow development when *we* do no publicity about development results prior beta - we can't expect users to follow hg checkin activity and bug reports.  mozillazine build channel is the closest thing we have to publicizing nightly activity, and there may be less than 100 followers - perhaps even fewer if you judge by posting activity)  Firefox has retuned how they handle nightly channel users. We should probably do something similar.

nits - I believe Aurora was meant primarily for localizers.  And I don't think addon developers every paid much attention to Aurora - I thought that there place was beta.  (Heck, many barely pay attention to beta.)
Flags: needinfo?(sancus)
(In reply to Wayne Mery (:wsmwk) from comment #39)
> nits - I believe Aurora was meant primarily for localizers.  And I don't
> think addon developers every paid much attention to Aurora

I don't know about the intent of the releases, but FWIW: I got on the Aurora (Earlybird) channel a few years back by googling for "Thunderbird Beta" and landing at this public download page (shown here via archive.org snapshot), which offers download links for Earlybird (first) and Beta (second):
  https://web.archive.org/web/20161130132239/https://www.mozilla.org/en-US/thunderbird/channel/

I installed Earlybird from that page because I'm a user who wants a latest-and-greatest experience from my mail client, and who is willing to encounter & file occasional bugs.  I would imagine that many of the stranded Aurora-users are in that same general user-category.
I'm not going to have any time to work on this.
Assignee: nthomas → nobody
I was following up on this, to see what it would take to get a nightly-signed beta. I think we would need to update https://dxr.mozilla.org/build-central/source/buildbotcustom/process/release.py#136 (plus appropriate configuration to make it conditional). We'd also need to use the BZ2/SHA-1 mar format to update from aurora-era builds.
Given that we do want update all users, and that the above mar works for linux/osx, do we want to update those users while we work out what to do about windows?
Flags: needinfo?(vseerror)
(In reply to Tom Prince [:tomprince] from comment #43)
> Given that we do want update all users, and that the above mar works for
> linux/osx, do we want to update those users while we work out what to do
> about windows?

sure. Some of the crashes have known fixes, so it will both good for the users, and good for us.

I estimate the Mac aurora population to be 100-200.  Windows maybe 250-500.  Linux probably very low.
Flags: needinfo?(vseerror)
I've updated the rule on auroratest to only apply for Linux/macOS. If we can verify that this updates Linux and macOS users, but doesn't trigger failures on windows, I'll move the change over to the aurora channel.
Flags: needinfo?(jsabash)
(In reply to Tom Prince [:tomprince] from comment #45)
> I've updated the rule on auroratest to only apply for Linux/macOS. If we can
> verify that this updates Linux and macOS users, but doesn't trigger failures
> on windows, I'll move the change over to the aurora channel.

I'm sure I can get a linux tester, Mac not so much.I guess the last tinderbox builds are Here:http://ftp.mozilla.org/pub/thunderbird/tinderbox-builds/comm-aurora-linux/1497944406/ ?
Flags: needinfo?(jsabash)
I think the latest release is http://ftp.mozilla.org/pub/thunderbird/nightly/2017/06/2017-06-20-00-40-06-comm-aurora/ (and earlier ones can be found at similar paths.
Success reported for linux, also does not serve updates to windows as desired.
The update for Linux/OSX should now be live on the aurora channel.
(In reply to Tom Prince [:tomprince] from comment #49)
> The update for Linux/OSX should now be live on the aurora channel.

Doesn't look like it, to me...

At least: I tried downloading 64-bit Linux Earybird from the link at Comment 47, and I started that Thunderbird build with a fresh profile, and then did "Help | About", and it says:
  54.0a2 (2017-06-20) (64-bit)
  EarlyBird is up to date

The tarball I used was:
https://ftp.mozilla.org/pub/thunderbird/nightly/2017/06/2017-06-20-00-40-06-comm-aurora/thunderbird-54.0a2.en-US.linux-x86_64.tar.bz2
It looks like I had the priorities in balrog reversed. I've updated them to be in the correct order now (hopefully!).
Nice. That seems to have fixed it. Specifically, now I'm offered 3 "updates" in series, when I run that build and do Help|About and then restart.

First I get a very small update (a few KB I think?) which I imagine just changes the update channel.  Then I restart, and then I get a 47MB update.  And then after I restart to apply that update, the branding in the toolbar changes to Thunderbird 56.0b3 (no longer Earlybird), and then Help|About immediately offers me another 37.3 MB update, which gets me to 58.0b2 (64-bit), at which point I'm fully updated it seems.

So I think we can call this FIXED?
This is fixed for Linux/macOS. Windows users still aren't being updated (see comment #25 for why).
Bulk change of QA Contact to :jlund, per https://bugzilla.mozilla.org/show_bug.cgi?id=1428483
QA Contact: catlee → jlund
(In reply to Nick Thomas [:nthomas] (UTC+12) from comment #25)
> ...
> AFAIK there is no way to update the maintenance service, so we're looking at
> significantly more work to create special repacks of a beta but with nightly
> signing. How many users are we talking about here ?

I think my estimate from comment 44 was on the low side, but it's not possible to put a statisticaly sound estimate on it anymore.   Extrapolating using several crash-stats fields the number is probably 500-1,000 total.  And many are on very old release which, presumably, the either don't want to update from, don't know how, or update is broke for them (in which case only a full download would help).  I think that may be as much as half the population.

In short, I don't think a programmed migration is worth the effort mentioned in comment 25. 

But I fullly support doing a new aurora start page (Andrei's comment 33), that might help wake some of them up :)
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → INVALID
(In reply to Wayne Mery (:wsmwk) from comment #55)
> ...
> But I fullly support doing a new aurora start page (Andrei's comment 33),
> that might help wake some of them up :)

https://github.com/thundernest/thunderbird-website/issues/39
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: