666046 - create custom snippets to migrate orphaned users to latest mozilla-aurora

Status: RESOLVED INVALID

(Release Engineering :: Release Requests, defect, P2)

Description

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Per discussion in aurora meeting, and https://wiki.mozilla.org/Firefox/Aurora/Grow_Usage_%28phase_1%29

1) we want more users on mozilla-aurora
2) we still like the users we have on mozilla-beta already, so do not want to migrate 
3) we see orphaned bunches of users on specific older beta/RCs who are unsafe, and also not helping by testing a supported aurora/beta. 

RelEng should target the users in https://wiki.mozilla.org/Firefox/Auror/Grow_Usage_%28phase_1%29#Potential_audience, taking into account state of release automation, as well as OS/hardware support limitations (ie 10.4, PPC, etc).

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

What is the timescale for this work ?

Comment 2

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to comment #1)
> What is the timescale for this work ?

People still in post-ff5.0 blur, so the request was semi-vague "sometime at/by 5july would be nice". Everyone is very conscious of all the release work done recently, so this is not a chemspill, but also wanting to increase the Aurora userbase (and testing) in a timely manner.

(also we should treat the wiki doc as suggestions - RelEng has more exposure to awkward update paths to avoid, like where OS dropped, or dialog boxes changed sizes. The overall goal is to find a way to increase Aurora userbase in a rational manner.)

Comment 3

[[e-mail] (:bhearsum)]

From the wiki (Prompt == Major update in RelEng terms):
Prompt or minor update?

    We will try the billboard first, in your face and we can have a message on it
    If that doesn't work, we will go to unprompted
    These users should already have a minor update sitting there waiting for them, not sure why they aren't installing/restarting 

Last prompt wasn't localized

    We should localize as much as possible for maximum impact

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Effectiveness of updates to 5.0b1

I understand the desire to increase the number of users on the Aurora channel but IMO the 4.0 betas aren't good place to source them from. Users there have shown a lack of interest in updating over a long time period - 4 months for 4.0b12, longer for the older releases - which makes it a lot harder to get them to move. 

Lets try to back that up with some data, first the tl;dr: The advertised update to 5.0b1 wasn't effective at getting a lot of users over from 4.0 betas. We did get some users, but no more than just publishing an automatic update would have.

The detail:
We offered an advertised update to the fake beta 5.0b1, from 4.0b1 - 4.0b12 (and 3.7a3 thru a5) for just the en-US locale. The other locales had an automatic update to 4.0.2 in place. The timeline was:
* 2011-05-04: enabled updates to 5.0b1 at 50% throttle
* 2011-05-09: updates were unthrottled (all requests offered 5.0b1)
* 2011-05-20: automatic update to 5.0b2 replaces 5.0b1, including locales
The billboard for 5.0b1 was https://www.mozilla.com/en-US/firefox/5.0beta/details/.

I've attached a couple of graphs of the trends around then. The first is the en-US ADU count for the four newest beta versions, 4.0b9 thru b12, from the start of April to the end of May. The second is the percentage of ADU which is en-US (for those versions and the same time period).

The points I'd make to them are
* broadly speaking, there's an exponential decay of ADU on old versions, this is particularly obvious for 4.0b11 and 12 in the first graph
* the billboard has no significant effect on that trend during the first half of May
* the en-US percentage is also flat across the first half of May

So what does that mean ? The billboard was as effective at getting people to 5.0b1 as the automatic updates were to 4.0.1. Which is to say, not very.


-------
For completeness, here are the metrics queries (MDX) I used to generate the graphs. It's blocklist data, limited to the beta channel.

# 4.0b vs time.png
select {[Products].[Firefox].[4.0].[4.0b12], [Products].[Firefox].[4.0].[4.0b11], [Products].[Firefox].[4.0].[4.0b10], [Products].[Firefox].[4.0].[4.0b9]} ON COLUMNS,
  NON EMPTY {[Date].[2011].[4].Children, [Date].[2011].[5].Children} ON ROWS
from [BlockList Analysis]
where Crossjoin({[Channels].[beta]}, {[Locale Codes].[en-US]})

# just the %age en-US
with member [Locale Codes].[en-US_percent] as 'IIf(([Locale Codes].[All] = 0.0), NULL, (([Locale Codes].[en-US] / [Locale Codes].[All]) * 100.0))'
select Crossjoin({[Products].[Firefox].[4.0].[4.0b12], [Products].[Firefox].[4.0].[4.0b11], [Products].[Firefox].[4.0].[4.0b10], [Products].[Firefox].[4.0].[4.0b10], [Products].[Firefox].[4.0].[4.0b9]}, {[Locale Codes].[en-US_percent]}) ON COLUMNS,
  NON EMPTY {[Date].[2011].[4].Children, [Date].[2011].[5].Children} ON ROWS
from [BlockList Analysis]
where {[Channels].[beta]}

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

So we can get a trickle of users by pointing 4.0 betas at Aurora, but what about something faster ? I think they need to be new people, or moved over from beta.

Since we no longer have channel switcher, to move people around in the update system we'll need:
* a custom mar file(s) that include a channel-prefs.js pointing to aurora
* snippets that point to those mar(s) for all the versions, platforms, and locales

How we implement that depends on the user experience we want versus the time it takes to set it up. Here are some options:

1, channel changing update
how: create a mar that only changes the channel, serve that with a billboard (or not)
UX: very quick download of channel change mar (~5KB), restart and still have the old build, get latest aurora build via complete mar on next update check. Probably need some first run page messaging to get users to check for updates again.
releng: this would be pretty quick to do. One mar is easy to make, and all the snippets are the same.

2, static update path 
how: repack a particular set of aurora mar files for platforms & locales.
UX: get the billboard and a complete update (15+MB), apply to get an older aurora build, get another complete update to latest aurora when next update check occurs
releng: about 350 mars to create (but scriptable), would need to use patcher to generate the snippets

3, regenerated update path (2 but always pointing to latest aurora builds)
how: script the generation of mar files and snippets
UX: get the billboard prompt and download a complete update, restart to have latest aurora version
releng: We don't have any scripts that do this now, so we'd have to create something that repacked mar files and modified snippets. Would take a while to stand that up and it would probably be fragile

I think I should stop now.

Comment 6

[LegNeato]

Hmmmm, I forgot that the channel wouldn't change after the update unless we manually hacked some mars.

My preference would be #2 followed very closely by #1.

Comment 7

[LegNeato]

We decided we want to do #2 above, please let me know if you have any questions.

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Sketch for option 1

I haven't tried this code so there are probably typos/gotchas/complete omissions, but there's quite a lot that should be helpful. 

Option #2 would be similar, except that it starts with downloading an extracting (mar -x) an existing mar file for each platform & locale, unique snippets for each of those.

Comment 9

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Found in triage. With our new components, with new scope, I *think* this bug belongs here?

Comment 10

[Nick Thomas [:nthomas] (UTC+12)]

Withered on the vine.