729654 - Update betas 4.0b7 through 9.0b6 to 11.0b5 with addon compatibility checks disabled

Status: RESOLVED FIXED

(Release Engineering :: Release Requests, defect, P2)

Description

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

from email thread w/akeybl, grace:

Background:
1) We always have major updates (MUs) available on the wire for existing beta users on old betas to choose to update forward to the newest FF11.0beta
2) We have prompted users on older betas to upgrade before. 
3) Despite this, we still have users on old betas, who:
3a) are not using latest/greatest betas (hence running without known security fixes)
3b) are not helping test newer betas
...so we want to try again to upgrade them to latest FF11.0 beta. One theory is that the MU billboard is causing a lot of users to defer. so this attempt is to be a minor update, without the MU billboard.


ToDo:
1) RelEng will create updates that would be visible to users of older betas (specifically FF4.0b7... FF11.0b3) to upgrade them to latest FF11.0beta
1a) Explicitly, this update is *not* for beta users on version FF4.0b6 and older. Those users need to upgrade to FF4.0b7 first - this impacts ~112k users and is needed to handle the updates in place to handle the OSX ppc/32 switchover to 32/64.
2) This would be a complete update, not partial update. Maybe assumed but stating here to be explicit.
3) These updates will not be offered to older nightly/aurora users, only older beta users (not changing update channels makes this project easier)
4) These updates would be offered to beta users who manually check-for-updates as well as to beta users who do nothing and get idle-time notified of an available update.
5) This updates will be minor updates - not MU.
5a) As part of this project, we are aware that this update will bring users forward across major versions, and could potentially cause addon update/change/breakage. However, the "default to compatible" change is expected to minimize this pain.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

This seems to have changed a bit from the email so I'm looking for some clarification.

(In reply to John O'Duinn [:joduinn] from comment #0)
> from email thread w/akeybl, grace:
> 
> Background:
> 1) We always have major updates (MUs) available on the wire for existing
> beta users on old betas to choose to update forward to the newest FF11.0beta

I'm confused by this. We've had automatic updates (aka minor or background) for all these betas as we've gone along.

> 3b) are not helping test newer betas
> ...so we want to try again to upgrade them to latest FF11.0 beta. One theory
> is that the MU billboard is causing a lot of users to defer. so this attempt
> is to be a minor update, without the MU billboard.

I thought that it was that users are told that some of their extensions will be disabled and chose not to update. Hence we need to fake the version in the update offer so that a compatibility check isn't done. (This is extv and/or appv in the snippets).
 
> ToDo:
> 1) RelEng will create updates that would be visible to users of older betas
> (specifically FF4.0b7... FF11.0b3) to upgrade them to latest FF11.0beta

Alex, is this what you intended ? If 10.0 has Default to Compatible (DTC) it seems that doing 4.0b7 through 9.0b6 all that's required. Possibly some of the early 10.0 betas if DTC wasn't enabled at merge time. Originally I thought we were talking just about the oldest version. Please clarify.

Comment 2

[Alex Keybl [:akeybl]]

(In reply to Nick Thomas [:nthomas] from comment #1)
> This seems to have changed a bit from the email so I'm looking for some
> clarification.
> 
> (In reply to John O'Duinn [:joduinn] from comment #0)
>
> > 3b) are not helping test newer betas
> > ...so we want to try again to upgrade them to latest FF11.0 beta. One theory
> > is that the MU billboard is causing a lot of users to defer. so this attempt
> > is to be a minor update, without the MU billboard.
> 
> I thought that it was that users are told that some of their extensions will
> be disabled and chose not to update. Hence we need to fake the version in
> the update offer so that a compatibility check isn't done. (This is extv
> and/or appv in the snippets).

These are both the correct motivations. We should make sure that the incompatible update check is suppressed, which Rob Strong mentioned would require a change to the app version.

> Alex, is this what you intended ? If 10.0 has Default to Compatible (DTC) it
> seems that doing 4.0b7 through 9.0b6 all that's required. Possibly some of
> the early 10.0 betas if DTC wasn't enabled at merge time. Originally I
> thought we were talking just about the oldest version. Please clarify.

This sounds like a good optimization. People on FF10 beta shouldn't yet be considered "left behind" we can address them in the future if necessary.

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

Just to make 100% sure we're on the same page here .... IIRC we've done one prompted update for 4.0b7+ users, otherwise there's always been an automatic update to the latest beta. I don't think a MU billboard can be a factor in those users not updating.

rs, could you confirm we should be setting appVersion to the same version as the in the query ? eg for 4.0b7 we set 4.0b7, for 9.0 betas we set 9.0 etc

Comment 4

[Robert Strong (they/them - no direct email)]

Yes, appVersion should be the same as %VERSION% in the update url
http://mxr.mozilla.org/mozilla-central/source/browser/app/profile/firefox.js#177

Comment 5

[Alex Keybl [:akeybl]]

(In reply to Nick Thomas [:nthomas] from comment #3)
> Just to make 100% sure we're on the same page here .... IIRC we've done one
> prompted update for 4.0b7+ users, otherwise there's always been an automatic
> update to the latest beta. I don't think a MU billboard can be a factor in
> those users not updating.

OK - understood. That leaves us with people who have updates disabled or have incompatible add-ons, the latter of which is being addressed here. Thanks Nick.

Comment 6

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to Alex Keybl [:akeybl] from comment #5)
> (In reply to Nick Thomas [:nthomas] from comment #3)
> > Just to make 100% sure we're on the same page here .... IIRC we've done one
> > prompted update for 4.0b7+ users, otherwise there's always been an automatic
> > update to the latest beta. I don't think a MU billboard can be a factor in
> > those users not updating.
> 
> OK - understood. That leaves us with people who have updates disabled or
> have incompatible add-ons, the latter of which is being addressed here.
Actually, there's also a third group. 

Nick's comment about only having done one prompted MU is important. Historically, we get a % of people who see but do not upgrade with each prompted MU. Of the group that doesnt upgrade, a % of those users will typically then upgrade with the next MU prompt. Usually after about 3? prompted MUs, we find ourselves left with the users who have updates disabled or wont upgrade because of addon/other incompatibilities. 

Is there anything to prevent us doing another prompted MU right now? 

A prompted MU is something which we can mechanically do trivially simply, and with little QA impact, while we figure out the details of this custom-update project.



> Thanks Nick.
Yes, indeed.

Comment 7

[Alex Keybl [:akeybl]]

(In reply to John O'Duinn [:joduinn] from comment #6)
> Actually, there's also a third group. 
> 
> Nick's comment about only having done one prompted MU is important.
> Historically, we get a % of people who see but do not upgrade with each
> prompted MU. Of the group that doesnt upgrade, a % of those users will
> typically then upgrade with the next MU prompt. Usually after about 3?
> prompted MUs, we find ourselves left with the users who have updates
> disabled or wont upgrade because of addon/other incompatibilities. 
> 
> Is there anything to prevent us doing another prompted MU right now? 

It's not clear to me that there is a third group. In comment#1, I thought Nick was saying that we've been offering updates all along (perhaps I'm wrong). If that's the case, whatever's preventing users from updating through automatic updates likely won't be solved by presenting them with a banner. I can only think of incompatible add-ons, an updater bug, or disabled add-ons being the cause for these large pockets on our beta channel.

Comment 8

[Grace Jimenez :grace]

Hi, 

Can someone please advise on timing of the project?
We have another merge/release in a couple of weeks, which will be a priority.
Is it still possible to this before the next merge/release?
Or can we do it with the Aurora 13 update?

The goal is to complete this project before the end of March.

Thanks everyone!

Comment 9

[Nick Thomas [:nthomas] (UTC+12)]

I'll start generating the update snippets today. My current plan is that 4.0b7 through to 9.0b6 will be updated to 11.0b5, with appropriate faking of the version to make Firefox think it doesn't need to check compatibility. This would stay the same as we generate new beta releases, so that we don't have to regenerate snippets every time. Will that be OK Alex ?

QA would have to speak to their availability, and to how testing here would differ from that done for DTC, but hopefully early next week to go live if all goes well.

Comment 10

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [cvs] Freeze 9.0b6 updates to 11.0b5 in patcher config

Comment 11

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [tools] Drop 9.0b6 from update verify too

Comment 12

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 602782 [details] [diff] [review]
[cvs] Freeze 9.0b6 updates to 11.0b5 in patcher config

Checking in mozBeta-branch-patcher2.cfg;
new revision: 1.56; previous revision: 1.55

Comment 13

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 602783 [details] [diff] [review]
[tools] Drop 9.0b6 from update verify too

http://hg.mozilla.org/build/tools/rev/9d35648e931b

Comment 14

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: patcher config for snippet generation

Take mozBeta-branch-patcher.cfg for 11.0b5, and then
cvs update -j1.56 -j1.55 mozBeta-branch-patcher2.cfg 
cvs update -j1.51 -j1.50 mozBeta-branch-patcher2.cfg
and fix conflict, to get back 4.0b8 through 9.0b6 back. Add 4.0b7 
from moz20-branch-patcher.cfg.

Comment 15

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: patcher config for snippet generation

Adds 4.0rc1, 4.0rc2, 4.0.1, 5.0, 5.0.1, 6.0 too, since they were published onto the beta channel.

Comment 16

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Script for snippet generation

rail, could you take a look at this script and the snippets to cross-check they're sane ?

Comment 17

[Nick Thomas [:nthomas] (UTC+12)]

Pushed live, QA signoff pending.

Comment 18

[Nick Thomas [:nthomas] (UTC+12)]

Tweaking summary to the final target of this update. We're seeing plenty of downloads of 11.0b5 but not a reduction in ADU of the older beta versions, which implies these people are downloading updates but failing to apply them. And destined to repeat that ad infinitum.

Comment 19

[Robert Strong (they/them - no direct email)]

Hopefully one of these people will file a report.

Comment 20

[Alex Keybl [:akeybl]]

(In reply to Robert Strong [:rstrong] (do not email) from comment #19)
> Hopefully one of these people will file a report.

Sadly we haven't heard of any "unexpected update" yet from 3.6, but I'm including Cheng to see if he can take another look.

Comment 21

[Nick Thomas [:nthomas] (UTC+12)]

Apparently I forgot to close this out.