625140 - Do major updates from 4.0beta[x] to 4.0beta[y] to make sure it all works and move users off old betas

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[LegNeato]

Do major updates from 4.0beta[x] to 4.0beta[y] to make sure it all works and move users off old betas. This is similar but different than bug 590953.

If there is a chemspill needed for 4.0.1 we'll want to know the machinery works for putting up an update dialog between minor versions.

As an added bonus, we can use this opportunity to try to urge users off old betas and onto the latest.

Comment 1

[chris hofmann]

sounds like code has changed in this area during the 4.0 development cycle so it definitely should be tested.  are there links to bugs for those changes around?

Comment 2

[LegNeato]

I believe the main changes have been around bug 530872

Comment 3

[Robert Strong (they/them - no direct email)]

bug 530872 is by far the main change to the UI along with bug 480178

If we do this to the beta audience I suggest only doing it for a few days and then going back to the regular snippet... I'm a tad concerned that one of the reasons people don't update is because we require UI for major updates (at least with the old / fallback snippet format) and if we pester some people too much they will just turn off auto app update.

Comment 4

[chris hofmann]

yeah,  definitely not wanting to pester, but we have +300k active daily users on beta7 (released nov 10) so getting those users moved up to beta 9 next week would pretty valuable.  does the new major update feature have variable messages with increasing levels of scariness?   the low level recommendation is the most appropriate message for this test.   beta testers will probably see an update reminder as a good thing since they are more active in trying out new stuff.

we could also dig out stats that helped to measure number of users that took heed of the message, number of users that continued to ignore it (and/or somehow didn't see it?),  and number of users that might have bailed on the betas.

Since we are dealing with a smaller and more active user population on beta7 (300k) ,beta8 (1.4 million) ,and beta9 (getting ready for push) it might be easier to observe and translate the different actions people take when they see the message.

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

Does this require bug 459972 ?

Comment 6

[Robert Strong (they/them - no direct email)]

Not as far as I am concerned. The fallback to the previous behavior is in place and as long as the new appVersion is not present the old behavior will take place.

Comment 7

[chris hofmann]

joduinn,  can you get someone assigned to this?

legnetto,  lets get it marked as a beta blocker.

Comment 8

[Robert Strong (they/them - no direct email)]

(In reply to comment #4)
> yeah,  definitely not wanting to pester, but we have +300k active daily users
> on beta7 (released nov 10) so getting those users moved up to beta 9 next week
> would pretty valuable.  does the new major update feature have variable
> messages with increasing levels of scariness?   the low level recommendation is
> the most appropriate message for this test.   beta testers will probably see an
> update reminder as a good thing since they are more active in trying out new
> stuff.
Doing this for the beta 7 systems makes sense and though I'd prefer that the beta 8 systems got the normal snippet to avoid update fatigue I'm ok either way.

The message is from remote html as it was previously so the message can be anything we want

> we could also dig out stats that helped to measure number of users that took
> heed of the message, number of users that continued to ignore it (and/or
> somehow didn't see it?),  and number of users that might have bailed on the
> betas.
We've done that a couple of times recently and there are definitely people that choose not to update when given the choice.

Comment 9

[[e-mail] (:bhearsum)]

Attachment: sample MU snippet

So...the way I'm reading this bug is that we want to do a MU from 4.0b7 -> 4.0b9 without using any of the new snippets variables. Attached is a sample snippet of what I think we're looking for here. Rob & Christian, can you guys confirm before I proceed further?

The details URL can be changed to anything, of course.

Also note that we will have no partial MARs for this, only completes.

Comment 10

[Robert Strong (they/them - no direct email)]

Comment on attachment 504702 [details] [diff] [review]
sample MU snippet

note: since we are using the fallback mechanism the detailsUrl needs to point to a billboard.

Comment 11

[LegNeato]

(In reply to comment #9)
> Created attachment 504702 [details] [diff] [review]
> sample MU snippet
> 
> So...the way I'm reading this bug is that we want to do a MU from 4.0b7 ->
> 4.0b9 without using any of the new snippets variables. Attached is a sample
> snippet of what I think we're looking for here. Rob & Christian, can you guys
> confirm before I proceed further?

I think we'll want to do b1-b7. That should just be copying the same snippets multiple places, correct? If it, more involved than that, not worth it, but we have ~20k people on every older beta < 6

> 
> The details URL can be changed to anything, of course.

let's point it at http://www.mozilla.com/en-US/firefox/4.0b9/details/. I'll work on getting a billboard up asap

> 
> Also note that we will have no partial MARs for this, only completes.

Yep, this isn't testing everything, just that we haven't painted ourselves into a corner.

Comment 12

[[e-mail] (:bhearsum)]

(In reply to comment #11)
> (In reply to comment #9)
> > Created attachment 504702 [details] [diff] [review]
> > sample MU snippet
> > 
> > So...the way I'm reading this bug is that we want to do a MU from 4.0b7 ->
> > 4.0b9 without using any of the new snippets variables. Attached is a sample
> > snippet of what I think we're looking for here. Rob & Christian, can you guys
> > confirm before I proceed further?
> 
> I think we'll want to do b1-b7. That should just be copying the same snippets
> multiple places, correct? If it, more involved than that, not worth it, but we
> have ~20k people on every older beta < 6

I think I can support them without much trouble. Let's assume we'll do it for all of those versions. I'll let you know if it ends up being very troublesome.

> > 
> > The details URL can be changed to anything, of course.
> 
> let's point it at http://www.mozilla.com/en-US/firefox/4.0b9/details/. I'll
> work on getting a billboard up asap

K

Comment 13

[[e-mail] (:bhearsum)]

Attachment: potential patcher config

Created this by copying and slightly tweaking moz2-branch-patcher2.cfg. Currently generating snippets with it.

Comment 14

[[e-mail] (:bhearsum)]

Attachment: update verify configs

Created these in the same way as the patcher config.

Comment 15

[[e-mail] (:bhearsum)]

Attachment: updated patcher config

I've got test snippets up on releasetest now.

For posterity, here's how I generated them:
- Ran MajorUpdateFactory on a staging master w/ the patcher config attached. I commented out the patcher config and update verify config portions of the factory
- After it finished, I removed the 4.0b7 partial.txt files from the directories uploaded to staging-stage
- Transferred the snippets to aus2-staging
- Ran backupsnip/pushsnip on the test snippets

I did a single test on my laptop w/ 4.0b7 64-bit Linux en-GB. Here's what happened:
- Clicked Help -> About Firefox
- Clicked "Apply Update" button
- Old-style "Software Update" dialog appeared and told me there was an update. No details were available, presumably because the details link is a 404.
- Clicked "Get the new version", it downloaded the update
- Clicked "Apply update", update applied successfully.

Rob, does that sound right? You should be able to try for yourself on any 3.7a1 -> 4.0b7 build

Comment 16

[Robert Strong (they/them - no direct email)]

(In reply to comment #15)
That sounds correct

Comment 17

[[e-mail] (:bhearsum)]

I had to push a new set of snippets because the original batch was missing "ja-JP-mac" on Mac.

This set appears to be fine from my perspective, and is ready for QA testing. I've sent mail to release-drivers to that effect.

Comment 18

[chris hofmann]

Users might be confused by what version they are coming from, or what version they are running after update so this feedback might not be reflective of what's actually happened, but there are couple of interesting comments on input and the labs feedback channel in the last hour that suggest some possible additional test cases.

http://groups.google.com/group/mozilla-labs-testpilot/browse_thread/thread/d4c6cdfb53553d70#

I like 3.6. But... when I opened my FF, it was highjacked by the beta version.
Yup, FF highjacked FF

--> any chance a 3.6 version could have matched on the upgrade offer?

watching comments for
http://input.mozilla.com/en-US/search/?product=firefox&q=update

I see this one too
http://input.mozilla.com/en-US/opinion/1150583

after upgrading to 4.0 beta 9, the auto update put be back at 3.16. Maybe should point out to new beta users to turn off auto update.

--> do we have test cases for looking at what happens if the update fails?

Comment 19

[Robert Strong (they/them - no direct email)]

There are unit tests for failed updates... don't know whether there are litmus tests.

I highly suspect the feedback had to do with either launching the Mozilla Firefox shortcut instead of the Mozilla Firefox Beta X shortcut or trying to launch the Beta while a release build was already running.

Comment 20

[LegNeato]

The billboard or this is now on production.

Comment 21

[Robert Strong (they/them - no direct email)]

Hey LegNeato, I thought you had used the sample billboard for the new format as a template... it requires a new attribute on the body tag for it to work due to the fix for bug 548061. Just add billboard="1" to the body as in
<body billboard="1">

Also, all images should be included as data urls since the user can disable remote image loading.

Comment 22

[LegNeato]

(In reply to comment #21)
> Hey LegNeato, I thought you had used the sample billboard for the new format as
> a template... it requires a new attribute on the body tag for it to work due to
> the fix for bug 548061. Just add billboard="1" to the body as in
> <body billboard="1">
> 
> Also, all images should be included as data urls since the user can disable
> remote image loading.

Done and pushed live in revision 81481.

Comment 23

[Robert Strong (they/them - no direct email)]

Awesome and it looks good!

Comment 24

[[e-mail] (:bhearsum)]

Christian, you mentioned the other day that we want to target this at b10 instead. Which versions do you want to receive it? I'm assuming we want everything at least up to b8 to get it, but what about b9?

Comment 25

[[e-mail] (:bhearsum)]

Attachment: updated patcher config targetting b10

Comment 26

[[e-mail] (:bhearsum)]

Attachment: updated again, with right updateType

Comment 27

[LegNeato]

(In reply to comment #24)
> Christian, you mentioned the other day that we want to target this at b10
> instead. Which versions do you want to receive it? I'm assuming we want
> everything at least up to b8 to get it, but what about b9?

Ideally everything < beta 9. I think we want beta 9 to still be minor to encourage people to update automatically. Correct me if I am wrong, but I think we can't have both an advertised MU and an unadvertised minor update targeting the same users active together.

I also just realized that it means we are actually testing the code in < beta 9 rather than beta 10 (due to updating)...we'll want to do this for RC as well (will likely make another bug).

Comment 28

[[e-mail] (:bhearsum)]

Alright, I've got MU snippets for 3.7a5*, 4.0b[1-8] -> 4.0b10 up and ready for QA testing on "releasetest". I'll send mail to release-drivers to that effect.

Comment 29

[chris hofmann]

Although I'd like to see us get the laggards off the old betas, beta 10 is turning out to not be a very good choice to move people up to.  These lagging beta users in large part probably became very comfortable with one of the older more stable betas and that's the reason they've stuck with it.

The mozcrt19.dll@0x1327f and maybe a few other crashes are keeping crash per user rates high on beta 10.  It might be a good idea to wait and have a look at beta 11 for this change.  Bug 628152 will be fie should also be  trying to get beta 11 out quickly to recover from beta 10 instability.

Comment 30

[LegNeato]

I see your point, though the main goal of this is to make sure the mechanisms work...getting users off old builds is just a side benefit of the framework/method we are using to test. I want to do this with 10 and leave the possibility open for doing other ones.

Comment 31

[[e-mail] (:bhearsum)]

Shipped yesterday.