Closed Bug 625140 Opened 11 years ago Closed 11 years ago

Do major updates from 4.0beta[x] to 4.0beta[y] to make sure it all works and move users off old betas

Categories

(Release Engineering :: General, defect, P2)

defect

Tracking

(blocking2.0 betaN+)

RESOLVED FIXED
Tracking Status
blocking2.0 --- betaN+

People

(Reporter: christian, Assigned: bhearsum)

References

Details

(Whiteboard: [release][aus][hardblocker])

Attachments

(2 files, 4 obsolete files)

Do major updates from 4.0beta[x] to 4.0beta[y] to make sure it all works and move users off old betas. This is similar but different than bug 590953.

If there is a chemspill needed for 4.0.1 we'll want to know the machinery works for putting up an update dialog between minor versions.

As an added bonus, we can use this opportunity to try to urge users off old betas and onto the latest.
Depends on: 590953
sounds like code has changed in this area during the 4.0 development cycle so it definitely should be tested.  are there links to bugs for those changes around?
I believe the main changes have been around bug 530872
Depends on: 530872
bug 530872 is by far the main change to the UI along with bug 480178

If we do this to the beta audience I suggest only doing it for a few days and then going back to the regular snippet... I'm a tad concerned that one of the reasons people don't update is because we require UI for major updates (at least with the old / fallback snippet format) and if we pester some people too much they will just turn off auto app update.
yeah,  definitely not wanting to pester, but we have +300k active daily users on beta7 (released nov 10) so getting those users moved up to beta 9 next week would pretty valuable.  does the new major update feature have variable messages with increasing levels of scariness?   the low level recommendation is the most appropriate message for this test.   beta testers will probably see an update reminder as a good thing since they are more active in trying out new stuff.

we could also dig out stats that helped to measure number of users that took heed of the message, number of users that continued to ignore it (and/or somehow didn't see it?),  and number of users that might have bailed on the betas.

Since we are dealing with a smaller and more active user population on beta7 (300k) ,beta8 (1.4 million) ,and beta9 (getting ready for push) it might be easier to observe and translate the different actions people take when they see the message.
Does this require bug 459972 ?
Not as far as I am concerned. The fallback to the previous behavior is in place and as long as the new appVersion is not present the old behavior will take place.
joduinn,  can you get someone assigned to this?

legnetto,  lets get it marked as a beta blocker.
blocking2.0: --- → ?
blocking2.0: ? → betaN+
(In reply to comment #4)
> yeah,  definitely not wanting to pester, but we have +300k active daily users
> on beta7 (released nov 10) so getting those users moved up to beta 9 next week
> would pretty valuable.  does the new major update feature have variable
> messages with increasing levels of scariness?   the low level recommendation is
> the most appropriate message for this test.   beta testers will probably see an
> update reminder as a good thing since they are more active in trying out new
> stuff.
Doing this for the beta 7 systems makes sense and though I'd prefer that the beta 8 systems got the normal snippet to avoid update fatigue I'm ok either way.

The message is from remote html as it was previously so the message can be anything we want

> we could also dig out stats that helped to measure number of users that took
> heed of the message, number of users that continued to ignore it (and/or
> somehow didn't see it?),  and number of users that might have bailed on the
> betas.
We've done that a couple of times recently and there are definitely people that choose not to update when given the choice.
Whiteboard: [release][aus]
Assignee: nobody → bhearsum
Attached patch sample MU snippet (obsolete) — Splinter Review
So...the way I'm reading this bug is that we want to do a MU from 4.0b7 -> 4.0b9 without using any of the new snippets variables. Attached is a sample snippet of what I think we're looking for here. Rob & Christian, can you guys confirm before I proceed further?

The details URL can be changed to anything, of course.

Also note that we will have no partial MARs for this, only completes.
Attachment #504702 - Flags: review?
Attachment #504702 - Flags: review?(robert.bugzilla)
Attachment #504702 - Flags: review?(clegnitto)
Attachment #504702 - Flags: review?
Whiteboard: [release][aus] → [release][aus][hardblocker]
Comment on attachment 504702 [details] [diff] [review]
sample MU snippet

note: since we are using the fallback mechanism the detailsUrl needs to point to a billboard.
Attachment #504702 - Flags: review?(robert.bugzilla) → review+
(In reply to comment #9)
> Created attachment 504702 [details] [diff] [review]
> sample MU snippet
> 
> So...the way I'm reading this bug is that we want to do a MU from 4.0b7 ->
> 4.0b9 without using any of the new snippets variables. Attached is a sample
> snippet of what I think we're looking for here. Rob & Christian, can you guys
> confirm before I proceed further?

I think we'll want to do b1-b7. That should just be copying the same snippets multiple places, correct? If it, more involved than that, not worth it, but we have ~20k people on every older beta < 6

> 
> The details URL can be changed to anything, of course.

let's point it at http://www.mozilla.com/en-US/firefox/4.0b9/details/. I'll work on getting a billboard up asap

> 
> Also note that we will have no partial MARs for this, only completes.

Yep, this isn't testing everything, just that we haven't painted ourselves into a corner.
Depends on: 626839
(In reply to comment #11)
> (In reply to comment #9)
> > Created attachment 504702 [details] [diff] [review]
> > sample MU snippet
> > 
> > So...the way I'm reading this bug is that we want to do a MU from 4.0b7 ->
> > 4.0b9 without using any of the new snippets variables. Attached is a sample
> > snippet of what I think we're looking for here. Rob & Christian, can you guys
> > confirm before I proceed further?
> 
> I think we'll want to do b1-b7. That should just be copying the same snippets
> multiple places, correct? If it, more involved than that, not worth it, but we
> have ~20k people on every older beta < 6

I think I can support them without much trouble. Let's assume we'll do it for all of those versions. I'll let you know if it ends up being very troublesome.

> > 
> > The details URL can be changed to anything, of course.
> 
> let's point it at http://www.mozilla.com/en-US/firefox/4.0b9/details/. I'll
> work on getting a billboard up asap

K
Attached file potential patcher config (obsolete) —
Created this by copying and slightly tweaking moz2-branch-patcher2.cfg. Currently generating snippets with it.
Attachment #504702 - Attachment is obsolete: true
Attachment #504702 - Flags: review?(clegnitto)
Created these in the same way as the patcher config.
Attached file updated patcher config (obsolete) —
I've got test snippets up on releasetest now.

For posterity, here's how I generated them:
- Ran MajorUpdateFactory on a staging master w/ the patcher config attached. I commented out the patcher config and update verify config portions of the factory
- After it finished, I removed the 4.0b7 partial.txt files from the directories uploaded to staging-stage
- Transferred the snippets to aus2-staging
- Ran backupsnip/pushsnip on the test snippets

I did a single test on my laptop w/ 4.0b7 64-bit Linux en-GB. Here's what happened:
- Clicked Help -> About Firefox
- Clicked "Apply Update" button
- Old-style "Software Update" dialog appeared and told me there was an update. No details were available, presumably because the details link is a 404.
- Clicked "Get the new version", it downloaded the update
- Clicked "Apply update", update applied successfully.

Rob, does that sound right? You should be able to try for yourself on any 3.7a1 -> 4.0b7 build
(In reply to comment #15)
That sounds correct
I had to push a new set of snippets because the original batch was missing "ja-JP-mac" on Mac.

This set appears to be fine from my perspective, and is ready for QA testing. I've sent mail to release-drivers to that effect.
Depends on: 627598
Users might be confused by what version they are coming from, or what version they are running after update so this feedback might not be reflective of what's actually happened, but there are couple of interesting comments on input and the labs feedback channel in the last hour that suggest some possible additional test cases.

http://groups.google.com/group/mozilla-labs-testpilot/browse_thread/thread/d4c6cdfb53553d70#

I like 3.6. But... when I opened my FF, it was highjacked by the beta version.
Yup, FF highjacked FF

--> any chance a 3.6 version could have matched on the upgrade offer?

watching comments for
http://input.mozilla.com/en-US/search/?product=firefox&q=update

I see this one too
http://input.mozilla.com/en-US/opinion/1150583

after upgrading to 4.0 beta 9, the auto update put be back at 3.16. Maybe should point out to new beta users to turn off auto update.

--> do we have test cases for looking at what happens if the update fails?
There are unit tests for failed updates... don't know whether there are litmus tests.

I highly suspect the feedback had to do with either launching the Mozilla Firefox shortcut instead of the Mozilla Firefox Beta X shortcut or trying to launch the Beta while a release build was already running.
The billboard or this is now on production.
Hey LegNeato, I thought you had used the sample billboard for the new format as a template... it requires a new attribute on the body tag for it to work due to the fix for bug 548061. Just add billboard="1" to the body as in
<body billboard="1">

Also, all images should be included as data urls since the user can disable remote image loading.
(In reply to comment #21)
> Hey LegNeato, I thought you had used the sample billboard for the new format as
> a template... it requires a new attribute on the body tag for it to work due to
> the fix for bug 548061. Just add billboard="1" to the body as in
> <body billboard="1">
> 
> Also, all images should be included as data urls since the user can disable
> remote image loading.

Done and pushed live in revision 81481.
Awesome and it looks good!
Christian, you mentioned the other day that we want to target this at b10 instead. Which versions do you want to receive it? I'm assuming we want everything at least up to b8 to get it, but what about b9?
Attachment #505072 - Attachment is obsolete: true
Attachment #505111 - Attachment is obsolete: true
Priority: -- → P2
Attachment #507101 - Attachment is obsolete: true
(In reply to comment #24)
> Christian, you mentioned the other day that we want to target this at b10
> instead. Which versions do you want to receive it? I'm assuming we want
> everything at least up to b8 to get it, but what about b9?

Ideally everything < beta 9. I think we want beta 9 to still be minor to encourage people to update automatically. Correct me if I am wrong, but I think we can't have both an advertised MU and an unadvertised minor update targeting the same users active together.

I also just realized that it means we are actually testing the code in < beta 9 rather than beta 10 (due to updating)...we'll want to do this for RC as well (will likely make another bug).
Alright, I've got MU snippets for 3.7a5*, 4.0b[1-8] -> 4.0b10 up and ready for QA testing on "releasetest". I'll send mail to release-drivers to that effect.
Although I'd like to see us get the laggards off the old betas, beta 10 is turning out to not be a very good choice to move people up to.  These lagging beta users in large part probably became very comfortable with one of the older more stable betas and that's the reason they've stuck with it.

The mozcrt19.dll@0x1327f and maybe a few other crashes are keeping crash per user rates high on beta 10.  It might be a good idea to wait and have a look at beta 11 for this change.  Bug 628152 will be fie should also be  trying to get beta 11 out quickly to recover from beta 10 instability.
I see your point, though the main goal of this is to make sure the mechanisms work...getting users off old builds is just a side benefit of the framework/method we are using to test. I want to do this with 10 and leave the possibility open for doing other ones.
Shipped yesterday.
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.