1335709 - Run another experiment with automigration on beta 52 to learn more about how successful the new undo method is

Status: RESOLVED FIXED

(Firefox :: Migration, defect)

Description

[:Gijs (he/him)]

We learned from previous versions of automigration experiments on beta that a significant proportion of users, whose data we automatically migrate from another browser, do not take active action to keep / not to keep that data, and instead have the option to do so removed because of changes to their bookmarks or passwords, or because they sign into sync (after which we could, at that time, no longer guarantee that removing data would not also lose "new" data).

We've since made changes to the automigration "undo" functionality such that it is now possible to "undo" those changes (ie choose "Don't Keep" for the data when asked by the notification bar) even after other changes to bookmarks, history and passwords are made (either by the user themselves or by them signing into a sync account).

We'd like to know what impact these changes have on the number of users that choose to keep or throw away the data we imported for them, by running another experiment on beta. We would also learn from this experiment how successful the new method of "undo" is (ie how often it encounters errors) and how fast/slow it is, in order to gauge how to prioritize making that process faster (and where we should do so).

Comment 1

[:Gijs (he/him)]

Attachment: Bug 1335709 - enable automigration for beta to test new-style 'undo' implementation,

Review commit: https://reviewboard.mozilla.org/r/108734/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/108734/

Comment 2

[Justin Dolske [:Dolske]]

Comment on attachment 8832470 [details]
Bug 1335709 - enable automigration for beta to test new-style 'undo' implementation,

https://reviewboard.mozilla.org/r/108734/#review111280

Comment 3

[:Gijs (he/him)]

Comment on attachment 8832470 [details]
Bug 1335709 - enable automigration for beta to test new-style 'undo' implementation,

Approval Request Comment
[Feature/Bug causing the regression]: automigration 'undo'
[User impact if declined]: see comment #1
[Is this code covered by automated tests?]: yes!
[Has the fix been verified in Nightly?]: n/a
[Needs manual test from QE? If yes, steps to reproduce]: no.
[List of other uplifts needed for the feature/fix]: bug 1333233 (already uplifted)
[Is the change risky?]: no
[Why is the change risky/not risky?]: we're enabling a feature on beta that we've enabled a few times before (and that we're also running in a funnelcake against 51 release). It won't be enabled for general release.
[String changes made/needed]: no.

To be clear, I'd like to land this in time for beta 5 (gtb Thursday) and then re-disable for beta 9 (Thu in 2 weeks) so we'd have 1 beta without this before the RC. That still gives us 2 weeks (4 betas) of data which will hopefully help work out if we need to spend more time working on undo error rates and/or performance before a general release, as well as providing some data on whether people's choices regarding 'undo' change now that bookmarks/passwords/sync no longer interfere with that possibility.

Comment 4

[:Gijs (he/him)]

Egh, fix comment link:

(In reply to :Gijs from comment #3)
> Comment on attachment 8832470 [details]
> Bug 1335709 - enable automigration for beta to test new-style 'undo'
> implementation,
> 
> Approval Request Comment
> [Feature/Bug causing the regression]: automigration 'undo'
> [User impact if declined]: see comment #0
> [Is this code covered by automated tests?]: yes!
> [Has the fix been verified in Nightly?]: n/a
> [Needs manual test from QE? If yes, steps to reproduce]: no.
> [List of other uplifts needed for the feature/fix]: bug 1333233 (already
> uplifted)
> [Is the change risky?]: no
> [Why is the change risky/not risky?]: we're enabling a feature on beta that
> we've enabled a few times before (and that we're also running in a
> funnelcake against 51 release). It won't be enabled for general release.
> [String changes made/needed]: no.
> 
> To be clear, I'd like to land this in time for beta 5 (gtb Thursday) and
> then re-disable for beta 9 (Thu in 2 weeks) so we'd have 1 beta without this
> before the RC. That still gives us 2 weeks (4 betas) of data which will
> hopefully help work out if we need to spend more time working on undo error
> rates and/or performance before a general release, as well as providing some
> data on whether people's choices regarding 'undo' change now that
> bookmarks/passwords/sync no longer interfere with that possibility.

Comment 5

[Julien Cristau [:jcristau]]

Comment on attachment 8832470 [details]
Bug 1335709 - enable automigration for beta to test new-style 'undo' implementation,

enable automigration experiment on beta52

Comment 6

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-beta/rev/b2fdacdd1a7e

Comment 7

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-esr52/rev/b2fdacdd1a7e

Comment 8

[:Gijs (he/him)]

Backed out again to allow beta 9 to ship without automigration, so we get 1 beta before rc that has it turned off again.

remote:   https://hg.mozilla.org/releases/mozilla-beta/rev/86178fe492254027e0cba462ccbdc7db8cb90817

Comment 9

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-esr52/rev/86178fe49225