1309613 - Test automigration on beta 51

Status: RESOLVED FIXED

(Firefox :: Migration, defect)

Description

[:Gijs (he/him)]

+++ This bug was initially created as a clone of Bug #1289172 +++

In bug 1289172, we assembled a test setup for automigration against beta 50. This has helped us to understand some factors about how it currently works, but there are a number of open questions that we would still like to address. In order to do this, we would like to run another test on beta 51. This bug will initially track work required to support that test in order to actually gather more useful data, and then have the (trivial copy of the 50) patch to actually run the experiment.

Comment 1

[:Gijs (he/him)]

Attachment: enable automigration on beta 51 for a few cycles, disable eslint checks for browser/branding/ pref files,

Comment 2

[:Gijs (he/him)]

Comment on attachment 8808599 [details] [diff] [review]
enable automigration on beta 51 for a few cycles, disable eslint checks for browser/branding/ pref files,

Approval Request CommentApproval Request Comment
[Feature/regressing bug #]: test for automatic migration of data from the default browser when a user first uses Firefox

[User impact if declined]: we won't know if/how effective the automatic migration and the UI for it are, meaning we would fly blind about making decisions on whether this is worth pursuing, how much time to invest in particular parts of it, etc.

We can't really do the same kind of testing on nightly/aurora because few (if any) users will run those builds as the first copy of Firefox that they ever run on their machine. We already have data that shows this (ie the volume of data for the "manual" migration telemetry we have on those builds is so low that by default it doesn't show up on telemetry.m.o). The data we do have for those trains is likely extremely skewed because of both volume and the 'type' of users that run aurora/nightly.

[Describe test coverage new/current, TreeHerder]: there is automated test coverage for most of the code that's enabled by this change.

[Risks and why]: low. We hope to land this on aurora, so that the first beta has this turned on, and then turn it off again on the beta train after several betas (we're thinking a few weeks) have gathered data about how this works out in practice. So the risk to release is 0 - we're just changing some prefs, and we'll revert those changes well before release.

[String/UUID change made/needed]: nope.


We did this before in bug 1289172 for beta 50. We're not aware of any issues that cropped up as a result. We've been prepping this for a bit by uplifting various new bits of telemetry to 51 aurora so that we get better data this time, as well as uplifting a number of fixes that improve the functionality so hopefully users who go through this on beta have a better experience.

Comment 3

[:Gijs (he/him)]

Comment on attachment 8808599 [details] [diff] [review]
enable automigration on beta 51 for a few cycles, disable eslint checks for browser/branding/ pref files,

Carrying over r+ because this is effectively the same patch as before, modulo me using the fix from bug 1303525 (landed on nightly a few weeks back) to avoid the #ifdefs in the branding files causing eslint issues, which happened last time we tried this.

Comment 4

[:Gijs (he/him)]

It seems that this didn't make it onto the relman radar. Please can we ensure this still makes b1? Thanks.

Comment 5

[Gerry Chang [:gchang]]

Comment on attachment 8808599 [details] [diff] [review]
enable automigration on beta 51 for a few cycles, disable eslint checks for browser/branding/ pref files,

Enable this automigration in beta 51. Aurora51+.

Comment 6

[:Gijs (he/him)]

remote:   https://hg.mozilla.org/releases/mozilla-aurora/rev/3b68ca65557a1eb724b3f9b3e0d1fe2a1a7574d7

Comment 7

[Cornel Ionce [:noni] [Hubs QA]]

Hi Gijs,

Could you please give us an estimate of how long this experiment will run on beta? 
We are testing migration once-per-cycle and this experiment is blocking us in running several of our tests.

Comment 8

[:Gijs (he/him)]

(In reply to Cornel Ionce [QA] (:cornel_ionce) from comment #7)
> Hi Gijs,
> 
> Could you please give us an estimate of how long this experiment will run on
> beta? 
> We are testing migration once-per-cycle and this experiment is blocking us
> in running several of our tests.

I'll look at turning this off after today's beta.

Comment 9

[:Gijs (he/him)]

https://hg.mozilla.org/releases/mozilla-beta/rev/ecdf085419775e4c24ae07b32967bc9ebf47b8f1