2088593 - Put upcoming import changes behind Nimbus switches so they can ship with a shared holdback

Status: NEW

(Firefox :: Migration, task, P1)

Description

[Ray Fambro]

From Fx163, we'd like each user-facing import change to land behind a variable on the existing migrationWizard Nimbus feature, so one long-running experiment can turn changes on for a delivery group and keep them off for a holdback. That lets us measure the combined effect of many small import fixes on retention, instead of each one being too small to read alone.

First candidates:

• per-type outcomes in the wizard (bug 2087593)
• the imported-history ranking fix (bug 1421485, already behind a pref)
• the New Tab shortcuts refresh after import (FIDE-4404)
• raising the history cap (bug 1871579)

Each default stays as it is today until the experiment turns it on.

Also asking: does the per-resource duration metric (bug 2080907) record on Nightly yet? Its fields appear in the Nightly metrics data, but I found no recorded values in the last 8 days. The resource_migrated event (bug 2080904) is firing on Fx161 Nightly.

Comment 1

[Ray Fambro]

Correction to comment 0: the per-resource duration metrics from bug 2080907 are recording on Nightly (about 110 metrics pings with history import durations in the last 8 days, 87 with logins and 44 with bookmarks). My first check looked for the wrong data shape. Also confirmed the migrationWizard Nimbus feature is still on main. The only open question here is which changes get a switch.

Comment 2

[Ray Fambro]

Update since comment 0 (Nov 13):

• The ranking fix (bug 1421485) landed Nov 3 with browser.migrate.chrome.history.visit.limit defaulting to 5, so it's on for everyone from Fx161. For this one the switch works in reverse: the holdback group gets 1 (today's behavior). There's no migrationWizard variable for it yet.
• The Safari history fix (bug 2089676) should get a switch too, once it lands.
• Later candidates, as they're built: Chrome passwords on Windows (FIDE-3615, Fx165), the Mac folder-access explainer (FIDE-4607), multi-profile import (FIDE-4605).
• The holdback runs as two experiments, Windows and Mac, starting with Fx163 (Jan 12). The first switches need to land by the Fx163 merge to beta on Dec 3.
• The full list is in the plan's measurement section (Mozilla-only): https://docs.google.com/document/d/18B173mGBrNSbd1uCLAY4tMzoiKed0nQ_pKAMXFOe9D0/edit

Comment 3

[Ray Fambro]

Update Nov 13, design change after talking with Jason:

• The holdback uses a new Nimbus feature (working name importHoldback) with a single on/off switch, instead of one switch per fix on migrationWizard. A profile can't be in two experiments on the same feature, so this keeps migrationWizard free for other import tests.
• The experiment launches once per OS (Windows, Mac) and isn't edited after launch. The holdback group has the switch on.
• Each import fix ships on by default and checks the switch in code. If it's on, the profile keeps today's behavior. That includes a follow-up to bug 1421485 so the holdback sets browser.migrate.chrome.history.visit.limit back to 1.
• A new exposure event fires when import runs, so the analysis can look at profiles that actually imported.
• Jason is taking this bug. The first pieces need to land by the Fx163 merge to beta on Dec 3.

This replaces the per-fix switches in comment 0.

Comment 4

[Jason Prickett]

Updated design after discussing with Ray:

• A new nimbus feature, importHoldback, with a single boolean. It is separate from migrationWizard intentionally, so the holdback doesn't block other import experiments
• Each independent fix will be on by default, but will also check the importHoldback's boolean, keeping the previous behavior when it is true. The holdback experiment will be launched once and will not need to be edited.
• Exposure will be recorded when the migration wizard is first shown. The existing migration_started_wizard and resource_migrated events can be used for further analysis.

For 163, we need the feature, the exposure event, and the check for bug 1421485 (and bug 2089676 once that lands).