Open Bug 1668315 Opened 4 years ago Updated 4 years ago

[Experiment] Message Router Content Experiment: Wizard-less Password import autocomplete Experiment Fx 82.0 to 83.0 Release

Categories

(Shield :: Shield Study, task, P3)

Product:
Shield
Component:
Shield Study
Type:
task

Tracking

(Not tracked)

People

(Reporter: experimenter, Unassigned)

References

(Blocks 1 open bug,
URL
)

Details

Wizard-less Password import autocomplete Experiment

We would like to run a study to see if showing the import wizard from a password autocomplete entry when Firefox knows there's a Chrome saved login for the site leads to growth.

Experimenter is the source of truth for details and delivery. Changes to Bugzilla are not reflected in Experimenter and will not change delivery configuration.

Data Science Issue: https://jira.mozilla.com/browse/DO-308
More information: https://experimenter.services.mozilla.com/experiments/wizard-less-password-import-autocomplete-experiment/

Summary: [Experiment]: Pref-Flip Experiment: Wizard-less Password import autocomplete Experiment → [Experiment] Message Router Content Experiment: Wizard-less Password import autocomplete Experiment Fx 82.0 to 83.0 Release

The experiment branches will collect event telemetry with category "exp_import" (viewable from about:telemetry -> Events) and also get enrolled viewable from about:telemetry -> Environment Data -> Experiments). The following normandy user_id values should get enrolled when testing against stage remote settings.

control:
app.normandy.user_id=32d112a3-35a2-3742-b10b-7bec02e89c38

treatment:
app.normandy.user_id=3f50c242-751d-8c40-842f-08863f554947

When not in the experiment, there will be the usual "form_autocomplete" events but not "exp_import"

notInExperiment:
app.normandy.user_id=0cb07ab0-e3e7-ca4d-a266-97af5b0946a2

E.g.,

XPCSHELL_TEST_PROFILE_DIR=foo /Applications/Firefox\ Beta.app/Contents/MacOS/firefox-bin -P profile-with-id-3f50c242-751d-8c40-842f-08863f554947 --first-startup

I was able to do some sanity tests on mac using nightly 83 and beta 82:

  1. prepare chrome single profile with importable password
  2. create new profile without actually starting firefox
  3. start firefox with XPCSHELL_TEST_PROFILE_DIR=foo ./firefox
  4. install remote settings devtools https://github.com/mozilla-extensions/remote-settings-devtools/releases/download/1.3.0-1/remote-settings-devtools.xpi
  5. open remote settings devtools
  6. select "Stage" from environment dropdown
  7. click "Force sync" for main/nimbus-desktop-experiments
  8. open about:config
  9. set app.normandy.user_id to appropriate value, e.g., 3f50c242-751d-8c40-842f-08863f554947 for treatment from comment 1
  10. quit firefox
  11. start firefox with ./firefox --first-startup
  12. go to site with importable password from a single profile
  13. click import suggestion

expect:

  • appropriate behavior: treatment = no wizard, control = wizard, not in experiment = wizard
  • about:telemetry -> Environment Data -> Experiments has the appropriate value, e.g., "password-autocomplete-wizardless.branch treatment "
  • about:telemetry -> Events shows "exp_import" related telemetry for treatment and control
Flags: needinfo?(cmuresan)

Wizard-less Password Import Autocomplete Experiment
Firefox Release 82, 83

We have finished testing the [PI-754] Wizard-less Password Import Autocomplete experiment.

Quality status: GREEN - SHIP IT

Why is this experiment green?

  • Delivery mechanism:
    • The only issue found during testing was fixed and verified.
  • Product specific:
    • During testing we have encountered the following known issues - https://mzl.la/31pws5C, however they do not affect user experience in a harmful way.

Testing Summary:

Tested Platforms:

  • Windows 10
  • Windows 7

Tested Firefox versions:

  • Firefox RC 82.0 (BuildID 20201014125134)
  • Firefox Beta 83.0b2 (BuildID 20201020173725)

Regards,
Ciprian Muresan
Ecosystem QA Team

Flags: needinfo?(cmuresan)

Documenting the telemetry that each branch sends (control sends more as the migration wizard has more telmetry):

exp_import#impression#formfill = 3 / {"autofillResult": "4", "loginsCount": "0"} both
exp_import#impression#popup = 3 / {"loginsCount": "0", "searchLength": "0"} both
exp_import#event#click = chrome / {"profilesCount": "1"} both
FX_MIGRATION_ENTRY_POINT control
FX_MIGRATION_SOURCE_BROWSER control
FX_MIGRATION_USAGE control
FX_MIGRATION_BOOKMARKS_IMPORT_MS control
FX_MIGRATION_HISTORY_IMPORT_MS control
FX_MIGRATION_LOGINS_IMPORT_MS both
FX_MIGRATION_LOGINS_JANK_MS both
FX_MIGRATION_BOOKMARKS_ROOTS control
FX_MIGRATION_BOOKMARKS_QUANTITY both
FX_MIGRATION_HISTORY_QUANTITY both
FX_MIGRATION_LOGINS_QUANTITY both
You need to log in before you can comment on or make changes to this bug.