1770737 - [Experiment] There are no telemetry pings displayed in the "Browser Console" for the "Import From Another Browser" flow

Status: RESOLVED INVALID

(Firefox :: Messaging System, defect, P2)

Description

[Marius Coman [:mcoman], Ecosystem QA]

Attachment: rec of the issue.gif

[Additional Notes]:

• On the "about:welcome" page for the "Import" screen a "migrate_close" event is displayed in the "Browser Console" after the "Import" process is completed or canceled.
• If the "Import From Another Browser" process is completed from the "File" menu, a "menu-importFromAnotherBrowser" scalar is displayed in the "about:telemetry" page.

[Affected versions]:

• Firefox Release 100.0.2 - Build ID: 20220519220738
• Firefox Beta 101.0b9 - Build ID: 20220519220322
• Firefox Nightly 102.0a1 - Build ID: 20220522190601

[Affected Platforms]:

• Windows 10 x64
• macOS 11.6.2

[Prerequisites]:

• Have the following prefs in the "about:config" page:
  • "nimbus.debug" pref set to true.
  • "browser.ping-centre.log" set to true.
• Have the "Browser Console" opened.

[Steps to reproduce]:

1. Open the browser with the profile from the prerequisites.
2. Navigate to the "about:studies?optin_slug=import-infrequent-eco2204-en&optin_branch=treatment-a&optin_collection=nimbus-preview" URL.
3. Restart the browser and click the spotlight's "Import to Firefox" button.
4. Complete or Cancel the "Import" flow and focus the "Browser Console".
5. Observe the displayed telemetry pings.

[Expected result]:

• A telemetry ping related to the "Import" flow is displayed.

[Actual result]:

• There are NO telemetry pings displayed for the "Import" flow.

[Additional Notes]:

• There are no "Import" pings/events displayed in the "about:telemetry" page after the "Import From Another Browser" flow is completed/canceled from the "Import" ECO spotlight.
• Attached a screen recording of the issue.

Comment 1

[Dan Mosedale (:dmosedale, :dmose)]

We do get Event Telemetry when clicking on the "Import from Firefox" button on the Spotlight itself.

As you've pointed out, we don't get Event Telemetry from the migration code itself. The migration code itself does send scalars for various things (eg FX_MIGRATION_BOOKMARKS_QUANTITY) as part of the Main Pings (not through Ping Centre / Event Telemetry at all). We might be able to use client_ids to correlate that tightly enough in time with the Spotlight telemetry to be able to use, but I suspect it would require waiting some amount of time before analysis, because I don't know how quickly/reliably the main pings tend to arrive. I believe Akash (whom I've NIed here) is reaching out to the data folks to understand how they've handled tracking migration experiments in the past (or if they think trying to use the non-event scalars is a good idea?). Akash, feel free to point the data folks to this bug, if that's helpful...

If the "Import From Another Browser" process is completed from the "File" menu, a "menu-importFromAnotherBrowser" scalar is displayed in the "about:telemetry" page.

This comes from Browser Usage Telemetry, which doesn't cover the migration UI. It's conceivable that we could extend it do so, though I haven't really looked into that. It would undoubtedly require riding the trains and/or uplifts.

On the "about:welcome" page for the "Import" screen a "migrate_close" event is displayed in the "Browser Console" after the "Import" process is completed or canceled.

That's implemented in code that has ridden the trains as part of AboutWelcome. If we graft (or make cleanly reusable and call) some of the related code as part of renderMultistage in spotlight.js. That too will need a train ride and/or uplift. If the data science folks don't pull any magic out of their hats, our best bet maybe to land this in 102 and run the experiment in 102. Or we try to squeeze it into 101 with an uplift.

Comment 2

[Akash Mathur [:akashmathur]]

Thanks for the flag on this - I've sent this bug to the data team as well, will update as soon as we know more

Comment 3

[Akash Mathur [:akashmathur]]

From Data Science:

Yea, tracking imports can be tricky. You’d need to measure the following:
(1) clicks on the message (2) import wizard opens (3) import wizard completions
and probably the
(4) import result, meaning did items things actually get imported
The first can be measured with the message telemetry, but 2-4 need to be measured with some, frankly, extremely messy and complex measurements inside the browser telemetry.
I’m not sure if there’s prior art built into mozanalysis or jetstream that would facilitate this

The recommendation is to open a DS ticket, and so this might need to land in 102

Comment 4

[Punam Dahiya [:pdahiya]]

NI @dmosedale to help figure telemetry ask after synch up with DS

Comment 5

[Dan Mosedale (:dmosedale, :dmose)]

https://mozilla-hub.atlassian.net/browse/DS-2015 has been filed, which effectively blocks this.

Comment 6

[Dan Mosedale (:dmosedale, :dmose)]

After talking to Su, the existing bookmarks telemetry is good enough to do analysis with, but it's complex and error prone analysis. My understanding is that he'd like DS to write the analysis code once (into the mozanalysis Python package) and have it reviewed, so they won't have to do this repeatedly rather than analyzing this experiment as another one-off. I'm going to repurpose DS-2015 for that work, and check with product on how best to get that work prioritized.

In other words, there's not an actual bug here -- we don't need any new telemetry, and they're happy to have us run the experiment now, but we won't be able to get it analyzed until the DS-2015 work is completed. I've just checked on my Mac that telemetry does show in about:telemetry (search for "migration" without the quotes) after importing into Chrome.

It's probably worth verifying that on Windows, and assuming it works, I think this bug can be closed as invalid.

Comment 7

[Dan Mosedale (:dmosedale, :dmose)]

https://searchfox.org/mozilla-central/source/toolkit/components/telemetry/Histograms.json#10091 is where you can find the sorts of histograms that should be visible.

Comment 8

[Marius Coman [:mcoman], Ecosystem QA]

Hi, Dan! I can confirm that the following Histograms and Keyed Histograms are displayed in the "about:telemetry" page after an import action was performed:

Histograms:

• FX_MIGRATION_ENTRY_POINT
• FX_MIGRATION_SOURCE_BROWSER

Keyed Histograms:

• FX_MIGRATION_USAGE
• FX_MIGRATION_BOOKMARKS_QUANTITY
• FX_MIGRATION_HISTORY_QUANTITY
• FX_MIGRATION_LOGINS_QUANTITY

I have tested this using the latest Firefox Release (101.0 Build ID - 20220526203855) on Windows 10 x64, macOS 11.6, and Linux Mint 20.2 x64.

Comment 9

[Dan Mosedale (:dmosedale, :dmose)]

The telemetry we've got should be sufficient; thanks for the recheck, Marius!