1819058 - [fxaButtonVisibility] The “Click” telemetry event and scalar are not registered when the FxA button from the toolbar is clicked while enrolled in the treatment branch of an experiment

Status: NEW

(Firefox :: Firefox Accounts, defect)

Description

[Alice Varro [avarro], Ecosystem QA]

[Affected versions]:

• Firefox Nightly 112.0a1 - Build ID: 20230226214053

[Affected Platforms]:

• Windows 10 x64.
• macOS 12.6.
• Linux Mint 20.2 x64.

[Prerequisites]:

• Have the “nimbus.debug” pref set to true on the “about:config” page.
• Have the browser console opened (Ctrl/Cmd + Shift +J).

[Steps to reproduce]:
1.Navigate to the “about:studies?optin_slug=firefox-accounts-toolbar-button-default-visibility&optin_branch=treatment-a&optin_collection=nimbus-preview” URL to force enroll in the Treatment branch of the “Firefox Accounts toolbar button default visibility” experiment.
2. Focus the Browser console and restart the browser (Ctrl/Cmd + Option + R).
3. Click the FxA button from the toolbar.
4. Navigate to the “about:telemetry#events-tab” page and observe the displayed events.
5. Navigate to the “about:telemetry#keyed-scalars-tab” page and search for the “fxa” keyword.

[Expected result]:

• A click event related to the FxA button is displayed on the “about:telemetry#events-tab” (e.g.: fxa_avatar_menu click login null {"fxa_status": "not_configured", "fxa_avatar": "true"}).
• A keyed scalar related to the FxA button is displayed on the “about:telemetry#keyed-scalars-tab” (e.g.: fxa_avatar_menu#click#login).

[Actual result]:

• No click events/scalars related to the FxA button are displayed on any of the pages.

[Regression Window]:

• Using Mozregression tools I have found the regression range. Here are the results:

  Last good revision: bd14887e94e405a5f90b9663160ce37919a98de5
  First bad revision: 35ebaf6616d7dab56b1ee131aff918fef827df2d
  Pushlog: link
  From this Pushlog it seems that Bug 1818570 may have introduced this behavior.

[Notes]:

• This issue is not reproducible for the Control branch of the experiment.
• This issue is also reproducible when the user is enrolled in the experiment using a “user.js” file.
• Attaching a screen recording of the issue here.

Comment 1

[Mike Conley (:mconley) (:⚙️)]

Thanks avarro!

The good news is that we do get a keyed scalar set when the user clicks on the toolbar button - it's under browser.ui.interaction.nav_bar with the key of fxa-toolbar-menu-button. So we're covered there.

For the event telemetry, you're right - we're no longer collecting that in this case (because here, panel is undefined), but the previously mentioned keyed scalar and the arguments we're passing in the sign-up/sign-in URL should be sufficient for our purposes here.

Comment 2

[BugBot [:suhaib / :marco/ :calixte]]

Setting Regressed by field after analyzing regression range found by mozregression in comment #0.

Comment 3

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1818570

Comment 4

[Donal Meehan [:dmeehan]]

Setting 111 to wontfix. The regressor was uplifted to beta but this non-user-facing bug that doesn't block the experiment

Comment 5

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1818570

Comment 6

[Shane Hughes [:aminomancer]]

Recent QA on FXE-561 found that this bug is still affecting FxA toolbarbutton telemetry. QA signed off on the experiment anyway since this was not considered a blocker for previous experiments. However, I can't figure out how keyed scalar fxa-toolbar-menu-button under browser.ui.interaction.nav_bar is recorded. Are we sure it's still recorded? Is there an example on redash or looker of a query for this scalar?

Thank you!