1932894 - [Experiment] The "customize homepage" button from the menu should not generate a telemetry event for the homepage button

Status: VERIFIED FIXED

(Fenix :: Experimentation and Telemetry, defect, P2)

Description

[Valentin Bandac [:vbandac], Ecosystem QA]

[Affected versions]:

• Firefox Nightly 134.0a1 (BUILD ID #2016057442), hg-9e9cd26f20ec+, GV: 134.0a1-20241121092921, AS: 134.20241120050405

[Affected Platforms]:

• Samsung A53 - Android 14
• OnePlus 6 - Android 11

[Prerequisites]:

• Have the latest Firefox Nightly 134.0a1 installed, but not opened.
• Have the latest version of nimbus-cli installed.
• Have the mobile device connected to a computer using adb.

[Steps to reproduce]:

1. Run the following command in a terminal: nimbus-cli --app fenix --channel nightly enroll preview/combined-android-toolbar-and-menu-experiment --branch treatment-a.
2. Close the browser and open it using a telemetry command (adb shell am start -n org.mozilla.fenix/mozilla.telemetry.glean.debug.GleanDebugActivity --ez logPings true --es sendPing events --es debugViewTag <chosen_tag>).
3. Tap the 3 dot menu on the Homepage and tap the “Customize homepage” button.
4. Navigate to the Glean viewer and observe the events displayed.

[Expected result]:

• Step 4: The following ping is displayed:

  "timestamp": 2116,
  "category": "app_menu",
  "name": "customize_homepage",
  "extra": {
  "glean_timestamp": "1732284150353"

[Actual result]:

• Step 4: The following ping is displayed:

{
"timestamp": 2116,
"category": "app_menu",
"name": "customize_homepage",
"extra": {
"glean_timestamp": "1732284150353"
}
},
{
"timestamp": 2117,
"category": "home_screen",
"name": "customize_home_clicked",
"extra": {
"glean_timestamp": "1732284150353"
}

[Notes]:

This issue is reproducible on the Treatment B branch as well.

Here is an example ping exposing the issue:
https://debug-ping-preview.firebaseapp.com/pings/vb4/eb84d1cf-a4ea-4efc-98fa-ae183e81f0d8

Comment 1

[Iorga Gabriel]

Attachment: Bug 1932894 - [Experiment] The "customize homepage" button from the menu should not generate a telemetry event for the homepage button.

Comment 2

[Gabriel Luong [:gl] (ΦωΦ)]

The expected and actual results provided in https://bugzilla.mozilla.org/show_bug.cgi?id=1932894#c0 seems to line up here so I am not sure what the problem is. Both of the telemetry probes of the "app_menu" and "home_screen" descriptions does describe opening customize homepage from the menu:
https://searchfox.org/mozilla-central/source/mobile/android/fenix/app/metrics.yaml#9992
https://searchfox.org/mozilla-central/source/mobile/android/fenix/app/metrics.yaml#7380

Comment 3

[Gabriel Luong [:gl] (ΦωΦ)]

Requesting further clarification given the expected and actual results are aligned other than the ordering.

Comment 4

[Valentin Bandac [:vbandac], Ecosystem QA]

Hello! It seems that my wording has caused some issues. I’ll change the report to hopefully clear things up.

The problem that I was trying to bring attention to is that tapping the “Customize homepage” button on the Main menu generates an additional telemetry event. This event is the same one that is fired when tapping on the “Customize homepage” button on the Home screen.

Comment 5

[Petru-Mugurel Lingurar [:petru]]

Looked also into this as one of the last issues regarding the menu redesign.

We have two places when on the homescreen from where users can go to the homepage customization settings:

• the "Customize homepage" button from the button of the homescreen

  • this already records HomeScreen.customizeHomeClicked.record(NoExtras()) from here

• the "Customize homepage" from menu

  • in the old menu this button records
    • AppMenu.customizeHomepage.record(NoExtras()) from here and
    • HomeScreen.customizeHomeClicked.record(NoExtras()) from here.
      This seems like an error. Why does the menu records telemetry for the homescreen and AppMenu.customizeHomepage is not enough?
  • in the new menu this button records
    • both AppMenu.customizeHomepage.record(NoExtras()) and HomeScreen.customizeHomeClicked.record(NoExtras()) from here.
      It's great that we're consistent with the "old" behaviour but not if that behaviour is wrong.

Looks to me like we have two different events for two different scenarios so each scenario should use just one probe.
Will ask product for the final decision.

Comment 6

[Petru-Mugurel Lingurar [:petru]]

Got a response from Channing, we should:

• keep HomeScreen.customizeHomeClicked for just the home screen interaction
• keep AppMenu.customizeHomepage for just the home menu interaction

Based on gl's comment 2 we should also update the documentation for HomeScreen.customizeHomeClicked to specify that this event relates to interaction with the "Customize homepage" from the home screen, not from the menu.

Comment 7

[Gabriel Luong [:gl] (ΦωΦ)]

One thing to consider is how this might look like less people are clicking on the the customize home button on the home screen when looking at any home screen telemetry dashboard. The caveat here is that the true value of how often someone is clicking on the customize home button homepage is subtracting AppMenu.customizeHomepage from HomeScreen.customizeHomeClicked to get the actual home screen customize home button count

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/62404a964303
[Experiment] The "customize homepage" button from the menu should not generate a telemetry event for the homepage button. r=android-reviewers,gl

Comment 9

[agoloman]

https://hg.mozilla.org/mozilla-central/rev/62404a964303

Comment 10

[BugBot [:suhaib / :marco/ :calixte]]

The patch landed in nightly and beta is affected.
:giorga, is this bug important enough to require an uplift?

• If yes, please nominate the patch for beta approval.
• If no, please set status-firefox134 to wontfix.

For more information, please visit BugBot documentation.

Comment 11

[Valentin Bandac [:vbandac], Ecosystem QA]

I have verified that this issue is no longer reproducible using Firefox Beta 135.0b3 (Build #2016067071, hg-0f3e062a57dc+, GV: 135.0-20250110123854, AS: 135.0) installed on OnePlus 6 (Android 11) and Pixel 8 (Android 14). I can confirm that the "home_screen.customize_home_clicked" telemetry event is no longer generated when tapping the “Customize homepage” button on the main menu.
Considering this I will mark the issue as Verified - Fixed.