1665391 - Add telemetry to measure how successful the bookmarks toolbar interaction changes are

Status: RESOLVED FIXED

(Firefox :: Bookmarks & History, enhancement, P3)

Description

[Jared Wein [:jaws] (please needinfo? me)]

• Identify newtab bookmark usage VS manually enabled toolbar usage
• Identify users importing bookmarks from the bookmarks toolbar. See https://sql.telemetry.mozilla.org/queries/74424/source that already has entry points for running the data migration tool
• Identify saved and opened bookmarks per location (bookmarks toolbar VS “Other bookmarks VS bookmarks menu VS other)
• Identify if the bookmark toolbar is enabled and why (user forced toolbar enablement through keyboard shortcut, user forced toolbar enablement through menu action, bookmark toolbar automatically appeared upon bookmark addition, toolbar never enabled, toolbar disabled post user forcing enablement, toolbar disabled post auto-enablement)

Comment 1

[:Gijs (he/him)]

Per some discussion today, Romain, can you add some more reasoning around why we want this data and what/how we should collect it?

Comment 2

[Jared Wein [:jaws] (please needinfo? me)]

Attachment: Bug 1665391 - Add telemetry to measure effectiveness of 2h2020 bookmarks changes.

Comment 3

[Jared Wein [:jaws] (please needinfo? me)]

Attachment: Data review form

Comment 4

[Chris H-C :chutten]

Comment on attachment 9188755 [details]
Data review form

PRELIMINARY NOTES:
For permanent collections it is best practice to add regression tests to ensure your collections collect the expected values when your instrumented code runs.

DATA COLLECTION REVIEW RESPONSE:

Is there or will there be documentation that describes the schema for the ultimate data set available publicly, complete and accurate?

Yes.

Is there a control mechanism that allows the user to turn the data collection on and off?

Yes. This collection is Telemetry so can be controlled through Firefox's Preferences.

If the request is for permanent data collection, is there someone who will monitor the data over time?

Yes, :jaws is responsible.

Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Category 2, Interaction.

Is the data collection request for default-on or default-off?

Default on for all channels.

Does the instrumentation include the addition of any new identifiers?

No.

Is the data collection covered by the existing Firefox privacy notice?

Yes.

Does there need to be a check-in in the future to determine whether to renew the data?

No. This collection is permanent.

---

Result: datareview+

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/2db51a17d69c
Add telemetry to measure effectiveness of 2h2020 bookmarks changes. r=Gijs

Comment 6

[Jared Wein [:jaws] (please needinfo? me)]

Comment on attachment 9188211 [details]
Bug 1665391 - Add telemetry to measure effectiveness of 2h2020 bookmarks changes.

Beta/Release Uplift Approval Request

• User impact if declined: No user impact, but Mozilla will be unable to understand the positive or negative user affect of the 2h2020 bookmarks change experiments.
• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: No
• Needs manual test from QE?: Yes
• If yes, steps to reproduce: 1a. Open the browser with each of the Bookmarks Toolbar settings (always, never, only on new tab).
  2a. Load about:telemetry
  3a. Click on Keyed Scalars
  4a. Search for "bookmark"
  5a. We should have a value of "bookmarks-bar_pinned_on", "bookmarks-bar_pinned_off", or "bookmarks-bar_pinned_newtab" that matches the setting from #1a

1b. Save a bookmark to the Bookmarks Toolbar
2b. Load about:telemetry
3b. Click on Scalars
4b. Search for "bookmark"
5b. We should have a key of "browser.engagement.bookmarks_toolbar_bookmark_added" that has a value of 1
6b. Move a bookmark to the Bookmarks Toolbar
7b. Repeat 2b-5b, the value should now be 2.

1c. Click on a bookmark in the Bookmarks Toolbar
2c. Load about:telemetry
3c. Click on Scalars
4c. Search for "bookmark"
5c. We should have a key of "browser.engagement.bookmarks_toolbar_bookmark_opened" that has a value of 1
6c. Click another bookmark in the Bookmarks Toolbar
7c. Repeat 2b-5b, the value should now be 2.

• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): No behavior change, just adding telemetry to track actions already implemented
• String changes made/needed: None

Comment 7

[Atila Butkovits]

https://hg.mozilla.org/mozilla-central/rev/2db51a17d69c

Comment 8

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 9188211 [details]
Bug 1665391 - Add telemetry to measure effectiveness of 2h2020 bookmarks changes.

Approved for 84.0b4.

Comment 9

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-beta/rev/d25f8c26e98a

Comment 10

[Daniel Cicas [:dcicas], Release QA]

Hello,

While testing these scenarios, for 5B when I reach that step its already at value 2 not 1.

Also after moving a bookmark to the toolbar (a preloaded bookmark "Customize Firefox" so as to not add another telemetry value) the value also goes up by 1 for moving a bookmark to the toolbar.

So because of this before reaching 7B, I have 3 as the value.

Comment 11

[Jared Wein [:jaws] (please needinfo? me)]

Hi Daniel,

If you load about:telemetry before step 1b is the value already at 1? You should be able to walk through the steps of comment 6 with about:telemetry open to the Scalars view with the "bookmark" filter applied and refresh the page between each step. Can you find when the value is initialized to 1? I am unable to reproduce the issue you are seeing.

Comment 12

[Daniel Cicas [:dcicas], Release QA]

Hello!

Sorry for the long delay I was on PTO.

I investigated this issue on 84.0b7 and I think I found where the issue is.

1. Start Firefox with fresh profile (at this moment browser.toolbars.bookmarks.2h2020 is set to false)
2. Reach about:telemetry and in Scalars search for bookmarks and you should see the telemetry ping having a value of 1 (Because the Getting Started default bookmark is counted).

Basically the default bookmark (Getting Started) is added to the count even before starting the step 1b.

Besides this issue, I have 3 more questions:

1. Is it intended for telemetry values to be recorded even if browser.toolbars.bookmarks.2h2020 is set to false?
2. For step 6B should moving a bookmark from Bookmarks Menu to Bookmarks Toolbar should that increase the count ?
3. It seems that moving one bookmark from one position on the toolbar to another also increases the count. Should I file a new bug for this issue?

Comment 13

[Jared Wein [:jaws] (please needinfo? me)]

Great, thanks for tracking that down.

In response to your questions:

1. Yes, telemetry should be recorded regardless of that pref value.
2. Yes, moving a bookmark from the Bookmarks Menu to Bookmarks Toolbar should increase the "browser.engagement.bookmarks_toolbar_bookmark_added" count.
3. This is a bug, can you please file a bug for this?

Thanks!

Comment 14

[:Gijs (he/him)]

(In reply to Jared Wein [:jaws] (please needinfo? me) from comment #13)

3. This is a bug, can you please file a bug for this?

Bouncing needinfo for this.

Comment 15

[Daniel Cicas [:dcicas], Release QA]

Hello,

I created Bug 1682553 for the issue in comment 13.