Closed Bug 1665391 Opened 4 years ago Closed 3 years ago

Add telemetry to measure how successful the bookmarks toolbar interaction changes are

Categories

(Firefox :: Bookmarks & History, enhancement, P3)

enhancement

Tracking

()

RESOLVED FIXED
85 Branch
Tracking Status
firefox84 --- fixed
firefox85 --- fixed

People

(Reporter: jaws, Assigned: jaws)

References

(Depends on 1 open bug)

Details

User Story

We will measure success of the bookmarking changes through these measures :
- increase retrieval of bookmarks through the bookmarks toolbar
- increase share of new users importing bookmarks from third party browsers
- no more than X% of users select “disable always” on the new bookmark toolbar selection menu

We hypothesize this will increase the following core metrics:
- increase new user retention for bookmark users
- increase new user activation for bookmark users

Telemetry probes needed to support these goals:
- Bookmark toolbar visibility indicator (enable always, disable always, enable only on new tab) (per profile per session)
- Bookmark import from bookmark toolbar indicator (Perhaps add "Bookmarks Toolbar" entry point to https://probes.telemetry.mozilla.org/?search=fx_migration_entry_point&view=detail&probeId=histogram%2FFX_MIGRATION_ENTRY_POINT)  (per profile per session)
- Counters for saved and opened bookmarks from the bookmarks toolbar  (per profile per session)

Attachments

(2 files)

  • Identify newtab bookmark usage VS manually enabled toolbar usage
  • Identify users importing bookmarks from the bookmarks toolbar. See https://sql.telemetry.mozilla.org/queries/74424/source that already has entry points for running the data migration tool
  • Identify saved and opened bookmarks per location (bookmarks toolbar VS “Other bookmarks VS bookmarks menu VS other)
  • Identify if the bookmark toolbar is enabled and why (user forced toolbar enablement through keyboard shortcut, user forced toolbar enablement through menu action, bookmark toolbar automatically appeared upon bookmark addition, toolbar never enabled, toolbar disabled post user forcing enablement, toolbar disabled post auto-enablement)

Per some discussion today, Romain, can you add some more reasoning around why we want this data and what/how we should collect it?

Flags: needinfo?(rtestard)
User Story: (updated)
Flags: needinfo?(rtestard)
User Story: (updated)
User Story: (updated)
User Story: (updated)
User Story: (updated)
Assignee: nobody → jaws
Status: NEW → ASSIGNED
Attached file Data review form
Attachment #9188755 - Flags: data-review?(chutten)

Comment on attachment 9188755 [details]
Data review form

PRELIMINARY NOTES:
For permanent collections it is best practice to add regression tests to ensure your collections collect the expected values when your instrumented code runs.

DATA COLLECTION REVIEW RESPONSE:

Is there or will there be documentation that describes the schema for the ultimate data set available publicly, complete and accurate?

Yes.

Is there a control mechanism that allows the user to turn the data collection on and off?

Yes. This collection is Telemetry so can be controlled through Firefox's Preferences.

If the request is for permanent data collection, is there someone who will monitor the data over time?

Yes, :jaws is responsible.

Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Category 2, Interaction.

Is the data collection request for default-on or default-off?

Default on for all channels.

Does the instrumentation include the addition of any new identifiers?

No.

Is the data collection covered by the existing Firefox privacy notice?

Yes.

Does there need to be a check-in in the future to determine whether to renew the data?

No. This collection is permanent.


Result: datareview+

Attachment #9188755 - Flags: data-review?(chutten) → data-review+
Pushed by jwein@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/2db51a17d69c
Add telemetry to measure effectiveness of 2h2020 bookmarks changes. r=Gijs

Comment on attachment 9188211 [details]
Bug 1665391 - Add telemetry to measure effectiveness of 2h2020 bookmarks changes.

Beta/Release Uplift Approval Request

  • User impact if declined: No user impact, but Mozilla will be unable to understand the positive or negative user affect of the 2h2020 bookmarks change experiments.
  • Is this code covered by automated tests?: Yes
  • Has the fix been verified in Nightly?: No
  • Needs manual test from QE?: Yes
  • If yes, steps to reproduce: 1a. Open the browser with each of the Bookmarks Toolbar settings (always, never, only on new tab).
    2a. Load about:telemetry
    3a. Click on Keyed Scalars
    4a. Search for "bookmark"
    5a. We should have a value of "bookmarks-bar_pinned_on", "bookmarks-bar_pinned_off", or "bookmarks-bar_pinned_newtab" that matches the setting from #1a

1b. Save a bookmark to the Bookmarks Toolbar
2b. Load about:telemetry
3b. Click on Scalars
4b. Search for "bookmark"
5b. We should have a key of "browser.engagement.bookmarks_toolbar_bookmark_added" that has a value of 1
6b. Move a bookmark to the Bookmarks Toolbar
7b. Repeat 2b-5b, the value should now be 2.

1c. Click on a bookmark in the Bookmarks Toolbar
2c. Load about:telemetry
3c. Click on Scalars
4c. Search for "bookmark"
5c. We should have a key of "browser.engagement.bookmarks_toolbar_bookmark_opened" that has a value of 1
6c. Click another bookmark in the Bookmarks Toolbar
7c. Repeat 2b-5b, the value should now be 2.

  • List of other uplifts needed: None
  • Risk to taking this patch: Low
  • Why is the change risky/not risky? (and alternatives if risky): No behavior change, just adding telemetry to track actions already implemented
  • String changes made/needed: None
Attachment #9188211 - Flags: approval-mozilla-beta?
Flags: qe-verify+
Status: ASSIGNED → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
Target Milestone: --- → 85 Branch

Comment on attachment 9188211 [details]
Bug 1665391 - Add telemetry to measure effectiveness of 2h2020 bookmarks changes.

Approved for 84.0b4.

Attachment #9188211 - Flags: approval-mozilla-beta? → approval-mozilla-beta+
QA Whiteboard: [qa-triaged]

Hello,

While testing these scenarios, for 5B when I reach that step its already at value 2 not 1.

Also after moving a bookmark to the toolbar (a preloaded bookmark "Customize Firefox" so as to not add another telemetry value) the value also goes up by 1 for moving a bookmark to the toolbar.

So because of this before reaching 7B, I have 3 as the value.

Flags: needinfo?(jaws)

Hi Daniel,

If you load about:telemetry before step 1b is the value already at 1? You should be able to walk through the steps of comment 6 with about:telemetry open to the Scalars view with the "bookmark" filter applied and refresh the page between each step. Can you find when the value is initialized to 1? I am unable to reproduce the issue you are seeing.

Flags: needinfo?(jaws) → needinfo?(daniel.cicas)

Hello!

Sorry for the long delay I was on PTO.

I investigated this issue on 84.0b7 and I think I found where the issue is.

  1. Start Firefox with fresh profile (at this moment browser.toolbars.bookmarks.2h2020 is set to false)
  2. Reach about:telemetry and in Scalars search for bookmarks and you should see the telemetry ping having a value of 1 (Because the Getting Started default bookmark is counted).

Basically the default bookmark (Getting Started) is added to the count even before starting the step 1b.

Besides this issue, I have 3 more questions:

  1. Is it intended for telemetry values to be recorded even if browser.toolbars.bookmarks.2h2020 is set to false?
  2. For step 6B should moving a bookmark from Bookmarks Menu to Bookmarks Toolbar should that increase the count ?
  3. It seems that moving one bookmark from one position on the toolbar to another also increases the count. Should I file a new bug for this issue?
Flags: needinfo?(daniel.cicas) → needinfo?(jaws)

Great, thanks for tracking that down.

In response to your questions:

  1. Yes, telemetry should be recorded regardless of that pref value.
  2. Yes, moving a bookmark from the Bookmarks Menu to Bookmarks Toolbar should increase the "browser.engagement.bookmarks_toolbar_bookmark_added" count.
  3. This is a bug, can you please file a bug for this?

Thanks!

Flags: needinfo?(jaws)

(In reply to Jared Wein [:jaws] (please needinfo? me) from comment #13)

  1. This is a bug, can you please file a bug for this?

Bouncing needinfo for this.

Flags: needinfo?(daniel.cicas)

Hello,

I created Bug 1682553 for the issue in comment 13.

Flags: needinfo?(daniel.cicas) → needinfo?(jaws)
Flags: needinfo?(jaws)
You need to log in before you can comment on or make changes to this bug.