Closed Bug 1480513 Opened 6 years ago Closed 6 years ago

Telemetry for Top Sites Search Shortcuts

Categories

(Firefox :: New Tab Page, enhancement, P1)

enhancement

Tracking

()

VERIFIED FIXED
Firefox 63
Iteration:
63.4 - Aug 20
Tracking Status
firefox62 --- verified
firefox63 --- verified

People

(Reporter: k88hudson, Assigned: nanj)

References

(Blocks 1 open bug)

Details

(Whiteboard: [has-pr])

Attachments

(1 file)

We will need the following telemetry implemented for Top Sites Search Shortcuts:

- clicks
- blocks
- adding/editing

Please include Maria (who is writing the PHD) and whoever the data scientist for this project to ensure we are covering everything we need. This will also need data review.
Iteration: --- → 63.4 - Aug 20
Assignee: nobody → najiang
Status: NEW → ASSIGNED
Priority: P2 → P1
1) What questions will you answer with this data?
To measure the user engagement of the search shortcut feature in Activity Stream, including how many clicks & blocks for each search vendor; how many users add & edit their search shortcuts. 

2) Why does Mozilla need to answer these questions?  Are there benefits for users? Do we need this information to address product or business requirements? Some example responses:
* Measure the search performance in Top Sites of Activity Stream.
* Determine whether those search shortcuts improve the search experience for Activity Stream.

3) What alternative methods did you consider to answer these questions? Why were they not sufficient?
There is no current instrumentation of this code.

4) Can current instrumentation answer these questions?
No.

5) List all proposed measurements and indicate the category of data collection for each measurement, using the Firefox [data collection categories](https://wiki.mozilla.org/Firefox/Data_Collection) on the Mozilla wiki.
* A new `card_type` called "search", and a new field `search_vendor` in the click/block user event for Top Sites, tracked in bug 1480513
* Two new user events for Top Sites, both tracked in bug 1480513:
  * Add a search shortcut
  * Edit a search shortcut

6) How long will this data be collected?  Choose one of the following:
I (:nanj) want to permanently monitor this data.

7) What populations will you measure?
All the channels for all the countries/locales.

8) If this data collection is default on, what is the opt-out mechanism for users?
The user can opt-out the data collection by either disabling the telemetry of Activity Stream or disabling the Firefox telemetry as a whole.

9) Please provide a general description of how you will analyze this data.
General reporting on Redash.

10) Where do you intend to share the results of your analysis?
This will be shared over Mozilla LDAP via Redash.
Flags: needinfo?(francois)
Depends on: 1480518
Hey Ryan, would you be able to look a the attached pull request / comments to see if the data collected as described here will be enough to satisfy the data analysis requirements for this upcoming search shortcut experiment?

Also, can you confirm is client_id is required or not to do the required analysis for this project?

Thanks!
Flags: needinfo?(rharter)
For further context, below if the PHD for the experiment:

https://docs.google.com/document/d/1mjNXo8Lddd0-kOr6yQ_x68z-gcZNYtjOopIMOByl-ko/edit#
Whiteboard: [has-pr]
(In reply to Nan Jiang [:nanj] from comment #2)

datareview+

1) Is there or will there be **documentation** that describes the schema for the ultimate data set available publicly, complete and accurate?

Yes, in https://github.com/mozilla/activity-stream/blob/master/docs/v2-system-addon/data_dictionary.md and https://github.com/mozilla/activity-stream/blob/master/docs/v2-system-addon/data_events.md.

2) Is there a control mechanism that allows the user to turn the data collection on and off?

Yes, either the usual telemetry setting, or the browser.newtabpage.activity-stream.telemetry pref to disable just the Activity Stream telemetry.

3) If the request is for permanent data collection, is there someone who will monitor the data over time?**

Yes, :nanj.

4) Using the **[category system of data types](https://wiki.mozilla.org/Firefox/Data_Collection)** on the Mozilla wiki, what collection type of data do the requested measurements fall under?  **

Category 2.

5) Is the data collection request for default-on or default-off?

Default ON.

6) Does the instrumentation include the addition of **any *new* identifiers** (whether anonymous or otherwise; e.g., username, random IDs, etc.  See the appendix for more details)?

No.

7) Is the data collection covered by the existing Firefox privacy notice?

Yes.

8) Does there need to be a check-in in the future to determine whether to renew the data?

No, permanent.
Flags: needinfo?(francois)
Commit pushed to master at https://github.com/mozilla/activity-stream

https://github.com/mozilla/activity-stream/commit/1f2cdf8858a9efacd74ada64adc05bab703b840a
Bug 1480513: Add telemetry for search shortcut (#4290)

Fix Bug 1480513 : part 2 - add telemetry for search shortcut
Blocks: 1482205
https://hg.mozilla.org/mozilla-central/rev/972b3c48c6e9
Status: ASSIGNED → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
Target Milestone: --- → Firefox 63
Thanks, Kate. I'll need more information about the experiment to be able to comment on whether that data collection is sufficient. Is there a PHD or a writeup of the experiment goals somewhere?
Flags: needinfo?(rharter) → needinfo?(khudson)
(In reply to Ryan Harter [:harter] from comment #8)
> Thanks, Kate. I'll need more information about the experiment to be able to
> comment on whether that data collection is sufficient. Is there a PHD or a
> writeup of the experiment goals somewhere?

Hi Ryan, the PHD doc was attached in comment 4.
Flags: needinfo?(khudson)
Thanks, Kate. I've added some comments to the PHD and below. Maybe we should schedule 30m to discuss?

It looks like the metrics we're interested in tracking are:

* Search Volume (from SAPs)
* Retention
* Usage
* Satisfaction

There are two experiments proposed in that document. For experiment 1 we should record whether we filtered any tiles at a per user level in experiment and whether we would have filtered tiles in control. This will allow us to measure the effect size over users that are affected by this change. 

Otherwise, search volume and retention are both defined over data available in main_summary. What are the definitions of Usage and Satisfaction? Have these been defined for previous AS studies?

The PHD mentions CTR for search tiles, but it's unclear whether this metric is a decision criterion. Is this the metric that's instrumented in the attached PR? I see an event ping which records user interactions with search tiles, but I don't see a count of how frequently these tiles are shown to users.

These experiments are listed as independent experiments. Is that accurate or is this actually a three branch experiment? E.g.:
* Control: {noDefaultSearchTile: False, topSiteSearchShortcuts: False}
* Expt 1 : {noDefaultSearchTile: True,  topSiteSearchShortcuts: False}
* Expt 2 : {noDefaultSearchTile: False, topSiteSearchShortcuts: True}
* Expt 3 : {noDefaultSearchTile: True,  topSiteSearchShortcuts: True}
Flags: needinfo?(khudson)
We don't currently have the means to measure whether we show an individual tile to particular users (or whether we would have) although we might still have time to implement it / get review. I think having a meeting to discuss this and the experiment variants (my understanding is that the two experiments would be tested separately) with Maria (product) sounds like a good idea.

Maria will follow up with a meeting request today. Thanks!
Flags: needinfo?(khudson)
Would this need to also land on 62?
(In reply to Liz Henry (:lizzard) (needinfo? me) from comment #12)
> Would this need to also land on 62?

Yes, all the uplifts for 62 are tracked in bug 1482398.
Blocks: 1482398
See Also: → 1483065
I have verified that the issue is no longer reproducible in the latest Nightly (63.0a1, Build ID 20180816100035) and Beta (62.0b18 Build ID 20180816151750) on Windows 10, Mac 10.13 and Arch Linux x64. Pings for Clicking, adding, removing, dismissing, pinning, unpinning have been added.
Status: RESOLVED → VERIFIED
Component: Activity Streams: Newtab → New Tab Page
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: