Closed
Bug 1480513
Opened 6 years ago
Closed 6 years ago
Telemetry for Top Sites Search Shortcuts
Categories
(Firefox :: New Tab Page, enhancement, P1)
Firefox
New Tab Page
Tracking
()
People
(Reporter: k88hudson, Assigned: nanj)
References
(Blocks 1 open bug)
Details
(Whiteboard: [has-pr])
Attachments
(1 file)
We will need the following telemetry implemented for Top Sites Search Shortcuts: - clicks - blocks - adding/editing Please include Maria (who is writing the PHD) and whoever the data scientist for this project to ensure we are covering everything we need. This will also need data review.
Updated•6 years ago
|
Iteration: --- → 63.4 - Aug 20
Updated•6 years ago
|
Assignee: nobody → najiang
Status: NEW → ASSIGNED
Priority: P2 → P1
Comment 1•6 years ago
|
||
Assignee | ||
Comment 2•6 years ago
|
||
1) What questions will you answer with this data? To measure the user engagement of the search shortcut feature in Activity Stream, including how many clicks & blocks for each search vendor; how many users add & edit their search shortcuts. 2) Why does Mozilla need to answer these questions? Are there benefits for users? Do we need this information to address product or business requirements? Some example responses: * Measure the search performance in Top Sites of Activity Stream. * Determine whether those search shortcuts improve the search experience for Activity Stream. 3) What alternative methods did you consider to answer these questions? Why were they not sufficient? There is no current instrumentation of this code. 4) Can current instrumentation answer these questions? No. 5) List all proposed measurements and indicate the category of data collection for each measurement, using the Firefox [data collection categories](https://wiki.mozilla.org/Firefox/Data_Collection) on the Mozilla wiki. * A new `card_type` called "search", and a new field `search_vendor` in the click/block user event for Top Sites, tracked in bug 1480513 * Two new user events for Top Sites, both tracked in bug 1480513: * Add a search shortcut * Edit a search shortcut 6) How long will this data be collected? Choose one of the following: I (:nanj) want to permanently monitor this data. 7) What populations will you measure? All the channels for all the countries/locales. 8) If this data collection is default on, what is the opt-out mechanism for users? The user can opt-out the data collection by either disabling the telemetry of Activity Stream or disabling the Firefox telemetry as a whole. 9) Please provide a general description of how you will analyze this data. General reporting on Redash. 10) Where do you intend to share the results of your analysis? This will be shared over Mozilla LDAP via Redash.
Flags: needinfo?(francois)
Reporter | ||
Comment 3•6 years ago
|
||
Hey Ryan, would you be able to look a the attached pull request / comments to see if the data collected as described here will be enough to satisfy the data analysis requirements for this upcoming search shortcut experiment? Also, can you confirm is client_id is required or not to do the required analysis for this project? Thanks!
Flags: needinfo?(rharter)
Comment 4•6 years ago
|
||
For further context, below if the PHD for the experiment: https://docs.google.com/document/d/1mjNXo8Lddd0-kOr6yQ_x68z-gcZNYtjOopIMOByl-ko/edit#
Reporter | ||
Updated•6 years ago
|
Whiteboard: [has-pr]
Comment 5•6 years ago
|
||
(In reply to Nan Jiang [:nanj] from comment #2) datareview+ 1) Is there or will there be **documentation** that describes the schema for the ultimate data set available publicly, complete and accurate? Yes, in https://github.com/mozilla/activity-stream/blob/master/docs/v2-system-addon/data_dictionary.md and https://github.com/mozilla/activity-stream/blob/master/docs/v2-system-addon/data_events.md. 2) Is there a control mechanism that allows the user to turn the data collection on and off? Yes, either the usual telemetry setting, or the browser.newtabpage.activity-stream.telemetry pref to disable just the Activity Stream telemetry. 3) If the request is for permanent data collection, is there someone who will monitor the data over time?** Yes, :nanj. 4) Using the **[category system of data types](https://wiki.mozilla.org/Firefox/Data_Collection)** on the Mozilla wiki, what collection type of data do the requested measurements fall under? ** Category 2. 5) Is the data collection request for default-on or default-off? Default ON. 6) Does the instrumentation include the addition of **any *new* identifiers** (whether anonymous or otherwise; e.g., username, random IDs, etc. See the appendix for more details)? No. 7) Is the data collection covered by the existing Firefox privacy notice? Yes. 8) Does there need to be a check-in in the future to determine whether to renew the data? No, permanent.
Flags: needinfo?(francois)
Comment 6•6 years ago
|
||
Commit pushed to master at https://github.com/mozilla/activity-stream https://github.com/mozilla/activity-stream/commit/1f2cdf8858a9efacd74ada64adc05bab703b840a Bug 1480513: Add telemetry for search shortcut (#4290) Fix Bug 1480513 : part 2 - add telemetry for search shortcut
Comment 7•6 years ago
|
||
https://hg.mozilla.org/mozilla-central/rev/972b3c48c6e9
Status: ASSIGNED → RESOLVED
Closed: 6 years ago
status-firefox63:
--- → fixed
Resolution: --- → FIXED
Target Milestone: --- → Firefox 63
Comment 8•6 years ago
|
||
Thanks, Kate. I'll need more information about the experiment to be able to comment on whether that data collection is sufficient. Is there a PHD or a writeup of the experiment goals somewhere?
Flags: needinfo?(rharter) → needinfo?(khudson)
Assignee | ||
Comment 9•6 years ago
|
||
(In reply to Ryan Harter [:harter] from comment #8) > Thanks, Kate. I'll need more information about the experiment to be able to > comment on whether that data collection is sufficient. Is there a PHD or a > writeup of the experiment goals somewhere? Hi Ryan, the PHD doc was attached in comment 4.
Reporter | ||
Updated•6 years ago
|
Flags: needinfo?(khudson)
Comment 10•6 years ago
|
||
Thanks, Kate. I've added some comments to the PHD and below. Maybe we should schedule 30m to discuss? It looks like the metrics we're interested in tracking are: * Search Volume (from SAPs) * Retention * Usage * Satisfaction There are two experiments proposed in that document. For experiment 1 we should record whether we filtered any tiles at a per user level in experiment and whether we would have filtered tiles in control. This will allow us to measure the effect size over users that are affected by this change. Otherwise, search volume and retention are both defined over data available in main_summary. What are the definitions of Usage and Satisfaction? Have these been defined for previous AS studies? The PHD mentions CTR for search tiles, but it's unclear whether this metric is a decision criterion. Is this the metric that's instrumented in the attached PR? I see an event ping which records user interactions with search tiles, but I don't see a count of how frequently these tiles are shown to users. These experiments are listed as independent experiments. Is that accurate or is this actually a three branch experiment? E.g.: * Control: {noDefaultSearchTile: False, topSiteSearchShortcuts: False} * Expt 1 : {noDefaultSearchTile: True, topSiteSearchShortcuts: False} * Expt 2 : {noDefaultSearchTile: False, topSiteSearchShortcuts: True} * Expt 3 : {noDefaultSearchTile: True, topSiteSearchShortcuts: True}
Flags: needinfo?(khudson)
Reporter | ||
Comment 11•6 years ago
|
||
We don't currently have the means to measure whether we show an individual tile to particular users (or whether we would have) although we might still have time to implement it / get review. I think having a meeting to discuss this and the experiment variants (my understanding is that the two experiments would be tested separately) with Maria (product) sounds like a good idea. Maria will follow up with a meeting request today. Thanks!
Flags: needinfo?(khudson)
Assignee | ||
Comment 13•6 years ago
|
||
(In reply to Liz Henry (:lizzard) (needinfo? me) from comment #12) > Would this need to also land on 62? Yes, all the uplifts for 62 are tracked in bug 1482398.
Comment 14•6 years ago
|
||
https://hg.mozilla.org/releases/mozilla-beta/rev/9e752b54a20f425449e1f00bf3adefa903f8051a
Comment 15•6 years ago
|
||
I have verified that the issue is no longer reproducible in the latest Nightly (63.0a1, Build ID 20180816100035) and Beta (62.0b18 Build ID 20180816151750) on Windows 10, Mac 10.13 and Arch Linux x64. Pings for Clicking, adding, removing, dismissing, pinning, unpinning have been added.
Updated•5 years ago
|
Component: Activity Streams: Newtab → New Tab Page
You need to log in
before you can comment on or make changes to this bug.
Description
•