Closed
Bug 1420949
Opened 8 years ago
Closed 8 years ago
Telemetry for user actions data review
Categories
(Firefox for iOS :: Telemetry, enhancement, P1)
Tracking
()
RESOLVED
FIXED
| Tracking | Status | |
|---|---|---|
| fxios | 11.0 | --- |
People
(Reporter: garvan, Assigned: liuche)
References
Details
Purpose: Understand how users are interacting with and using Focus.
We need data on:
- Settings
- UI Event Telemetry
Ping Overview:
https://docs.google.com/a/mozilla.com/document/d/1csZgXQOPOLXfp0TuHIkSuP2T3A0TQnNq4JQuiYCewv8
Data Review:
https://docs.google.com/a/mozilla.com/document/d/1mYq6T88fJUaOSqF4nPO98QX3lJf-Hymu85MQZkPw-Lg
Rebecca: requesting data review, we copied the format of bug 1347266
Updated•8 years ago
|
Assignee: rweiss → liuche
Updated•8 years ago
|
Flags: needinfo?(rweiss) → needinfo?(liuche)
Comment 2•8 years ago
|
||
After talking to Rebecca she suggest to move this to Chenxia instead.
| Assignee | ||
Comment 3•8 years ago
|
||
I will get to this in the next 1-2 days!
It looks like your google docs match the new data review questions, so I'll look over them, copy the content to a comment (so it's public and won't be lost in googledocs).
I'll need you to flag me for review again once you've added public in-tree documentation, and when you start adding individual pings as well, but for now I'll make sure this makes sense to collect, and flag you with any questions I have. Keeping NI for myself.
| Assignee | ||
Comment 4•8 years ago
|
||
Hi all, sorry for the delay, this is the downside of having not enough data peers. (Actually maybe I think there are no upsides :) )
This is a new telemetry system, but it's very similar to the telemetry on Firefox Android and Focus Android, and for now is only collecting settings events and a few usage events (opening app, reader mode/list, bookmarks). There is no existing telemetry on Firefox iOS, so there's a need to collect usage telemetry.
I have a few more questions, some of which I've listed in the doc so I'll copy them here:
- Is this data collection opt-in or opt-out? I wasn't able to find this in the documentation
- How can a user control whether/how data is being collected with this new telemetry system?
- What are the existing telemetry systems on Firefox iOS, are they just Adjust and Leanplum?
- Where does the clientid come from (is it from the existing telemetry systems, like Adjust?)
- (Specifically for settings) Could there be free-form user-entered settings that are collected? (e.g., default search engine, and the user can set a custom search engine)
Flags: needinfo?(liuche)
| Assignee | ||
Comment 5•8 years ago
|
||
Here is the data request as submitted via googledoc, copying here to keep a public record:
Request for new data and review
Sponsor: Stefan Arentz
Bug 1420949 adds the analysis plan as an attached file,
TODO: ask for three r?s: analysis review, data review, and implementation review.
The r?s can be filed in parallel, but analysis review will need an r+ before data review and implementation review.
Analysis Plan Template
Role call (one person can fit more than one role)
Sponsor for new data collection: Stefan Arentz
Data Analyst(s): Frank Bertsch/Stefan Arentz
Data Engineer(s): Frank Bertsch
Data QA: ?
Product/Engineering Manager: Stefan Arentz
Other Data POI(s): Garvan Keeley/Justin D’Arcangelo
Overview
What is the motivating question you want to answer with this data?
How are people using mobile Firefox?
How is the app setup (settings)?
How do they progress through the app?
Which features do they use?
What are the benefits of answering this question?
Which features do people use that most impact retention?
Will use this information to influence future product directions and new features
Knowing what blocklists or settings the user turns on/off helps us understand if we might want to add or remove specific blocklists or settings, or that need further explanation or breakdown.
We want to understand how often, and how many users utilize the Bookmarks feature on mobile.
We have heavily invested in this area in the form of Mobile Bookmark Sync and we need to know if those efforts resulted in higher usage of bookmarks on mobile. It will also allow us to do proper prioritization of additional work in this area and see what its impact will be.
We want to understand how often, and how many users utilize the Reading List feature on mobile.
We need to understand if this feature drives retention
We need to understand if promoting / improving this feature will increase retention
We want to understand how often, and how many users utilize the Reader Mode feature on mobile.
We need to understand if this feature drives retention
We need to understand if promoting / improving this feature will increase retention
Describe your instrumentation. List all the measures you are taking. High-level descriptions are sufficient.
We have a ping overview document that describes the specific measures.
Some dimensions, including: client id, os, locale
Settings changes - the old value and the new value
Events - bookmark usage, reading list usage, reader mode usage
Describe your implementation. Provide engineering descriptions about how these measures will be developed.
Data will be recorded in a telemetry histogram (main ping)
Data will be recorded in a telemetry scalar (main ping)
Data will be added to the telemetry environment
Data will be sent in a custom telemetry ping
Other (please describe in extensive detail below)
This data will be direct-to-parquet initially, and queried in STMO.
We are following the standard event telemetry format, with the possibility of utilizing whatever tools have formed around that standard. It is possible we could use third party tools for this analysis as well (e.g., funnel analysis).
Eventually we will probably have some sort of ETL job to pull out event data in some fashion.
Describe the analysis you will perform on the collected data to answer the motivating question.
The product manager will answer questions in STMO, where feature usage can be associated with cohorts and user types
Technicals
What populations do you need to these measurements for?
Which release channels?
Release and Beta
Which countries?
All countries
Which locales?
All locales
Any other filters?
No
How long do you want this data to be collected?
We will use this data for as long as we are developing mobile Firefox. It will need to be monitored to ensure we need all parts to answer questions that we have, we could always add/remove events or settings in the future.
What data platform do you want to collect and ingest your measurements?
Unified Telemetry
From which endpoints do you want this data be made available?
Re:dash (s.t.m.o)
Spark (a.t.m.o)
If you intend to use s.t.m.o., t.m.o., or a.t.m.o. in order to access your data, you will need to identify where the tools will be drawing data from, usually in the form of a rollup, a database table (e.g. in Redshift), or some other form of data store (e.g. a Parquet file).
This data will be direct-to-parquet initially, so one row per ping. We may add another ETL job to unroll or aggregate event data.
Are your measurements going to require its own rollup, table, or other data store, or do you want this measurement added to an existing rollup, table, or other data store?
New direct-to-parquet table (easy to implement, just need the schema)
Planning to query this with Athena
(Possible) New Derived Dataset: a flattened table of event details.
Compliance
Does your instrumentation include the addition of any new identifiers (whether anonymous or otherwise; e.g., username, random IDs, etc.)?
No. We will use client-id.
If you are asking for measurements to be added to the entire Firefox population, explain why you can’t use a sample for this question.
The mobile Firefox user base is a much smaller population than desktop Firefox.
What alternative methods did you consider to answer this question? Why were they not sufficient? Is this the least-risky way to solve your problem?
Adjust - but we want to move away from using adjust in Mobile
To what extent does current instrumentation allow you to answer this question?
It doesn’t really. Firefox for iOS has never had event telemetry.
Where are you going to share the results of your analysis?
Private internal reports
Artifacts
Provide an example of the measurement artifact (e.g. JSON blob) that will be produced by the implementation described in the overview.
{
"v": 1, // ping format version
"clientId": <string>, // client id, e.g. "c641eacf-c30c-4171-b403-f077724e848a",
"seq": <positive integer>, // running ping counter, e.g. 3
"locale": <string>, // application locale, e.g. "en-US"
"os": <string>, // OS name.
"osversion": <string>, // OS version.
“created”: <Timestamp>
“tz”: <String>
“settings”: {<string>:<string>} // last setting and value, e.g. {“ad_trackers”:”true”}
“events”: [[<int>, <string>, <string>, <string>, (optional) <string>, (optional) <json>], ...]
// see events below
}
If you require a Derived Dataset, provide its schema:
Exactly as above, just in parquet format. We may actually use the new schema-service Rob Hudson is building.
If you require an amendment to a Core Data Set, describe how and what will be added:
No amendments are required.
Comment 6•8 years ago
|
||
(In reply to Chenxia Liu [:liuche] - not actively working on Fennec from comment #4)
> - Is this data collection opt-in or opt-out? I wasn't able to find this in
> the documentation
Opt-out.
> - How can a user control whether/how data is being collected with this new
> telemetry system?
In the app settings menu, the "Send Usage Data" toggle is already used by Mozilla Unified Telemetry for turning the core ping on and off. Also, Adjust, Leanplum and Sentry all honor this setting regarding collection and transmission of usage data.
> - What are the existing telemetry systems on Firefox iOS, are they just
> Adjust and Leanplum?
Firefox for iOS uses Mozilla Unified Telemetry for the core ping and Adjust for install tracking. Also, Sentry is used for crash reporting. There is currently no system in place for UI event telemetry.
> - Where does the clientid come from (is it from the existing telemetry
> systems, like Adjust?)
This is the same `clientid` that the Mozilla Unified Telemetry core ping uses. It is a one-time randomly-generated UUID.
> - (Specifically for settings) Could there be free-form user-entered settings
> that are collected? (e.g., default search engine, and the user can set a
> custom search engine)
I'm not entirely sure what you mean? The ping format is flexible enough that it can accommodate any data type, but I believe the settings we are interested in sending (at least for now in "v1") are all boolean (i.e.: is night mode enabled).
Updated•8 years ago
|
Priority: -- → P1
| Assignee | ||
Comment 7•8 years ago
|
||
> > - (Specifically for settings) Could there be free-form user-entered settings
> > that are collected? (e.g., default search engine, and the user can set a
> > custom search engine)
>
> I'm not entirely sure what you mean? The ping format is flexible enough that
> it can accommodate any data type, but I believe the settings we are
> interested in sending (at least for now in "v1") are all boolean (i.e.: is
> night mode enabled).
Followed up on Slack, documenting here: this ping sends the string "custom" if the value is not one of the defaults. Concern was not that the ping would be so flexible as to accommodate any data type, but rather that it is not sending any identifiable information.
| Assignee | ||
Comment 8•8 years ago
|
||
> Is there or will there be documentation that describes the schema for the ultimate data set available publicly, complete and accurate? (see here, here, and here for examples). Refer to the appendix for "documentation" if more detail about documentation standards is needed.
Documentation will be added in Firefox iOS's telemetry wiki: https://github.com/mozilla-mobile/firefox-ios/wiki/Telemetry
> Is there a control mechanism that allows the user to turn the data collection on and off? (Note, for data collection not needed for security purposes, Mozilla provides such a control mechanism) Provide details as to the control mechanism available.
Yes, in the "Send usage data" setting in Firefox iOS
> If the request is for permanent data collection, is there someone who will monitor the data over time?**
Stefan Arentz (sarentz@mozilla.com) and Frank Bertsch (fbertsch@mozilla.com)
> Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under? **
Type 2, usage of product and user settings, reader mode, reading list, bookmarking usage
> Is the data collection request for default-on or default-off?
Default on
> Does the instrumentation include the addition of any new identifiers (whether anonymous or otherwise; e.g., username, random IDs, etc. See the appendix for more details)?
No, clientId only from coreping
> Is the data collection covered by the existing Firefox privacy notice?
Yes
> Does there need to be a check-in in the future to determine whether to renew the data? (Yes/No) (If yes, set a todo reminder or file a bug if appropriate)**
No
| Assignee | ||
Comment 9•8 years ago
|
||
Data-review+
Justin, Stefan, can you make sure the documentation on the Firefox iOS telemetry wiki ( https://github.com/mozilla-mobile/firefox-ios/wiki/Telemetry ) is updated? (I noticed that some of the links inside the wiki are broken.)
Ideally you can document the pings and possible values in the wiki, per the Event Ping Format googledoc ( https://docs.google.com/document/d/1csZgXQOPOLXfp0TuHIkSuP2T3A0TQnNq4JQuiYCewv8/edit ), something like the Focus Android ping doc: https://github.com/mozilla-mobile/focus-android/wiki/Telemetry
Flags: needinfo?(sarentz)
Flags: needinfo?(jdarcangelo)
Comment 10•8 years ago
|
||
(In reply to Chenxia Liu [:liuche] - not actively working on Fennec from comment #9)
> Data-review+
>
> Justin, Stefan, can you make sure the documentation on the Firefox iOS
> telemetry wiki (
> https://github.com/mozilla-mobile/firefox-ios/wiki/Telemetry ) is updated?
> (I noticed that some of the links inside the wiki are broken.)
>
> Ideally you can document the pings and possible values in the wiki, per the
> Event Ping Format googledoc (
> https://docs.google.com/document/d/
> 1csZgXQOPOLXfp0TuHIkSuP2T3A0TQnNq4JQuiYCewv8/edit ), something like the
> Focus Android ping doc:
> https://github.com/mozilla-mobile/focus-android/wiki/Telemetry
Done.
Flags: needinfo?(sarentz)
Flags: needinfo?(jdarcangelo)
Updated•8 years ago
|
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
You need to log in
before you can comment on or make changes to this bug.
Description
•