Closed Bug 1420949 Opened 8 years ago Closed 8 years ago

Telemetry for user actions data review

Categories

(Firefox for iOS :: Telemetry, enhancement, P1)

Other
iOS
enhancement

Tracking

()

RESOLVED FIXED
Tracking Status
fxios 11.0 ---

People

(Reporter: garvan, Assigned: liuche)

References

Details

Purpose: Understand how users are interacting with and using Focus. We need data on: - Settings - UI Event Telemetry Ping Overview: https://docs.google.com/a/mozilla.com/document/d/1csZgXQOPOLXfp0TuHIkSuP2T3A0TQnNq4JQuiYCewv8 Data Review: https://docs.google.com/a/mozilla.com/document/d/1mYq6T88fJUaOSqF4nPO98QX3lJf-Hymu85MQZkPw-Lg
See Also: → 1347266
Assignee: nobody → rweiss
Rebecca: requesting data review, we copied the format of bug 1347266
Flags: needinfo?(rweiss)
Assignee: rweiss → liuche
Flags: needinfo?(rweiss) → needinfo?(liuche)
After talking to Rebecca she suggest to move this to Chenxia instead.
I will get to this in the next 1-2 days! It looks like your google docs match the new data review questions, so I'll look over them, copy the content to a comment (so it's public and won't be lost in googledocs). I'll need you to flag me for review again once you've added public in-tree documentation, and when you start adding individual pings as well, but for now I'll make sure this makes sense to collect, and flag you with any questions I have. Keeping NI for myself.
Hi all, sorry for the delay, this is the downside of having not enough data peers. (Actually maybe I think there are no upsides :) ) This is a new telemetry system, but it's very similar to the telemetry on Firefox Android and Focus Android, and for now is only collecting settings events and a few usage events (opening app, reader mode/list, bookmarks). There is no existing telemetry on Firefox iOS, so there's a need to collect usage telemetry. I have a few more questions, some of which I've listed in the doc so I'll copy them here: - Is this data collection opt-in or opt-out? I wasn't able to find this in the documentation - How can a user control whether/how data is being collected with this new telemetry system? - What are the existing telemetry systems on Firefox iOS, are they just Adjust and Leanplum? - Where does the clientid come from (is it from the existing telemetry systems, like Adjust?) - (Specifically for settings) Could there be free-form user-entered settings that are collected? (e.g., default search engine, and the user can set a custom search engine)
Flags: needinfo?(liuche)
Here is the data request as submitted via googledoc, copying here to keep a public record: Request for new data and review Sponsor: Stefan Arentz Bug 1420949 adds the analysis plan as an attached file, TODO: ask for three r?s: analysis review, data review, and implementation review. The r?s can be filed in parallel, but analysis review will need an r+ before data review and implementation review. Analysis Plan Template Role call (one person can fit more than one role) Sponsor for new data collection: Stefan Arentz Data Analyst(s): Frank Bertsch/Stefan Arentz Data Engineer(s): Frank Bertsch Data QA: ? Product/Engineering Manager: Stefan Arentz Other Data POI(s): Garvan Keeley/Justin D’Arcangelo Overview What is the motivating question you want to answer with this data? How are people using mobile Firefox? How is the app setup (settings)? How do they progress through the app? Which features do they use? What are the benefits of answering this question? Which features do people use that most impact retention? Will use this information to influence future product directions and new features Knowing what blocklists or settings the user turns on/off helps us understand if we might want to add or remove specific blocklists or settings, or that need further explanation or breakdown. We want to understand how often, and how many users utilize the Bookmarks feature on mobile. We have heavily invested in this area in the form of Mobile Bookmark Sync and we need to know if those efforts resulted in higher usage of bookmarks on mobile. It will also allow us to do proper prioritization of additional work in this area and see what its impact will be. We want to understand how often, and how many users utilize the Reading List feature on mobile. We need to understand if this feature drives retention We need to understand if promoting / improving this feature will increase retention We want to understand how often, and how many users utilize the Reader Mode feature on mobile. We need to understand if this feature drives retention We need to understand if promoting / improving this feature will increase retention Describe your instrumentation. List all the measures you are taking. High-level descriptions are sufficient. We have a ping overview document that describes the specific measures. Some dimensions, including: client id, os, locale Settings changes - the old value and the new value Events - bookmark usage, reading list usage, reader mode usage Describe your implementation. Provide engineering descriptions about how these measures will be developed. Data will be recorded in a telemetry histogram (main ping) Data will be recorded in a telemetry scalar (main ping) Data will be added to the telemetry environment Data will be sent in a custom telemetry ping Other (please describe in extensive detail below) This data will be direct-to-parquet initially, and queried in STMO. We are following the standard event telemetry format, with the possibility of utilizing whatever tools have formed around that standard. It is possible we could use third party tools for this analysis as well (e.g., funnel analysis). Eventually we will probably have some sort of ETL job to pull out event data in some fashion. Describe the analysis you will perform on the collected data to answer the motivating question. The product manager will answer questions in STMO, where feature usage can be associated with cohorts and user types Technicals What populations do you need to these measurements for? Which release channels? Release and Beta Which countries? All countries Which locales? All locales Any other filters? No How long do you want this data to be collected? We will use this data for as long as we are developing mobile Firefox. It will need to be monitored to ensure we need all parts to answer questions that we have, we could always add/remove events or settings in the future. What data platform do you want to collect and ingest your measurements? Unified Telemetry From which endpoints do you want this data be made available? Re:dash (s.t.m.o) Spark (a.t.m.o) If you intend to use s.t.m.o., t.m.o., or a.t.m.o. in order to access your data, you will need to identify where the tools will be drawing data from, usually in the form of a rollup, a database table (e.g. in Redshift), or some other form of data store (e.g. a Parquet file). This data will be direct-to-parquet initially, so one row per ping. We may add another ETL job to unroll or aggregate event data. Are your measurements going to require its own rollup, table, or other data store, or do you want this measurement added to an existing rollup, table, or other data store? New direct-to-parquet table (easy to implement, just need the schema) Planning to query this with Athena (Possible) New Derived Dataset: a flattened table of event details. Compliance Does your instrumentation include the addition of any new identifiers (whether anonymous or otherwise; e.g., username, random IDs, etc.)? No. We will use client-id. If you are asking for measurements to be added to the entire Firefox population, explain why you can’t use a sample for this question. The mobile Firefox user base is a much smaller population than desktop Firefox. What alternative methods did you consider to answer this question? Why were they not sufficient? Is this the least-risky way to solve your problem? Adjust - but we want to move away from using adjust in Mobile To what extent does current instrumentation allow you to answer this question? It doesn’t really. Firefox for iOS has never had event telemetry. Where are you going to share the results of your analysis? Private internal reports Artifacts Provide an example of the measurement artifact (e.g. JSON blob) that will be produced by the implementation described in the overview. { "v": 1, // ping format version "clientId": <string>, // client id, e.g. "c641eacf-c30c-4171-b403-f077724e848a", "seq": <positive integer>, // running ping counter, e.g. 3 "locale": <string>, // application locale, e.g. "en-US" "os": <string>, // OS name. "osversion": <string>, // OS version. “created”: <Timestamp> “tz”: <String> “settings”: {<string>:<string>} // last setting and value, e.g. {“ad_trackers”:”true”} “events”: [[<int>, <string>, <string>, <string>, (optional) <string>, (optional) <json>], ...] // see events below } If you require a Derived Dataset, provide its schema: Exactly as above, just in parquet format. We may actually use the new schema-service Rob Hudson is building. If you require an amendment to a Core Data Set, describe how and what will be added: No amendments are required.
(In reply to Chenxia Liu [:liuche] - not actively working on Fennec from comment #4) > - Is this data collection opt-in or opt-out? I wasn't able to find this in > the documentation Opt-out. > - How can a user control whether/how data is being collected with this new > telemetry system? In the app settings menu, the "Send Usage Data" toggle is already used by Mozilla Unified Telemetry for turning the core ping on and off. Also, Adjust, Leanplum and Sentry all honor this setting regarding collection and transmission of usage data. > - What are the existing telemetry systems on Firefox iOS, are they just > Adjust and Leanplum? Firefox for iOS uses Mozilla Unified Telemetry for the core ping and Adjust for install tracking. Also, Sentry is used for crash reporting. There is currently no system in place for UI event telemetry. > - Where does the clientid come from (is it from the existing telemetry > systems, like Adjust?) This is the same `clientid` that the Mozilla Unified Telemetry core ping uses. It is a one-time randomly-generated UUID. > - (Specifically for settings) Could there be free-form user-entered settings > that are collected? (e.g., default search engine, and the user can set a > custom search engine) I'm not entirely sure what you mean? The ping format is flexible enough that it can accommodate any data type, but I believe the settings we are interested in sending (at least for now in "v1") are all boolean (i.e.: is night mode enabled).
Priority: -- → P1
> > - (Specifically for settings) Could there be free-form user-entered settings > > that are collected? (e.g., default search engine, and the user can set a > > custom search engine) > > I'm not entirely sure what you mean? The ping format is flexible enough that > it can accommodate any data type, but I believe the settings we are > interested in sending (at least for now in "v1") are all boolean (i.e.: is > night mode enabled). Followed up on Slack, documenting here: this ping sends the string "custom" if the value is not one of the defaults. Concern was not that the ping would be so flexible as to accommodate any data type, but rather that it is not sending any identifiable information.
> Is there or will there be documentation that describes the schema for the ultimate data set available publicly, complete and accurate? (see here, here, and here for examples). Refer to the appendix for "documentation" if more detail about documentation standards is needed. Documentation will be added in Firefox iOS's telemetry wiki: https://github.com/mozilla-mobile/firefox-ios/wiki/Telemetry > Is there a control mechanism that allows the user to turn the data collection on and off? (Note, for data collection not needed for security purposes, Mozilla provides such a control mechanism) Provide details as to the control mechanism available. Yes, in the "Send usage data" setting in Firefox iOS > If the request is for permanent data collection, is there someone who will monitor the data over time?** Stefan Arentz (sarentz@mozilla.com) and Frank Bertsch (fbertsch@mozilla.com) > Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under? ** Type 2, usage of product and user settings, reader mode, reading list, bookmarking usage > Is the data collection request for default-on or default-off? Default on > Does the instrumentation include the addition of any new identifiers (whether anonymous or otherwise; e.g., username, random IDs, etc. See the appendix for more details)? No, clientId only from coreping > Is the data collection covered by the existing Firefox privacy notice? Yes > Does there need to be a check-in in the future to determine whether to renew the data? (Yes/No) (If yes, set a todo reminder or file a bug if appropriate)** No
Data-review+ Justin, Stefan, can you make sure the documentation on the Firefox iOS telemetry wiki ( https://github.com/mozilla-mobile/firefox-ios/wiki/Telemetry ) is updated? (I noticed that some of the links inside the wiki are broken.) Ideally you can document the pings and possible values in the wiki, per the Event Ping Format googledoc ( https://docs.google.com/document/d/1csZgXQOPOLXfp0TuHIkSuP2T3A0TQnNq4JQuiYCewv8/edit ), something like the Focus Android ping doc: https://github.com/mozilla-mobile/focus-android/wiki/Telemetry
Flags: needinfo?(sarentz)
Flags: needinfo?(jdarcangelo)
(In reply to Chenxia Liu [:liuche] - not actively working on Fennec from comment #9) > Data-review+ > > Justin, Stefan, can you make sure the documentation on the Firefox iOS > telemetry wiki ( > https://github.com/mozilla-mobile/firefox-ios/wiki/Telemetry ) is updated? > (I noticed that some of the links inside the wiki are broken.) > > Ideally you can document the pings and possible values in the wiki, per the > Event Ping Format googledoc ( > https://docs.google.com/document/d/ > 1csZgXQOPOLXfp0TuHIkSuP2T3A0TQnNq4JQuiYCewv8/edit ), something like the > Focus Android ping doc: > https://github.com/mozilla-mobile/focus-android/wiki/Telemetry Done.
Flags: needinfo?(sarentz)
Flags: needinfo?(jdarcangelo)
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.