1580649 - Data Collection Review of new UI telemetry for Fennec's AwesomeScreen "Sign up" and "Sign in" buttons

Status: RESOLVED FIXED

(Firefox for Android Graveyard :: General, task, P1)

Description

[Chris Peterson [:cpeterson]]

[Tracking Requested - why for this release]:

For reference, Grisha added a Leanplum event (in bug 1568605) to report when a Fennec user connects (i.e. "signup", "signin", or "reconnect") to a Firefox Account. We might be able to use that same Leanplum event but annotate it with a UI source:

• "Sign up" button on Onboarding card #1
• "Sign up" button on Onboarding card #2
• "Sign up" button on Onboarding card #3
• "Sign up" button on New Tab
• "Sign in" link on New Tab
• "Sign in" settings menu item

I'm tentatively assigning this bug to Vlad because he is implementing the Onboarding buttons (bug 1570878) and New Tab button (bug 1570880).

This Amplitude dashboard shows Fennec FxA telemetry for logins and registrations by entrypoint. After Fennec ESR 68.1 was released, there was an unexpected increase in FxA registrations from a so-called "(none)" entrypoint. Is that a Fennec bug? Shouldn't all entrypoints be annotated with a name?

https://analytics.amplitude.com/mozilla-corp/dashboard/jac3w56

Comment 1

[Chris Peterson [:cpeterson]]

Attachment: Fennec_AwesomeScreen_FxA_Buttons_Data_Collection_Review_form.txt

Bug 1570880 added "Sign up" and "Sign in" buttons to Fennec's AwesomeScreen. We would like to record UI telemetry measuring how frequently these buttons are used.

We already collect FxA UI telemetry when users sign in from Fennec's first-run Onboarding screens or the Settings menu. That telemetry was added four years ago in bug 1205706 using Amplitude. This new telemetry just adds probes for the AwesomeScreen's new "Sign up", "Sign in", and FxA dismiss buttons.

Comment 2

[Chris H-C :chutten]

Comment on attachment 9098689 [details]
Fennec_AwesomeScreen_FxA_Buttons_Data_Collection_Review_form.txt

Could you point me to the documentation where this collection is described? Firefox Telemetry usually has descriptions in their definitions files (which gets put into the Probe Dictionary), and github-resident mozilla projects often have a TELEMETRY.md or similar. I'm not sure where leanplum collections would be described.

Comment 3

[Chris Peterson [:cpeterson]]

(In reply to Chris H-C :chutten from comment #2)

Could you point me to the documentation where this collection is described?
Firefox Telemetry usually has descriptions in their definitions files (which
gets put into the Probe Dictionary), and github-resident mozilla projects
often have a TELEMETRY.md or similar. I'm not sure where leanplum
collections would be described.

Here is the wiki introduction to Fennec's UI telemetry: https://wiki.mozilla.org/Mobile/Projects/Telemetry_probes_for_Fennec_UI_elements

Looks like Fennec UI telemetry events are supposed to be documented in https://searchfox.org/mozilla-esr68/source/mobile/android/docs/uitelemetry.rst, but it only lists the generic UI telemetry categories (like TelemetryContract.Event.ACTION and TelemetryContract.Method.BUTTON) and none of the specific events in each category. An example: "suggestions_optin_yes" and "suggestions_optin_no" events recording whether the user opted into search suggestions:

https://searchfox.org/mozilla-esr68/rev/61d633e53a2bb8b66832cccb1baf7f5a6ae2c3c6/mobile/android/base/java/org/mozilla/gecko/home/BrowserSearch.java#918

What do you suggest we do? Document these new specific telemetry events (like our call to Telemetry.sendUIEvent(TelemetryContract.Event.ACTION, TelemetryContract.Method.BUTTON, "awesomescreen-signup");) in uitelemetry.rst, even though none of the other TelemetryContract.Method.BUTTON events are documented like that?

Comment 4

[Chris Peterson [:cpeterson]]

What do you suggest we do? Document these new specific telemetry events (like our call to Telemetry.sendUIEvent(TelemetryContract.Event.ACTION, TelemetryContract.Method.BUTTON, "awesomescreen-signup");) in uitelemetry.rst, even though none of the other TelemetryContract.Method.BUTTON events are documented like that?

Chenxia recommended on Slack that we document our new FxA button probes in https://searchfox.org/mozilla-esr68/source/mobile/android/docs/uitelemetry.rst. I will post a doc patch here.

Comment 5

[Chris Peterson [:cpeterson]]

Attachment: Bug 1580649 - Document Fennec's FxA "Sign in" button telemetry events. r?VladBaicu

Bug 1570880 added "Sign up" and "Sign in" buttons to Fennec's AwesomeScreen and UI telemetry events measuring how frequently these buttons are used. We already collect FxA UI telemetry when users sign in from Fennec's first-run Onboarding screens or the Settings menu. This patch documents all of these FxA UI telemetry events.

Comment 6

[Chris H-C :chutten]

Comment on attachment 9098689 [details]
Fennec_AwesomeScreen_FxA_Buttons_Data_Collection_Review_form.txt

DATA COLLECTION REVIEW RESPONSE:

    Is there or will there be documentation that describes the schema for the ultimate data set available publicly, complete and accurate?

Yes. This collection is documented in the esr68 repo [UITelemetry docs](https://hg.mozilla.org/releases/mozilla-esr68/file/tip/mobile/android/docs/uitelemetry.rst).

    Is there a control mechanism that allows the user to turn the data collection on and off?

Yes. This collection is Telemetry so can be controlled through Firefox's Preferences.

    If the request is for permanent data collection, is there someone who will monitor the data over time?

Yes, Chris Peterson is responsible.

    Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Category 2, Interaction.

    Is the data collection request for default-on or default-off?

Default on for pre-release channels only.

    Does the instrumentation include the addition of any new identifiers?

No.

    Is the data collection covered by the existing Firefox privacy notice?

Yes.

    Does there need to be a check-in in the future to determine whether to renew the data?

No. This collection is permanent.

---
Result: datareview+

Comment 7

[Chris Peterson [:cpeterson]]

Comment on attachment 9099460 [details]
Bug 1580649 - Document Fennec's FxA "Sign in" button telemetry events. r?VladBaicu

ESR Uplift Approval Request

• If this is not a sec:{high,crit} bug, please state case for ESR consideration: This is not a sec bug.
• User impact if declined: This is a telemetry doc change requested by data-review to document new telemetry probes reported by Fennec. If this patch is declined, end users won't be affected but users who want to audit Fennec's telemetry probes won't see the new probes. OTOH, other Fennec telemetry probes have been added over the last couple years without updating the docs.
• Fix Landed on Version: None because this is a patch for code that is only in mozilla-esr68.
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): This patch is not risky because it only touches a doc text file (https://searchfox.org/mozilla-esr68/source/mobile/android/docs/uitelemetry.rst).
• String or UUID changes made by this patch: No UI strings were changed, just en-US docs.

Comment 8

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 9099460 [details]
Bug 1580649 - Document Fennec's FxA "Sign in" button telemetry events. r?VladBaicu

Doesn't really need approval since it's just a documentation update and NPOTB, but I'll approve for posterity :-)

Comment 9

[Chris Peterson [:cpeterson]]

Ryan, how do I land on mozilla-esr68? IIUC, neither Lando nor regular folks can push to mozilla-esrXX.

Comment 10

[Ryan VanderMeulen [:RyanVM]]

Now that it has a regular approval, sheriffs should pick it up like usual.

Comment 11

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-esr68/rev/3eae025d6aeb

Comment 12

[Chris Peterson [:cpeterson]]

Thanks, Ryan!

I'm resolving this bug as FIXED because chutten has granted data-review+ and our telemetry doc fix has landed.