1621703 - Windows Default Browser Agent - Phase 2 - Toast notification telemetry

Status: RESOLVED FIXED

(Toolkit :: Default Browser Agent, enhancement)

Description

[Nick Alexander :nalexander [he/him]]

This ticket tracks delivering telemetry about interactions with any displayed toasts.

Philosophically such interactions are events, which have special handling in Glean. I'm not sure if it's feasible or desirable to try to mimic this event-y flow; part of this ticket is working out such details.

Phase 1 already includes regularly scheduled telemetry so we'll stick to the same, or a very similar, technical implementation.

Comment 1

[Robin Steuber (she/her) [:bytesized]]

Attachment: Bug 1621703 - Windows Default Browser Agent Telemetry r=mhowell,agashlin,nalexander

Comment 2

[Robin Steuber (she/her) [:bytesized]]

Comment on attachment 9147771 [details]
Bug 1621703 - Windows Default Browser Agent Telemetry r=mhowell,agashlin,nalexander

This patch requires data review. It adds fields to the existing windows default browser agent ping, which is external to Firefox. These are the fields that are being added:

• notification_type - A string indicating whether the notification that was possibly shown was the initial notification, or the followup notification.
• notification_shown - A string indicating whether a notification was shown, not shown, or resulted in an error.
• notification_action - A string indicating what interaction occurred with notification (which button was pressed, how the notification was dismissed).
• client_id - A UUID stored in HKEY_CURRENT_USERS (i.e. it is a per-Windows-user ID). It does not match the UUID used for any other client ID in any other ping.

We are hoping to merge this pretty soon so, if possible, a quick turnaround would be greatly appreciated. Thanks!

Comment 3

[Robin Steuber (she/her) [:bytesized]]

Telemetry fields have been added to the pipeline schema in commit 45df778ecc8460f7333c126de808c1c9281457e9.

Comment 4

[Robin Steuber (she/her) [:bytesized]]

Attachment: wdba_data_review_form.md

Comment 5

[Robin Steuber (she/her) [:bytesized]]

Data review fixed.

Comment 6

[Rachel Tublitz [:rachel]]

Chutten: when you do a data review here, we'd specifically like your thoughts on whether or not we should be adding a client_id.

Comment 7

[Chris H-C :chutten]

Comment on attachment 9149503 [details]
wdba_data_review_form.md

First and foremost, I'm afraid it's `data-review-` right off the bat as data collection review must be done in public so that users subject to collection can be directed to the data collection review for the data we're collecting from their Firefox.

Secondly, since you asked my opinion on the client id (normally Data Stewards' roles are limited to "does this new collection require additional review or is it good to go?"), to my knowledge the questions being posed in Q1 do not require a client identifier. A "settings changed" identifier (like a session id) generated when the settings change was detected and cleared when the final notification is set or the settings are changed again would be more in keeping with Mozilla's approach to Lean Data.

Any identifier requires Trust review on top of Data Review. Persistent identifiers also generally need to be communicated to the pipeline for self-serve data deletion requests when/if the user opts out of data collection.

Third, data collections must be documented in order to be reviewed. I've mentioned this in the code review, but the ["default browser" ping docs](https://firefox-source-docs.mozilla.org/toolkit/components/telemetry/data/default-browser-ping.html) need to be updated for any new collection.

Comment 8

["Saptarshi Guha[:joy]"]

Despite the requester's client_id field not being the same as telemetry client_id, there is likely no need for this field.

• to answer Q1 for a given day, the number of pings where default browser is not Firefox and last browser is Firefox which doesnt require a UUID.

To answer this question for period larger than day e.g. one week a UUID would be required, but is this a required question?

Comment 9

[Ryan Harter [:harter]]

(In reply to "Saptarshi Guha[:joy]" from comment #8)

Despite the requester's client_id field not being the same as telemetry client_id, there is likely no need for this field.

Documenting some context from past conversations. The client_id is a suggestion I made on the Phase 1 of this telemetry. IIRC, the goal of Phase 1 was to be able to faster identify retention events caused by forced changes to the default browser.

to answer Q1 for a given day, the number of pings where default browser is not Firefox and last browser is Firefox which doesnt require a UUID.

My initial concern for the Phase 1 telemetry was that we'd have some clients sending many pings per day. This is a problem we face with our main-ping. Usually, I'd just spot check the ping counts against our main-ping to confirm they're similar, but default browser telemetry will likely report very different user volumes than the main-ping.

Again, this is just for context. My review of the Phase 1 ping was very cursory, so I defer to Saptarshi.

Comment 10

[Romain Testard [:RT]]

Here is a summary of discussions with Rachel and Saptarshi regarding whether we need a client_id:
1 Questions we need answered with the data
a Is there a trend shift where users change defaults to Browser X? We should be able to do this without risk of false positives like "1 client generates many pings". Clients can send more than one ping a day and bots might send higher counts.
b What is the share of Firefox users with Firefox set as default moving to Browser X? User was previously interpreted as DAU although recent data explorations in share of MAU that are inactive showed that "Share of DAU that have a characteristic" can be very different from "Share of MAU that have a characteristic". Given churn is measured using MAU it sounds appropriate to use MAU for this measure - i.e What is the share of Firefox MAU with Firefox set as default moving to Browser X?

2 Do a and b above require a client_id?
a does not require a client_id. The team suggested usage of a registry entry on the machine for last ping sent - then the client could check it and not send another until 24 hours have passed. Ensures only one ping per machine is sent without extra telemetry

b MAU measurement requires a client_id. Although do we need MAU measurements, or is DAU good enough?:

• We may need MAU because we measure activity and churn from MAU and measuring only "share of DAU switching defaults" will give us a fairly noisy signal given the fact that MAU and DAU populations are so different.
• We may not need MAU because there is probably a bias around the fact that these users don't necessarily open Firefox on the day they send a ping - which probably implies that this is yet another cohort that will behave differently from DAU and MAU so attempting to map this behaviour to the one of MAU may be bound to failure. In this case there is probably a good argument that observing the trend of DAU that open their computers and switch defaults is probably as good of an approximation as MAU who open their computer and switch defaults when trying to use this as a signal that MAU churn may get impacted.

In summary:

• We'll move forward with the proposal to use a registry entry on the machine for last ping sent in order to avoid daily duplicates and comply with the 'lean data' principle
• We'll be observing the daily share of Firefox installs with Firefox set as default moving to Browser X and sending a daily ping as opposed to making this observation monthly given it's a leaner approach that does not require a client_id and that monthly observations would also not necessarilly be a better approximation. If we find that the daily the share is not a good enough signal we may revisit the decision at this point

Comment 11

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla/bigquery-etl/pull/1036

Comment 12

[BugBot [:suhaib / :marco/ :calixte]]

There's a r+ patch which didn't land and no activity in this bug for 2 weeks.
:bytesized, could you have a look please?
For more information, please visit auto_nag documentation.

Comment 13

[Robin Steuber (she/her) [:bytesized]]

Yeah, these patches are waiting for some other work to be ready before they can merge.

Comment 14

[Robin Steuber (she/her) [:bytesized]]

Attachment: wdba_data_review_form.md

Hopefully things look better this time around. The bug is now public and there is no longer a client id field being sent in the telemetry.

Comment 15

[Robin Steuber (she/her) [:bytesized]]

Attachment: Bug 1621703 - Migrate prefixed CurrentDefault registry value name to an unprefixed one r=mhowell,agashlin,nalexander

This patch also adds an install-directory-prefixed "Installed" value since RemoveAllRegistryEntries relies on prefixed value names and we no longer use any other install-directory-prefixed value names in the WDBA.

Comment 16

[Robin Steuber (she/her) [:bytesized]]

Attachment: Bug 1621703 - Add mutex to protect the registry r=mhowell,agashlin,nalexander

Since the WDBA now accesses registry values whose names are not prefixed, it could have concurrency problems when running at the same time as agents from other installations. This patch adds a mutex to address that problem.

Depends on D80706

Comment 17

[Robin Steuber (she/her) [:bytesized]]

Attachment: Bug 1621703 - Only send one ping per day r=mhowell,nalexander,agashlin

Depends on D80707

Comment 18

[Robin Steuber (she/her) [:bytesized]]

Attachment: Bug 1621703 - Replace variable with existing macro with the same value r=mhowell

Comment 19

[Chris H-C :chutten]

Comment on attachment 9158595 [details]
wdba_data_review_form.md

DATA COLLECTION REVIEW RESPONSE:

    Is there or will there be documentation that describes the schema for the ultimate data set available publicly, complete and accurate?

Yes. This collection is documented in its [in-tree documentation](https://firefox-source-docs.mozilla.org/toolkit/components/telemetry/data/default-browser-ping.html).

    Is there a control mechanism that allows the user to turn the data collection on and off?

Yes. This collection can be controlled through the Data Choices Preferences in Firefox.

    If the request is for permanent data collection, is there someone who will monitor the data over time?

Yes, Romain Testard is responsible.

    Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Category 2, Interaction.

    Is the data collection request for default-on or default-off?

Default on for all channels.

    Does the instrumentation include the addition of any new identifiers?

No.

    Is the data collection covered by the existing Firefox privacy notice?

Yes.

    Does there need to be a check-in in the future to determine whether to renew the data?

No. This collection is permanent.

---
Result: datareview+

Comment 20

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/c313c7e92aad
Windows Default Browser Agent Telemetry r=mhowell,agashlin,nalexander,chutten
https://hg.mozilla.org/integration/autoland/rev/abfba63bc3c1
Replace variable with existing macro with the same value r=mhowell,nalexander
https://hg.mozilla.org/integration/autoland/rev/edfbd5a25c68
Migrate prefixed CurrentDefault registry value name to an unprefixed one r=mhowell,agashlin,nalexander
https://hg.mozilla.org/integration/autoland/rev/2fedf44d9f2a
Add mutex to protect the registry r=mhowell,agashlin,nalexander
https://hg.mozilla.org/integration/autoland/rev/63c0fc65889b
Only send one ping per day r=mhowell,nalexander,agashlin

Comment 21

[Razvan Maries]

https://hg.mozilla.org/mozilla-central/rev/c313c7e92aad
https://hg.mozilla.org/mozilla-central/rev/abfba63bc3c1
https://hg.mozilla.org/mozilla-central/rev/edfbd5a25c68
https://hg.mozilla.org/mozilla-central/rev/2fedf44d9f2a
https://hg.mozilla.org/mozilla-central/rev/63c0fc65889b