1749775 - Add user telemetry for 98 VPN experiment promo

Status: VERIFIED FIXED

(Firefox :: Messaging System, enhancement, P1)

Description

[Andrei Oprea [:andreio]]

This is going to happen only in experiments.

Comment 1

[Andrei Oprea [:andreio]]

Attachment: Bug 1749775 - Add user telemetry for 98 VPN promo, r?pdahiya,andreio

Comment 2

[Andrei Oprea [:andreio]]

Attachment: WIP: Bug 1749775 - Remove the privatebrowsing Nimbus feature

Comment 3

[Dan Mosedale (:dmosedale, :dmose)]

Attachment: data-review.md

This just recently ended up on my plate, and so it's coming close to the end of Nightly 98. We'd like to do an experiment using this code in 98 Release, so I'm hoping it's practical to turn this around (unreasonably?) quickly, for which I apologize.

As you'll notice in the data review, there are a few things that I will be verifying/clarifying early on Thursday, Pacific time. I'm posting this tonight, as I want to get it on folks' radar sooner rather than later, given that we're hoping to land this patch Thursday and/or Friday.

Needinfos on pdahiya, as a quick reviewer to make sure that what I've written fits with the code we hope to landing very soon, and on timhuang, to make sure that this all sounds appropriate and kosher given that this touches about:privatebrowsing, which people are likely to feel particularly sensitive about.

Thanks for all youre input!

Comment 4

[Punam Dahiya [:pdahiya]]

LGTM! For "12) Is there a third-party tool (i.e. not Telemetry) that you are proposing to use for this data collection? If so:" answer should be 'No' as we are using Telemetry to store 'dismiss_btn' clicks similar to 'info_link' and 'promo_link' clicks

https://searchfox.org/mozilla-central/source/browser/actors/AboutPrivateBrowsingChild.jsm#45

Comment 5

[Tim Huang[:timhuang]]

In general, I think this is fine because we will only use internal tools to process data and the data doesn't contain sensitive information that can be used to identify users in PB mode.

But, I think we need to make sure we don't persist any Telemetry data in disk for PB mode. Maybe Chutten can answer this for me.

Comment 6

[Chris H-C :chutten]

(In reply to Tim Huang[:timhuang] from comment #5)

In general, I think this is fine because we will only use internal tools to process data and the data doesn't contain sensitive information that can be used to identify users in PB mode.

But, I think we need to make sure we don't persist any Telemetry data in disk for PB mode. Maybe Chutten can answer this for me.

Firefox's and Mozilla's stance on data collection about and within Private Browsing Mode remains unchanged from when Ehsan laid out this framework on the governance list back in 2018. It's a good read, so consider giving it a skim in its entirety, but for brevity's sake the relevant passage is towards the top:

Going forward, we'd like to modify this policy and will no longer do things to conceal, from ourselves or from local adversaries, the fact that the user was in private browsing.

As an example, the uri count we use to qualify whether a day's activity in the browser, y'know, had some activity in it? The load of any uris in PBM counts.

Comment 7

[Chris H-C :chutten]

Comment on attachment 9262045 [details]
data-review.md

PRELIMINARY NOTES:

In the answer to Q12 there is no third-party tool being used or considered. Even PingCenter [sic] uses Mozilla tools (it uses TelemetrySend on the client, and The Data Platform on the client).

DATA COLLECTION REVIEW RESPONSE:

Is there or will there be documentation that describes the schema for the ultimate data set available publicly, complete and accurate?

Yes.

Is there a control mechanism that allows the user to turn the data collection on and off?

Yes. This collection is Telemetry so can be controlled through Firefox's Preferences.

If the request is for permanent data collection, is there someone who will monitor the data over time?

Yes, Dan Mosedale is responsible.

Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Category 2, Interaction.

Is the data collection request for default-on or default-off?

Default on for all channels.

Does the instrumentation include the addition of any new identifiers?

No.

Is the data collection covered by the existing Firefox privacy notice?

Yes.

Does the data collection use a third-party collection tool?

No.

---

Result: datareview+

Comment 8

[Dan Mosedale (:dmosedale, :dmose)]

Attachment: data-review.md

data-review updated as proposed by Punam; carrying forward the r+.

Comment 9

[Dan Mosedale (:dmosedale, :dmose)]

[Tracking Requested - why for this release]:

This is required to run a planned experiment on the private browsing page in 98 release.

Comment 10

[Andrei Oprea [:andreio]]

Attachment: WIP: Bug 1749775 - Finish 98 VPN experiment promo impl: user telemetry + string workaround

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/094fa0ecd170
Add user telemetry for 98 VPN promo, r=andreio,pdahiya

Comment 12

[Noemi Erli[:noemi_erli]]

https://hg.mozilla.org/mozilla-central/rev/094fa0ecd170

Comment 13

[Dan Mosedale (:dmosedale, :dmose)]

Steps to verify the telemetry:

https://docs.google.com/document/d/1NZHLHY-Su16J7cTnkdVd_GB9IO_eQ3qFShh4xq8Wuvk/edit#bookmark=id.q0zw069y78yu

Comment 14

[Ciprian Muresan [:cmuresan], Ecosystem QA]

I have verified that the expose (Nimbus), click - promo_link, and click - dismiss_button events are successfully displayed in about:telemetry#events-tab when the VPN promo is displayed, the CTA button clicked, and the x button clicked using the latest Firefox Nightly (99.0a1, BuildID 20220213214259) on Windows 10, macOS 11.2.6, Ubuntu 20.04.
I've also verified that the pings can successfully reach a server using a local telemetry server.

Comment 15

[Dan Mosedale (:dmosedale, :dmose)]

Thanks so much, Ciprian! You're the best!

Comment 16

[Dan Mosedale (:dmosedale, :dmose)]

Comment on attachment 9258907 [details]
Bug 1749775 - Add user telemetry for 98 VPN promo, r?pdahiya,andreio

Beta/Release Uplift Approval Request

• User impact if declined: We won't be able to run a test about:privatebrowsing VPN promo experiment in Beta 98, and we won't be able to run the real experiment on 98 release.
• Is this code covered by automated tests?: No
• Has the fix been verified in Nightly?: Yes
• Needs manual test from QE?: Yes
• If yes, steps to reproduce: Listed at https://docs.google.com/document/d/1NZHLHY-Su16J7cTnkdVd_GB9IO_eQ3qFShh4xq8Wuvk/edit#bookmark=id.q0zw069y78yu
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): Only touches telemetry that is not currently in use.
• String changes made/needed: None

Comment 17

[Pascal Chevrel:pascalc]

Comment on attachment 9258907 [details]
Bug 1749775 - Add user telemetry for 98 VPN promo, r?pdahiya,andreio

Approved for 98 beta 5, thanks.

Comment 18

[Pascal Chevrel:pascalc]

https://hg.mozilla.org/releases/mozilla-beta/rev/160f955e94bd

Comment 19

[Ciprian Muresan [:cmuresan], Ecosystem QA]

I have verified that the expose, promo_link, and dismiss_button events are successfully registered on the latest Firefox Beta 98.0b5 (BuildID 20220215094522) using Windows 10, macOS 11.6.3, Ubuntu 20.04 and also verified that the same pings successfully reach a local telemetry server.