1722834 - add telemetry for captive portal vpn promo for targeting and opportunity sizing validation

Status: RESOLVED FIXED

(Core :: Networking, task, P1)

Description

[Dan Mosedale (:dmosedale, :dmose)]

Needed for 92: Opportunity sizing and experiment interaction validation numbers

networking.captive-portal.login_infobar_displayed (event)

• will help us with population sizing for upcoming experiments (need to get DS confirmation on this)
• driven by code in the front-end

networking.captive-portal.login_button_pressed (event)

• will help us validate that our impression targetting is correct
• driven by code in the front end

networking.captive-portal.login_successful: (event)

• will help us validate that our impression targetting is correct
• driven by captive-portal-login-success observer

There are attachments in https://bugzilla.mozilla.org/show_bug.cgi?id=1697222 that show reasonable examples of similar data collection and a data review form that may be useful in implementing this.

Comment 1

[Dan Mosedale (:dmosedale, :dmose)]

There is an existing semi-spec for general captive portal telemetry in bug 1315968. This is comprehensive enough that I don't think we want to do all this stuff just for our experimentation, but putting it here for reference.

Comment 2

[Dan Mosedale (:dmosedale, :dmose)]

To clarify, I still think we should implement our own simple experiment-specific telemetry here, since we want to turn something around fast for 92.

Comment 3

[Punam Dahiya [:pdahiya]]

It will be good to collect networking.login_successful_captive_portal: (event) , that helps define captive portal usage success/failures.

Specific to experiment promo UI, If we use messaging system infobar we should be able to leverage existing pings https://firefox-source-docs.mozilla.org/browser/components/newtab/docs/v2-system-addon/data_events.html#infobar-pings

Comment 4

[Dan Mosedale (:dmosedale, :dmose)]

:pdahiya, if we do leverage the infobar code that sends existing pings, I assume we still need to get the specifics of the pings for our new infobar a data-review. Is that correct?

Comment 5

[Dan Mosedale (:dmosedale, :dmose)]

Some discussion from slack:

pdahiya: yeah it will be good to dig into CaptivePortalWatcher, will probably be easier to add firefox telemetry similar to upgrade dialog

barret: @dmosedale so for event telemetry we need to group it under a category, for which we have to manually turn on/off collection of all events in that category. e.g. Services.telemetry.setEventRecordingEnabled("networking")
Maybe we should name the category something more specific like networking.captive_portal or networking.vpn_experiment?

I'd say "networking.captive_portal", since this is really just a subset of the stuff in bug 1315968.

I also feel like there's pretty high value in also implementing the "failed to connect" event from bug 1315968, since that would be a great baseline to help identify the (believed low likelihood) case where we inadvertently break some captive portal-ly thing.

I'll sleep on this and have more thoughts Friday. In the meantime, opinions very much solicited...

Comment 6

[Dan Mosedale (:dmosedale, :dmose)]

A pointer to the newtab infobar telemetry, from :pdahiya: https://firefox-source-docs.mozilla.org/browser/components/newtab/docs/v2-system-addon/data_events.html#infobar-pings

Comment 7

[Punam Dahiya [:pdahiya]]

(In reply to Dan Mosedale (:dmose, :dmosedale) from comment #4)

:pdahiya, if we do leverage the infobar code that sends existing pings, I assume we still need to get the specifics of the pings for our new infobar a data-review. Is that correct?

@dmosedale my understanding we get data review for only new telemetry, info bar existing pings had been reviewed and should be safe to reuse. https://firefox-source-docs.mozilla.org/browser/components/newtab/docs/v2-system-addon/data_events.html#infobar-pings

Comment 8

[Valentin Gosu [:valentin] (he/him)]

(In reply to Dan Mosedale (:dmose, :dmosedale) from comment #5)

barret: @dmosedale so for event telemetry we need to group it under a category, for which we have to manually turn on/off collection of all events in that category. e.g. Services.telemetry.setEventRecordingEnabled("networking")

We already use the "network." namespace for some probes, so we may want to use the same namespace. I support using more specific names for the probe, such as network.vpn_experiment, but as long as it doesn't interfere with existing telemetry I don't have a strong opinion.

Comment 9

[Andrei Oprea [:andreio]]

Telemetry probes are documented in newtab/data_events because pings are sent using PingCentre not the regular telemetry pipeline; newtab is the only client of PingCentre.

Comment 10

[Dan Mosedale (:dmosedale, :dmose)]

We've shifted the form of what we're doing slightly: for this first experiment, the specific goals are to validate our targeting and give us an approximate opportunity size for the en-US market.

As a result, we're shifting the purpose of this bug a bit. The infobar telemetry review (if needed) will happen as part of bug 1722990, and we get the telemetry implementation simply by using the infobar, so it's not really separable from the UX as a whole.

Furthermore, the risk described earlier of us inadvertently breaking something captive-portally is now believed to be exceedingly low, to the point of not controlling for, as I now understand that this is one of normal ways that UJET does experiments: target off an existing event, and inject just such an infobar without changing the feature implementation itself.

This bug is now going to host implementing the small subset of captive-portal telemetry that will help validate targetting data and the opportunity size for the experiment that we're going to be hosting.

Comment 11

[Dan Mosedale (:dmosedale, :dmose)]

In response to https://bugzilla.mozilla.org/show_bug.cgi?id=1315968#c6 (another bug, but I'm moving this part of the conversation here, as it's where I expect the implementation to happen.

• would you accept a patch implementing this? If so, where would you prefer it be collected? browser-captivePortal.js?

I assume telemetry recording interactions with the captive portal bar would go there, yes.
Otherwise recording the observer notifications could be done in the CaptivePortalService or in a separate module. I don't have a strong opinion on that.

OK, this is useful input. My suspicion is that the easiest thing to do this week will be to just have the observer notifications done in browser-captivePortal.js, but I'll let :barret make a proposal on that front.

:valentin, I don't suppose you happen to know whether the CP infobar will always be displayed before the captive-portal-login-success is sent?

Comment 12

[Dan Mosedale (:dmosedale, :dmose)]

So, what I think we want now is this:

opt-out for release (since other channels are likely to have quite different selection bias)

networking.captive-portal.login_infobar_displayed (event)

• this will help with population sizing for our experiment (I'm not sure quite how, though; :andreio can you confirm and explain or deny this?)
• driven by code in the front-end

networking.captive-portal.login_successful: (event)

• driven by captive-portal-login-success observer
• will help us validate that our targetting code will is selecting the right set of folks
• :andreio, I'm assuming that we'll be able to count the number of displays in en-US only and compare it to the number of promo infobar impressions and they should be equal. Does that sound right to you?

and I've updated comment 0 to capture it.

This will require a data review, and we could write our specific use case and some UJET related email addresses to monitor it. HOWEVER, I'm assuming this is stuff that the CP folks will want to keep as part of the Firefox Health Report data indefinitely, since it's a subset of the bigger set things they care about. So maybe the rationale in the data-review really wants to be about the needs and intent of the CP team, and have a email addresses related to them hooked up...

:valentin, what do you think?

Comment 13

[Valentin Gosu [:valentin] (he/him)]

(In reply to Dan Mosedale (:dmose, :dmosedale) from comment #11)

:valentin, I don't suppose you happen to know whether the CP infobar will always be displayed before the captive-portal-login-success is sent?

I'm not 100% sure. The CP checker could complete very quickly and the window may not be visible when the CP login process is happening ( see here )

(In reply to Dan Mosedale (:dmose, :dmosedale) from comment #12)

:valentin, what do you think?

Sounds good. I think it's worth also recording an event for when the user actually clicks the button to open the login page, since a login could also happen outside of Firefox.

Comment 14

[Dan Mosedale (:dmosedale, :dmose)]

Based on discussion with :valentin in the tech doc, it sounds like we're likely to only want to display the promo infobar if the login button is pressed in the firefox captive portal infobar, since otherwise the user may have logged in some other way (presumably using the OS UI?), leaving the user potentially without context. I've made a note in the tech plan to consider this in more detail in v2.

So I've added this to the spec in comment 0:

networking.captive-portal.login_button_pressed (event)

• will help us validate that our impression targetting is correct
• driven by code in the front end

Comment 15

[Beth Rennie [:beth] (she/her)]

Attachment: 1722834-data-review.md

Comment 16

[Beth Rennie [:beth] (she/her)]

Attachment: Bug 1722834 - Implement telemetry for captive portal detection and login r?valentin

Comment 17

[Chris H-C :chutten]

Comment on attachment 9234546 [details]
1722834-data-review.md

DATA COLLECTION REVIEW RESPONSE:

Is there or will there be documentation that describes the schema for the ultimate data set available publicly, complete and accurate?

Yes.

Is there a control mechanism that allows the user to turn the data collection on and off?

Yes. This collection is Telemetry so can be controlled through Firefox's Preferences.

If the request is for permanent data collection, is there someone who will monitor the data over time?

No. This collection will expire in Firefox 99.

Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Category 2, Interaction.

Is the data collection request for default-on or default-off?

Default on for all channels.

Does the instrumentation include the addition of any new identifiers?

No.

Is the data collection covered by the existing Firefox privacy notice?

Yes.

Does the data collection use a third-party collection tool?

No.

---

Result: datareview+

Comment 18

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/08b6e297674f
Implement telemetry for captive portal detection and login r=valentin

Comment 19

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/08b6e297674f