1471318 - Add "reach" telemetry for Messaging Experiments

Status: VERIFIED FIXED

(Firefox :: Messaging System, enhancement, P1)

Description

[Tim Spurway [:tspurway]]

Let's add a field to Activity Stream session ping to capture if the user meets any targeting criteria of any active AS Router messages.  This value should be a JSON array of AS Router message ids that matched for the user regardless of whether or not an actual message was shown to the user during the session.  

Note that if the user has disabled any Message Providers (such as Snippets), then the messages from provider should *not* be included in the session ping

Comment 1

[Kate Hudson :k88hudson]

Some further questions for this bug:

- do we need just a number of unique users for reach or also the number of estimated total possible impressions/sessions?
- is a 24-hour cycle sufficiently accurate for looking at the targeting criteria?

Comment 2

[Tim Spurway [:tspurway]]

The goal of calculating the reach of a targeting expression would be to answer the question:  "how many unique users could possibly be targeted each day by this expression?".   

I think it would suffice to have a daily ping with each user reporting if they matched each expression at any time during that day.  The caveat here is that we be careful about 'day part' targeting, which targets a certain time of day.  We will have to make sure that we take this into consideration when we are evaluating the expressions for the daily ping.

Comment 3

[Kate Hudson :k88hudson]

De-prioritizing this in favour of other more important bugs for 63

Comment 4

[Andrei Oprea [:andreio]]

Message ids are category 3 information and cannot be linked with session id.
We have two options:
- Discuss the objective of this bug, see what we're trying to measure and see if we can get that metric through some other means
- Report message ids without session information. We would keep the 24hr requirements and send out regular pings of the messages a user has seen but we won't be able to link that information to anything.

Note this issue is similar to pocket impressions which were not allowed to be linked to clients and we had to come up with a proxy id to work around it.

Comment 5

[Kate Hudson :k88hudson]

Is this something we still care about and are willing to prioritize?

Comment 6

[Nan Jiang [:nanj]]

Reset the priority and assignee as now this ping regains the interest.

Comment 7

[Nan Jiang [:nanj]]

Attachment: Bug 1471318 - Add reach ping for Messaging Experiments r=andreio,k88hudson

Comment 8

[Nan Jiang [:nanj]]

Attachment: data_review_reach_ping.txt

Hey chutten, could you take a look at this new collection?

You can find the document here.

Thanks!

Comment 9

[Tim Smith (inactive) 👨‍🔬 [:tdsmith]]

Hi; I'm a data scientist working on automated experiment analysis infrastructure, and this provides a critical improvement in our ability to analyze these experiments quickly and rigorously, so thank you for working on it.

I left some comments in phabricator with a couple of small questions about the documentation, but I had a few broader questions. I'm not sure whether a distinction is being made between cohorts and branches; can you explain? What does it mean to qualify for a cohort? What's the difference between qualification and experiment enrollment?

For this to be useful, this event has to be sent at the moment when a client in the control branch would have seen a message, if they had been enrolled in any of the variant branches.

Comment 10

[Nan Jiang [:nanj]]

(In reply to Tim Smith 👨‍🔬 [:tdsmith] from comment #9)

I left some comments in phabricator with a couple of small questions about the documentation, but I had a few broader questions. I'm not sure whether a distinction is being made between cohorts and branches; can you explain?

Hi Tim, thanks for the comments/questions! There is no distinction between them, we use them interchangably. But I agree, we should be more consistent on the wording. Will review the event schema with our DS (:shong), and unify the wording in the code/docs.

What does it mean to qualify for a cohort? What's the difference between qualification and experiment enrollment?

There are two levels of targeting for Messaging Experiments (including Firstrun (about:welcome), CFR, Moments Page, What's New Panel, and Snippets). The first level is for the experiment enrollment, like those filters defined in the Normandy experiment recipe. Here is an example for a CFR experiment, this is the enrollment targeting. The second level is for message targeting. For instance, we only show a recommendation to users who have met certain conditions. Note that each branch has its own targeting, whether or not to show the message is up to which branch that user was enrolled in.

For this to be useful, this event has to be sent at the moment when a client in the control branch would have seen a message, if they had been enrolled in any of the variant branches.

Agreed. The current implementation will report all the qualified branches (including the enrolled one) from all the users enrolled in the experiment. Will confirm that with Su to ensure we're collecting the right thing.

Comment 11

[Tim Smith (inactive) 👨‍🔬 [:tdsmith]]

Thanks for the examples; that's really helpful. This addresses message targeting, right? And enrollment targeting and telemetry is handled by ExperimentManager?

The two branches here seem to have identical targeting and trigger expressions; is it the case that all clients would send a reach event containing both branch names when the CFR is displayed? That would be great.

I'm concerned that there's an 80 character limit on event values; firing a separate event for each branch instead of concatenating branch names would avoid the risk of failing to send telemetry for complex experiments with many branches, and also has better query ergonomics for the use cases I have in mind.

Comment 12

[Nan Jiang [:nanj]]

(In reply to Tim Smith 👨‍🔬 [:tdsmith] from comment #11)

Thanks for the examples; that's really helpful. This addresses message targeting, right? And enrollment targeting and telemetry is handled by ExperimentManager?

Exactly, this ping only cares about the message targeting. The enrollment targeting and telemetry are indeed handled by ExperimentManager, which, in turn, relies on Normandy. I've added this bit in the patch.

The two branches here seem to have identical targeting and trigger expressions; is it the case that all clients would send a reach event containing both branch names when the CFR is displayed? That would be great.

Indeed, their targeting are exactly the same. Since we wanted to run this as an A/A test for ExperimentManager.

I'm concerned that there's an 80 character limit on event values; firing a separate event for each branch instead of concatenating branch names would avoid the risk of failing to send telemetry for complex experiments with many branches, and also has better query ergonomics for the use cases I have in mind.

Yeah, we're aware of those length limitations of Events telemetry, and understanding the risk of string truncation. Will finalize the strategy with Su for a more robust solution. Thanks for the reminder!

Comment 13

[Nan Jiang [:nanj]]

Have gone through the proposed schema and other details with Su, and we believe that the current implementation is sufficient for us to get started.

We will also be using the comma-separated string to record the qualified branch IDs for now, and we will mitigate the truncation risk by experiment review and monitoring.

Comment 14

[Chris H-C :chutten]

Comment on attachment 9144342 [details]
data_review_reach_ping.txt

DATA COLLECTION REVIEW RESPONSE:

    Is there or will there be documentation that describes the schema for the ultimate data set available publicly, complete and accurate?

Yes. This collection is documented in its [in-tree documentation](https://firefox-source-docs.mozilla.org/browser/components/newtab/docs/v2-system-addon/data_events.html).

    Is there a control mechanism that allows the user to turn the data collection on and off?

Yes. This collection can be controlled through Firefox's Preferences.

    If the request is for permanent data collection, is there someone who will monitor the data over time?

Yes, Nan Jiang is responsible.

    Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Category 1, Technical.

    Is the data collection request for default-on or default-off?

Default on for all channels.

    Does the instrumentation include the addition of any new identifiers?

No.

    Is the data collection covered by the existing Firefox privacy notice?

Yes.

    Does there need to be a check-in in the future to determine whether to renew the data?

No. This collection is permanent.

---
Result: datareview+

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/8d47bff523b1
Add reach ping for Messaging Experiments r=k88hudson

Comment 16

[Bogdan Tara[:bogdan_tara | bogdant]]

https://hg.mozilla.org/mozilla-central/rev/8d47bff523b1

Comment 17

[Marius Coman [:mcoman], Ecosystem QA]

I have verified that the "messaging_experiments#reach#cfr" scalar is successfully displayed in the "about:telemetry" page using the following steps:

[Steps to reproduce]:

1. Open the browser with the profile from prerequisites.
2. Open a new tab am click the "Facebook" top site.
3. Repeat the previous step 4 times.
4. Navigate to the "about:telemetry" page and go to the "Keyed Scalars" section.
5. Observe the scalars displayed in the "telemetry.event_counts" section.

Verified using the Firefox Nightly 77.0a1 and latest Firefox Nightly (78.0a1 Build ID - 20200504205324) installed on Windows 10 x64, Mac 10.15.4, and Ubuntu 18.04 x64.