1621289 - Use stableSample in FxMessaging System

Status: RESOLVED FIXED

(Firefox :: Messaging System, enhancement, P1)

Description

[Kate Hudson :k88hudson]

In order to support releasing messages to a random sample of users via Messaging System, we'd like to use the stableSample jexl transform via FilterExpression.jsm.

Example usage:

We want to release LOGIN_CFR to a random sample of 50% of users who have Firefox accounts in order to:

• Measure click-through rate
• Limit exposure to the experimental message

{
  id: "LOGIN_CFR",
  content: {...},
  targeting: "[userId, 'LOGIN_CFR'].stableSample(0.5) && usesFirefoxSync"
}

Note that:

• the stableSample code is shared/identical to Normandy's implementation
• targeting is evaluated at the moment when the message will be shown, rather than at start-up
• branch enrolment / experiments are out of scope for this use case

Questions:

1. We don't have access to normandy.userId in messaging system, is that what we should use for input or is something else appropriate?
2. Is it correct to assume the ordering of randomization / other targeting doesn't matter?

Is this approach sound from an analysis perspective, and if not, is there a better approach?

Comment 1

[Tim Smith (inactive) 👨‍🔬 [:tdsmith]]

1. client_id is equally useful as an input.

2. If you want to measure the effect of altering an experience that's delivered after some predicate event, randomizing in advance or once the predicate event has occurred should be equivalent. [1]

I understand that this is for a use case where you have no intention of doing A/B comparisons and just want to limit the exposed population. If an absolute CTR number is the only analysis you need, and you don't need to e.g. compare CTRs of different messages, this seems right, yeah. Do you have a use case in mind for this or is the point to prove a concept?

Is this the feedback you were looking for?

A thing I've reflected on lately, thinking about Normandy preference rollouts, is that including the population-size filter in the JEXL filter expression makes it harder for Normandy to emit telemetry that says "I matched this filter expression, except for the random filter, so I was not randomized into the rollout," which would be useful in the inevitable case that someone someday tries to do an A/B test this way, or wishes they had run one after the fact.

[1] When you enroll in advance, you measure the effect of deploying the intervention; when you enroll after the predicate event, you measure the effect of experiencing the intervention. Understanding how often the predicate event happens lets you calculate one from the other.

Comment 2

[Kate Hudson :k88hudson]

This is super helpful and exactly what I'm looking for, thanks!

There is a specific use case here (https://bugzilla.mozilla.org/show_bug.cgi?id=1620313) which will show a CFR-style survey UI (a little pill in the address bar) to a sample of users so that we can get a quick sense if the click-through rate is in the realm of viability for surveys before investing in further testing, which would involve building out out the rest of the doorhanger.

I agree that it's worth looking dealing with the population-size filter separately – I'll capture that in our separate discussion around experiments code / targeting.

Comment 3

[Kate Hudson :k88hudson]

Attachment: Bug 1621289 - Add userId to ASRouterTargeting.jsm

Comment 4

[Kate Hudson :k88hudson]

Ok, one more point after discussing this – we might want to use some kind of "experiment identifier" instead of the message ID as an input to the sample, in case we need to run the sample again.

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/d359693f52b1
Add userId to ASRouterTargeting.jsm r=andreio

Comment 6

[Kate Hudson :k88hudson]

[Tracking Requested - why for this release]: Required for the targeting expression of an experiment in 75 (Bug 1620313)

Comment 7

[Daniel Varga [:dvarga]]

https://hg.mozilla.org/mozilla-central/rev/d359693f52b1

Comment 8

[Robert (rrayborn, he/him)]

I'm not sure if the scope of this is short or long term, but for a long term solution I'd like to chime in with a few comments.

Having a consistent user ID will be important for Normandy experiment follow up surveys. It's fine if that ID is not the Normandy ID (although that would require education and changes in how we deploy experiments with survey follow ups). Or we can use legacy Heartbeat for that purpose.

Re-targeting users within our survey tooling is rare but helpful.

Thanks and great thread!

Comment 9

[Kate Hudson :k88hudson]

Comment on attachment 9132693 [details]
Bug 1621289 - Add userId to ASRouterTargeting.jsm

Beta/Release Uplift Approval Request

• User impact if declined: Unable to sample a new heartbeat message type via messaging to a sub-set of users, which would mean having to release it to 100% of the population or use a more expensive approach
• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: Yes
• Needs manual test from QE?: No
• If yes, steps to reproduce:
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): Adds targeting with no impact to user experience directly. Note that this functionality will be part of the QA scope of releasing any off-train messages that use it.
• String changes made/needed:

Comment 10

[Julien Cristau [:jcristau]]

Comment on attachment 9132693 [details]
Bug 1621289 - Add userId to ASRouterTargeting.jsm

approved for 75.0b5

Comment 11

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/releases/mozilla-beta/rev/5447953ca2dc