Commenting on the adding
trigger to this ping:
Currently, this reach ping is reporting the users that could have seen a given CFR (match the targeting criteria), but that is different from the users that would have seen a CFR (since that depends on user behavior, i.e. hitting a certain website).
What that gives us is something like this (each category is superset of the one below):
- experiment population
- filter eligible [what we have]
- users who send reach ping
- (includes those who hit trigger those who don't)
- (for control, that would be both users who would have seen the CFR in question, as well as those who wouldn't)
- (for the treatment of the CFR in question, that would be users who send impression / see the CFR as well as those who don't)
- filter and trigger eligible [we don't have]
- users who send reach v2 ping described in section below
- (only includes users who have or would have seen CFR in question)
(IMO) I think it's fine to go forward for now with the reach ping as it's currently implemented and evaluate it's usefulness in reducing noise in experiments.
If this allows us to reduce the analysis populations so that it's mostly reflective of the population we want (for example, once we reduce it, then 90% of the reduced population would have seen the CFR, estimated from the impressions in treatment) and we're not concerned about noise washing out experiment effects, then we should be good as is.
If not, however, then we would need a more specific reach ping taking triggers into account. I've laid out some specs or what that would look like below:
The requirements for a reach ping v2 that includes the trigger are thus:
- It should tell us who would have seen a given CFR for an experiment (filter and trigger match)
- It should tell us when they would have seen a given CFR (since we'll need to be able to differentiate pre-exposure and post exposure timelines / data)
- It should have coverage for every CFR defined in the experiment, for every user enrolled in the experiment
- note: careful consideration should be made to distinguish between branches with the same CFR that has different parameters, as well as when a treatment branch is modifying / removing an existing CFR (in which case we'll need a reach ping in the treatment for the unmodified / unremoved CFR).
That would require some significant re-design (changing the checking timing, etc.) so I think it would make sense to implement as a new ping (as opposed to modifying the existing reach ping).