I met DS and survey folks on Friday to discuss the survey for Interventions, and they have a few asks for the survey.
They've asked that we trigger the survey directly after the user types one of the identified searches. Since we're already recording event telemetry on each of the control and variant branches, I hope we can do this relatively easily.
They've asked that we show the survey to users in three distinct cases:
Treatment: When they engage with an intervention
Treatment: When they are shown an intervention but ignore it
Control: When they would have seen an intervention but we didn't show it
Survey group size
We need to aim for 250+ responses per survey group to get useful results. Typically we'll get about 2% of users to respond to a survey, so that means we'll need to get at least 12500 users to click on an intervention. We'll see how the experiment goes, but we can keep it running if we don't get enough results.
As the first case will be the least common, we'll probably want to prompt 100% of the time. For the other cases, we'll want to randomly sample so we get about the same number of surveys in each group. As an example we think we'll see a CTR of 2%, we should survey about 2% of the other groups to keep it balanced. We don't need to be perfect here, we just need to make sure we hit 250+ without over-surveying the second and third groups.
To implement, the add-on needs to know the URLs to open. Related:
To implement the "random sample" part for the other two cases, the add-on needs to know the number of users in the experiment.