1456141 - Document exact targeting/frequency parameters for Onboarding experiment

Status: RESOLVED FIXED

(Firefox :: Messaging System, enhancement, P2)

Description

[Kate Hudson :k88hudson]

For our upcoming onboarding experiment, we should define exactly what the targeting parameters should be. These will be expressed as JEXL in the message definitions. 

For example:
- should there be a delay before the message is shown?
- for how long / how many times should the message be shown?
- are there restrictions on profile age?

Comment 1

[Kate Hudson :k88hudson]

Also: how does this interact with the about:welcome stuff?

Comment 2

[Ursula Sarracini (:ursula)]

Here's a doc I put together for some preliminary onboarding targeting parameters: https://docs.google.com/document/d/1ivWv3wRg3Juih_IeGdRCtgZcFAsrIogsdjf8ufb_QUg/edit#heading=h.m13lpag72vpa . Key take away here is that it will have the same criteria as the about:welcome new first run experience, and so we'll need to work out the parameters for running those experiments. 
Chris, can you take a look at this and see that I've gotten everything right? As well as provide some insight about how the about:welcome experiment will run, and how that fits into the experiment we want to run with the new onboarding tour?

Comment 3

[Chris More [:cmore]]

(In reply to Ursula Sarracini (:ursula) from comment #2)
> Here's a doc I put together for some preliminary onboarding targeting
> parameters:
> https://docs.google.com/document/d/
> 1ivWv3wRg3Juih_IeGdRCtgZcFAsrIogsdjf8ufb_QUg/edit#heading=h.m13lpag72vpa .
> Key take away here is that it will have the same criteria as the
> about:welcome new first run experience, and so we'll need to work out the
> parameters for running those experiments. 
> Chris, can you take a look at this and see that I've gotten everything
> right? As well as provide some insight about how the about:welcome
> experiment will run, and how that fits into the experiment we want to run
> with the new onboarding tour?

Makes sense to me. Where do we define the variations of the A/B test at? The variations should be:

a: control (current experience)
b: about:welcome
c: no first run experience at all (control of the control)

That will give us how much retention impact the /firstrun/ has over doing nothing (variation C) and how much about:welecome retention impact has compared to /firstrun/ and doing nothing.

Thanks!

Comment 4

[Ursula Sarracini (:ursula)]

Thanks Chris. I'll document the variations for both experiments in the doc. It looks like we have a clear path forward for the new onboarding tour and about:welcome in 62. In addition, the document in comment 2 has the specific targeting parameters for the new onboarding experience. I think we can close this now.

Comment 5

[Ed Lee :Mardak]

https://hg.mozilla.org/mozilla-central/rev/79ab9d325994