1515360 - [Experimentation Support] Research Population Management In Experiment Series

Status: RESOLVED FIXED

(Data Science :: Investigation, task)

Description

[Su-Young Hong]

Brief description of the request:

* For the newtabs experiments, for each experiment, we need to draw 'fresh' study subjects for each experiment (haven't seen an experiment variant yet). however, since we'll be running multiple experiments, how will our subject poplution 'shrink' over time, and how can we manage their use? 
* deliverable is report so we can understand this
* followup is talk to experiments team (mythmon or rob helmer) about how to implement longitudinal management of subjects

Link to any assets:

* https://trello.com/c/oSwlKUGn/15-experiment-design

Is there a specific data scientist you would like or someone who has helped to triage this request:

* assigned to me from me

Comment 1

[Su-Young Hong]

update, spoke to Nick Chapman and it sounds like the timeline for [start experimentation in release] to [rollout to general population] is up to 3 release cycles (approximately 6x3 weeks) 

calculate based on that longest period scenario assumption

Comment 2

[Su-Young Hong]

V1 of population management simulation here:

https://dbc-caf9527b-e073.cloud.databricks.com/#notebook/64292/command/64899

Comment 3

[Su-Young Hong]

Update:

So changing the scope of this bug a little bit.

Originally, it was to double check what the addressable population of Newtab Experiment series was and how that shifted over a time period as we ran more experiments[1], so we could be informed about what our experimentation sampling population would look like throughout the experiment series.

Those estimates are available here (these notebooks are not polished, they're working edits)

[newtab-experiments]layouts-test-sizing-and-power-V1(
[newtab-experiments]population-management-investigation
[newtab-experiments]population-management-investigation-pt2

---

Now, I'm going to update the scope of this bug so the goal is:

• Report describing how a given "addressable population" shifts over time given no-resampling[1].
• Generalized report or functions that allow one to do this with different filtering criteria
• Writeup of why this is useful and use cases for this

Next Steps:

• Decide where this will "live"
  • Candidates: functions in mozanalysis, notebook in an experiments resources repo, ???
• Think of use cases and how it fits into an experiments workflow

[1] With the condition that profiles enrolled in past experiments would not be allowed to enroll in new experiments.

Comment 4

[Su-Young Hong]

walking this back to the original scope (get estimates of eligible population run over the Newtab experiment series, done).

The expanded scope (generalize methodology, develop functions to do this in future experiments, report for why this is important) can be revisited later in reopened/new bug.

closing this now.