1499651 - Develop Methods for statistical inference on growth metrics

Status: RESOLVED FIXED

(Data Science :: Investigation, task)

Description

[Jesse McCrosky]

Brief description of the request:

Determine an appropriate (and hopefully general) method for calculating confidence intervals for and performing hypothesis tests on growth metrics.

Link to any assets:

Summary document: https://docs.google.com/document/d/1Q7fmMoEV9KC9KtQlZnyaOsIFw6EGOq-VIuOvRQ55Mt4/

There has been significant discussion on #growth-metrics-ci on slack and I will paste that conversation in an update once I confirm there are no objections.

Is there a specific data scientist you would like or someone who has helped to triage this request:

Jesse McCrosky initiated this project and is working on it with Tim Smith

Comment 1

[Jesse McCrosky]

In case people are interested, this is the first “production” implementation of the methods I proposed.  In the dashboard we can see that in the small-sample arms of the experiment there are differences in retention, but the confidence intervals make clear that these differences are just noise.  The hypothesis tests a the top show that none of the arms has a significant difference in retention https://dbc-caf9527b-e073.cloud.databricks.com/#notebook/39538/dashboard/46095

Comment 2

[Jess Mokrzecki [:jmok]]

Is this ticket still needed or can it be closed? Please note if we do not receive a response within 7 days we will close the ticket.

Comment 3

[Jesse McCrosky]

This is still ongoing work.

Comment 4

[Jesse McCrosky]

These methods are now released in GUD. Further validation could be done, but is not a priority, so closing this.