Closed Bug 1545649 Opened 6 years ago Closed 5 years ago

[Pocket] Design Process for Experiment Self Sufficiency

Categories

(Data Science :: General, task)

Product:
Data Science
Component:
General
Platform:
x86_64
Unspecified
Type:
task
Priority:
Not set
Severity:
normal
Points:
3

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: shong, Assigned: shong)

References

Details

Summary:

This bug is to track the process of eventually getting the Pocket team to have self-sufficiency in defining, launching, and running their own Normandy experiments in Firefox.

Currently, I'm taking on a very hands on role which involves consulting on the experiment planning and design, shepherding the experiment through the Experimenter process, and running and reporting the analysis.

The long terms goal is for the Pocket team to take on a more hands on role in the above steps, and for Data Science to take on more of a consulting / review role.

The strategy for getting to that point is to:

  • Map out the process (including the Experimenter process) for experiment design, launch, monitoring, and analysis
  • Describe that process through a comprehensive set of comprehensive templates, checklists, and instructions (with clear roles assigned)
  • Train the Pocket PMs and Data Scientists on how to use above materials

Update:

Started drafts of some process materials. Note, these are not finalized materials yet (WIP)

Experiment Description Template

  • This is a template for PMs to define what the newtab variants they want to test are. This template requires:
    • The pref being flipped
    • The value of the pref for that variant (the endpoint)
    • The payload that the endpoint delivers
    • Screenshot of the newtab experience the variant produces

Outline of Experiment Launching Process

  • This is an outline describing in depth the steps involved in launching an Experimenter ticket, with responsibilities and dependencies for each stage.

Experiment Specs Sheet

  • This document outlines possible specs in an experiment that need to be determined for an experiment to be fully defined. Includes requirements (hard and soft) for the experiment to work with newtab / discovery stream.

Experiment Monitoring Template

  • This is a template dashboard for monitoring an experiment once it has launched. For future experiments, fork the underlying queries with parameterized experiment_slug and submission_date_s3 and date, and rebuild dashboard from the forked queries.

Analysis Templates

  • Template notebooks for experiment analysis. This will probably be modified in the future.

I've updated the reference documents for helping stakeholders navigate the experiment process:

  • Experiment Process Overview For Stakeholders
    • overview of the process for initiating, launching, and analyzing an experiment.
    • explains the steps and roles of each party (PM, ENG, DS) at each step
  • Experiment Fact Finding
    • documentation resource / template for getting necessary information from stakeholders for defining and scoping an experiment
  • Experiment Branches
    • documentation resource / template for documenting what the branches in an experiment are and defining their technical specs

Specifically for pocket, I've provided experiment gotcha's specific to their needs for launching successful experiments

These resources have been shared and communicated with the Pocket team throughout H1.

Closing ticket now.

Status: ASSIGNED → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.