1353102 - (singlepref-experiment-cep-monitoring) Monitoring Status of Deployed Quantum (pref-flip) Experiments should be awesome

Status: RESOLVED INVALID

(Data Platform and Tools :: Monitoring & Alerting, enhancement, P2)

Description

[Gregg Lind (Fx Strategy and Insights - Shield - Heartbeat )]

# Problem Statement

After deploy / turn on pref-flip experiments (from Normandy) is SHOULD be EASY to know if they are working [1].

## AUDIENCE
- Not the public
- Engineer / pm / data comfortable


## [1] "Working" means
- ENABLED: pings are arriving for all branches
- BRANCH RATIO: with right ratio / frequency
- SAMPLING IS OK (i..e, we aren't getting A MILLION clients, when we expect 300)
- NOTHING IS ON FIRE
  - crashes aren't super super inflated right away
  - (other ideas welcome)

## proposed solution
- IN A WEB PAGE, display [2]
- by BRANCH and EXPERIMENT NAME 
  - total pings
  - # of unique clients
  - optional, but awesome 
    - # of crashes, so we can know
    - have fantasy cutoffs that automagically email someone for some reasons?  
      - first day summary report?
      - OHNO TOO MANY CRASHES
      - NO PINGS, but we expect them

## extrinsic claims
- there will be iterations, and 1st cupcake small, gross, fragile (but working) is better than waiting at all.

## [2] Display
- could be fancy, but a CSV or JSON table is fine.

## Timeline
- experiment rollout: which will be within 3 weeks
- better if it's a earlier, s.t. others can iterate

## Expected work product
- the display (page)
- links to code where the change happened, so that others can read/contrib

Comment 1

[:shell escalante]

Quantum is so done.