1513319 - Allow rapid experimentation with appropriate features, cohorts, telemetry, etc.

Status: RESOLVED FIXED

(Firefox :: New Tab Page, enhancement, P1)

Description

[Ed Lee :Mardak]

We want to be able to experiment with a large number (~10/week?) of potential new tab / pocket designs to figure out which lead to better engagement with low overhead and trusted results.

For example, we probably want to try multiple layouts at the same time to minimize the effects of stories differing each day.

One idea is to adjust the server endpoint for various cohorts of users for an ongoing study that allows the server to adjust the layout at whatever rate without needing additional experimentation signoff.

To be able to differentiate the multiple experiments for a given cohort, we'll want to make sure our telemetry can differentiate them, which probably involves just sending back some experiment id.

Examples of `stories_endpoint` of browser.newtabpage.activity-stream.feeds.section.topstories.options

default / control: https://getpocket.cdn.mozilla.net/v3/firefox/global-recs?version=3&consumer_key=$apiKey&locale_lang=en-US&feed_variant=default_spocs_on
cohort 1: …&feed_variant=cohort_1
cohort 2: …&feed_variant=cohort_2
…

The current response: { stories:
   { status: 1,
     settings:
      { spocsPerNewTabs: 1,
        domainAffinityParameterSets: [Object],
        timeSegments: [Array],
        recsExpireTime: 5400,
        version: 'bc184fc1a56dde7e794af29f070f3fdb1dd5d9a2' },
     spocs: [Array],
     recommendations: [Array],

Potentially we could just echo back the "settings.version" value in pocket related telemetry?

Comment 1

[Ed Lee :Mardak]

In terms of trusted results, we'll want to design the experiments to minimize unintended conflicting/missing data discrepancies and get product and experimentation sign off.

Comment 2

[Ed Lee :Mardak]

Experiment Design has raised concerns of reusing cohorts as there will be confounding factors of people not going from a control-UX to experiment-UX (and instead going from experiment1-UX to experiment2-UX). Potentially to get high rates of experiments with non-overlapping populations is to get release/QA sign offs for a family of experiments instead of needing sign-offs for each experiment.

Comment 3

[Nan Jiang [:nanj]]

All the experiment related telemetry is identified and tracked at

https://docs.google.com/document/d/1H7VAh2kc0Xhnl7GvWT64x5xTZNUc3r_hKYoM25N7XqA/edit?usp=sharing

Comment 4

[Tim Spurway [:tspurway]]

marking this done