Closed Bug 1513319 Opened 2 years ago Closed 2 years ago

Allow rapid experimentation with appropriate features, cohorts, telemetry, etc.

Categories

(Firefox :: New Tab Page, enhancement, P1)

enhancement

Tracking

()

RESOLVED FIXED
Iteration:
67.2 - Feb 11 - 24

People

(Reporter: Mardak, Assigned: tspurway)

References

Details

We want to be able to experiment with a large number (~10/week?) of potential new tab / pocket designs to figure out which lead to better engagement with low overhead and trusted results.

For example, we probably want to try multiple layouts at the same time to minimize the effects of stories differing each day.

One idea is to adjust the server endpoint for various cohorts of users for an ongoing study that allows the server to adjust the layout at whatever rate without needing additional experimentation signoff.

To be able to differentiate the multiple experiments for a given cohort, we'll want to make sure our telemetry can differentiate them, which probably involves just sending back some experiment id.

Examples of `stories_endpoint` of browser.newtabpage.activity-stream.feeds.section.topstories.options

default / control: https://getpocket.cdn.mozilla.net/v3/firefox/global-recs?version=3&consumer_key=$apiKey&locale_lang=en-US&feed_variant=default_spocs_on
cohort 1: …&feed_variant=cohort_1
cohort 2: …&feed_variant=cohort_2
…

The current response: { stories:
   { status: 1,
     settings:
      { spocsPerNewTabs: 1,
        domainAffinityParameterSets: [Object],
        timeSegments: [Array],
        recsExpireTime: 5400,
        version: 'bc184fc1a56dde7e794af29f070f3fdb1dd5d9a2' },
     spocs: [Array],
     recommendations: [Array],

Potentially we could just echo back the "settings.version" value in pocket related telemetry?
See Also: → 1513312
In terms of trusted results, we'll want to design the experiments to minimize unintended conflicting/missing data discrepancies and get product and experimentation sign off.
Experiment Design has raised concerns of reusing cohorts as there will be confounding factors of people not going from a control-UX to experiment-UX (and instead going from experiment1-UX to experiment2-UX). Potentially to get high rates of experiments with non-overlapping populations is to get release/QA sign offs for a family of experiments instead of needing sign-offs for each experiment.
All the experiment related telemetry is identified and tracked at

https://docs.google.com/document/d/1H7VAh2kc0Xhnl7GvWT64x5xTZNUc3r_hKYoM25N7XqA/edit?usp=sharing
Depends on: 1514006
Assignee: nobody → tspurway
Severity: normal → enhancement
Iteration: --- → 66.1 - Dec 10-23
Priority: -- → P1
Iteration: 66.1 - Dec 10-23 → 66.2 - Dec 24 - Jan 6
Iteration: 66.2 - Dec 24 - Jan 6 → 66.3 - Jan 7 - 20
Iteration: 66.3 - Jan 7 - 20 → 66.4 - Jan 21 - 27
Iteration: 66.4 - Jan 21 - 27 → 67.1 - Jan 28 - Feb 10
Iteration: 67.1 - Jan 28 - Feb 10 → 67.2 - Feb 11 - 24

marking this done

Status: NEW → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED
Component: Activity Streams: Newtab → New Tab Page
You need to log in before you can comment on or make changes to this bug.