Closed Bug 1655977 Opened 5 years ago Closed 5 years ago

[Experiment] Pref-Flip Experiment: activity-stream-pocket-new-tab-dynamic-video-card-release-test-relaunch Fx 78.0 to 79.0 Release

Categories

(Shield :: Shield Study, task, P3)

Product:
Shield
Component:
Shield Study
Type:
task

Tracking

(firefox79+ fixed)

Status:
RESOLVED FIXED
Tracking Flags:
Tracking Status
firefox79 + fixed

People

(Reporter: experimenter, Assigned: digitalnewslab)

References

(
URL
)

Details

User Story

Delivery Type: Pref Flip Experiment

    What is the preference we will be changing

None

    What are the branches of the experiment and what values should
    each branch be set to?

- Treatment Variant 50%:

Value: None

With video buttons
        
- Control Control 50%:

Value: None

Without video buttons
        

    What version and channel do you intend to ship to?

1% of Release Firefox 78.0 to 79.0

    Are there specific criteria for participants?

Preset Common Pocket 1
Countries: United States of America (US) 

Locales: English (US) (en-US) 

    What is your intended go live date and how long will the experiment run?

Aug 04, 2020 - Sep 08, 2020 (35 days)

    What is the main effect you are looking for and what data will you use to
    make these decisions?

The main effect we're looking for is whether a dynamic video card will have a higher CTR than regular story cards. Our main success metric is high CTR with no corresponding decline in engagement with stories and SPOCS.

Do you plan on surveying users at the end of the delivery?  No.
Strategy and Insights can help create surveys if needed

    Who is the owner of the data analysis for this experiment?

kirill@getpocket.com

    Will this experiment require uplift?

False

    QA Status of your code:

This experiment was signed off as GREEN. For further information see:
https://experimenter.services.mozilla.com/experiments/activity-stream-pocket-new-tab-dynamic-video-card-release-test-v2/#comment1057

    Link to more information about this experiment:

https://experimenter.services.mozilla.com/experiments/activity-stream-pocket-new-tab-dynamic-video-card-release-test-v2/

activity-stream-pocket-new-tab-dynamic-video-card-release-test-relaunch

New Tab Story Card with a dynamic video button

Experimenter is the source of truth for details and delivery. Changes to Bugzilla are not reflected in Experimenter and will not change delivery configuration.

Data Science Issue: https://jira.mozilla.com/browse/DO-352
More information: https://experimenter.services.mozilla.com/experiments/activity-stream-pocket-new-tab-dynamic-video-card-release-test-v2/

[Tracking Requested - why for this release]:

tracking-firefox79: --- → ?

New Tab Dynamic Video Card
Firefox Release 78
Firefox Release 79
Firefox Release 80

We have finished testing the [PI-746] RElaunch - New Tab Dynamic Video Card - Firefox Release 79 experiment.

Quality status: GREEN - SHIP IT

Why is this experiment green?

  • During our testing we’ve encountered 1 known issue. However, the issue is not specific to the experiment and it does not impact the user experience in a harmful way.

Testing Summary:

  • Test Suite: TestRail
  • Verified that SPOCs appeared with the US override (browser.newtabpage.activity-stream.discoverystream.spocs-endpoint set to https://spocs.getpocket.com/spocs?site=1124115&country=US&region=CA for the Variant branch, and
    https://spocs.getpocket.com/spocs?site=1070098&country=US&region=CA for the Control branch).
  • Verified the following by using a VPN to simulate the US geo-location:
    • Verified that the enroll/unenroll works correctly by using a Normandy recipe.
    • Verified that the “Recommended by Pocket” is correctly displayed.
    • Verified that the “Sponsored Cards” are correctly displayed and they are working correctly.
    • Verified that the correct telemetry pings are displayed in both browser console and in the “about:telemetry” page.

Tested Platforms:

  • Windows 10 x64
  • Ubuntu Linux 18.04
  • macOS 10.15

Tested browser versions:

  • Firefox Release 78.0.2 en-US (Build ID: 20200708170202)
  • Firefox Release 79.0 en-US (Build ID: 20200720193547)
  • Firefox Beta 79.0b8 en-US (for the browser update scenario)
  • Firefox Beta 80.0b2 en-US (Build ID:20200730152159)

Regards,
Robert Martin
Ecosystem QA

User Story: (updated)
Summary: [Experiment]: Pref-Flip Experiment: activity-stream-pocket-new-tab-dynamic-video-card-release-test-relaunch → [Experiment] Pref-Flip Experiment: activity-stream-pocket-new-tab-dynamic-video-card-release-test-relaunch Fx 78.0 to 79.0 Release
status-firefox79: --- → affected
tracking-firefox79: ? → +
User Story: (updated)
Start Date: 2020-08-04 End Date: 2020-09-08
Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.