Closed Bug 1646755 Opened 6 years ago Closed 6 years ago

[Experiment] Pref-Flip Experiment: Amazon Top Sites tile study- US Fx 78.0 to 79.0 Release

Categories

(Shield :: Shield Study, task, P3)

Product:
Shield
Component:
Shield Study
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: experimenter, Assigned: mikedeboer)

References

(
URL
)

Details

User Story

Delivery Type: Pref Flip Experiment

    What is the preference we will be changing

amazon-tile-pref-name

    What are the branches of the experiment and what values should
    each branch be set to?

- Control Control 50%:

Value: None

Control Branch
        
- Treatment Treatment-admarketplace 50%:

Value: None

Changes Amazon tile to a URL and sends a background request to ad market place
        

    What version and channel do you intend to ship to?

10% of Release Firefox 78.0 to 79.0

    Are there specific criteria for participants?

None
Countries: United States of America (US) 

Locales: all

    What is your intended go live date and how long will the experiment run?

Jul 07, 2020 - Aug 18, 2020 (42 days)

    What is the main effect you are looking for and what data will you use to
    make these decisions?

What is the main effect you are looking for and what data will  you use to make these decisions? What metrics are you using to measure success

Our main data source is coming from Amazon (and 3rd party's AdmarketPlace and Sovrn) campaign data, showing us how many searches occurred in both branches. We don't have in-content telemetry for Amazon at the moment, so we need to rely in this data for now.

Also, we plan to implement and look at:
 * Amazon tile impressions / clicks (on newtab and in search suggestions)
 * Backend error rates through this mechanism, i.e. how often does attribution fail when routing users directly to Amazon?

Survey = No.
No, we do not plan on surveying users at the end of the delivery?


bmiroglio@mozilla.com July 6, 2020, 8:13 p.m.

Confirming teons comment above. After discussing withe broader team, we will keep this at 10%, which will give us ~1100 daily searches in US (https://sql.telemetry.mozilla.org/queries/72538/source). This isn't ideal sizing, and we will only have the capability to identify large changes, however 10% is already uncommonly-large experiment sample for release.

    Who is the owner of the data analysis for this experiment?

bmiroglio@mozilla.com

    Will this experiment require uplift?

False

    QA Status of your code:

This experiment was signed-off as Yellow. For further information see below:

_Modify Amazon Search Shortcut to a direct link to Amazon.com_

_Firefox Beta 78.0bx_

We have finished testing the [PI-606] Modify Amazon Search Shortcut to a direct link to Amazon.com experiment on en-US build. 
However, in order to test the Control branch, we had to bypass Bug 1645420 and Normandy Issue 2236 (limitation), by not setting any preferences in the “Control” branch section from the Normandy stage recipe.

**Quality status:** YELLOW - SHIP IT CONDITIONALLY

**Why is this experiment yellow?:**

- During testing we have encountered 2 known issues: 
- Bug 1597252 is normal and it is not affecting the user experience in a harmful way. 
- Bug 1644449 is causing some default search engines to be wrongly removed when the browser points to the Normandy Stage server, but we are unsure if this will happen in production too as we cannot create a testing recipe on production.
- There is a concern that due to Issue 2236 limitation, the only remaining viable way to launch this study might affect the gathered/read telemetry data (no Control branch prefs). 
- Given the short amount of time for testing, we only managed to start and finish a full test run on Windows 10 x64, and only managed to perform exploratory testing using Firefox Beta 78.0b7 on macOS 10.15 and Ubuntu Linux 18.04.

**What needs to be done?**

- [Engineering] Investigate Bug 1644449 because we can reliably reproduce it on the “stage” environment even with VPN and other Normandy recipes, which could possibly point towards a Normandy issue. That being said, we cannot be certain that the users will not be affected in the “production” environment.
- [QA] If possible, provide more testing time in order to be confident in the test results.

**Testing Summary:**

- Testing results: TestRail (Windows 10)
- Performed exploratory testing on macOS and Ubuntu Linux based on the scenarios from Test Cases.

**Tested Platforms:**

- Windows 10 x64
- macOS 10.15
- Ubuntu 18.04 x64

**Tested Firefox versions:**

- Firefox Beta 78.0b7 en-US

Regards,
Robert Martin
Ecosystem QA Team

    Link to more information about this experiment:

https://experimenter.services.mozilla.com/experiments/amazon-top-sites-tile-study-v2/

Amazon Top Sites tile study V2

Does the search attribution flow break if we change the target the Amazon tile from prefilling the URLBar with '@amazon ' to a URL?

Experimenter is the source of truth for details and delivery. Changes to Bugzilla are not reflected in Experimenter and will not change delivery configuration.

Data Science Issue: https://jira.mozilla.com/browse/DS-673
More information: https://experimenter.services.mozilla.com/experiments/amazon-top-sites-tile-study-v2/

User Story: (updated)
Summary: [Experiment]: Pref-Flip Experiment: Amazon Top Sites tile study V2 → [Experiment] Pref-Flip Experiment: Amazon Top Sites tile study- US Fx 78.0 to 79.0 Release
User Story: (updated)
Start Date: 2020-07-14 End Date: 2020-08-25
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.