Closed Bug 1660819 Opened 4 years ago Closed 4 years ago

[Experiment] Pref-Flip Experiment: Amazon Search Replacement study - DE Fx 81.0 to 82.0 Beta

Categories

(Shield :: Shield Study, task, P3)

Product:
Shield
Component:
Shield Study
Type:
task

Tracking

(firefox81+ fixed, firefox82+ fixed)

Status:
RESOLVED FIXED
Tracking Flags:
Tracking Status
firefox81 + fixed
firefox82 + fixed

People

(Reporter: experimenter, Assigned: mikedeboer)

References

(
URL
)

Details

(Whiteboard: [no-nag])

User Story

Delivery Type: Pref Flip Experiment

    What is the preference we will be changing

browser.search.experiment

    What are the branches of the experiment and what values should
    each branch be set to?

- Control Amazon Partner URL 50%:

Value: unused

Existing Amazon URL
        
- Treatment Ad Marketplace URL 50%:

Value: amazon-de-adm

Changes search url to use Ad Marketplace codes, and adds a background attribution request
        

    What version and channel do you intend to ship to?

10% of Beta Firefox 81.0 to 82.0

    Are there specific criteria for participants?

None
Countries: Germany (DE) 

Locales: all

    What is your intended go live date and how long will the experiment run?

Sep 08, 2020 - Oct 06, 2020 (28 days)

    What is the main effect you are looking for and what data will you use to
    make these decisions?

We will measure the search attribution via the existing telemetry methods of search telemetry reporting. We will also ensure that the reported attributions via the separate proxy URL are in line with the telemetry reported searches on the Amazon server.

    Who is the owner of the data analysis for this experiment?

None

    Will this experiment require uplift?

False

    QA Status of your code:

QA Status: GREEN. For more information, see: https://experimenter.services.mozilla.com/experiments/amazon-search-replacement-study-de/#comment1132

    Link to more information about this experiment:

https://experimenter.services.mozilla.com/experiments/amazon-search-replacement-study-de/

Amazon Search Replacement study - DE

This experiment covers two questions:

  1. Does the Amazon experience change if we change the search engine URL included in the Amazon engine?
  2. Does the search attribution flow break if we change Amazon search engine URL?

Experimenter is the source of truth for details and delivery. Changes to Bugzilla are not reflected in Experimenter and will not change delivery configuration.

Data Science Issue: https://jira.mozilla.com/browse/DO-317
More information: https://experimenter.services.mozilla.com/experiments/amazon-search-replacement-study-de/

[Tracking Requested - why for this release]:

status-firefox81: --- → affected
status-firefox82: --- → affected
tracking-firefox81: --- → ?
tracking-firefox82: --- → ?

Amazon Search Engine Replacement
Firefox Beta 81 DE
Firefox Beta 81 en-US
Firefox Beta 81 en-GB

We have finished testing the [PI-784] Amazon Search Engine Replacement experiments.

Quality status: GREEN - SHIP IT

Why are these experiments green?

  • During the first round of testing we’ve encountered 1 major issue. Given the severity of the issue, we’ve decided together with the Search team to stop testing until the issue is fixed.
  • After the issue was fixed, we’ve started another round of testing and we have not encountered any issues.

Testing Summary:

Tested Platforms:

  • Windows 10 x64
  • Windows 7 x64
  • Ubuntu Linux 20.04
  • Linux Mint 20
  • macOS 10.15
  • macOS 10.14

Tested browser versions:

  • Firefox Beta 81.0b6 en-US (Build ID: 20200903205131)
  • Firefox Beta 81.0b6 DE (Build ID: 20200903205131)
  • Firefox Beta 81.0b6 en-GB (Build ID: 20200903205131)
  • Firefox Nightly 82.0a1 en-US (for the update scenario)
  • Firefox Nightly 82.0a1 DE (for the update scenario)
  • Firefox Nightly 82.0a1 en-GB (for the update scenario)

Regards,
Robert Martin
Ecosystem QA

Whiteboard: [no-nag]
User Story: (updated)
Summary: [Experiment]: Pref-Flip Experiment: Amazon Search Replacement study - DE → [Experiment] Pref-Flip Experiment: Amazon Search Replacement study - DE Fx 81.0 to 82.0 Beta
Start Date: 2020-09-10 End Date: 2020-10-08
Assignee: nobody → mdeboer
User Story: (updated)
Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.