Closed Bug 1669364 Opened 4 years ago Closed 4 years ago

[Experiment] Pref-Flip Experiment: Verify changing location service providers still gives accurate results. V2 Fx 82.0 to 83.0 Release

Categories

(Shield :: Shield Study, task, P3)

Product:
Shield
Component:
Shield Study
Type:
task

Tracking

(firefox81 ?, firefox82 ?)

Status:
RESOLVED FIXED
Tracking Flags:
Tracking Status
firefox81 --- ?
firefox82 --- ?

People

(Reporter: experimenter, Assigned: daleharvey)

References

(
URL
)

Details

User Story

Delivery Type: Pref Flip Experiment

    What is the preference we will be changing

geo.provider.network.compare.url

    What are the branches of the experiment and what values should
    each branch be set to?

- Control Control 50%:

Value: None

Control branch
        
- Treatment Treatment 50%:

Value: None

Compare GLS againt MLS
        

    What version and channel do you intend to ship to?

1% of Release Firefox 82.0 to 83.0

    Are there specific criteria for participants?

Prefs:

Experiments:

Any additional filters:
Countries: all

Locales: all

    What is your intended go live date and how long will the experiment run?

Oct 20, 2020 - Nov 24, 2020 (35 days)

    What is the main effect you are looking for and what data will you use to
    make these decisions?

What is the main effect you are looking for and what data will
you use to make these decisions? What metrics are you using to measure success

The pref flip will enable telemetry to be collected about the accuracy of results which will be analysed alongside server log data


Do you plan on surveying users at the end of the delivery? Yes/No.
No

Strategy and Insights can help create surveys if needed

    Who is the owner of the data analysis for this experiment?

tbrooks@mozilla.com

    Will this experiment require uplift?

False

    QA Status of your code:

We discussed in lightning advising of orig experiment(V1)- Krupa, bkal, mikedeboer, mythmon, shell - and determined that the benefit / real testability is very low for MLS (had previous experience with this feature).  Search team will be checking telemetry after this goes live to look for indications of issues.  Worst case is relaunch - as there is feature fall-back so not customer breaking.

QA Status: GREEN. For more information see https://experimenter.services.mozilla.com/experiments/verify-changing-location-service-providers-still-gives-accurate-results-v2/#comment1180.

    Link to more information about this experiment:

https://experimenter.services.mozilla.com/experiments/verify-changing-location-service-providers-still-gives-accurate-results-v2/

Verify changing location service providers still gives accurate results. V2

Verify changing location service providers still gives accurate results.

Experimenter is the source of truth for details and delivery. Changes to Bugzilla are not reflected in Experimenter and will not change delivery configuration.

Data Science Issue: https://jira.mozilla.com/browse/DS-269
More information: https://experimenter.services.mozilla.com/experiments/verify-changing-location-service-providers-still-gives-accurate-results-v2/

verify-location-service-providers-change V2
Firefox Release 82

We have finished testing the [PI-824] verify-location-service-providers-change V2 experiment.

Quality status: GREEN - SHIP IT

Why is this experiment green?

  • We haven’t found any issues during testing.

Testing Summary:

Tested Platforms:

  • Windows 10
  • macOS 10.15
  • LinuxMX 4.19/Ubuntu 20.04

Tested Firefox versions:

  • Firefox RC 82.0 en-US (BuildID 20201012131351)
  • Firefox Nightly 83.0a1 (for the update test only)

Regards,
Ciprian Muresan
Ecosystem QA Team

status-firefox81: --- → ?
status-firefox82: --- → ?
User Story: (updated)
Summary: [Experiment]: Pref-Flip Experiment: Verify changing location service providers still gives accurate results. V2 → [Experiment] Pref-Flip Experiment: Verify changing location service providers still gives accurate results. V2 Fx 82.0 to 83.0 Release
Start Date: 2020-10-20 End Date: 2020-11-24
Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.