Closed Bug 1552783 Opened 5 years ago Closed 5 years ago

[Experiment] Add-On: DNS over HTTPS Experiment #7 - Gather Statistics on the Effectiveness of Heuristics Fx 68.0 Release

Categories

(Shield :: Shield Study, defect)

Product:
Shield
Component:
Shield Study
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(firefox68+ fixed)

Status:
RESOLVED FIXED
Tracking Flags:
Tracking Status
firefox68 + fixed

People

(Reporter: experimenter, Assigned: wthayer)

References

Details

User Story

Experiment Type: Add-on experiment

    What are the branches of the experiment:

- Treatment active 100%:

Single-branch study
        

    What version and channel do you intend to ship to?

0.5% of Release Firefox 68.0

    Are there specific criteria for participants?


Prefs: - none

Studies: -none

Any additional filters: - no

Countries: United States of America (US) 

Locales: 

    What is your intended go live date and how long will the experiment run?

Jul 30, 2019 - Aug 06, 2019 (7 days)

    What is the main effect you are looking for and what data will you use to
    make these decisions?

What is the main effect you are looking for and what data will you use to make these decisions? What metrics are you using to measure success?

This is not a hypothesis-driven study; we intend to collect additional telemetry in release in order to inform product heuristics for DNS over HTTPS.

Do you plan on surveying users at the end of the study?

No.

    Who is the owner of the data analysis for this experiment?

tdsmith

    Will this experiment require uplift?

False

    QA Status of your code:



    Link to more information about this experiment:

https://experimenter.services.mozilla.com/experiments/dns-over-https-experiment-7-gather-statistics-on-the-effectiveness-of-heuristics/

DNS over HTTPS Experiment #7 - Gather Statistics on the Effectiveness of Heuristics

We would like to study the effectiveness of heuristics aimed at not enabling DNS over HTTPS (DoH) in situations where it will impact users. In our current implementation, when a lookup over DoH fails, we automatically fall back to the OS for DNS resolution. However, there are cases where DoH can provide different results than the user intends to receive for a particular DNS query. We are concerned about (at least) the following three categories:
DNS Filtering - users or administrators can configure their systems to use special DNS services that intentionally fail to correctly resolve categories of content such as adult content. DoH should not override the system DNS settings in this scenario.
Parental controls - administrators may set up special rules via DNS that (for example) block certain services at certain times, or after a certain period of usage. DoH should not override the system DNS settings in this scenario.
Corporate “split-horizon” DNS - organizations may set up their DNS systems to resolve normally unresolvable names such as “https://intranet”. Our OS fallback will fix this type of problem with DoH, but if an organization overrides a publicly resolvable name internally, DoH should not override the system DNS settings.

More information: https://experimenter.services.mozilla.com/experiments/dns-over-https-experiment-7-gather-statistics-on-the-effectiveness-of-heuristics/

tracking-firefox68: --- → ?
Summary: [Shield] Add-On Study: DNS over HTTPS Experiment #7 - Gather Statistics on the Effectiveness of Heuristics → [Shield] Add-On Study: DNS over HTTPS Experiment #7 - Gather Statistics on the Effectiveness of Heuristics, 68, Release
User Story: (updated)
Summary: [Shield] Add-On Study: DNS over HTTPS Experiment #7 - Gather Statistics on the Effectiveness of Heuristics, 68, Release → [Experiment] Add-On: DNS over HTTPS Experiment #7 - Gather Statistics on the Effectiveness of Heuristics Fx 68.0 Release
User Story: (updated)
Depends on: 1553700

DNS over HTTPS Experiment #7 - Gather Statistics on the Effectiveness of Heuristics
Firefox 68 Release

We have finished testing the DNS over HTTPS Experiment #7 - Gather Statistics on the Effectiveness of Heuristics experiment.

QA’s recommendation: GREEN - SHIP IT

Reasoning:

  • We haven’t found new issues during testing of the latest 1.0.4 version of the add-on.

Testing Summary:

Tested Platforms:

  • Windows 10 x64
  • macOS 10.14

Tested Firefox versions:

  • Firefox Unbranded Release 68.0

DNS over HTTPS Experiment #7 - Gather Statistics on the Effectiveness of Heuristics
Firefox 68 Release

We have finished testing the DNS over HTTPS Experiment #7 - Gather Statistics on the Effectiveness of Heuristics experiment.

QA’s recommendation: GREEN - SHIP IT

Reasoning:

  • The only issue (#3) found during testing was fixed and verified in the latest version (1.0.6) of the add-on and no new issues were uncovered in this version.

Testing Summary:

Tested Platforms:

  • Windows 10 x64
  • macOS 10.14

Tested Firefox versions:

  • Firefox Unbranded Release 68.0.1
User Story: (updated)
Start Date: 2019-07-31 End Date: 2019-08-07
Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
status-firefox68: affected → fixed
User Story: (updated)
You need to log in before you can comment on or make changes to this bug.