1647305 - [Experiment] Pref-Flip Experiment: Comcast Steering Experiment Fx 78.0 to 80.0 Release

Status: RESOLVED FIXED

(Shield :: Shield Study, task, P3)

Description

[Experimenter Bot]

Comcast Steering Experiment

We are testing the use of Comcast's DNS over HTTPS service.

Experimenter is the source of truth for details and delivery. Changes to Bugzilla are not reflected in Experimenter and will not change delivery configuration.

Data Science Issue: https://jira.mozilla.com/browse/DS-751
More information: https://experimenter.services.mozilla.com/experiments/comcast-steering-experiment/

Comment 1

[Carmen Fat [:cfat] - Ecosystem QA]

DNS Over HTTPS Comcast Experiment
Firefox Release 78-80

We have finished testing the [PI-625] - DNS Over HTTPS Comcast Experiment.

Quality status: YELLOW - SHIP IT CONDITIONALLY

Why is this experiment yellow?

• All the filed issues were either fixed and verified or addressed. However, we are concerned about the following ones:
  1. Bug 1646387 [PI-625] "disable_doh" telemetry events are wrongly displayed when disabling and re-enabling the network adapter - major
     -> This issue is only reproducible on Windows, however it results in invalid telemetry events that might affect the outcome of the collected telemetry data.
     -> From the QA’s point of view we are unsure how this bug will affect the population of the experiment.
  2. Bug 1646383 [PI-625] Multiple "netChange" telemetry events are sent when re-enabling the network - major
     -> This issue is not specific to the experiment as it is also reproducible with the DoH feature.
  3. Bug 1646968 [PI-625] The "steeredProvider: comcast" extra is displayed in the "policy_without_doh" events when DoH is disabled by policies not related to DoH - normal
     -> The behavior of the reported issue was addressed as expected. However, uplifting patches this late in the cycle without performing regression testing, might create regressions in the behavior of the experiment.

What needs to be done?
1. Bug 1646387
[Engineering] [Data Science] - Assess how damaging the issue is and resolve it.
[QA] - Verify the issue after it is fixed.
2. Bug 1646383
[Engineering] [Data Science] - This report should be addressed by the Engineering and Data Science teams since they have a better knowledge on how extra telemetry events might impact the results of the collected data.
3. Bug 1646968
[Engineering] - Since removing the telemetry event mentioned in comment 3 is not an urgent task and does not impact the user in any way, we recommend pushing this change for a further release.

Testing Summary:

• Test Runs: TestRail (https://testrail.stage.mozaws.net/index.php?/plans/view/30574)

Tested Platforms:

• Windows 10 x64
• Ubuntu 16.04 x64
• macOS 10.15

Tested browser versions:

• Firefox Beta 78.0b9 (Build ID: 20200619002543)
• Firefox Beta 78.0b7 (for the browser update scenario)

Comment 2

[Tony Cinotto [:tcinotto] (UTC-8)]

[Tracking Requested - why for this release]: Experiment will follow after results of Comcast 1% US rollout.

Comment 3

[Experimenter Bot]

Start Date: 2020-08-11 End Date: 2020-09-08