1613481 - [Experiment] Staged Rollout: DoH NIGHTLY Rollout to All US Desktop Users Fx 74.0 to 80.0 Nightly

Status: RESOLVED FIXED

(Shield :: Shield Study, task, P3)

Description

[Experimenter Bot]

DoH NIGHTLY Rollout to All US Desktop Users

We plan to leverage Normandy to enable the DoH rollout add-on (in-tree) for 100% of NIGHTLY users in the US (by geoIP).

Note: (*) Enable means setting the preference doh-rollout.enabled to true, which will be read by the add-on to run heuristics to determine if DoH should be used

Experimenter is the source of truth for details and delivery. Changes to Bugzilla are not reflected in Experimenter and will not change delivery configuration.

More information: https://experimenter.services.mozilla.com/experiments/doh-nightly-rollout-to-all-us-desktop-users/

Comment 1

[Carmen Fat [:cfat] - Ecosystem QA]

DoH NIGHTLY and BETA Rollout to All US Desktop Users
Firefox NIGHTLY 74 and BETA 73

We tested the DoH NIGHTLY and BETA Rollout to All US Desktop Users rollout.

Quality status: GREEN - SHIP IT

Why is this feature green?

• The blocker issue related to Rollback was fixed and verified.
• We had 2 additional days to test the remaining scenarios and properly investigate the potential issues.
• The functionality part of the add-on seems to work as expected while performing regression testing.
• However, the only concern from the QA point of view is that there are still 2 issues open, both targeting the inconsistent behavior of telemetry heuristics. We already discussed these two issues with the engineering team and reached the conclusion that they don’t have a big impact on the rollout. In addition, the engineering team already intends to fix this inconsistent behavior in the future (see the bug report here)

Testing Summary:

• We verified all the scenarios described in the previous sign-off email.
• We verified that setting a non DoH enterprise policy doesn’t enable DNS over HTTPS.
• We verified that setting a DoH disabled enterprise policy doesn’t enable DNS over HTTPS.
• We verified that DoH rollout remains enabled and runs heuristics after updating to a later version of Firefox.
• We verified that DoH rollout remains enabled and runs heuristics after upgrading to a newer Firefox version.
• We verified that the DoH door hanger is correctly displayed on an old profile.
• We verified that the DoH door hanger is correctly displayed in a new profile after setting the “doh-rollout.profileCreationThreshold” pref to a future date through Normandy testing recipe.
• We verified that the “OK, got it” and “Disable” buttons from the door hanger correctly enable or disable the DoH feature.
• We verified that the “Learn More” link from the door hanger redirects to the correct page.
• We verified that on a macOS machine that has Parental control enabled, DoH feature is disabled.

Tested Platforms:

• Windows 10 x64
• Mac 10.15.2
• Mac 10.14.6

Tested Firefox versions:

• Firefox Nightly 74.0a1 (2020-02-10) en-US
• Firefox RC 73.0 (build 3) en-US

Comment 2

[Carmen Fat [:cfat] - Ecosystem QA]

DoH NIGHTLY and BETA Rollout to All US Desktop Users
Firefox NIGHTLY 75 and BETA 74

We tested the DoH NIGHTLY and BETA Rollout to All US Desktop Users rollout on Linux.

Quality status: GREEN - SHIP IT

Why is this feature green?

• We haven't found any new issues on Linux OS.

Testing Summary:

• The same scenarios mentioned in the 2 previous sign off emails were tested on Linux OS as well.

Tested Platforms:

• Linux Debian 9 x64
• Linux Ubuntu 18.04 x64
• Linux Ubuntu 16.04 x64

Tested Firefox versions:

• Firefox Nightly 75.0a1 (Build ID 20200210213347) en-US
• Firefox Beta 74.0b1 (Build ID 20200210140608 ) en-US

Comment 3

[Experimenter Bot]

Start Date: 2020-02-11 End Date: 2020-12-07

Comment 4

[Troy Janda]

This is still active in the latest Nightly

![Screen] (https://i.postimg.cc/mggNLnbL/error.png)