1533397 - [Shield] Pref-Flip Study: Retention/Engagement impact of enabling the Enterprise roots feature in the presence of an AV, 66, release

Status: RESOLVED FIXED

(Shield :: Shield Study, enhancement)

Description

[Experimenter Bot]

Retention/Engagement impact of enabling the Enterprise roots feature in the presence of an AV

Several AVs recently broke HTTPs with their HTTPs scanning features that require their certs to be added to our cert store (Avast on bug 1523701, BitDefender on bug 1508624, Kaspersky on bug 1449115). The security team confirmed that having the preference security.enterprise_roots.enabled set to true would have fixed all of these issues without known regressions and we want to validate that in the presence of an AV, enabling this preference would have a positive impact on retention and engagement (we cannot detect a change in certificate error page displays through telemetry since telemetry is sent over https, that breaks in these instances....).
Description of the impacted population:

• Win 10 and Win 8 release users (the API allowing detection of an AV registered with the system was only available since Win8)
• An AV is registered with the system and IS NOT Windows Defender. This information is available on telemetry under "sec.antivirus"

Test cohort: Win 8+ users on release 66 with an AV registered that is not Windows Defender and don't have security.enterprise_roots.enabled set to true. Set security.enterprise_roots.enabled to true.

Control cohort: Win 8+ users on release 66 with an AV registered that is not Windows Defender and don't have security.enterprise_roots.enabled set to true.

More information: https://experimenter.services.mozilla.com/experiments/retentionengagement-impact-of-enabling-the-enterprise-roots-feature-in-the-presence-of-an-av/

Comment 1

[Romain Testard [:RT]]

[Tracking Requested - why for this release]:

Comment 2

[Ben Bucksch (:BenB)]

This is highly dangerous.

It makes our management of root CAs much more difficult. See e.g. Symantec case, the ongoing DarkMatter discussion of the inclusion criteria, etc. The entire discussion is about whether or not to include certain CAs. If you wholesale include all CAs from Windows, you defer the discussion to Microsoft and make the entire Root CA policy that we have completely moot.

https://www.mozilla.org/en-US/about/governance/policies/security-group/certs/
The way I see it, this effort is in direct contradiction to our established root CA policy and rules.

Comment 3

[Wayne Thayer]

(In reply to Ben Bucksch (:BenB) from comment #2)

This is highly dangerous.

It makes our management of root CAs much more difficult. See e.g. Symantec case, the ongoing DarkMatter discussion of the inclusion criteria, etc. The entire discussion is about whether or not to include certain CAs. If you wholesale include all CAs from Windows, you defer the discussion to Microsoft and make the entire Root CA policy that we have completely moot.

https://www.mozilla.org/en-US/about/governance/policies/security-group/certs/
The way I see it, this effort is in direct contradiction to our established root CA policy and rules.

This is a misunderstanding of what the 'enterprise roots' pref does. It does not import the entire OS root store (if it did I would agree with these statements). It detects roots that have been added to the default root store by the user or an admin and imports ONLY those added roots to the NSS root store.

Comment 4

[Experimenter Bot]

    Experiment Type: Pref Flip Study

    What is the preference we will be changing

security.enterprise_roots.enabled

    What are the branches of the study and what values should
    each branch be set to?

- Treatment Enterprise root set to true 50%:

Value: true

Win 8+ users on release 66 with an AV registered that is not Windows Defender who don't have security.enterprise_roots.enabled set to true. Set security.enterprise_roots.enabled to true.
        
- Treatment Enterprise root set to false 50%:

Value: false

Win 8+ users on release 66 with an AV registered that is not Windows Defender who don't have security.enterprise_roots.enabled set to true.
        

    What version and channel do you intend to ship to?

0.67% of Release Firefox 66.0

    Are there specific criteria for participants?

Locales: All

Geographic regions: All

Prefs: Include only users with security.enterprise_roots.enabled set to False (which is the default)

Studies:

Any additional filters:

- Win 8+
- sec.antivirus reports an AV used that is not Windows Defender

    What is your intended go live date and how long will the study run?

Mar 25, 2019 - Apr 08, 2019 (14 days)

    What is the main effect you are looking for and what data will you use to
    make these decisions?

What is the main effect you are looking for and what data will
you use to make these decisions? What metrics are you using to measure success

We are hoping not to see regressions in retention and markers of engagement.

# Power analysis
A study population of 130,000 per branch (260,000 total) should be sufficient to detect a

- 1% decrease in retention
- 3% decrease in total active hours
- 3% decrease in total subsession hours
- 4% decrease in total URIs

This corresponds to 0.67% of the population with the required OS and antivirus filters.

Do you plan on surveying users at the end of the study? No.

    Who is the owner of the data analysis for this study?

Chris Beard

    Will this experiment require uplift?

False

    QA Status of your code:



    Link to more information about this study:

https://experimenter.services.mozilla.com/experiments/retentionengagement-impact-of-enabling-the-enterprise-roots-feature-in-the-presence-of-an-av/

Comment 5

[Laurențiu Nicola]

It detects roots that have been added to the default root store by the user or an admin and imports ONLY those added roots to the NSS root store.

Or by an employer, antivirus software or malware.

What is the main effect you are looking for and what data will you use to make these decisions?
What metrics are you using to measure success

We are hoping not to see regressions in retention and markers of engagement.

"Not seeing any regressions" doesn't exactly answer those questions.

What exactly are you trying to achieve? Reduce the number of TLS errors that the users will encounter? Sure, I can understand that, but you're doing it at the expense of the users' security.

Comment 6

[Wayne Thayer]

(In reply to Laurentiu Nicola from comment #5)

It detects roots that have been added to the default root store by the user or an admin and imports ONLY those added roots to the NSS root store.

Or by an employer, antivirus software or malware.

What is the main effect you are looking for and what data will you use to make these decisions?
What metrics are you using to measure success

We are hoping not to see regressions in retention and markers of engagement.

"Not seeing any regressions" doesn't exactly answer those questions.

What exactly are you trying to achieve? Reduce the number of TLS errors that the users will encounter? Sure, I can understand that, but you're doing it at the expense of the users' security.

We know that turning the pref on by default will reduce the number of TLS errors. We believe it will in turn increase user engagement, but proving that in a study would be difficult, so our aim is to prove that enabling the pref by default will not harm engagement - hence "Not seeing any regressions".

We assume that an adversary that can install a root in the OS root store can also install a root directly into the Firefox root store. Under this threat model, enabling the enterprise roots pref does not compromise the user's security.

Comment 7

[Liz Henry (:lizzard Please n-i to RyanVM, jcristau, or pascal)]

Untracking since we have this information easily findable now in Experimenter.

Comment 8

[Liz Henry (:lizzard Please n-i to RyanVM, jcristau, or pascal)]

Note: the experiment went live April 1 and will run till April 15.