1503572 - Enable SSL_CERT_VERIFICATION_ERRORS histogram on release

Status: RESOLVED FIXED

(Firefox :: Security, defect, P1)

Description

[Romain Testard [:RT]]

When analyzing certificate error page displays we realized that over 20% of Beta users see a cert error page in a week. 
SSL_CERT_VERIFICATION_ERRORS is not enabled on release although we want to better understand SSL cert verification error correlations with AV vendors (do AVs cause higher rate of cert error page displays?), overall user retention and churn since cert error page displays can potentially drive users off of Firefox to another browser.

Histogram details: https://telemetry.mozilla.org/probe-dictionary/?search=ssl_cert_verification_errors&searchtype=in_name&channel=beta

In 64 we also plan on running a shield pref flip study analyzing churn/engagement impact that the new cert error page design have - this study is the immediate driver for this request but we also want to keep the probe after the study to trend SSL cert verification errors over time.

Comment 1

[Romain Testard [:RT]]

Hi Chutten, my understanding is that I need to get data collection review for this, would you please confirm if you can help with this or help identify someone who could?

Request for data collection review form:
1 What questions will you answer with this data?
- Are users with specific AVs more likely to encounter cert errors?
- How are cert errors trending over time?
- How do SSL cert error occurrences impact churn and engagement
- How do new certificate error page design impacts churn and engagement?

2 Why does Mozilla need to answer these questions? Are there benefits for users? Do we need this information to address product or business requirements? 
- This data will help identify AV vendors with poor Firefox integration causing user experience degradation
- This data will help identify SSL cert verification errors that drive churn, helping us prioritize better certificate error UIs to mitigate this
- This data will help confirm if we can ship the updated cert error page design to release

3 What alternative methods did you consider to answer these questions? Why were they not sufficient?
We considered using Beta data although data analysts suggested Beta population is not representative when compared to release for this type of user experience issues.

4 Can current instrumentation answer these questions?
None that I'm aware of

5 List all proposed measurements and indicate the category of data collection for each measurement, using the Firefox data collection categories on the Mozilla wiki.
Make existing histogram available on release: SSL_CERT_VERIFICATION_ERRORS
Category 2 “Interaction data”

6 How long will this data be collected? Choose one of the following:
 want to permanently monitor this data.

7 What populations will you measure?
Release, all countries, all locales

8 If this data collection is default on, what is the opt-out mechanism for users?
Disable telemetry

9 Please provide a general description of how you will analyze this data.
Redash dashboard

10 Where do you intend to share the results of your analysis?
Redash dashboard

Comment 2

[Chris H-C :chutten]

I am a Data Steward and can provide Data Collection Review. It works out better if the request is in an attachment (as documented in https://wiki.mozilla.org/Firefox/Data_Collection#Step_1:_Submit_Request) so it shows at the top and has review flags and stuff... but I can do it here, too.

DATA COLLECTION REVIEW RESPONSE:

    Is there or will there be documentation that describes the schema for the ultimate data set available publicly, complete and accurate?

Yes. Standard Telemetry mechanisms apply.

    Is there a control mechanism that allows the user to turn the data collection on and off?

Yes. Standard Telemetry mechanisms apply.

    If the request is for permanent data collection, is there someone who will monitor the data over time?

:rtestard has volunteered to permanently monitor this collection.

    Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Category 1, Technical (Cert verification errors seem more technical than interaction to me)

    Is the data collection request for default-on or default-off?

Default on for all channels.

    Does the instrumentation include the addition of any new identifiers (whether anonymous or otherwise; e.g., username, random IDs, etc. See the appendix for more details)?

No.

    Is the data collection covered by the existing Firefox privacy notice? 

Yes.

    Does there need to be a check-in in the future to determine whether to renew the data?

No. This collection is permanent.

---
Result: datareview+

Comment 3

[Johann Hofmann [:johannh]]

Attachment: Bug 1503572 - Enable SSL_CERT_VERIFICATION_ERRORS histogram on release. r=janerik

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/e5cd93207b3a
Enable SSL_CERT_VERIFICATION_ERRORS histogram on release. r=janerik

Comment 5

[Cristian Brindusan [:cbrindusan]]

Backed out changeset e5cd93207b3a (Bug 1503572) for build bustages on TelemetryHistogramEnums.h

Backout: https://hg.mozilla.org/integration/autoland/rev/1e0f0a17143f07a323d4f166327ae2c9d164d393

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&resultStatus=pending%2Crunning%2Csuccess%2Ctestfailed%2Cbusted%2Cexception&revision=e5cd93207b3afab072daed9c7aeb216123b92797&selectedJob=209886314

Failure log: https://treeherder.mozilla.org/logviewer.html#?job_id=209886314&repo=autoland&lineNumber=3530

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/245c309e5a26
Enable SSL_CERT_VERIFICATION_ERRORS histogram on release. r=janerik

Comment 7

[Natalia Csoregi [:nataliaCs]]

https://hg.mozilla.org/mozilla-central/rev/245c309e5a26

Comment 8

[Romain Testard [:RT]]

This is marked as FIXED in 65 although the ask was to get it done on 64 to support a shield study.
Do we need a separate bug for uplift to 64 or will this be done on this bug?

Comment 9

[Johann Hofmann [:johannh]]

Comment on attachment 9022546 [details]
Bug 1503572 - Enable SSL_CERT_VERIFICATION_ERRORS histogram on release. r=janerik

[Beta/Release Uplift Approval Request]

Feature/Bug causing the regression: None

User impact if declined: No user impact either way, we want to be able to measure certificate errors in release 64

Is this code covered by automated tests?: Unknown

Has the fix been verified in Nightly?: No

Needs manual test from QE?: No

If yes, steps to reproduce: 

List of other uplifts needed: None

Risk to taking this patch: Low

Why is the change risky/not risky? (and alternatives if risky): Just updating the opt-out status of a telemetry probe.

String changes made/needed: None

Comment 10

[Julien Cristau [:jcristau] (back April 22)]

Comment on attachment 9022546 [details]
Bug 1503572 - Enable SSL_CERT_VERIFICATION_ERRORS histogram on release. r=janerik

move a telemetry probe to opt-out; approved for 64.0b8

Comment 11

[Eliza Balazs [:ebalazs_]]

https://hg.mozilla.org/releases/mozilla-beta/rev/aaa628592fed