Enable SSL_CERT_VERIFICATION_ERRORS histogram on release

RESOLVED FIXED in Firefox 64

Status

()

defect
P1
normal
RESOLVED FIXED
6 months ago
5 months ago

People

(Reporter: RT, Assigned: johannh)

Tracking

(Blocks 1 bug)

64 Branch
Firefox 65
Points:
---
Bug Flags:
qe-verify -

Firefox Tracking Flags

(firefox64 fixed, firefox65 fixed)

Details

Attachments

(1 attachment)

(Reporter)

Description

6 months ago
When analyzing certificate error page displays we realized that over 20% of Beta users see a cert error page in a week. 
SSL_CERT_VERIFICATION_ERRORS is not enabled on release although we want to better understand SSL cert verification error correlations with AV vendors (do AVs cause higher rate of cert error page displays?), overall user retention and churn since cert error page displays can potentially drive users off of Firefox to another browser.

Histogram details: https://telemetry.mozilla.org/probe-dictionary/?search=ssl_cert_verification_errors&searchtype=in_name&channel=beta

In 64 we also plan on running a shield pref flip study analyzing churn/engagement impact that the new cert error page design have - this study is the immediate driver for this request but we also want to keep the probe after the study to trend SSL cert verification errors over time.
(Reporter)

Comment 1

6 months ago
Hi Chutten, my understanding is that I need to get data collection review for this, would you please confirm if you can help with this or help identify someone who could?

Request for data collection review form:
1 What questions will you answer with this data?
- Are users with specific AVs more likely to encounter cert errors?
- How are cert errors trending over time?
- How do SSL cert error occurrences impact churn and engagement
- How do new certificate error page design impacts churn and engagement?

2 Why does Mozilla need to answer these questions? Are there benefits for users? Do we need this information to address product or business requirements? 
- This data will help identify AV vendors with poor Firefox integration causing user experience degradation
- This data will help identify SSL cert verification errors that drive churn, helping us prioritize better certificate error UIs to mitigate this
- This data will help confirm if we can ship the updated cert error page design to release

3 What alternative methods did you consider to answer these questions? Why were they not sufficient?
We considered using Beta data although data analysts suggested Beta population is not representative when compared to release for this type of user experience issues.

4 Can current instrumentation answer these questions?
None that I'm aware of

5 List all proposed measurements and indicate the category of data collection for each measurement, using the Firefox data collection categories on the Mozilla wiki.
Make existing histogram available on release: SSL_CERT_VERIFICATION_ERRORS
Category 2 “Interaction data”

6 How long will this data be collected? Choose one of the following:
 want to permanently monitor this data.

7 What populations will you measure?
Release, all countries, all locales

8 If this data collection is default on, what is the opt-out mechanism for users?
Disable telemetry

9 Please provide a general description of how you will analyze this data.
Redash dashboard

10 Where do you intend to share the results of your analysis?
Redash dashboard
Flags: needinfo?(chutten)
(Assignee)

Updated

6 months ago
Component: Shell Integration → Security
(Assignee)

Updated

6 months ago
Assignee: nobody → jhofmann
Status: NEW → ASSIGNED
Priority: -- → P1

Comment 2

6 months ago
I am a Data Steward and can provide Data Collection Review. It works out better if the request is in an attachment (as documented in https://wiki.mozilla.org/Firefox/Data_Collection#Step_1:_Submit_Request) so it shows at the top and has review flags and stuff... but I can do it here, too.

DATA COLLECTION REVIEW RESPONSE:

    Is there or will there be documentation that describes the schema for the ultimate data set available publicly, complete and accurate?

Yes. Standard Telemetry mechanisms apply.

    Is there a control mechanism that allows the user to turn the data collection on and off?

Yes. Standard Telemetry mechanisms apply.

    If the request is for permanent data collection, is there someone who will monitor the data over time?

:rtestard has volunteered to permanently monitor this collection.

    Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Category 1, Technical (Cert verification errors seem more technical than interaction to me)

    Is the data collection request for default-on or default-off?

Default on for all channels.

    Does the instrumentation include the addition of any new identifiers (whether anonymous or otherwise; e.g., username, random IDs, etc. See the appendix for more details)?

No.

    Is the data collection covered by the existing Firefox privacy notice? 

Yes.

    Does there need to be a check-in in the future to determine whether to renew the data?

No. This collection is permanent.

---
Result: datareview+
Flags: needinfo?(chutten)
(Reporter)

Updated

6 months ago
Blocks: 1501004

Comment 4

5 months ago
Pushed by jhofmann@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/e5cd93207b3a
Enable SSL_CERT_VERIFICATION_ERRORS histogram on release. r=janerik
(Assignee)

Updated

5 months ago
Flags: needinfo?(jhofmann)
Attachment #9022546 - Attachment description: Bug 1503572 - Enable SSL_CERT_VERIFICATION_ERRORS histogram on release. r=chutten → Bug 1503572 - Enable SSL_CERT_VERIFICATION_ERRORS histogram on release. r=janerik

Comment 6

5 months ago
Pushed by jhofmann@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/245c309e5a26
Enable SSL_CERT_VERIFICATION_ERRORS histogram on release. r=janerik

Comment 7

5 months ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/245c309e5a26
Status: ASSIGNED → RESOLVED
Last Resolved: 5 months ago
Resolution: --- → FIXED
Target Milestone: --- → Firefox 65
(Reporter)

Comment 8

5 months ago
This is marked as FIXED in 65 although the ask was to get it done on 64 to support a shield study.
Do we need a separate bug for uplift to 64 or will this be done on this bug?
(Assignee)

Comment 9

5 months ago
Comment on attachment 9022546 [details]
Bug 1503572 - Enable SSL_CERT_VERIFICATION_ERRORS histogram on release. r=janerik

[Beta/Release Uplift Approval Request]

Feature/Bug causing the regression: None

User impact if declined: No user impact either way, we want to be able to measure certificate errors in release 64

Is this code covered by automated tests?: Unknown

Has the fix been verified in Nightly?: No

Needs manual test from QE?: No

If yes, steps to reproduce: 

List of other uplifts needed: None

Risk to taking this patch: Low

Why is the change risky/not risky? (and alternatives if risky): Just updating the opt-out status of a telemetry probe.

String changes made/needed: None
Attachment #9022546 - Flags: approval-mozilla-beta?
Comment on attachment 9022546 [details]
Bug 1503572 - Enable SSL_CERT_VERIFICATION_ERRORS histogram on release. r=janerik

move a telemetry probe to opt-out; approved for 64.0b8
Attachment #9022546 - Flags: approval-mozilla-beta? → approval-mozilla-beta+
You need to log in before you can comment on or make changes to this bug.