Closed Bug 1526154 Opened 10 months ago Closed 8 months ago

DigiCert: Missed Underscore Certificate Revocations


(NSS :: CA Certificate Compliance, task)

Not set


(Not tracked)



(Reporter: brenda.bernal, Assigned: brenda.bernal)


(Whiteboard: [ca-compliance])

1.How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in, a Bugzilla bug, or internal self-audit), and the time and date.

A third party report was sent to members of our team notifying DigiCert of existing underscore certs that were not revoked or included in our incident reports.

2.A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

February 1, 2019 – A third party reported that they found additional certificates that have not been revoked, nor included in our incident reports to Mozilla. Digicert acknowledged receipt of this notification on the same day.

Febryary 4, 2019- – Concluded that a subset of the certificates identified were pre-certificates that do not have a valid cert associated.

February 5, 2019 – Confirmed that the remaining certs were either valid certificates or were already included in our incident reports for underscore certificates. We did not intend to revoke the latter post the January 14, 2019 deadline, as stated in those incident reports. Notifications to impacted customers also commenced.

February 6, 2019, -- DigiCert ran another data report to ensure we did not miss any other certificates that were not revoked. We also revoked remaining valid certificates with underscores, as reported.

February 7, 2019 – DigiCert is revoking pre-certificates found with no valid certificates (for housekeeping purposes).

3.Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

October 1, 2018 – DigiCert ceased issuance of underscore certificates in light of the CAB/F discussion on this topic.

January 14, 2019 – DigiCert revoked valid underscore certificates given the SC12 ballot deadline, with the exception of 7 customer accounts that requested an extension. We are systematically revoking those remaining underscore certificates as published in those individual incident reports.

4.A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

There were 17 valid certificates with underscores that did not get revoked on January 14, 2019.
The first certificate was issued on February 9, 2016 and the last certificate was issued on September 11, 2018

5.The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

Additionally, we are revoking pre-certificates for housekeeping purposes; these certs are not associated with valid certificates. We will post the links as soon as we complete the revocations.

6.Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

We had a flaw in our script logic to extract the certificate data for the mass revocation we were actioning for the January 14, 2019 deadline. We used the order number as the primary key vs. the unique serial number, and the status of either the original certificate or re-issued certificate to identify valid certificates we needed to revoke. The script logic ended up missing certificates as part of the order. There was one instance for an account where we missed the certificate due to a data glitch in the reporting.

7.List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

DigiCert will make substantial improvements in our data reporting to ensure the thoroughness and accuracy when we are required to revoke certificates related to ballots and related incidents. As part of this improvement, we are ensuring all of our certificate data from our various systems are included in the data lake we have been building and refining.

Additionally, we plan to conduct a more extensive QA of revocation scripts to ensure that criteria and logic will not inadvertently exclude problem certificates that need to be included in the scope of an investigation and/or revocation.

Assignee: wthayer → brenda.bernal
Ever confirmed: true
QA Contact: kwilson → wthayer
Whiteboard: [ca-compliance]

Hi Wayne, any update or other information you require for this incident report?

Hi Brenda, thanks for checking in on this. I do think there is a common thread in this issue and a few of other CAs related to the difficulty of getting a complete list of affected certificates. You suggested that "ensuring all of our certificate data from our various systems are included in the data lake we have been building and refining" would reduce the risk of future issue of this nature for DigiCert. What is the status of that effort?

Flags: needinfo?(brenda.bernal)

Hi Wayne, We have a comprehensive source that draws from the various systems where we can query and report, as needed, for incidents and ballot impact assessment. Additionally, we have improved our quality control on scripts that we need to run to ensure completeness and accuracy in reporting moving forward.

Flags: needinfo?(brenda.bernal)

Brenda: I'm interpreting your answer to mean that the data lake is at a state where DigiCert is confident that it will prevent future issues of this nature.

I believe that this bug can be resolved.

Closed: 8 months ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.