1797165 - DigiCert: Delayed Revocation of ~5.5 hours

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Jeremy Rowley]

Attachment: bugzilla_1794050_delayed.csv

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.36

Steps to reproduce:

This is an extension of Bug 1794050. All timelines are in MDT.

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

While investigating comment 4, we discovered an issue from when the first batch of certificates were detected and when revocation occurred. Specifically, revocation occurred within 5 days of DigiCert gathering the validation information but 5 days and 5 hours after we verified which certificates were impacted by the validation issue.

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

09:11 Oct 5, 2022 - Compliance performed a sweep to determine the final list of impacted certificates with a count of 118.

15:52 Oct 5, 2022 – We grabbed the cert contact information and submitted the cert problem report. Although the contact data usually happens at the same time as the final sweep, the two were roughly 5.5 hours apart in this case.

16:39 Oct 6, 2022 - Bug 1794050 was logged.

14.24 Oct 10, 2022 - The certificates were revoked.

10.23 Oct 19, 2022 - Comment 4 added to Bug 1784050

Oct 20-21, 2022 - Investigation underway but could not confirm there was an issue.

12:40 Oct 24, 2022 - Confirmed there was an issue but different than in the comment The issue was with the initial 118 revocations.

3. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

118 certs as identified in sheet.

5 .The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

See attached. This is a subset of the list on Bug 1794050.

4. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

Historically, we grabbed org contact information at the same time as when we confirmed certificate issues. However, we moved to using our Business Intelligence team for org information and compliance for incident investigation. In this case, there was a delay between getting the org information for all of the certificates compared to when compliance confirmed mis-issuance. The result was a five hour delay in revocation.

7. List of steps your CA is taking to resolve the situation and ensure that such situation or incident will not be repeated in the future, accompanied with a binding timeline of when your CA expects to accomplish each of these remediation steps.

Going forward, we moved where the revocation timeline starts when the SHA2 cert fingerprint is available and not when org information is gathered. This aligns the process and systems for internal problem reports and external problem reports as external problem reports were always tied to submission of the cert data, not the org info.

Comment 1

[Jeremy Rowley]

Any additional questions?

Comment 2

[Ben Wilson]

I don't have any additional questions. Unless there are questions from others, I intend to close this on Friday, 4-Nov-2022.