Open Bug 1927506 Opened 6 days ago Updated 5 days ago

DigiCert: Incorrect OrgID in S/MIME certificates for one customer

Categories

(CA Program :: CA Certificate Compliance, task)

Tracking

(Not tracked)

ASSIGNED

People

(Reporter: tim.hollebeek, Assigned: tim.hollebeek)

Details

(Whiteboard: [ca-compliance] [smime-misissuance])

Steps to reproduce:

Incident Report

Summary

During the collection of sample certificates for our auditors during our annual Webtrust audit a similar issue to bug https://bugzilla.mozilla.org/show_bug.cgi?id=1898986 was found.

Impact

70,204 SMIME Certificates for a single customer.

Timeline

All times are UTC.

24 May 2024 21:51: Patch rolled out and validation restarted . This patch blocks a value the system does not recognize as a valid registration number from being entered if the NTR registration number is selected.

25 May 2024 23:08: https://bugzilla.mozilla.org/show_bug.cgi?id=1898986 bug posted.

21 June 2024 16:12: Bug 1898986 closed.

24 October 2024 19:00: Based on a potential discrepancy noticed while collecting Webtrust audit samples, our data analytics team scanned for certificates for NTRUS and asked the internal audit team to investigate. Audit team investigated and noted they believed this to be in error and sent it to the leadership team to confirm.

25 October 2024 16:56: Incident was confirmed and certificates scheduled for revocation.

25 October 2024 23:20: The data team re-ran the scan using using just NT and Gov. This scan covered all regions, no additional certs were found.

30 October 2024 16:00: All remaining impacted certificates are scheduled for revocation.

Root Cause Analysis

During the remediation of 1898986, we patched the validation system to prevent inclusion of new identifiers with “NTR“+ “Government entity”. We also searched for all associated validations.

Unfortunately, when that search was done the scan was for "NTRDE" + "Government entity" as that was the context of the bug. The data team did not fully consider that other countries could have this same problem. This recent finding was for one with NTRUS+CA-Government Entity

This time, we corrected the scanning logic to include “NT” + “Gov” across the full system. We are adding additional upstream blocks in the validation system and in pkilint so that in the future there aren’t single points of failure that can cause this issue to recur.

Lessons Learned

What went well

  • The error was caught internally during a normal review process .
  • All certificates will be revoked within BR timelines.

What didn't go well

  • The initial scan during 1898986 did not take into consideration countries other than Germany.

Where we got lucky

  • The incorrect organization identifier impacted only 1 further customer.
  • Customer is using our automation product so these certificates can be replaced silently with limited impact to end users.

Action Items

Action items and an affected certificates list will be provided in the full report.

Assignee: nobody → tim.hollebeek
Status: UNCONFIRMED → ASSIGNED
Type: defect → task
Ever confirmed: true
Whiteboard: [ca-compliance] [smime-misissuance]
You need to log in before you can comment on or make changes to this bug.