1866091 - SwissSign: EV JurisdictionStateOrProvinceName - one certificate not selected for revocation

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Roman Fischer]

Incident Report

Summary

In the recent incident regarding EV certificates using codes for jurisdictionSateOrProvinceName instead of full names, one certificate was not selected for revocation because it was in "domain validation pending" state. The domain validation was completed by the customer after the fix was implemented but the certificate was issued with the code instead of the full name of the jurisdictionStateOrProvinceName.

Impact

One EV certificate was mis-issued after implementation of the fix for Bugzilla https://bugzilla.mozilla.org/show_bug.cgi?id=1860750.

Timeline

2023-10-20:

• 12:30 UTC A SwissSign internal employee raises a question about EV juristictionStateOrProvince Name field -> This caused Bugzilla https://bugzilla.mozilla.org/show_bug.cgi?id=1860750 to be created.

2023-11-21:

• 15:31 UTC A SwissSign developer working on an updated self-audit script notices the mis-issued certificate and informs Compliance team via email

2023-11-22:

• 07:30 UTC Compliance confirms mis-issuance and starts mis-issuance process
• 08:15 UTC Affected customer and audit body is informed
• 08:45 UTC Affected certificate revoked by customer
• 15:50 UTC Bugzilla posted

Root Cause Analysis

Our new CA system uses different internal states than the self-developed old one. When compiling the list of certificates affected by Bugzilla 1860750, we had to re-write the script that selects the affected certificates and creates the to-be-revoked list because it was the first large-scale mis-issuance that we couldn't handle manually on the new CA system.
This script did not take into account that there may be certificate-request with "pending domain verification". Such requests should have been cancelled before implementing any changes affecting certificate content. This was not done. So the request remained in the system and when the customer eventually verified the domain, it got issued.

Lessons Learned

What went well

• Internal self-audit picked up the mis-issued certificate

What didn't go well

• The pending certificate request was not cancelled when implementing the fix for Bugzilla 1860750

Where we got lucky

• Only one certificate request was pending at the time of the incident

Action Items

Action Item	Kind	Due Date
Revoke mis-issued certificate -> Done	Mitigate	2023-11-22
Checked that no other certificate requests are pending with wrong information -> Done	Mitigate	2023-11-22
Clarify with vendor of CA system that no other states or border-cases need to be considered when doing mass-revocation	Prevent	2023-11-30
Update internal process description for mass-revocation on the new CA system to include cancelling of pending certificate requests	Prevent	2023-11-30

Appendix

Details of affected certificates

Serial: 4747bc74c34e1b0ab7235c1d223342d743936d51
SHA 256: 2A09929B6014AFCD60C0A9F712715A64700C6A2A77C65FA95B3033E62D603E2D
Link: https://crt.sh/?id=10904522597

Comment 1

[Roman Fischer]

Update on action items

• Clarification with vendor of CA system done.
• Internal proces description for mass-revocation updated.

Are there any open questions regarding this incident?

Comment 2

[Roman Fischer]

As there seem to be no more questions... can we close this Bugzilla?

Comment 3

[Ben Wilson]

I will close this on Friday, 8-Dec-2023, unless there are any questions or concerns.