1862082 - D-Trust: Delay beyond 5 days in revoking misissued certificate

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Enrico Entschew]

Incident Report

This is a preliminary report.

Summary

Within the scope of the incident report https://bugzilla.mozilla.org/show_bug.cgi?id=1861069, 15 affected DV certificates were to be revoked until October 30, 2023. Unfortunately, one certificate was not revoked.

The certificate will be revoked as soon as possible. The same applies to the start of the thorough root cause analysis.

Certificate that has not been revoked:
https://crt.sh/?sha256=57E5E699D847DA98F58AED2098CD1237925A6FEC03871B69B920BDFB425C92B2

Comment 1

[Enrico Entschew]

The affected certificate was successfully revoked today at 08:09 UTC. Analysis of the root cause is still running at the moment. An update of this ticket will follow soon.

Comment 2

[Enrico Entschew]

Incident Report

Summary

Within the scope of the incident report https://bugzilla.mozilla.org/show_bug.cgi?id=1861069, 15 affected DV certificates were to be revoked until October 30, 2023. Unfortunately, one certificate was not revoked on time.

It was successfully revoked on October 31, 2023 at 08:09 UTC.

Impact

One DV certificate was not revoked on time.

Certificate that has not been revoked on time:
https://crt.sh/?sha256=57E5E699D847DA98F58AED2098CD1237925A6FEC03871B69B920BDFB425C92B2

Timeline

All times are UTC.

2023-10-25:

• 13:10 Manual check of DV certificates issued after September 15, 2023. Discovery that misissuance has occurred. Please see https://bugzilla.mozilla.org/show_bug.cgi?id=1861069
• 14:00 Decision to revoke affected certificates within 5 days

2023-10-30:

• Until 13:00 revocation of 14 remaining unrevoked deficient DV certificates
• 21:10 Discovery that 1 deficient DV certificate remained unrevoked
• 22:37 Opening the incident report at bugzilla on the delay of the revocation

2023-10-31:

• 08:00 Start of investigation of root cause
• 08:09 Revocation of affected certificate
• 17:33 Informing Conformity Assessment Body about the issue

2023-11-03:

• 09:00 End of investigation of root cause

Root Cause Analysis

The IT team usually performs a revocation of multiple certificates in the form of a "mass revocation". This process is divided into several steps: Preparation of a list of the certificates to be revoked, development of a revocation script, execution of the revocation and verification of the revocation success across all certificates.

This time we used a manual revocation process. This was because only 9 certificates were affected. The manual process consists of the following steps: Preparation of a list of certificates to be revoked, selection of the certificates to be revoked (which must be done one by one by the first revocation officer), and revocation of the selected certificates (which must be done one by one manually by the second revocation officer). A manually performed random check did not result in any objections. In this way, the error that a certificate had not been revoked was not noticed.

Lessons Learned

What went well

What didn't go well

• A success test of the revocation was not performed on all affected certificates.

Where we got lucky

• The error was discovered quite promptly by processing the Bugzilla ticket https://bugzilla.mozilla.org/show_bug.cgi?id=1861069.

Action Items

Action Item	Kind	Due Date
From now on, as a final step, we will perform a full check on all certificates dedicated for revocation to ensure that they have been successfully revoked as a final step.	Prevent	2023-11-01

Appendix

Details of affected certificates

List of all affected certificates

https://crt.sh/?sha256=57E5E699D847DA98F58AED2098CD1237925A6FEC03871B69B920BDFB425C92B2

Comment 3

[Enrico Entschew]

Two quick additions to avoid misunderstandings:

1. The certificate that was not revoked was noticed during the processing of incident https://bugzilla.mozilla.org/show_bug.cgi?id=1861069.
2. 15 certificates were to be revoked. 6 certificates were revoked by the customers in time. Therefore, 9 certificates had to be revoked by D-Trust in order to meet the deadline of 5 days after discovery of the misissuance.

Comment 4

[Enrico Entschew]

Quick update: There is nothing new to report.

Comment 5

[Enrico Entschew]

Quick Update: There is nothing new to report.

Comment 6

[Enrico Entschew]

We have no further additions to this ticket. Unless there are any further questions we respectfully ask that this incident be closed.

Comment 7

[Ben Wilson]

I will close this on Wed. 13-Dec-2023 unless there are questions or concerns expressed.

Comment 8

[amir]

The answer to what went wrong isn't sufficient in my opinion here.

Why did you not use your mass revocation flow that seemingly has worked in the past?

To me, so far, this just reads as "human error". Adding an extra step to manually review does not create a technical solution on preventing missing revocations.

What I'm looking for is:

• What type of technical controls (e.g. not extra human review) are you planning on putting in place to prevent this from happening in the future?
• Why did you diverge from your usual revocation strategy?
  • On this note, you also stated, "development of a revocation script". Does that imply that for every incident involving mass revocation, you're writing a new script to handle revoking these certificates?

Comment 9

[Enrico Entschew]

Thank you very much for your comments and questions. I will answer them under the respective questions and comments.
The answer to what went wrong isn't sufficient in my opinion here.

Why did you not use your mass revocation flow that seemingly has worked in the past?

Due to the low revocation volume, it was decided that manual revocation should be carried out. The manual process can only be used if the number of certificates to be revoked is very low (up to a low two-digit number). As the preparation and execution of scripts in high-security infrastructures is critical, the decision is made in consultation with management depending on the situation.

To me, so far, this just reads as "human error". Adding an extra step to manually review does not create a technical solution on preventing missing revocations.

In order to prevent such an incident in the future, we have introduced a complete technical check of the successful revocation of all affected certificates for the manual process as well. In our view, we have thus sufficiently ensured for the manual revocation process that such a case should no longer occur.

What I'm looking for is:
• What type of technical controls (e.g. not extra human review) are you planning on putting in place to prevent this from happening in the future?
• Why did you diverge from your usual revocation strategy?
o On this note, you also stated, "development of a revocation script". Does that imply that for every incident involving mass revocation, you're writing a new script to handle revoking these certificates?

"Development of a revocation script" means the preparation of the existing script with a reference to the list of affected certificates, the testing of the script and the approval by the management.

Comment 10

[amir]

Thank you for your responses! I don't have any further questions.