1893610 - D-Trust: Notice to affected Subscriber and person filing CPR not sent within 24 hours

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Enrico Entschew]

Preliminary Incident Report

Summary

This is a preliminary report.

As part of the investigation into the Bugzilla incident 1891225 (https://bugzilla.mozilla.org/show_bug.cgi?id=1891225) it was determined that there was a non-conformity of the requirements of TLS BR 4.9.5.

The requirements regarding the notification of the person who filed the Certificate Problem Report and the affected Subscriber weren’t met in time.

The initial result of the investigation is that the internal e-mail forwarding to the responsible department did not work.

We are currently investigating this incident.

Impact

The intended recipients of the preliminary incident report were notified with delay. However, this did not delay the actual revocation of the certificates concerned.

Timeline

2024-03-13:
The internal central incident email address had a spelling mistake after e-mail forwarding configurations were changed in the e-mail system. This caused the notification to be delayed.

2024-04-09:

• 08:15 Email via [e-mail] about a potential non-conformity of the BRs regarding the order of the subject attributes (RDN)

2024-04-10:

• 13:49 Time when the email actually reached the relevant department
• 14:00 Start of investigation on Bugzilla Incident 1891225 (https://bugzilla.mozilla.org/show_bug.cgi?id=1891225)
• 16:08 An email was sent to the person who filed the Certificate Problem Report with the information that we’re investigating the reported problem.

2024-04-11:

• 08:40 All affected customers were informed.

2024-04-12:

• 14:35 The person who filed the Certificate Problem Report was informed about the fact that all affected certificates were revoked and about the bug ID of the preliminary incident report.

2024-04-22:

• 09:00 Start of investigation as to what caused the delay in our internal communication.

Root Cause Analysis

D-Trust uses the services of Bundesdruckerei's call and support center for first level support. Our service provider made changes to the internal email forwarding configuration that we were not aware of. They deactivated previous forwarders in the course of an email system change. Afterwards the dedicated internal central email address in the new forwarding configurations was entered with a spelling mistake.

When the Certificate Problem Report arrived, it was processed by a support agent and forwarded to the incorrect email address. Unfortunately, it was only realized late on that there was an error message stating that the email had not been delivered.

The support agent then informed the support manager responsible for D-Trust, who in turn contacted the relevant department.

Unfortunately, this process took over 24 hours.

Lessons Learned

• Will be provided with another update

What didn't go well

• Will be provided with another update

Where we got lucky

• Will be provided with another update

Action Items

Action Item	Kind	Due Date
Set up our own internal e-mail distribution list, which can no longer be influenced by the service provider.	Prevent	2024-05-10
Intensify regular testing of the e-mail distribution list and the responsiveness of the responsible teams (previously quarterly, now monthly)	Prevent	2024-04-23

Appendix

n.a.

Based on Incident Reporting Template v. 2.0

Comment 1

[Enrico Entschew]

We will publish the final incident report on 5/8/2024 at the latest.

Comment 2

[Leyla Sahin]

Final Incident Report

Summary

This is the final report.

As part of the investigation into the Bugzilla incident 1891225 (https://bugzilla.mozilla.org/show_bug.cgi?id=1891225) it was determined that there was a non-conformity of the requirements of TLS BR 4.9.5.

The requirements regarding the notification of the person who filed the Certificate Problem Report and the affected Subscriber weren’t met in time.

The initial result of the investigation is that the internal email forwarding to the responsible department did not work.

Impact

The intended recipients of the preliminary incident report were notified with delay. However, this did not delay the actual revocation of the certificates concerned.

Timeline

2024-03-13:
The internal central incident email address had a spelling mistake after email forwarding configurations were changed in the email system. This caused the notification to be delayed.

2024-04-09:

• 08:15 Email via [e-mail] about a potential non-conformity of the BRs regarding the order of the subject attributes (RDN)

2024-04-10:

• 13:49 Time when the email actually reached the relevant department
• 14:00 Start of investigation on Bugzilla Incident 1891225 (https://bugzilla.mozilla.org/show_bug.cgi?id=1891225)
• 16:08 An email was sent to the person who filed the Certificate Problem Report with the information that we’re investigating the reported problem.

2024-04-11:

• 08:40 All affected customers were informed.

2024-04-12:

• 14:35 The person who filed the Certificate Problem Report was informed about the fact that all affected certificates were revoked and about the bug ID of the preliminary incident report.

2024-04-22:

• 09:00 Start of investigation as to what caused the delay in our internal communication.

2024-04-30:

• 11:05 We have changed the email address for submitting Certificate Problem Reports. The service provider’s email address has been replaced with a D-Trust email address.

2024-05-08:

• 11:00 End of investigation and preparation of the final incident report

Root Cause Analysis

D-Trust uses the services of Bundesdruckerei's call and support center for first level support. Our service provider made changes to the internal email forwarding configuration that we were not aware of. They deactivated previous forwarders in the course of an email system change. Afterwards the dedicated internal central email address in the new forwarding configurations was entered with a spelling mistake.

When the Certificate Problem Report arrived, it was processed by a support agent and forwarded to the incorrect email address. Unfortunately, it was only realized late on that there was an error message stating that the email had not been delivered.

The support agent then informed the support manager responsible for D-Trust, who in turn contacted the relevant department.

Unfortunately, this process took over 24 hours.

Lessons Learned

• The reason the subscriber and the person who filed the Certificate Problem Report were not informed on time was that we used an email address from the service provider. We were not aware that the service provider made changes to the internal email forwarding configuration. To avoid future problems like this we removed the email address of the service provider and replaced it with a D-Trust email address.
• Previously we tested the incident issue emailing address quarterly. Now we will test it more often. We are considering an automation of the test.

What didn't go well

• The process took more than 24 hours for the report to reach the relevant department at D-Trust.

Where we got lucky

• We had already lost a lot of time, so we made solving the problem our highest priority.
• We were lucky that only a few customers were affected.
• We were also lucky that we were able to reach the customers quickly and that they were cooperative.
• Despite the delay in forwarding the issue email to the relevant department, this did not delay the actual revocation of the certificates concerned.

Action Items

Action Item	Kind	Due Date
Set up our own email distribution list, which can no longer be influenced by the service provider.	Prevent	2024-04-30
Intensify regular testing of the email distribution list and the responsiveness of the responsible teams (previously quarterly, now monthly)	Prevent	2024-04-23

Appendix

n.a.

Comment 3

[Enrico Entschew]

We have implemented all measures. They have already been successfully applied. Please see https://bugzilla.mozilla.org/show_bug.cgi?id=1896190
I kindly ask you to check whether this incident can be closed.

Comment 4

[Ben Wilson]

Are there any comments, questions, or suggestions from the community? If not, I'll plan to close this sometime next week (May 27-31).