1875820 - Telekom Security: TLS certificates with basicConstraints not marked as critical

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Arnold Essing]

Telekom Security has identified certificates with basicConstraints in the certificate that are not marked as critical. This is a violation of BR 7.1.2.7.6
This is a preliminary report. We are investigating and will update with a full incident report soon.

Comment 1

[Arnold Essing]

Summary
In the TLS certificates under one of our TLS-CAs (TeleSec Business TLS-CA 2022, https://crt.sh/?caid=242556) we had set the basicConstraints but didn't mark them as critical.

Impact
The incident impacted 816 TLS certificates since September 15, 2023.
From our point of view, this incident didn't impact the usage of the certificates, since the used coding is still conform to RFC5280 and we have not yet received any questions or problem reports from customers or software suppliers.
Furthermore, we do not see any security breaches or restriction of trust in the certificates, since there is no incorrect or invalid data included.

Timeline
All times are UTC.

2024-01-22:
08:25 The error message in crt.sh was found in our weekly checks
08:30 Video conference of the root and compliance team to analyze the error
09:15 Video conference between the root and compliance team and the solution management of the affected solution to discuss the root cause and the next steps
09:45 Video conference with our software supplier concerning zlint
10:30 Video conference with the management of the Trust Center
10:45 The issuance of the affected certificates was stopped
11:03 An incident was opened in the incident- and change management tool
11:28 The affected certificate template as the cause of the error was corrected and a certificate for checking was issued
11:37 The list of the affected certificates and customers was provided
15:27 The preliminary bug was filed in bugzilla
16:10 The information for the affected customers was finalized
16:37 The affected customers were informed and were asked to exchange and revoke the affected certificates as soon as possible but within 5 days at the latest

Root Cause Analysis
The root cause of the error was, that a change in the TLS BR version 2.0.0 was overseen and accordingly that change was not implemented, i.e. the certificates were issued as before with the basicConstraints not marked as critical.
The TLS BR version 2.0.0 included an entirely revised chapter 7 with many changes to the format but only a few changes to the content. Thus, the change in the criticality of the basicConstraints was overseen, although the changed definitions of the certificate profiles were checked by several persons.
The reason why this error was only found four months after the change in BR is that zlint also only has implemented this change in version 3.6 which was provided on January 7 and which was planned to be tested and implemented in our environment in the next weeks. Also crt.sh had not issued an error until last week. For this reason, the error was not detected during our weekly crt.sh checks from mid-September until last week.
Note: crt.sh was not running stable in the last weeks, so you could not fully rely on the crt.sh checks.

Lessons learned
What went well?
Our weekly checks have made us aware of the error rather quickly after it was discovered by crt.sh. After the error message in crt.sh was seen, the cause of the error was soon analyzed and fixed in about three hours

What didn’t go well?
The changes to the BR were very extensive, but despite the complete restructuring, there were only a few actual content changes to the already existing profile requirements and that is where the human error on our side occurred despite the involvement of multiple persons. For us, it would have most likely been very helpful, if the actual content changes had been described separately or marked more clearly so that they do not get lost in the flood of structural changes and new chapters. This could be achieved, for example, by separating ballots regarding bigger structural changes from those regarding content changes.
Linting certificates shortly after changes cannot always be relied upon, as the linting tools only check new requirements with a time delay.

Where we got lucky
From our point of view, this incident did not result in any security breaches or in any problems with applications.

Action Items
| Action Item | Kind | Due Date |
| Renewed alignment of all requirements of the BR in 4EP | Detect | 2024-01-31 |
| Update of zlint in the productive environment | Prevent | February 2024|

Appendix
The list of affected certificates with links to crt.sh (Pre-certificates, as not all leaf certificates are published in crt.sh).

Comment 2

[Arnold Essing]

Attachment: certificatesSHA256.csv

Comment 3

[Arnold Essing]

As indicated in the timeline above, we informed all affected customers about the bug on January 22nd and asked them to revoke their certificates as soon as possible, but within 5 days at the latest.
Based on feedback from some customers (e.g. bank, police, toll system) who use the affected certificates in their critical infrastructures and were not able to replace all certificates within 5 days, we carried out a risk assessment of the impact that a revocation would have on these infrastructures whose functionality depends on certificates that could not be replaced in time.
For this reason, we decided on January 27 not to revoke the affected certificates of these critical infrastructures after the 5-day period, but to give the affected customers more time to replace the certificates in order to ensure the continuity of their critical infrastructures.
On January 27 336 of the affected certificates were not yet revoked. Accordingly we have filed a delayed revocation bug, see https://bugzilla.mozilla.org/show_bug.cgi?id=1877388.

Comment 4

[Antonis]

Hello Arnold,

Thanks for your incident report! I have a few questions regarding that:

What is the "solution management" team? Is this Customer Support / Account Managers / etc. that interface with customers regarding certificates? Are they CA Staff?

Is there a reason they were involved in the incident response process before Management, Bugzilla, Vendor Techs, etc.? This seems like incorrect prioritization during an emergency.

---

Also, can you clarify who is the supplier (what do they supply?) in the following?

09:45 Video conference with our software supplier concerning zlint

Do you use a third party offering for zlint, or are you running it yourselves?

---

Finally, in the quote below you mention asking customers to revoke the certificates themselves:

16:37 The affected customers were informed and were asked to exchange and revoke the affected certificates as soon as possible but within 5 days at the latest

Did you have plans for performing a mass revocation yourself, should customers not get back to you? Is there a mass revocation procedure that's updated and tested regularly, where you could have performed everything yourself, without relying on your customers to do it themselves?

Thanks!

Comment 5

[Arnold Essing]

Hello Antonis,
Thanks for your questions, to which we would like to give feedback.

What is the "solution management" team? Is this Customer Support / Account Managers / etc. that interface with customers regarding certificates? Are they CA Staff?
Is there a reason they were involved in the incident response process before Management, Bugzilla, Vendor Techs, etc.? This seems like incorrect prioritization during an emergency.

Our "Solution Management" is the CA staff responsible for the affected trust service, i.e. for both technical and organizational matters. They are the ones required to get information about database entries, certificate templates, affected customers etc. Therefore, after the error message in crt.sh was reviewed by the Compliance Team, the Solution Management was required (and responsible) to perform further analysis, including the confirmation of the error from our side, evaluating the impact and the root cause and assess the situation before we can brief the management and plan next steps.

Also, can you clarify who is the supplier (what do they supply?) in the following?
09:45 Video conference with our software supplier concerning zlint
Do you use a third party offering for zlint, or are you running it yourselves?

Software supplier refers to the developer of our CA software. In addition to the CA software, our software supplier also provides other tools, e.g. for linting certificates, in which zlint is integrated, among other things. Accordingly, zlint runs directly in our environment.

Finally, in the quote below you mention asking customers to revoke the certificates themselves:
16:37 The affected customers were informed and were asked to exchange and revoke the affected certificates as soon as possible but within 5 days at the latest.
Did you have plans for performing a mass revocation yourself, should customers not get back to you? Is there a mass revocation procedure that's updated and tested regularly, where you could have performed everything yourself, without relying on your customers to do it themselves?

Yes, we did have plans for mass revocations, in the event that customers had not reacted in a timely manner or had not credibly explained the reasons for not revoking certificates. As stated in the timeline of our revocation delay bug, shortly before the 5 day period ended (2024-01-27), we evaluated whether all certificates were revoked or whether plausible reasons had been provided by the customers to delay the revocation (including the risk assessment from our side). Luckily, all customers had responded until then and no (mass) revocation was necessary.
The procedures for mass revocation are defined, tested and, as they are part of our emergency management, regularly revised.

Comment 6

[Stefan Kirch]

The last certificates were revoked today, i.e. all certificates affected by this bug are revoked now.

Comment 7

[Stefan Kirch]

We are monitoring this bug for feedback. Please let us know if there are any further comments or questions.

Comment 8

[Stefan Kirch]

Yesterday we have successfully updated zlint in the production environment.
Please let us know if there are any further comments or questions.

Comment 9

[Stefan Kirch]

We are monitoring this bug for feedback. Please let us know if there are any further comments or questions.

Comment 10

[Stefan Kirch]

We are monitoring this bug for feedback. Please let us know if there are any further comments or questions.

Comment 11

[Stefan Kirch]

We are monitoring this bug for feedback. Please let us know if there are any further comments or questions.

Comment 12

[Arnold Essing]

We are monitoring this bug for feedback. Please let us know if there are any further comments or questions.

Comment 13

[Arnold Essing]

We are monitoring this bug for feedback. Please let us know if there are any further comments or questions.

Comment 14

[Stefan Kirch]

If there are no further questions, we kindly ask that this bug can be closed.

Comment 15

[Ben Wilson]

I will close this on Friday, 3-May-2024, unless there are any questions or concerns.