1649941 - T-Systems: Incorrect OCSP Delegated Responder Certificate

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Ryan Sleevi]

The following was originally reported to m.d.s.p. at https://www.mail-archive.com/[e-mail]/msg13493.html

T-Systems has issued one or more OCSP Delegated Responders, as defined within RFC 6960, Section 2.6 and Section 4.2.2.2, without including the id-pkix-ocsp-nocheck response, as required by the Baseline Requirements, Version 1, Section 13.2.5 through Version 1.7.0, Section 4.9.9

Example certificate: https://crt.sh/?id=1303235837

Please provide an incident report, including the timeline for revocation.

Comment 1

[Arnold Essing]

Deutsche Telekom Securtiy GmbH confirms receipt of this report and is investigating the issue.

Comment 2

[Arnold Essing]

1.How your CA first became aware of the problem, and the time and date.
Telekom Security were initially informed of the problem via the email from Ryan Sleevi to m.d.s.p. (2020-07-01 23:06 CEST) as well as via the automated information about the Bugzilla in relation to that email (2020-07-02 03:23 CEST). The information was first seen by Telekom Security around 2020-07-02 07:05 CEST.

2.A timeline of the actions your CA took in response.
2020-07-02 07:05 CEST: First inspection, assessment and forwarding of the information.
2020-07-02 09:00 CEST: First conference in which the actual understanding of the problem was established (affected certificates are not the OCSP Delegated Responders of the Issuing CAs but the Issuing CAs themselves). Additionally, two more certificates were found to be affected. All affected certificates are operated by Telekom Security without the involvement of a third party.
2020-07-02 10:19 CEST: Management as well as personnel responsible for the affected certificates were informed about the potential severity of the problem.
2020-07-02 13:00 CEST: Second conference including management, responsible personnel and ISMS to discuss and evaluate the problem as well as a first impact analysis.
2020-07-02 14:38 CEST: Confirmation of the receipt of the Bugzilla (see above).
2020-07-02 – 2020-07-06: Since some aspects were still unclear to us, we followed the discussion on m.d.s.p. closely and continuously analysed the problem. In parallel, vendors have been contacted and technical tests were performed to analyse the applicability of different actions (immediate revocation, renewal, re-key etc.). These tests also comprised technical aspects on how to operate the affected Issuing CAs without the EKU.
2020-07-06 ~16:30 CEST: Based on the results and based on the understanding of the security issue, the decision to revoke and destroy the keys as quick as possible (see 7.) was made. However, a revocation within the required seven days was determined to be too disruptive for the subscribers/involved parties to be bearable.
2020-07-06 – ongoing: We are currently evaluating further possibilities to remediate or mitigate the risk that results from the delayed revocation in regard to the webPKI.
2020-07-08: Post of this Incident Report.
2020-07-09: Planned: Issuance of new CA certificates with new keys.

3.Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.
At the time being, there were no problematic certificates to be issued. Future certificates will not contain the OCSPSigning EKU.

4.A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.
Deutsche Telekom AG Issuing CA
Deutsche Telekom AG secure email CA E02
TeleSec PKS CA 8

Additional Certificates that have not been disclosed/are not included in https://misissued.com/batch/138/
Deutsche Telekom AG secure email SN: 75 81 aa 9f 98 30 a3 ab bf 5b b6 9f 84 d8 56 (name constrained)
Deutsche Telekom AG secure email SN: 15 31 b1 a1 34 7c 85 a9 7a 37 f6 0e bb 50 fd 86 (name constrained)

5.The complete certificate data for the problematic certificates.
Deutsche Telekom AG Issuing CA 01 https://crt.sh/?id=40463077
Deutsche Telekom AG secure email CA E02 https://crt.sh/?id=2517734973
Deutsche Telekom AG secure email SN: 75 81 aa 9f 98 30 a3 ab bf 5b b6 9f 84 d8 56 (name constrained)
Deutsche Telekom AG secure email SN: 15 31 b1 a1 34 7c 85 a9 7a 37 f6 0e bb 50 fd 86 (name constrained)
TeleSec PKS CA 8 https://crt.sh/?id=1303235837

The two undisclosed, name constrained certificates will be attached to this Bugzilla. The other three certificates can be viewed via crt.sh.

6.Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
As thoroughly discussed in the m.d.s.p., we also assumed that the EKU OCSPSigning would be required in order to issue actual OCSP Delegated Responder certificates. This assumption was further supported by the technical inability of our CA to issue OCSP Delegated Responder certificates without the corresponding EKU being included in the Issuing CA certificate.
In addition to that: we misinterpreted the wording in RFC6960 to be a one-way implication instead of the actual equivalence. This understanding was corrected as mentioned above.

7.List of steps your CA is taking to resolve the situation and ensure that such situation or incident will not be repeated in the future, accompanied with a binding timeline of when your CA expects to accomplish each of these remediation steps.
Regarding the compliance issue:
Telekom Security will revoke the affected certificates as soon as possible. However, the revocation will not be performed within the seven days as required by the Baseline Requirements 4.9.1.2. The decision to not revoke the affected certificates within seven days is based on the impact this would have on the subscribers. There are currently more than 230.000 S/MIME and signing-certificates issued under the affected CA certificates. It will require approximately five months to reissue these EE-certificates under a new Issuing CA.
Telekom Security plans to take the following steps to solve the compliance issue:

• New Issuing CAs with new key pairs and without the OCSPSigning-EKU will be issued within the next days (Planned: 2020-07-09).
• Regarding TeleSec PKS CA 8, the new Issuing CA will be under a different root CA that is not affiliated with the webPKI. Regarding the remaining affected certificates only one new certificate will be issued under the same Root CA.
• Migration: all affected EE-certificates will be reissued under the new Issuing CAs (all S/MIME certificates).
• Revocation and key destruction: after all EE-certificates from an affected Issuing CA have been reissued, that CA will then be revoked and the key will be destroyed.
• This process requires approximately five months to be completed for all affected certificates.

We intend to provide a more precise timeline regarding the revocation of each certificate and the destruction of the corresponding private key.

Regarding the security issue:
Telekom Security understands that the mere revocation of the affected certificates is not sufficient to solve the security issue of this problem. As mentioned above, the private keys related to the affected certificates will be destroyed as soon as possible. Until the destruction, additional measures against the misuse of these keys will be implemented.
We also want to inform you that all affected Issuing CA certificates are under our complete control, operated solely by ourselves with the key material being inside FIPS140-2 L3 certified HSMs. It is planned to provide special attestations by auditors regarding the current network controls as well as the current HSM-controls. These attestations are currently in the planning stage.

We are aware that these information, including the planned attestations, do not suffice to fully remediate the risk as seen by the users of the webPKI. In order to mitigate this risk as much as possible, we are eager to provide additional evidence in further updates to this Bugzilla. The evaluation of the possibilities on how to provide such additional assurances is ongoing and we will continue to follow the m.d.s.p. closely as well as observe the related Bugzillas for further information and inspiration for further mitigation measures.

Comment 3

[Arnold Essing]

Attachment: cPKI.7z

Comment 4

[Arnold Essing]

Yesterday we also created a ticket https://bugzilla.mozilla.org/show_bug.cgi?id=1651487 for the delayed revocation.

Comment 5

[Jan Völkel]

Arnold Essing is currently not available so other team members will be providing updates to this Bugzilla on his behalf and until his return. The content of this specific update comprises a status update and more details regarding the risk assessment.

Status Update
As planned, issuance of new ICA certificates with new keys was carried out 2020-07-09.
The successor of the TeleSec PKS CA 8 has been issued under a different root (non-TLS) and is currently being distributed to the relying parties. Reissuance of end-entity certificates will be available to customers next Monday (2020-07-20) and the former ICA certificate is planned to be revoked 2020-08-04 with a destruction of the keys witnessed by external auditors shortly after. A more precise date regarding the key destruction will be communicated as soon as agreed upon.
The successor of Deutsche Telekom AG Issuing CA 01, Deutsche Telekom AG secure email (SN: 75 81 …), Deutsche Telekom AG secure email (SN: 15 31 …) and Deutsche Telekom AG secure email E02 is only one single ICA certificate (Deutsche Telekom AG secure email E03). This certificate was issued by the TeleSec GlobalRoot Class 2 and is currently being distributed to the relevant systems by the customer. Reissuance of end-entity certificates will start next Monday (2020-07-20), focussing on certificates issued by the Deutsche Telekom AG Issuing CA 01 first. Dates for revocation and key destruction will be communicated within the next weeks, depending on the progress made.

Information regarding the risk assessment
As seen in other Bugzillas and explicitly stated by Ryan, one “goal of this report should be to holistically help browsers and relying parties to understand our architecture” and, thereby, understand the resulting risk of the security issue (https://bugzilla.mozilla.org/show_bug.cgi?id=1649945#c8).
Telekom Security operates and maintains all components in relation to Root, OCSP, affected ICAs, support systems, etc. on its own. The tasks are performed by trusted personnel only. No third parties are involved. The systems have no direct connection to the internet or the intranet of the company and are all operated in high security zones with corresponding security controls.
The private keys of all CAs are generated and protected over their entire lifecycle by FIPS140-2 L3-certified HSMs. Therefore, all cryptographic activities regarding the private keys can only be performed by the HSMs themselves. This includes the signing of OCSP responses. Access to the corresponding partitions of the HSMs is restricted by firewalls and the manufacturer’s security system, which only allows specific servers (i.e. the servers with the CA-software) to communicate with the relevant partitions of these HSMs. Any communication of software with the partition is required to be authenticated based on a client certificate and passphrase.
The only software able to do so is the CA-software itself, which is not capable of generating OCSP-responses. No other software has a valid certificate or access to the passphrase. Additionally, any unnecessary software, services and other unrequired functionalities have been removed from the servers. This configuration, other security configurations and the integrity of the software are continuously monitored so that any changes will be immediately detected, and an alarm is triggered.
All log data is stored in tamper-proof log-appliances and continuously monitored for suspicious patterns.
The described setup is technically and organizationally secured against unauthorised changes.
We concluded that a misuse of the CA keys for the generation of (malicious) OCSP responses is highly interlinked with a compromise of the CA itself, for which security controls are in place. Based on the architecture and controls implemented (as described above and audited without a gap in the covered periods), we assess the probability of the security issue being exploited by an external or an internal attacker as very low.
On the other hand, there are more than 230.000 certificates used for S/MIME, qualified signatures and more. Close to all of these certificates are currently stored on (Q)SCD devices and, therefore, cannot be replaced within a short period of time. Due to this, a revocation of the affected certificates within seven days would have a severe impact on the operability of our customers who are also providing services for health care, the public sector and other critical infrastructures.

Based on our results, the decision to delay the revocation was made in order to provide our customers with more time to reissue the affected end-entity certificates under new ICAs. Nonetheless, the revocation and key destruction shall be carried out as soon as possible for each affected ICA-certificate in order to remove the very low but still existent residual risk.

Future compliance with CA/B Forum Baseline Requirements
Telekom Security is aware that measures are required so that a revocation within seven days will be possible in future incidents. The evaluation of the possibilities regarding this specific topic has been started and is ongoing. Currently, however, the focus lies on solving the compliance/security issue of the affected certificates.
Since it was mentioned in quite a few Bugzillas of other affected CAs, we would like to ask whether you prefer the delayed revocation to be discussed within this Bugzilla or in the corresponding “Delayed revocation”-Bugzilla?

Please let us know if you have any comments regarding the information provided.
Should there be any updates on progress, additional mitigation measures or other relevant information, we will share these as soon as possible.
You will hear from us regardless at the latest next Friday.

Comment 6

[Ryan Sleevi]

Thanks Jan!

I think your Comment #5 is quite useful for describing how the system is designed. I think it's equally important (and this is why I mentioned Detailed Control Reporting, in Bug 1649945) to also understand how this is verified. What tests the auditor performs to verify each of these claims and/or ensure that there aren't bypasses, for example. The role of the auditor, at least with respect to WebTrust, is to begin by assuming what the CA says isn't true, until they have obtained sufficient evidence that leads them to believe it is fairly stated. ETSI's approach is quite different here, as it relates to the underlying ISO standard, so understanding the process for auditing and verifying these claims is equally relevant.

With respect to revocation delays, you're correct that Bug 1651487 is the better bug to discuss those systemic changes.

Comment 7

[Stefan Kirch]

The migration was started this week as planned. As suggested by Ryan updates to the migration progress will be posted in Bug 1651487.
This Bug will be updated at least once a week or as soon as new measures are planned/implemented or if we have other relevant information.

Comment 8

[Arnold Essing]

On Thursday July 23, 2020, the auditor conducted the initial on site checks of the HSM and network controls.
For the weekly revocation delay update please see Bugzilla https://bugzilla.mozilla.org/show_bug.cgi?id=1651487

Comment 9

[Ben Wilson]

I am willing to close this bug and consolidate further discussion of this issue under Telekom Security's bug for delayed revocation, Bug #1651487. The comments in this bug contain many valuable disclosures and observations which are preserved for cross-reference in that bug. However, before I close this bug, I would like to understand what steps Telekom Security takes to ensure it is following and participating in discussions that highlight these types of issues. For instance, I assume that Telekom Security follows and occasionally participates in discussions about the CA incidents of other CAs in Bugzilla; changes and discussions of CA/Browser Forum Guidelines, and in particular, certificate profiles; and m.d.s.p posts, which discuss developments and interpretations of the applicable standards for CAs. (Section 2.1 of the Mozilla Root Store Policy states, "CAs MUST follow and be aware of discussions in the mozilla.dev.security.policy forum, where Mozilla's root program is coordinated. They are encouraged, but not required, to contribute to those discussions.") I think that understanding what additional steps you take to ensure this expectation is met, in order to detect and prevent future issues, is useful to ensure that no future interpretation issues arise.

Comment 10

[Arnold Essing]

Yes, we are actively following all the forums and mailing lists in which we are familiar with (e.g.: CAB Public list, Server Certificate Working Group, Mozilla Mailing list, CT-policy Mailing list, S/Mime Working Group, …) including all Bugs from other CAs on a daily basis (workdays).
When points of interest arise, we check immediately to see if they have any relevance for us. These checks are documented and improvements are made when appropriate.
In addition to our ongoing status calls, we also have a weekly QA call of the core-team to discuss all current points. Along with several other checks, this is documented in detail in our check-lists.
We have not yet actively participated in discussions due to our lack of membership in the CA/Browser Forum. In the future we aspire to actively participate, which is why we took part in the 49th & 50th CAB Forum F2F meetings. Since January we have added two new experts (Jan and Stefan) to the core team.

Comment 11

[Ben Wilson]

Closing - please refer to Bug 1651487.