1455137 - T-Systems: Undisclosed Intermediate certificate

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Wayne Thayer]

T-Systems has failed to disclose the following intermediate CA certificate in CCADB as required by section 5.3 of the Mozilla root store policy:

https://crt.sh/?sha256=c6193077c6189d1dfbbf813b87dc7cbf0498acf727887bc7ec54320906de9bc8&opt=mozilladisclosure

Please disclose the certificate, and provide an incident report, as described here:
https://wiki.mozilla.org/CA/Responding_To_A_Misissuance#Incident_Report
The incident report should be posted to the mozilla.dev.security.policy forum and added to this bug.

Comment 1

[Bernd]

Hello Wayne,

we replaced the former technically constrained Sub-CA "Deutsche Telekom AG Issuing CA 01" (TCSC: EKU and business controls; Certificate Serial Number: ee:db:86:0e:52:3b:2e:43) by a new technically constrained Sub-CA (TCSC: EKU and Name Constrains). The old Sub-CA is deactivated, but used for revocation purposes - so we cannot revoke it. 

Please add the Sub-CA "Deutsche Telekom AG Issuing CA 01; Serial Number: ee:db:86:0e:52:3b:2e:43" to OneCRL.

Best regards,
Bernd

Comment 2

[Wayne Thayer]

Bernd: Please explain in more detail why it is not possible to revoke the "Deutsche Telekom AG Issuing CA 01" certificate.

Simply adding this certificate to OneCRL is not an acceptable solution because OneCRL only affects TLS certificates in Firefox, and this intermediate is restricted to emailProtection.

Comment 3

[Bernd]

The last certificate issued by this Sub-CA is valid until April 19, 2021 and we have to create the daily certificate revocation list (CRL) until this point in time. That's the only reason, we cannot revoke the Sub-CA before that date.

Comment 4

[Wayne Thayer]

Bernd: why can't you revoke only the unconstrained version of the certificate?

If you cannot revoke the unconstrained version of this certificate, then it needs to be disclosed and audited, or removed from the Mozilla root store. How would you like to proceed?

Comment 5

[Bernd]

Wayne: please remove the certificate "Deutsche Telekom AG Issuing CA 01" (SN: eedb860e523b2e43, SHA2-FP: C6193077C6189D1DFBBF813B87DC7CBF0498ACF727887BC7EC54320906DE9BC8) from the Mozilla root store. 
We cannot revoke the old unconstrained certificate because the new constrained certificate was issued with a different commonName. The new constrained certificate’s audit date is scheduled for August 2018. After this, we would like this certificate to be "reincluded" into the Mozilla root store.

Comment 6

[Wayne Thayer]

(In reply to Bernd from comment #5)
> Wayne: please remove the certificate "Deutsche Telekom AG Issuing CA 01"
> (SN: eedb860e523b2e43, SHA2-FP:
> C6193077C6189D1DFBBF813B87DC7CBF0498ACF727887BC7EC54320906DE9BC8) from the
> Mozilla root store. 
> We cannot revoke the old unconstrained certificate because the new
> constrained certificate was issued with a different commonName. The new
> constrained certificate’s audit date is scheduled for August 2018. After
> this, we would like this certificate to be "reincluded" into the Mozilla
> root store.

Bernd: we cannot remove an intermediate certificate because we only ship your root certificate in the Mozilla trust store.

I think you may be confused about the subject of this issue. There are 4 "Deutsche Telekom AG Issuing CA 01" certificates using this keypair:

https://crt.sh/?ski=8645e44842a86cb84e2dc576abc08cf66551fcf3

The one that is undisclosed is the first in the list, signed by the "T-TeleSec GlobalRoot Class 2" root. I suspect that your audits do cover this certificate (the current audits do not list intermediate certificates, but the same audit is ientified as covering the other three "Deutsche Telekom AG Issuing CA 01" in CCADB.

Therefore, I think that you simply need to disclose the "Deutsche Telekom AG Issuing CA 01" intermediate signed by the "T-TeleSec GlobalRoot Class 2" root in CCADB. Will you please add this certificate to CCADB: https://crt.sh/?id=40463077 ?

Comment 7

[Bernd]

Wayne: The certificate "Deutsche Telekom AG Issuing CA 01" (SN: eedb860e523b2e43) is added to CCADB.

Comment 8

[Wayne Thayer]

Bernd: we're almost there, but you need to enter the information for the audit that covers this certificate in the CCADB record. Once you do that, this certificate will no longer appear under "disclosure incomplete" at https://crt.sh/mozilla-disclosures (it usually takes a day for crt.sh to detect CCADB changes)

Comment 9

[Ryan Sleevi]

Wayne: It appears this information was updated, as far as I can tell?

It also does not appear that an incident report, as requested in Comment #0, has been provided.

Comment 10

[Wayne Thayer]

Arnold: please supply the incident report describing why this happened and how it will be prevented in the future: https://wiki.mozilla.org/CA/Responding_To_A_Misissuance#Incident_Report

Comment 11

[Ryan Sleevi]

Arnold, Bernd: Please provide an update.

Comment 12

[Arnold Essing]

1. How your CA first became aware of the problem, and the time and date.
   On 18/04/2018 at 15:09 PDT, a bug was posted by Mozilla that the Intermediate Certification Authority certificate "Deutsche Telekom AG Issuing CA 01" with the serial number ee:db:86:0e:52:3b:2e:43 under the Root CA "T-TeleSec GlobalRoot Class 2" in the CCADB was not disclosed as required by section 5.3 of the Mozilla Root Store Policy.

2. A timeline of the actions your CA took in response.
   Certificate issuance under the sub-CA "Deutsche Telekom AG Issuing CA 01" with the serial number ee:db:86:0e:52:3b:2e:43 was deactivated on 18.04.2018. From this point on, only revocation lists were issued by the CA.
   The previous technically restricted sub-CA "Deutsche Telekom AG Issuing CA 01" with the serial number: ee: db: 86: 0e: 52: 3b: 2e: 43 was replaced by the SUB CA "Deutsche Telekom AG secure email CA" with the serial number 75:81:aa:9f:98:30:a3:ab:bf:5b:b6:9f:84:d8:56 with domain constraints (EKU and name restrictions).
   Based on the European standards "ETSI EN 319 411-1, V1.1.1 (2016-02)", "ETSI EN 319 401, V2.1.1 (2016-02)" and taking into account the requirements from "ETSI EN 319 403, V2. 2.2 (2015-08)", discussions were heard with the auditor from 23.04.2018 to 26.04.2018 to conduct an audit
   The audit was completed on 27.11.2018 at the locations Netphen and Frankfurt, Germany and covers the timeframe from 28.11.2017 to 27.11.2018.
   In the audit, it was determined by the auditor that the CA "Deutsche Telekom AG Issuing CA 01"; Serial number: ee: db: 86: 0e: 52: 3b: 2e: 43 had ceased issuing certificates on April 18, 2018. Furthermore, it was determined that the CA only issued information on revocation status.

3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.
   The issuance of certificates from the CA "Deutsche Telekom AG Issuing CA 01"; Serial number: ee: db: 86: 0e: 52: 3b: 2e: 43 was stopped on 18 April 2018.

4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.
   Sub-CA "Deutsche Telekom AG Issuing CA 01“; Serial number: ee: db: 86: 0e: 52: 3b: 2e: 43"

5. The complete certificate data for the problematic certificates.
   https://crt.sh/?id=40463077

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
   The switch to a new sub-CA certificate "Deutsche Telekom AG secure email CA" with the serial number 75:81:aa:9f:98:30:a3:ab:bf:5b:b6:9f:84:d8:56 with EKU and name restrictions (Domain constraints) was scheduled for January 2018. The new CA had to be commissioned on new systems at a new Trust Center location.
   Due to delays in the deployment of the new system, the new CA was not operational until April 2018.

7. List of steps CA is taking to resolve the situation and ensure it will not be repeated.
   In order to avoid such problems in the future for the subsequent "Deutsche Telekom AG secure email CA", in addition to the technical constraint, name restriction (domain constraint) has been incorporated into the certificate.
   In addition, the cPKI from DTAG will be audited based on the European Standards "ETSI EN 319 411-1, V1.1.1 (2016-02)" and "ETSI EN 319 401, V2.1.1 (2016-02)" taking into account the requirements from "ETSI EN 319 403, V2.2.2 (2015-08)".

Comment 13

[Ryan Sleevi]

Arnold, thanks for responding, but I don't think this incident report really demonstrates an understanding of the issue or concerns.

On 2018-04-18, when Wayne created this bug, T-Systems was out of compliance with Mozilla policy, by failing to disclose an intermediate.
Subsequently, on 2018-06-29 (Comment #8), it was pointed out that the audit information had not been disclosed.

Understanding why the unconstrained version was not disclosed is core to this incident report.

Comment 14

[Arnold Essing]

Our Intermediate-CA "Deutsche Telekom AG Issuing CA 01“; Serial number: ee: db: 86: 0e: 52: 3b: 2e: 43" was not yet audited and for this reason it was not disclosed.
In the beginning of 2018, after we were unable to put our technical and name constraint Intermediate-CA "Deutsche Telekom AG secure email CA" 75:81:aa:9f:98:30:a3:ab:bf:5b:b6:9f:84:d8:56 into operation as planned, we decided to have both CAs audited so that we can publish them in the CCADB.
We successfully completed the audit on November 27, 2018. The audit covers the period from November 28, 2017 to November 27, 2018. As soon as we have the Certificate and the Browser Attestation for our Intermediate-CA "Deutsche Telekom AG Issuing CA 01“; Serial number: ee: db: 86: 0e: 52: 3b: 2e: 43", we will post them in the CCADB.

Comment 15

[Ryan Sleevi]

I'm going to defer this to Wayne, to see if there's a better way of phrasing the question.

I don't believe this really inspires confidence; it sounds like the decision not to disclose was that you created an intermediate that was not yet audited. Yet Mozilla policy is fairly clear on the expectations and timing from disclosure. It also doesn't sound like there have been meaningful process improves to correct this going forward, likely because it sounds like there's still some confusion about the expectations. That this intermediate was 'deactivated' after the report is similarly... concerning.

Comment 16

[Wayne Thayer]

I'm guessing that T-Systems assumed that they needed to complete the audit information in CCADB as part of the disclosure?

Arnold: is that correct? If so, why was that incorrect assumption made? The "Deutsche Telekom AG Issuing CA 01“; Serial number: ee: db: 86: 0e: 52: 3b: 2e: 43" is marked in CCADB as "audit same as parent", so it would have been easy enough to disclose it within one week of issuance in compliance with policy section 5.3.2.

Also, as Ryan asked in comment #13, I am still waiting for an explanation of what T-Systems has done / will do to ensure that Mozilla requirements, including intermediate disclosure, are met in the future.

Comment 17

[Arnold Essing]

Ryan, you probably misinterpreted our description. We did not recreate the intermediate "Deutsche Telekom AG Issuing CA 01". The intermediate "Deutsche Telekom AG Issuing CA 01" existed for a long time and was technically limited. At the beginning of 2018 we wanted to exchange the certificate of the "Deutsche Telekom AG Issuing CA 01" for a new certificate which contains the additional domain limitation which is in accordance with the Mozilla Root Policy. For technical reasons the exchange could not be implemented. Since we could not withdraw the certificate of the "Deutsche Telekom AG Issuing CA 01", we realized that we had to enter the "Deutsche Telekom AG Issuing CA 01" in the audit process, even though we discontinued issuance of certificates by this CA in mid-April. As you know, the preparation of an audit is a very busy process, which then, including the audit itself, lasted until November 27, 2018. We expect to receive the Audit results documents in the beginning of February.

Wayne, after the discussion with you (Comment # 6 and Comment # 7), we inserted the "Deutsche Telekom AG Issuing CA 01" in the CCADB. Yes, it is true that we were indeed uncertain as how to disclose the "Deutsche Telekom AG Issuing CA 01" without already having the audit results. When entering the data into the CCADB, we received an error message if the marker was not set to "audit same as parent" and we could not yet enter the Audit data as the Audit was not completed until November 27, 2018. For this reason, the marker was set incorrectly. We would like to emphasize here that our intention was merely the registration of the CA in the CCADB and was under no circumstances an intent to deceive or cover anything up. We have now deleted the marker in the CCADB and will add the pending data immediately upon receipt of the Audit documents.

To ensure that Mozilla requirements, including intermediate disclosure are met in the future, we replaced the staff responsible for the root program at T-Systems. The work guidelines for creating new Intermediate-CAs or changes at existing Intermediate-CAs are being taught in dedicated sessions. We have enhanced our internal audit process with regular and event-based inspections of the accuracy and consistency of the CCADB. Changes in the CCADB are made on the principle of dual control (4-eyes principle).

Comment 18

[Ryan Sleevi]

Arnold, thanks for clarifying that there's some confusion around your description.

I think it'd be helpful to revisit the timeline in Comment #12 if you believe there are additional details. For example, spelling out when you created the first version of this intermediate (technically constrained or otherwise) to your deactivation event.

I have trouble understanding the explanations provided on this issue and the related facts. For example, serial ee:db:86:0e:52:3b:2e:43 ( https://crt.sh/?sha256=c6193077c6189d1dfbbf813b87dc7cbf0498acf727887bc7ec54320906de9bc8 ) was first seen on 2016-10-05, judging by CT logs. There were three other certificates ( https://crt.sh/?id=19542940 , https://crt.sh/?id=12722136 , https://crt.sh/?id=12624834 ) that share that same subject and SPKI, and none appear to be technically constrained.

So I'm trying to understand the timeline of events from 2016-10-05 to 2018-04-18, and comment #12 is quite confusing to understand.

Comment 19

[Arnold Essing]

Ryan, you are right. Here is the timeline for the intermediate certificate „Deutsche Telekom AG Issuing CA 01“, serial number „‎00 ee db 86 0e 52 3b 2e 43“.

The internal Company PKI (cPKI) of Deutsche Telekom issues exclusively email certificates. The CA is technical constraint on email certificates for captive domains only.

2016-07-13 14:51:48 UTC: Generation of the intermediate certificate „Deutsche Telekom AG Issuing CA 01“, serial number „‎00 ee db 86 0e 52 3b 2e 43“ and the EKU E-Mail Protection.

2016-11-08 21:47:25 UTC: first issuing of an ee-certificate from the intermediate „Deutsche Telekom AG Issuing CA 01“, serial number „‎00 ee db 86 0e 52 3b 2e 43“

2017-11-29 15:27 UTC: First information to Mozilla root program that we might have problems to fulfill the requirements for name constraints with the intermediate „Deutsche Telekom AG Issuing CA 01, serial number „‎00 ee db 86 0e 52 3b 2e 43“.

2018-01-10 09:30:00 UTC: The exchange of the intermediate „Deutsche Telekom AG Issuing CA 01, serial number „‎00 ee db 86 0e 52 3b 2e 43“ with a new intermediate with name constraints was denied due to technical reasons. Management decision to build up a new PKI in a different datacenter.

2018-01-11: Begin with preparation of auditing the intermediate „Deutsche Telekom AG Issuing CA 01, serial number „‎00 ee db 86 0e 52 3b 2e 43“ and begin planning to build up a new PKI.

2018-01-18 11:04:58 UTC: generation of the intermediate certificate „Deutsche Telekom AG secure email CA“, serial number „75 81 aa 9f 98 30 a3 ab bf 5b b6 9f 84 d8 56“ and the EKU E Mail Protection and name constraints for the new PKI.

2018-04-18 18:03:02 UTC: last issuing of an ee-certificate from the intermediate „Deutsche Telekom AG Issuing CA 01“, serial number „‎00 ee db 86 0e 52 3b 2e 43“ from then on only issuing revocation list.

2018-04-18 18:14:04 UTC: first issuing of an ee-certificate from the intermediate „Deutsche Telekom AG secure email CA“, serial number „75 81 aa 9f 98 30 a3 ab bf 5b b6 9f 84 d8 56“.

From 2018-04-23 to 2018-04-26: First onsite meeting with the auditor.

2018-06-29: Accidentally incorrect entry of the of the intermediate certificate „Deutsche Telekom AG Issuing CA 01“, serial number „‎00 ee db 86 0e 52 3b 2e 43” into the CCADB.

2018-11-27: Audit completed successfully. The audit period is from 28.11.2017 to 27.11.2018.

2019-01-23: Issuing of the certification document for the intermediate certificate „Deutsche Telekom AG Issuing CA 01“, serial number „‎00 ee db 86 0e 52 3b 2e 43”.

Comment 20

[Ryan Sleevi]

Wayne: This issue has languished a bit, but I'm not terribly confident that we're making progress on understanding the systemic issues or what's being taken to address it, going back to the 11 months ago in Comment #0. Comment #17 has the mitigations - namely, an entirely new team is now in place - and while I'm not entirely confident that this will systemically address the issue, I also suspect that the consequence is that we will not be able to understand more about the failures that lead to this.

Comment 21

[Wayne Thayer]

Arnold: please confirm that T-Systems understands that all new unconstrained intermediate certificates must be disclosed in CCADB within one week of issuance, regardless of the availability of an audit, to comply with Mozilla policy section 5.3.

Comment 22

[Arnold Essing]

Hello Wayne, Yes, T-Systems clearly understands that all new unconstrained intermediate certificates must be disclosed in the CCADB within one week of issuance, regardless of the availability of an audit.

Comment 23

[Wayne Thayer]

Arnold" thank you for the response. I believe this issue is now resolved.