1507376 - TÜViT: Issues with T-Systems Audits

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Wayne Thayer]

T-Systems was found to have misissued 100% of their EV certificates beginning in January 2018, as documented in bug #1498463. Note that there was some initial confusion about the bad encoding being a BR violation.

The attestation statement from TÜViT covering the period from May 18, 2017 to May 17, 2018 [1] did not report this issue. Further, given that T-Systems continued to issue these certificates until notified via the mozilla.dev.security.policy mailing list, it is apparent that T-Systems did not detect the problem, a fact that they have since confirmed.

In response to a complaint made by Ryan Sleevi, TÜViT declined to revoke T-Systems' certificate on the grounds that the bad encoding was a minor non-conformity, and thus the CA has 3 months to correct it.

Ryan then posted a description of the issue and his concerns with TÜViT [2] to the mozilla.dev.security.policy mailing list. TÜViT also posted a response.

This bug is for documentation purposes. Representatives of TÜViT are welcome, but not required to comment.

[1] https://www.tuvit.de/fileadmin/Content/TUV_IT/zertifikate/en/AA2018072004_Audit_Attestation_E_T-TeleSec-GlobalRoot-Class-3_20180723_s.pdf
[2] https://groups.google.com/d/msg/mozilla.dev.security.policy/Q9whve-HJfM/RIrLjzruBgAJ

Comment 1

[Matthias Wiedenhorst]

Thanks Wayne. I am going to provide a detailed incident report as soon as possible, hopefully not later than next Monday (2018-11-26).

Comment 2

[Matthias Wiedenhorst]

Remark: The term “certificate” has a double meaning in this context, unfortunately. It can either mean “X.509 certificate” as issued by the TSP or “ETSI certificate” as issued by certification bodies like TÜViT. I hope that context makes clear which of both is meant in the different places.

1. How TÜViT first became aware of the problem 
On 2018-10-15, 05:42 UTC Mr. Ryan Sleevi sent a complaint by email.

2. A timeline of the actions you took in response
In January 2018, T-Systems began issuing certificates with the incorrect qcStatement.
In the time periods of April 23-26, 2018  and May 22-24, 2018 the yearly audit was performed but the incorrect qcStatement was not detected.
2018-10-15, 05:42 UTC: Mr. Ryan Sleevi sent a complaint by email, notifying us about the problematic certificates and the audit problem.
2018-10-16, 08:05 UTC: Email from TÜViT to T-Systems informing them of the complaint and requesting information regarding the matter.
2018-10-16, 08:08 UTC: Email from T-Systems informing TÜViT about the incident and providing the link to the corresponding Bugzilla entry (https://bugzilla.mozilla.org/show_bug.cgi?id=1498463).
Note: We have reason to believe that this mail has been written in parallel to our mail, not as a response to our mail from 3 minutes before.
2018-10-16, 11:25 UTC: Response from T-Systems to TÜViT’s mail from 08:05, providing further info on the nature of the bug.
2018-10-19, 08:30 UTC: Telco with T-Systems, explanation and discussion of the incident, confirmation that T-Systems had stopped issuance on 2018-10-04.
Subsequent check on crt.sh, which showed no certificates issued between 2018-10-04 and 2018-10-15, verification that T-System began to issue corrected certificates on 2018-10-16.
=> Preliminary classification of the qcStatement issue by TÜViT as nonconformity type B (as no immediate security risk had been identified, the CA had stopped issuance and had already deployed a fix).
=> Preliminary classification of the failure to revoke as nonconformity type B (due to the classification of the triggering issue and due to the explicit mention of the possibility to delay revocation at https://wiki.mozilla.org/CA/Responding_To_An_Incident#Revocation).
Note: Non-conformity type B is a pending non-conformity for which corrective actions have to be implemented within 3 month according to ETSI EN 319 403, 7.6.

2018-10-25, 13:00 UTC: Conference call with T-Systems and Supervisory Body (SB) “Bundesamt für Sicherheit in der Informationstechnik”. SB acknowledged the classification as non-conformity type B. SB explicitly pointed out that remediation / revocation plan shall also take into account availability issues for the affected customers and that from their point of view, a 3-month revocation timeline would be sufficient. T-Systems stated that they plan to implement a significantly faster timeline, not using the full 3 month.
=> Classification of both issues (qcStatement and revocation) by TÜViT as nonconformity type B.
2018-10-26: A re-check of the audit trail of all certifications issued by TÜViT for QCP-w policy has been performed. No other errors with the qcStatement have been detected.
2018-10-31, 13:00 UTC: Telco with T-Systems about status of remediation. Info that it has been decided to revoke all remaining affected certificates no later than 2018-11-12.
2018-11-06, 10:30 UTC: Telco with T-Systems about status of remediation. Confirmation by T-Systems that the last remaining affected certificates will be revoked on 2018-11-12.
2018-11-12, 13:00 UTC: Last remaining affected certificates have been revoked by T-Systems.
Scheduled for 2018-11-27: Audit of the relevant parts of the TSP with regard to the non-conformities and issuance of a corresponding audit attestation. 

3. Whether you have stopped, or have not yet stopped, issuing certificates with the problem. 
The ETSI certificates / audit attestations listed under point 5 were the only issued certificates with this problem. 

4. A summary of the problematic certificates. 
ETSI certification issued to T-Systems according to policy EVCP and QCP-w, certificate numbers TUVIT-CA67110 and TUVIT.97111.TSP as well as the corresponding audit attestation.
No other certifications issued for policy QCP-w have been found to be affected.

5. The complete certificate data for the problematic certificates. 
https://www.tuvit.de/fileadmin/Content/TUV_IT/zertifikate/en/67110UE_s.pdf
https://www.tuvit.de/fileadmin/Content/TUV_IT/zertifikate/en/97111UE_s.pdf
https://www.tuvit.de/fileadmin/Content/TUV_IT/zertifikate/en/AA2018072004_Audit_Attestation_E_T-TeleSec-GlobalRoot-Class-3_20180723_s.pdf

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
During audits, certificates are checked using the possibilities of crt.sh. A number of certificates is chosen from the CA’s database of issued certificates. For these certificates, the evidence trail generated during application, registration and issuance is inspected. In addition, these certificates are identified on crt.sh and the available linters are applied. 
This was performed as random sample. In case of the questioned audit the initial sample size was 5 certificates. As no problems had been identified with these certificates (neither in the records nor on crt.sh) it was decided to use this sample.
It was known, that crt.sh would not check the qcStatement at all. Hence, the auditor was required to perform a manual check of it. Therefore, one certificate was opened in an ASN.1 Editor and it was looked for the OIDs “0.4.0.1862.1.1”, “0.4.0.1862.1.5” and “0.4.0.1862.1.6.3”. It was also ensured that the OID “0.4.0.1862.1.4” was not present, as it would not have been allowed in such a certificate.
The error occurred, as the auditor accepted the “0.4.0.1862.1.6.3 “ without realizing that it was self-standing and not as a value under “0.4.0.1862.1.6“ as it would have been correct. 

7. List of steps you are taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when you expect to accomplish these things.
We are strongly recommending the implementation of a pre-issuance linting to customers since mid-2017 and we will continue to do so.
Re-Training of the auditors regarding the qcStatements was done on 2018-10-26.
We are going to require a crt.sh check of all certificates issued within the audit period and known to crt.sh against both available linters in order to identify profile issues within the complete set of issued certificates rather than relying on a random sample. Due to CT requirements, this should cover all BR / EV relevant certificates. Work instructions are planned to be distributed to all auditors by the end of November.
In addition, we are evaluating the possibilities for a linting solution separate from crt.sh.

Comment 3

[Wayne Thayer]

Matthias: can you provide an update on the status of the actions described in your response to question 7?

Comment 4

[Matthias Wiedenhorst]

Yes, sure. Sorry for delay.

Auditors have been instructed to perform crt.sh checks as described above in a team meeting on December 6th, 2018.
Distribution of the written procedure has been delayed due to holiday season, but have finally been sent to all auditors on January 8th, 2019.

The research for additional linting possibilities has not yet resulted in anything worth reporting.

Comment 5

[Wayne Thayer]

Thanks Matthias.

Ryan - do you have any further questions or updates to provide based on your inquiries to the Supervisory and National Accreditation bodies?

Comment 6

[Ryan Sleevi]

I have yet to receive the "Final" response from the Conformance Assessment Body regarding this incident.

Comment 7

[Ben Wilson]

Hello Matthias,
Is there a final statement from TUVit?
Thanks,
Ben

Comment 8

[Matthias Wiedenhorst]

Hello Ben,
the final response of the Conformity Assessment Body regarding his complaint has been sent to Ryan on 2019-01-29.
Best regards
Matthias

Comment 9

[Ben Wilson]

Closing this bug based on response in Comment 8 above.