1498463 - T-Systems: Improperly encoded QCStatements extension

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Wayne Thayer]

The following issue was reported on the mozilla.dev.security.policy mailing list:

A number of Qualified Web Authentication Certificates have been issued
with incorrect qcStatements encoding. A small survey displays that all
certificates issued by a specific SubCA are affected by this issue
(https://crt.sh/?CN=%25&iCAID=1481). The CA has been notified about
this, but more than a week has passed and it has not yet provided any
feedback, while it continues to issue such malformed certificates (e.g.
https://crt.sh/?id=816495298).

Technical details
-----------------

According to ETSI EN 319 412-5 (Electronic Signature and Infrastructure
(ESI); Certificate Profiles; Part 5: QCStatements) section 4.2.3
(QCStatement claiming that the certificate is a EU qualified certificate
of a particular type), the QCStatement QcType with OID
id-etsi-qcs-QcType (0.4.0.1862.1.6) declares that a certificate has been
issued for a particular purpose (e-sign, e-seal, qualified web
authentication certificate). Every certificate containing this
QCStatement must have a SEQUENCE OF OBJECT IDENTIFIER which declares the
purpose, e.g. id-etsi-qct-web (0.4.0.1862.1.6.3).
T-Systems International GmbH has failed to follow this specification,
and instead issues certificates having id-etsi-qct-web as a QCStatement.
Such a certificate can be found at https://crt.sh/?asn1=795148644. You
can compare this with https://crt.sh/?asn1=844599393 which has the
QcType QCStatement correctly encoded.

Disclosure to CA timeline
-------------------------

- 2 October 2018: First notification to the CA, with a detailed
description of the issue.
- 2 October 2018: Reply by a CA representative that they will look at it.
- 8 October 2018: Second notification and request for feedback.

No further communication has taken place.

This is a violation of the BR requirement to adhere to RFC 5280 and Mozilla policy section 5.2.

Failure to revoke these certificates within 24 hours of the notification is a violation of BR section 4.9.1.1(9)

Please provide an incident report for this problem, as described here:
https://wiki.mozilla.org/CA/Responding_To_A_Misissuance#Incident_Report
The incident report should be posted to the mozilla.dev.security.policy forum and added to this bug.

Comment 1

[Arnold Essing]

After receiving notification of a possible error and after the first analyzation of the error, we classified the error as minor and not security crucial.  The error does not present a security problem, nevertheless issuing of these certificates has been deactivated as of October 4, 2018.  The detailed analyzation confirmed our classification.  As part of our company internal incident process, the error has been corrected and an incident report has been created.  From our perspective, the error is not in violation of the Baseline Requirements or Mozilla Root Store Policy, therefore a Mozilla-Bug was not created.  We would like to explicitly thank Mr. Fotis Loukos for pointing out this error.  The error was not detected despite the creation of templates using the 4-eyes principle, the tests and the QS performed by the external auditor.  The affected customers will be informed.  Therefore we kindly ask you to close this bug.

Comment 2

[Ryan Sleevi]

(In reply to Arnold Essing from comment #1)
> After receiving notification of a possible error and after the first
> analyzation of the error, we classified the error as minor and not security
> crucial.  The error does not present a security problem, nevertheless
> issuing of these certificates has been deactivated as of October 4, 2018. 
> The detailed analyzation confirmed our classification.  As part of our
> company internal incident process, the error has been corrected and an
> incident report has been created.  From our perspective, the error is not in
> violation of the Baseline Requirements or Mozilla Root Store Policy,
> therefore a Mozilla-Bug was not created.  We would like to explicitly thank
> Mr. Fotis Loukos for pointing out this error.  The error was not detected
> despite the creation of templates using the 4-eyes principle, the tests and
> the QS performed by the external auditor.  The affected customers will be
> informed.  Therefore we kindly ask you to close this bug.

This does not meet the expectations for a CA incident report.

At minimum, you cannot acknowledge this as an incident while also failing to provide an incident report. More significantly and more troubling, however, is the statement that

> From our perspective, the error is not in violation of the Baseline Requirements or Mozilla Root Store Policy, therefore a Mozilla-Bug was not created.

As described in https://groups.google.com/d/msg/mozilla.dev.security.policy/x3s3CSKdIlY/o-oBBD7eAgAJ , if you are positing an alternative explanation, please provide clear textual citations as to why you believe this. Otherwise, please provide an incident report in the format requested and as expected.

> The error was not detected despite the creation of templates using the 4-eyes principle, the tests and the QS performed by the external auditor.

This significantly calls into question the competency of the auditor and the auditor's certification under ETSI EN 319 403.

1) Have you reported this incident to your auditor, consistent with the certification scheme used?
2) Have you reported this incident to your supervisory body?
3) When did you report these incidents? Using the timeline of the incident report, please include these events, if they occurred, in the timeline of events?
4) Has there been any determination by the accrediting CAB to suspend and/or terminate the certification of your CA?

Comment 3

[Arnold Essing]

Incident report:

1.	How your CA first became aware of the problem.
	
	Tuesday, October 2, 2018 at 5:55 p.m. CET:	Our hotline received an email from Mr. Fotis Loukos 

2.	A timeline of the actions your CA took in response.

	Tuesday, October 2, 2018 at 7:36 p.m. CET:	Replied to email from Mr. Loukus.
	Tuesday, October 2, 2018			An investigation was launched. 
	
	The analysis revealed: 
	- that no attack potential can be derived from the incorrect coding of the QC statement. 
	- it is not relevant to BR 4.9.1.1 or Root program requirements.  
	- Internal lint check revealed no error message.
	- External lint check (crt.sh) revealed no error message
	- Throughout the period there was no usage of the feature by the customer and thus no risk
	- We are not aware of any problems with any client applications 

	Therefore, we came to the conclusion that there is no reason to revoke these certificates within the time period of the BR requirements.

	Thursday, October 4, 2018 at 4:00 p.m. CET:	Due to our internal quality rules we stopped the issuance of EV-certificates. 
	Friday, October 5, 2018				Bugfix was commissioned.
	Thursday, October 11, 2018:			Bugfix was delivered and test phase started.
	Friday, October 12, 2018:			Bugzilla Incident was opened. 
	Friday, October 12, 2018:			First response posted in Bugzilla. 
	Monday, October 15, 2018:			During test phase two additional erroneous certificates were issued and immediately disabled.
	Monday, October 15, 2018:			Implementation of final configuration. 
	Tuesday, October 16, 2018:			Supervisory body and auditor were informed.
	Tuesday, October 16, 2018:			Restart of regular operation. 

3.	Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem.
 	
	Thursday, October 4, 2018 at 4:00 p.m. CET:	Issuance of certificates were stopped.


4.	A summary of the problematic certificates.
	
	- Number of certs:		218 affected EV-certificates 
	- Date of first certificate:	January 17, 2018
	- Date of last certificate:	October 4, 2018
	
	- Number of certs:		2 affected and immediately revoked certificates during testing
	- Date of first certificate:	October 15, 2018
	- Date of last certificate:	October 15, 2018



5.	The complete certificate data for the problematic certificates.	The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.
	
	See attached csv-file


6.	Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

	There was a mistake in the software process while creating the ASN.1 structure. The underlying specification was erroneous because we misinterpreted the ETSI specification.  After the implementation of the feature, we checked it against our incorrect specification and therefore could not detect the error.  This feature was first activated on January 17, 2018.  It was not detected by any customers because no customer needed this feature.  The current lint tools do not check for this feature.  Even the auditor did not notice the error.
	

7.	List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

	- We have corrected our specification according to the ETSI guideline ETSI EN 319 412-5.
	- The CA software was developed according to our specifications.
	- The new software is in active operation since October 16, 2018. For this reason the error will no longer occur.
	- We are in close contact with the auditor and supervisory body to clarify how to deal with the affected certificates going forward.

We wish to thank Mr. Fotis Loukos  for bringing this problem to our attention.

Comment 4

[Arnold Essing]

Attachment: T-Systems_certificates.csv

Comment 5

[Ryan Sleevi]

Thank you for adopting the error reporting template and providing more details. I have further questions inline.

(In reply to Arnold Essing from comment #3)
> Incident report:
> 
> 1.	How your CA first became aware of the problem.
> 	
> 	Tuesday, October 2, 2018 at 5:55 p.m. CET:	Our hotline received an email
> from Mr. Fotis Loukos 
> 
> 2.	A timeline of the actions your CA took in response.
> 
> 	Tuesday, October 2, 2018 at 7:36 p.m. CET:	Replied to email from Mr. Loukus.
> 	Tuesday, October 2, 2018			An investigation was launched. 

Your reply identifies that for the period of 2018-01-17 until this incident, your system was actively misissuing certificates and you were violating your CP/CPS and the requirements of the relevant accreditation schemes.

As part of understanding how this issue happened, so that the ecosystem can be improved, it's important to better understand what controls existed and failed.

For example, when was this configuration profile introduced, what was the process for reviewing it, why did that process fail to detect the material and significant non-conformity?

Similarly, when your CAB performed an assessment against ETSI EN 319 411-2, why did they fail to detect the material non-compliance? What steps were taken with the CAB to demonstrate your compliance with this scheme, why did those steps fail to detect the issue, and what proposals for the future to ensure more reliable detection of 'similar' issues?

Similarly, this violates your CP/CPS. What steps are taken to ensure compliance with your CP/CPS, why did those steps fail, and what steps are being take to detect and mitigate future issues?

Understanding concrete timelines associated with these sorts of events - when profiles were introduced, when assessments were performed, what evidence was examined, and what secondary controls such as review were exercised, can all help to build confidence in the operation of T-Systems.

As it stands, this is such a significant and material non-conformity that it raises question about the entire process - both from the TSP side and the CAB side. The incident report is a means of demonstrating that the TSP and the CAB took reasonable steps, as judged by the community, to reduce such risks, as well as to demonstrate a holistic evaluation of why those steps failed and how they can be systemically improved.

> 	The analysis revealed: 
> 	- that no attack potential can be derived from the incorrect coding of the
> QC statement. 

Do you agree or disagree that CAs that issue certificates in material non-conformance with the audit schemes, and in clear violation of their CP/CPS, present an immediate and significant risk to the CA ecosystem?

> 	- it is not relevant to BR 4.9.1.1 or Root program requirements.  

Do you agree or disagree that such certificates fail to adhere to ETSI EN 319 412-5, thus fail to adhere to your CP/CPS, and thus MUST be revoked consistent with 4.9.1.1 (9)?

> 	- Internal lint check revealed no error message.

What internal lint process do you use? What steps have you taken to ensure this incident is identified in the future by the internal lint check? What steps have you taken to review why these checks did not already exist with your internal lint check? What steps have you taken to ensure that all of the internal lint checks are conformant with the BRs?

> 	- External lint check (crt.sh) revealed no error message

What steps have you taken to remediate this?

> 	- Throughout the period there was no usage of the feature by the customer
> and thus no risk

There is a substantive difference between no usage and no risk. How has T-Systems determined that no customer relied on these certificates complying with ETSI EN 319 412-5 and ETSI EN 319 412? Given the compatibility risk these certificates pose, to what end does T-Systems claim there is no risk?

> 6.	Explanation about how and why the mistakes were made or bugs introduced,
> and how they avoided detection until now.
> 
> 	There was a mistake in the software process while creating the ASN.1
> structure. The underlying specification was erroneous because we
> misinterpreted the ETSI specification.  After the implementation of the
> feature, we checked it against our incorrect specification and therefore
> could not detect the error.  This feature was first activated on January 17,
> 2018.  It was not detected by any customers because no customer needed this
> feature.  The current lint tools do not check for this feature.  Even the
> auditor did not notice the error.

What caused the mistake to be made? How can such mistakes be prevented in the future? The incident response process tries to address these. It's not clear why the software process failed, and thus not clear why it's reasonable to believe it will not fail again. It's not clear why the auditor did not notice the error, and thus not clear why it's reasonable to believe the auditor will notice other material non-conformities in the future.

Comment 6

[u616638]

6.	Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

In the summer of 2016 we created the specifications for software development in accordance to ETSI EN 319 412-5.

In this specification, the optional mapping of the QCtype was incorrectly defined, so that instead of the QCstatement QCtype (0.4.0.1862.1.6) the QCtype id-etsi-qct-web (0.4.0.1862.1.6.3) was encoded.  
This error occurred while writing the development specification.

The initiated investigation of the process revealed that when the specification was created, the assessment of the planned customization (security risk, criticality, damage potential, complexity, compatibility effects, etc.) 
was assessed as low overall. The optional inclusion of further technical support was therefore considered not necessary.

For the reasons stated, the specification was therefore released by the standard review process.

In the further course the finished software development was tested against the faulty specification. Therefore, the error was also not detected by our acceptance tests. Because of the faulty specification the correct QCstatement is also absent from the relevant CP/CPS. 

The internally used lint tools (CERTlint, CABlint, zLint) in the acceptance tests do not evaluate the QC statement and thus could not detect this error.

On January 16, 2018 (after the registration of our SubCA on November 23, 2017 in the EU Trusted List), we activated the QC feature as part of our change process.  
Naturally, our tests showed no deviation from the unfortunately faulty specification.

7.	List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, 
        accompanied with a timeline of when your CA expects to accomplish these things.

As we have outlined, the bug in the specification, CP/CPS and software has been corrected since October 16, 2018.

To avoid future errors of this type, we put together an entire package of measures:

- We have expanded our internally used lint-tools to check the QCstatement.
- A ZLint module that checks for the QCstatement is in planning and will be provided for the community.
- We have now extended our review process for CA documents for development specifications.
  It is essential for the rating of the planned changes and development contracts.
  The release requires further technical assessment immediately (for example by cryptography-specialist).
  The corresponding working instructions will be updated latest by November 15, 2018.
- We have extended our internal quality rules to the effect that in future adaptations of certificate profiles, a corresponding 
  test option (generally via a lint tool) must always be present in the issuing process of a certificate.
- We started to exchange the affected certificates on October 19, 2018, although there has not yet been a formal stipulation 
  imposed by the state supervisory authority concerning the certificates. The customers were requested in writing to immediately 
  renew and revoke the respective certificates.

Comment 7

[Arnold Essing]

7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

Update:
- As of Monday November 12, 2018 13:00 UTC all affected certificates have been revoked.
- We have updated our corresponding working instructions.

Comment 8

[Wayne Thayer]

Please update this bug when the ZLint module referenced in comment 6 is completed and linting T-Systems certificates.

Comment 9

[paul.leo.steinberg]

(In reply to Arnold Essing from comment #4)
> Created attachment 9017850 [details]
> T-Systems_certificates.csv

Quite a few of these certificates (3fcfcaf480d8cfa33454cb711bcffed50b3a5a1c, 8fb46aafa3c806dfefdaa95b571991b40899862e, 79332eabb32d4cfd0622c9f51063ddf377c121f1, 342bb872de3252044ae50002fdb5f0f14b376194, possibly others, too) are neither in crt.sh nor in censys.io.

How should anyone trust you with certificates (let alone extended validation) if you cannot even write an incident report with the necessary diligence?

Comment 10

[Arnold Essing]

Please have a look at https://wiki.mozilla.org/CA/Communications -> 2.1  Sept. 2018 Responses ->  Responses to Action 7.  Therefore are some certificates not in crt.sh.

Comment 11

[paul.leo.steinberg]

(In reply to Arnold Essing from comment #10)
> Please have a look at https://wiki.mozilla.org/CA/Communications -> 2.1 
> Sept. 2018 Responses ->  Responses to Action 7.  Therefore are some
> certificates not in crt.sh.

And you please have a look at <https://wiki.mozilla.org/CA/Responding_To_An_Incident#Incident_Report>: "complete certificate data for the problematic certificates". You MUST disclose all problematic certificates as there's no use in telling people that you misissued certificates without letting them see what was wrong. You cannot asking for a general absolution without this (and that you are trying, is quite telling about your understanding of a transparent process)!

The information you provided does not even allow anyone to verify that you have actually revoked these certificates (judging from the number of revocations on your CRL in the given timeframe, one might come to the conclusion that you have, in fact, not revoked all certificates!).

Comment 12

[Arnold Essing]

Attachment: T-Systems_certificates_20181130.csv

Comment 13

[Arnold Essing]

Attachment: T-Systems_certificates_ECC-root.zip

Comment 14

[Arnold Essing]

Enclosed the information to all 220 problematic certificates. Four Test certificates are attached separately in a zip-file, these are not included in crt.sh because their ECC root has not yet been published.

Comment 15

[Ryan Sleevi]

I may be missing something, but from the CSV attached, it still seems some certificates are not disclosed.

Hashes such as:
39da03bf1924e15f0a61ece7fac421de293b9fcf
9041b34d222ec763ae978dbc317964c1fbe67ede
790c326430dd5c7e089bd82c96422bfb09048dc4
b65071076f483ce0d18ce0acbbb99d6dcf9ec6b6
b85c991db9f668211e6de546d66c9d606a6e681a

These also don't seem to line up with the test certificates. Am I holding something wrong?

Comment 16

[Arnold Essing]

Attachment: T-Systems_certificates_20190107_orig.csv

Comment 17

[Arnold Essing]

Not all Leaf-Certificates are available in crt.sh, so we added the missing PreCert-IDs in the updated CSV (Comment 16).

Comment 18

[Ryan Sleevi]

Arnold: Thanks for providing the latest CSV

In trying to understand the delta between the CSVs in Comment #4, Comment #12, and Comment #16 (which I'll refer to as CSV's 1/2/3)

Present in 1, not present in 2/3:
A) 1e9dab9b8d2849d9b1bf430f6f6278c1ef618610
B) 371e9dcd5c5b1b0b11c4ab16fd2733dd1c718951
C) 584b74e2bdb5b07cb12a1d1058512e935d066f85
D) 79332eabb32d4cfd0622c9f51063ddf377c121f1
E) 8633e7734cc0fa6ee3c1fb5829e62d1dcf452246
F) a3d7c0308441cd2d5887ff5cb3c49e2cec0fac7 (probably omitted the leading 0?)
G) cd5511105e9892df0ac9f2d8c0d413ae916ecaba
H) dd3d8bbf7a4520fc70e470e0c03e33d3ced25192

Present in 2/3, not present in 1:
I) cd5511105e9892df0ac9f2d8c0d413ae916ecaba
J) 2be181b4aae000e5038d7412258057a59a01eb50
K) df7038bba2d53ca0e58de8a16daa93c6f0f28967

When trying to understand this a bit more:
F and I are likely the same cert; F being the precert hash (typo'd), I being the full cert
J and E are likely the same cert; E being the precert hash, J being the full cert
K and A are likely the same cert; A being the precert hash, K being the full cert

So what I'm not sure of is:
What's up with B, C, D, G, and H? Why are they seemingly not present in the new CSV? If this has already been answered in your data, hoping you can just point it out. I'm wanting to make sure I'm not missing anything in this disclosure.

Comment 19

[Arnold Essing]

B) 371e9dcd5c5b1b0b11c4ab16fd2733dd1c718951
-> Hash of test certificate “servpass-259305.G2.pem.cer”. It can be found in attachment “T-Systems_certificates_ECC-root.zip”.

C) 584b74e2bdb5b07cb12a1d1058512e935d066f85
-> Hash of test certificate “servpass-259299.G2.pem.cer”. It can be found in attachment “T-Systems_certificates_ECC-root.zip”.

D) 79332eabb32d4cfd0622c9f51063ddf377c121f1
-> Hash of test certificate “servpass-262155.G2.pem.cer”. It can be found in attachment “T-Systems_certificates_ECC-root.zip”.

H) dd3d8bbf7a4520fc70e470e0c03e33d3ced25192
-> Hash of test certificate “servpass-262154.G2.pem.cer”. It can be found in attachment “T-Systems_certificates_ECC-root.zip”.

The four test certificates have been attached separately in the zip-file ‘T-Systems_certificates_ECC-root.zip’ with comment 13. They are not included in crt.sh because the ECC root has not been published yet. That’s why we removed the four certificates from attachments 2 and 3.

A) 1e9dab9b8d2849d9b1bf430f6f6278c1ef618610
K) df7038bba2d53ca0e58de8a16daa93c6f0f28967
-> yes, K and A are the same cert; A is the precert crt.sh ID=863797957, K is the full cert

E) 8633e7734cc0fa6ee3c1fb5829e62d1dcf452246
J) 2be181b4aae000e5038d7412258057a59a01eb50
-> yes, J and E are the same cert; E is the precert crt.sh ID=863797726, J is the full cert

F) a3d7c0308441cd2d5887ff5cb3c49e2cec0fac7
-> Present in all attachments 1/2/3
-> yes, we omitted the leading 0 in attachment 1 [details] [diff] [review]
-> precert crt.sh ID=596346268, F is the full cert crt.sh ID=613080100
-> Hash of the precert 333483232F5402AFE40A5C4633C9ED72E2EE3A98

I) cd5511105e9892df0ac9f2d8c0d413ae916ecaba
G) cd5511105e9892df0ac9f2d8c0d413ae916ecaba
-> Present in all attachments 1/2/3 (with leading whitespace in attachments 2/3)
-> precert crt.sh ID=368730770, I and G are the full cert crt.sh ID=380760128
-> Hash of the precert 09A3217FB1EEE92C9ABFD6183473BA9DEDC87B95

Comment 20

[Arnold Essing]

7.List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

Update:
Our pre-issuance linting tool including QCStatement checks was set up for the Intermediate CA TeleSec ServerPass Extended Validation Class 3 CA.

Comment 21

[Ryan Sleevi]

Is this in reply to Comment #8? Are there any other details to add before I summarize this issue?

Comment 22

[Arnold Essing]

Yes, Comment #20 was in reply to Comment #8.
Publication of a zlint module is currently in preparation.
No, there are no other details to be added.

Comment 23

[Arnold Essing]

We have created the pull request for the publication of a zlint module. https://github.com/zmap/zlint/pull/250

Comment 24

[Wayne Thayer]

It appears that all questions have been answered and remediation steps completed.