1391074 - T-Systems: Non-BR-Compliant Certificate Issuance

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Kathleen Wilson]

The following problems have been found in certificates issued by your CA, and reported in the mozilla.dev.security.policy forum. Direct links to those discussions are provided for your convenience.

To continue inclusion of your CA’s root certificates in Mozilla’s Root Store, you must respond in this bug to provide the following information:
1) How your CA first became aware of the problems listed below (e.g. via a Problem Report, via the discussion in mozilla.dev.security.policy, or via this Bugzilla Bug), and the date.
2) Prompt confirmation that your CA has stopped issuing TLS/SSL certificates with the problems listed below.
3) Complete list of certificates that your CA finds with each of the listed issues during the remediation process. The recommended way to handle this is to ensure each certificate is logged to CT and then attach a CSV file/spreadsheet of the fingerprints or crt.sh IDs, with one list per distinct problem.
4) Summary of the problematic certificates. For each problem listed below: number of certs, date first and last certs with that problem were issued.
5) Explanation about how and why the mistakes were made, and not caught and fixed earlier.
6) List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.
7) Regular updates to confirm when those steps have been completed.

Note Section 4.9.1.1 of the CA/Browser Forum’s Baseline Requirements, which states:
“The CA SHALL revoke a Certificate within 24 hours if one or more of the following occurs: …
9. The CA is made aware that the Certificate was not issued in accordance with these Requirements or the CA’s Certificate Policy or Certification Practice Statement; 
10. The CA determines that any of the information appearing in the Certificate is inaccurate or misleading; …
14. Revocation is required by the CA’s Certificate Policy and/or Certification Practice Statement; or 
15. The technical content or format of the Certificate presents an unacceptable risk to Application Software Suppliers or Relying Parties (e.g. the CA/Browser Forum might determine that a deprecated cryptographic/signature algorithm or key size presents an unacceptable risk and that such Certificates should be revoked and replaced by CAs within a given period of time).

However, it is not our intent to introduce additional problems by forcing the immediate revocation of certificates that are not BR compliant when they do not pose an urgent security concern. Therefore, we request that your CA perform careful analysis of the situation. If there is justification to not revoke the problematic certificates, then explain those reasons and provide a timeline for when the bulks of the certificates will expire or be revoked/replaced. 

We expect that your forthcoming audit statements will indicate the findings of these problems. If your CA will not be revoking the certificates within 24 hours in accordance with the BRs, then that will also need to be listed as a finding in your CA’s BR audit statement.

We expect that your CA will work with your auditor (and supervisory body, as appropriate) and the Root Store(s) that your CA participates in to ensure your analysis of the risk and plan of remediation is acceptable. If your CA will not be revoking the problematic certificates as required by the BRs, then we recommend that you also contact the other root programs that your CA participates in to acknowledge this non-compliance and discuss what expectations their Root Programs have with respect to these certificates.


The problems reported for your CA in the mozilla.dev.security.policy forum are as follows:

** Failure to respond within 24 hours after Problem Report submitted
https://groups.google.com/d/msg/mozilla.dev.security.policy/PrsDfS8AMEk/w2AMK81jAQAJ
The problems were reported via your CA’s Problem Reporting Mechanism as listed here:
https://ccadb-public.secure.force.com/mozilla/CAInformationReport
Therefore, if this is the first time you have received notice of the problem(s) listed below, please review and fix your CA’s Problem Reporting Mechanism to ensure that it will work the next time someone reports a problem like this.

** Invalid dnsNames (e.g. invalid characters, internal names, and wildcards in the wrong position)
https://groups.google.com/d/msg/mozilla.dev.security.policy/CfyeeybBz9c/lmmUT4x2CAAJ
https://groups.google.com/d/msg/mozilla.dev.security.policy/D0poUHqiYMw/Pf5p0kB7CAAJ

** Certificates with metadata-only subject fields (at least one subject field that only contains ASCII punctuation characters)
https://groups.google.com/d/msg/mozilla.dev.security.policy/Sae5lpT02Ng/-lsC11JnBwAJ
Prevent further issuance of certs with N/A and other metadata but revocation not necessary in this case.

Comment 1

[Jonathan Rudenberg]

Additionally, I was unable to find a public incident report sent to Mozilla for certificates with invalid dnsNames containing "double dots"  detailed and listed in the links below.

https://groups.google.com/d/topic/mozilla.dev.security.policy/5bpr9yBgaYo/discussion
https://misissued.com/batch/13/

Comment 2

[Lothar Eickholt]

We became aware of one of the current problems on July 20th. As a result we revoked these certificates. 
We were made aware of the other problems from the Bug report 1391074.

We will confirm, that we stopped the further issuance of certificates with these problems. 
The registration staff has been informed and the validation procedure was updated. 

We are currently in the development of an appropriate tool to thoroughly check our certificates databases.

Due to specification changes, problems may occur with older certificates due to the usage of special characters.  
Although these problems are of BR relevance, they had to be prioritized. From our point of view the listed problems do not directly endanger the security infrastructure, issuance processes or certificate acceptance. Major changes in the validation and issuing procedures may call for a great deal of time for planning, testing and implementation. 

The existing validation rules (e.g. syntax) will be further improved and the registration staff will be more sensitive to such cases. 
The established, regular internal audits will be expanded. Findings and their reliable counter measures will be reported and monitored in the internal audit tracking tools. 

This is a tried and proven method within the CA.  Customers were contacted and miss issued certificates were exchanged and the older certificates has been revoked.

We will share the results with you by COB August 25th

Comment 3

[Jonathan Rudenberg]

I emailed a problem report about the certificate with an invalid dnsName, can you please investigate and explain why it was not received?

Here are the relevant message headers:

  From: Jonathan Rudenberg <[e-mail]>
  Subject: Misissuance - invalid dnsNames
  Message-Id: <[e-mail]>
  Date: Sun, 13 Aug 2017 00:48:20 -0400
  To: [e-mail], [e-mail]

Comment 4

[Ryan Sleevi]

(In reply to Lothar Eickholt from comment #2)
> This is a tried and proven method within the CA.  Customers were contacted
> and miss issued certificates were exchanged and the older certificates has
> been revoked.

Thanks for responding. I think it's still necessary to provide additional detail.

That is, we can look at the problem from two dimensions: The problem itself, and the systemic issues that allowed the problem to manifest. Your description focuses on the resolution of the problem, but doesn't indicate any systemic changes have been made. As a consequence, it does not help the community feel that your CA has taken steps to reduce the risk of future violations (of any requirement) in the future. That is, one dimension is "Did you fix the bug", but another dimension is "How was the bug introduced, why was it not detected, and what steps are you taking to prevent future bugs"

To understand how you might approach this problem, consider https://groups.google.com/d/msg/mozilla.dev.security.policy/vl5eq0PoJxY/W1D4oZ__BwAJ and how it provided a timeline of events, the steps that were already in place (and with substantial detail), where there were controls missing or mistakes made, details about the steps being taken (e.g. "We fixed the bug" is not sufficient detail to understand), and a holistic, systemic awareness of how the CA is managed and the opportunities for errors.


For this specific case:
- Your description does not explain how this problem happened. That is, what was the underlying cause of this issue, how was it introduced, and how was it mitigated? There are many things one can infer from your response, but unfortunately, the details are not clear. One way to ensure there's a complete response is to use the 'Reply' button next to Comment #0 to each of the requested items.
- Your response implies there may have been pre-existing awareness of some of these issues. Can you indicate where you acknowledged and/or communicated with the community regarding this?
- You indicate your registration staff have been informed. What were they informed of? Is this a manual process of entering domains, or do you have systems in place to detect and prevent such human error?
- You indicated there may be other problems with your issuance. Could you expand upon that? Given that the Baseline Requirements have been in place since 2012, and RFC 5280 normatively references the specifications such as RFC 1034, the issuance processing requirements have been long established by the industry. Why were these not integrated into your issuance pipeline?
- Please provide greater details about the validation rules in place, why they failed, and how they're being improved.
- Please provide greater details about how internal audits are conducted and how they're being expanded in a way that would detect and/or mitigate this issue.
- Please provide greater details about how reliable counter measures will be reported and monitored.


As you share your responses, using the 'Reply' button next to Comment #0 and this comment, Comment #4, will allow you to quote the relevant messages and provide the details inline, to ensure that each specific request has been identified and resolved.

Comment 5

[Lothar Eickholt]

(In reply to Kathleen Wilson from comment #0)
> The following problems have been found in certificates issued by your CA,
> and reported in the mozilla.dev.security.policy forum. Direct links to those
> discussions are provided for your convenience.
> 
> To continue inclusion of your CA’s root certificates in Mozilla’s Root
> Store, you must respond in this bug to provide the following information:
> 1) How your CA first became aware of the problems listed below (e.g. via a
> Problem Report, via the discussion in mozilla.dev.security.policy, or via
> this Bugzilla Bug), and the date.
We were mainly aware of the current issues with communication in Bug 1391074 on 2017-08-07. Investigation began to get consistent and unified understanding of the problems to subsequently work on them. Issue 111449358 we became aware on 2017-08-14 and processed. Issue 1258252 we became aware on 2017-07-19 witch was reported in posting of mozilla.dev.security.policy. Issue 4598076 we became aware with email from Mr. Rudenberg on 2017-08-13. Dealing with it began delayed delayed. Further isssues we became aware on mississued.com until 2017-08-15. Details, remediations of these mentioned and further issues please find in our statements in #3 and following.


> 2) Prompt confirmation that your CA has stopped issuing TLS/SSL certificates
> with the problems listed below.
> 3) Complete list of certificates that your CA finds with each of the listed
> issues during the remediation process. The recommended way to handle this is
> to ensure each certificate is logged to CT and then attach a CSV
> file/spreadsheet of the fingerprints or crt.sh IDs, with one list per
> distinct problem.
> 4) Summary of the problematic certificates. For each problem listed below:
> number of certs, date first and last certs with that problem were issued.
> 5) Explanation about how and why the mistakes were made, and not caught and
> fixed earlier.
> 6) List of steps your CA is taking to resolve the situation and ensure such
> issuance will not be repeated in the future, accompanied with a timeline of
> when your CA expects to accomplish these things.
> 7) Regular updates to confirm when those steps have been completed.
> 
> Note Section 4.9.1.1 of the CA/Browser Forum’s Baseline Requirements, which
> states:
> “The CA SHALL revoke a Certificate within 24 hours if one or more of the
> following occurs: …
> 9. The CA is made aware that the Certificate was not issued in accordance
> with these Requirements or the CA’s Certificate Policy or Certification
> Practice Statement; 
> 10. The CA determines that any of the information appearing in the
> Certificate is inaccurate or misleading; …
> 14. Revocation is required by the CA’s Certificate Policy and/or
> Certification Practice Statement; or 
> 15. The technical content or format of the Certificate presents an
> unacceptable risk to Application Software Suppliers or Relying Parties (e.g.
> the CA/Browser Forum might determine that a deprecated
> cryptographic/signature algorithm or key size presents an unacceptable risk
> and that such Certificates should be revoked and replaced by CAs within a
> given period of time).
> 
> However, it is not our intent to introduce additional problems by forcing
> the immediate revocation of certificates that are not BR compliant when they
> do not pose an urgent security concern. Therefore, we request that your CA
> perform careful analysis of the situation. If there is justification to not
> revoke the problematic certificates, then explain those reasons and provide
> a timeline for when the bulks of the certificates will expire or be
> revoked/replaced. 
> 
> We expect that your forthcoming audit statements will indicate the findings
> of these problems. If your CA will not be revoking the certificates within
> 24 hours in accordance with the BRs, then that will also need to be listed
> as a finding in your CA’s BR audit statement.
> 
> We expect that your CA will work with your auditor (and supervisory body, as
> appropriate) and the Root Store(s) that your CA participates in to ensure
> your analysis of the risk and plan of remediation is acceptable. If your CA
> will not be revoking the problematic certificates as required by the BRs,
> then we recommend that you also contact the other root programs that your CA
> participates in to acknowledge this non-compliance and discuss what
> expectations their Root Programs have with respect to these certificates.
> 
> 
> The problems reported for your CA in the mozilla.dev.security.policy forum
> are as follows:
> 
> ** Failure to respond within 24 hours after Problem Report submitted
> https://groups.google.com/d/msg/mozilla.dev.security.policy/PrsDfS8AMEk/
> w2AMK81jAQAJ
> The problems were reported via your CA’s Problem Reporting Mechanism as
> listed here:
> https://ccadb-public.secure.force.com/mozilla/CAInformationReport
> Therefore, if this is the first time you have received notice of the
> problem(s) listed below, please review and fix your CA’s Problem Reporting
> Mechanism to ensure that it will work the next time someone reports a
> problem like this.
> 
> ** Invalid dnsNames (e.g. invalid characters, internal names, and wildcards
> in the wrong position)
> https://groups.google.com/d/msg/mozilla.dev.security.policy/CfyeeybBz9c/
> lmmUT4x2CAAJ
> https://groups.google.com/d/msg/mozilla.dev.security.policy/D0poUHqiYMw/
> Pf5p0kB7CAAJ
> 
> ** Certificates with metadata-only subject fields (at least one subject
> field that only contains ASCII punctuation characters)
> https://groups.google.com/d/msg/mozilla.dev.security.policy/Sae5lpT02Ng/-
> lsC11JnBwAJ
> Prevent further issuance of certs with N/A and other metadata but revocation
> not necessary in this case.

Comment 6

[Lothar Eickholt]

(In reply to Kathleen Wilson from comment #0)
> The following problems have been found in certificates issued by your CA,
> and reported in the mozilla.dev.security.policy forum. Direct links to those
> discussions are provided for your convenience.
> 
> To continue inclusion of your CA’s root certificates in Mozilla’s Root
> Store, you must respond in this bug to provide the following information:
> 1) How your CA first became aware of the problems listed below (e.g. via a
> Problem Report, via the discussion in mozilla.dev.security.policy, or via
> this Bugzilla Bug), and the date.
> 2) Prompt confirmation that your CA has stopped issuing TLS/SSL certificates
> with the problems listed below.
> 3) Complete list of certificates that your CA finds with each of the listed
> issues during the remediation process. The recommended way to handle this is
> to ensure each certificate is logged to CT and then attach a CSV
> file/spreadsheet of the fingerprints or crt.sh IDs, with one list per
> distinct problem.
> 4) Summary of the problematic certificates. For each problem listed below:
> number of certs, date first and last certs with that problem were issued.

https://crt.sh/?id=1258252&opt=cablint f11-apache-2.fak11.uni-muenchen.de
Issue B: rfc822Name in server certificate
Issue C: DNSNames is not FQDN (double dots in DNSName www-test.psy..uni-muenchen.de)
Reporter: Posting to mozilla.dev.security.policy
First reported to DFN-PKI: 2017-07-19
Investigation started at 2017-07-19 and concentrated on double dots in DNSNames, as that was the
complaint on mozilla.dev.security.policy. Certificate holder could not be reached directly per phone,
so an email was sent and a 24h grace period was granted. T-Systems as root was informed of the
findings via email on 2017-07-20.
Revoked: 2017-07-20

https://crt.sh/?id=4598076&opt=cablint www.pcnm.uni-osnabrueck.de
Issue B: rfc822Name in server certificate
Issue D: DNSName is not in preferred syntax (trailing dot in DNSName advancedmaterials.uniosnabrueck.
de.)
Reporter: Mail from Jonathan Rudenberg to mail address [e-mail]
First reported to DFN-PKI: 2017-08-13
Investigation started at 2017-08-15. The mail address [e-mail] is not an official problem reporting channel, so it is not continuously monitored, and investigation was delayed. Investigation concentrated on invalid DNSNames syntax, as that was the complaint from Jonathan Rudenberg. Unfortunately it took some time to understand the problem with „DNSName is not in preferred syntax“, as we misinterpreted RFC5280 and its link to RFC 1034. Certificate holder could not be reached directly per phone, so an email was sent and a 24h grace period was granted.
Revoked: 2017-08-18

https://crt.sh/?id=84167105&opt=cablint share.hs-duesseldorf.de
https://crt.sh/?id=140554773&opt=cablint share.hs-duesseldorf.de
https://crt.sh/?id=130818697&opt=cablint share.hs-duesseldorf.de
https://crt.sh/?id=23313232&opt=cablint share.hs-duesseldorf.de
Issue A: Metadata in OU
Reporter: misissued.com
First reported to DFN-PKI: 2017-08-15
Apparently, those certificates were uploaded to misissued.com on 2017-08-12. We noticed them on 2017-08-15 on misissued.com. Certificate holder was contacted, and needed more time than 24h to revoke which was granted.
Revoked: 2017-08-17

https://crt.sh/?id=111449358&opt=cablint bragi.helmholtz-hzi.de
Issue A: Metadata in OU
Reporter: T-Systems (Root CA)
First reported to DFN-PKI: 2017-08-11
Investigation started at 2017-08-14. T-Systems had used two personal email addresses of DFN employees on vacation to report the issue, therefore there was a delay until the issue was addressed correctly.
Certificate holder could not be reached directly by phone, so an email was sent and a 24h grace period was granted.
Revoked: 2017-08-15

https://crt.sh/?id=4975967&opt=cablint
https://crt.sh/?id=10029013&opt=cablint
https://crt.sh/?id=6086077&opt=cablint
https://crt.sh/?id=8246413
https://crt.sh/?id=11698289
https://crt.sh/?id=42470123
Issue: ERROR Unknown TLD in SAN
Reporter: https://misissued.com/batch/9/
Reporter: https://bugzilla.mozilla.org/show_bug.cgi?id=1391074 
First reported: 2017-08-13
The problem was realized first on 2017-0815. Through an major update in validation process on 2016-10-10 , there was the remediation without knowledge of this issue . Our inquiries shows, that there were no issuance with this issue after this date. The certificate holders were contacted. First certificates revoked on 2017-08-16. Some certificate holders needed more time. The last certificate was revoked on 2017-08-23

https://crt.sh/?id=110333785
https://crt.sh/?id=142043470
https://crt.sh/?id=10566368
https://crt.sh/?id=10705000
https://crt.sh/?id=23313232
https://crt.sh/?id=84167105
https://crt.sh/?id=130818697
https://crt.sh/?id=140554773
Reporter: https://misissued.com/batch/5/
Reporter: https://bugzilla.mozilla.org/show_bug.cgi?id=1391074
The problem was realized with bugzilla id = 1391074 on 2017-0817. Certificate holder were contacted. Five certificates revoked on 2017-08-17. One certificate holder needs more time. That certificate was revoked on 2017-08-21. One certificate holder with the two last certificates needed more time than 24h to revoke which was granted until 2017-08-23 13:00.

https://crt.sh/?id=48401159
Reporter: https://bugzilla.mozilla.org/show_bug.cgi?id=1391074
The problem was realized with bugzilla id = 1391074 on 2017-0817. Certificate holder was contacted and certificate was revoked 2017-08-17

There are some additional certificates reported on misissued.com that were issued in or before 2011 when the BRs clearly did not apply, which contain multiple additional noncompliances. Those non-compliances are expected. All those certificates are expired, so we don't think there's a necessity to comment on them.


> 5) Explanation about how and why the mistakes were made, and not caught and
> fixed earlier.
> 6) List of steps your CA is taking to resolve the situation and ensure such
> issuance will not be repeated in the future, accompanied with a timeline of
> when your CA expects to accomplish these things.
> 7) Regular updates to confirm when those steps have been completed.
> 
> Note Section 4.9.1.1 of the CA/Browser Forum’s Baseline Requirements, which
> states:
> “The CA SHALL revoke a Certificate within 24 hours if one or more of the
> following occurs: …
> 9. The CA is made aware that the Certificate was not issued in accordance
> with these Requirements or the CA’s Certificate Policy or Certification
> Practice Statement; 
> 10. The CA determines that any of the information appearing in the
> Certificate is inaccurate or misleading; …
> 14. Revocation is required by the CA’s Certificate Policy and/or
> Certification Practice Statement; or 
> 15. The technical content or format of the Certificate presents an
> unacceptable risk to Application Software Suppliers or Relying Parties (e.g.
> the CA/Browser Forum might determine that a deprecated
> cryptographic/signature algorithm or key size presents an unacceptable risk
> and that such Certificates should be revoked and replaced by CAs within a
> given period of time).
> 
> However, it is not our intent to introduce additional problems by forcing
> the immediate revocation of certificates that are not BR compliant when they
> do not pose an urgent security concern. Therefore, we request that your CA
> perform careful analysis of the situation. If there is justification to not
> revoke the problematic certificates, then explain those reasons and provide
> a timeline for when the bulks of the certificates will expire or be
> revoked/replaced. 
> 
> We expect that your forthcoming audit statements will indicate the findings
> of these problems. If your CA will not be revoking the certificates within
> 24 hours in accordance with the BRs, then that will also need to be listed
> as a finding in your CA’s BR audit statement.
> 
> We expect that your CA will work with your auditor (and supervisory body, as
> appropriate) and the Root Store(s) that your CA participates in to ensure
> your analysis of the risk and plan of remediation is acceptable. If your CA
> will not be revoking the problematic certificates as required by the BRs,
> then we recommend that you also contact the other root programs that your CA
> participates in to acknowledge this non-compliance and discuss what
> expectations their Root Programs have with respect to these certificates.
> 
> 
> The problems reported for your CA in the mozilla.dev.security.policy forum
> are as follows:
> 
> ** Failure to respond within 24 hours after Problem Report submitted
> https://groups.google.com/d/msg/mozilla.dev.security.policy/PrsDfS8AMEk/
> w2AMK81jAQAJ
> The problems were reported via your CA’s Problem Reporting Mechanism as
> listed here:
> https://ccadb-public.secure.force.com/mozilla/CAInformationReport
> Therefore, if this is the first time you have received notice of the
> problem(s) listed below, please review and fix your CA’s Problem Reporting
> Mechanism to ensure that it will work the next time someone reports a
> problem like this.
> 
> ** Invalid dnsNames (e.g. invalid characters, internal names, and wildcards
> in the wrong position)
> https://groups.google.com/d/msg/mozilla.dev.security.policy/CfyeeybBz9c/
> lmmUT4x2CAAJ
> https://groups.google.com/d/msg/mozilla.dev.security.policy/D0poUHqiYMw/
> Pf5p0kB7CAAJ
> 
> ** Certificates with metadata-only subject fields (at least one subject
> field that only contains ASCII punctuation characters)
> https://groups.google.com/d/msg/mozilla.dev.security.policy/Sae5lpT02Ng/-
> lsC11JnBwAJ
> Prevent further issuance of certs with N/A and other metadata but revocation
> not necessary in this case.

Comment 7

[Lothar Eickholt]

(In reply to Kathleen Wilson from comment #0)
> The following problems have been found in certificates issued by your CA,
> and reported in the mozilla.dev.security.policy forum. Direct links to those
> discussions are provided for your convenience.
> 
> To continue inclusion of your CA’s root certificates in Mozilla’s Root
> Store, you must respond in this bug to provide the following information:
> 1) How your CA first became aware of the problems listed below (e.g. via a
> Problem Report, via the discussion in mozilla.dev.security.policy, or via
> this Bugzilla Bug), and the date.
> 2) Prompt confirmation that your CA has stopped issuing TLS/SSL certificates
> with the problems listed below.
> 3) Complete list of certificates that your CA finds with each of the listed
> issues during the remediation process. The recommended way to handle this is
> to ensure each certificate is logged to CT and then attach a CSV
> file/spreadsheet of the fingerprints or crt.sh IDs, with one list per
> distinct problem.
> 4) Summary of the problematic certificates. For each problem listed below:
> number of certs, date first and last certs with that problem were issued.
> 5) Explanation about how and why the mistakes were made, and not caught and
> fixed earlier.
> 6) List of steps your CA is taking to resolve the situation and ensure such
> issuance will not be repeated in the future, accompanied with a timeline of
> when your CA expects to accomplish these things.
#5 and #6
Metadata in OU
This issue has three different causes:
1. Improper validation by RAs. All certificate subject fields are pre-approved by trusted personnel, like C, ST, L, O, OU authorization domain names. 
The RAs who inspected the OU fields before approving did not catch the improper values.

=> Remediation: Retrain RAs to raise metadata constellation awareness for OU validation. 
Timeframe: Next 2 weeks.

2. Our PKI systems currently doesn't prevent metadata-only OUs.
=> Remediation: Change PKI system to prevent common cases of metadata-only values.
Timeframe: Next 4 weeks.

3. Monitoring of issued certificates did not catch the problem. CAs have a deep 3% audit (as required per BR), and lighter audit of all issued certificates (where certificates are manually inspected, including the OU field). This problem was not caught by both 3% and manual full audit.

=> Remediation: Check why the problem was not caught by manual full audit.
Timeframe: 1 week

DNSName is not in preferred syntax (trailing dot)
4 further certificates contain DNSName subject alternative names with trailing dots. One of those certificates contains three DNSNames with this error, the others each only one.
Up to 2015, one PKI system allowed to append a trailing dot to a domain name. This was changed in March 2015. The change was solely motivated to make name handling simpler; it was not detected that trailing dots are actually forbidden according to the BRs via RFC5280=>RFC1034 and trailing dots are technically correct domain names but not “preferred syntax”.
Remediation: Since 2015-03-10 the PKI software prevents domain names with trailing dots.
Handling of existing certificates: This issue does not impair the security of affected certificates and thus does not necessarily warrant revocation in our opinion.




> 7) Regular updates to confirm when those steps have been completed.
> 
> Note Section 4.9.1.1 of the CA/Browser Forum’s Baseline Requirements, which
> states:
> “The CA SHALL revoke a Certificate within 24 hours if one or more of the
> following occurs: …
> 9. The CA is made aware that the Certificate was not issued in accordance
> with these Requirements or the CA’s Certificate Policy or Certification
> Practice Statement; 
> 10. The CA determines that any of the information appearing in the
> Certificate is inaccurate or misleading; …
> 14. Revocation is required by the CA’s Certificate Policy and/or
> Certification Practice Statement; or 
> 15. The technical content or format of the Certificate presents an
> unacceptable risk to Application Software Suppliers or Relying Parties (e.g.
> the CA/Browser Forum might determine that a deprecated
> cryptographic/signature algorithm or key size presents an unacceptable risk
> and that such Certificates should be revoked and replaced by CAs within a
> given period of time).
> 
> However, it is not our intent to introduce additional problems by forcing
> the immediate revocation of certificates that are not BR compliant when they
> do not pose an urgent security concern. Therefore, we request that your CA
> perform careful analysis of the situation. If there is justification to not
> revoke the problematic certificates, then explain those reasons and provide
> a timeline for when the bulks of the certificates will expire or be
> revoked/replaced. 
> 
> We expect that your forthcoming audit statements will indicate the findings
> of these problems. If your CA will not be revoking the certificates within
> 24 hours in accordance with the BRs, then that will also need to be listed
> as a finding in your CA’s BR audit statement.
> 
> We expect that your CA will work with your auditor (and supervisory body, as
> appropriate) and the Root Store(s) that your CA participates in to ensure
> your analysis of the risk and plan of remediation is acceptable. If your CA
> will not be revoking the problematic certificates as required by the BRs,
> then we recommend that you also contact the other root programs that your CA
> participates in to acknowledge this non-compliance and discuss what
> expectations their Root Programs have with respect to these certificates.
> 
> 
> The problems reported for your CA in the mozilla.dev.security.policy forum
> are as follows:
> 
> ** Failure to respond within 24 hours after Problem Report submitted
> https://groups.google.com/d/msg/mozilla.dev.security.policy/PrsDfS8AMEk/
> w2AMK81jAQAJ
> The problems were reported via your CA’s Problem Reporting Mechanism as
> listed here:
> https://ccadb-public.secure.force.com/mozilla/CAInformationReport
> Therefore, if this is the first time you have received notice of the
> problem(s) listed below, please review and fix your CA’s Problem Reporting
> Mechanism to ensure that it will work the next time someone reports a
> problem like this.
> 
> ** Invalid dnsNames (e.g. invalid characters, internal names, and wildcards
> in the wrong position)
> https://groups.google.com/d/msg/mozilla.dev.security.policy/CfyeeybBz9c/
> lmmUT4x2CAAJ
> https://groups.google.com/d/msg/mozilla.dev.security.policy/D0poUHqiYMw/
> Pf5p0kB7CAAJ
> 
> ** Certificates with metadata-only subject fields (at least one subject
> field that only contains ASCII punctuation characters)
> https://groups.google.com/d/msg/mozilla.dev.security.policy/Sae5lpT02Ng/-
> lsC11JnBwAJ
> Prevent further issuance of certs with N/A and other metadata but revocation
> not necessary in this case.

Comment 8

[Lothar Eickholt]

(In reply to Kathleen Wilson from comment #0)
> The following problems have been found in certificates issued by your CA,
> and reported in the mozilla.dev.security.policy forum. Direct links to those
> discussions are provided for your convenience.
> 
> To continue inclusion of your CA’s root certificates in Mozilla’s Root
> Store, you must respond in this bug to provide the following information:
> 1) How your CA first became aware of the problems listed below (e.g. via a
> Problem Report, via the discussion in mozilla.dev.security.policy, or via
> this Bugzilla Bug), and the date.
> 2) Prompt confirmation that your CA has stopped issuing TLS/SSL certificates
> with the problems listed below.
> 3) Complete list of certificates that your CA finds with each of the listed
> issues during the remediation process. The recommended way to handle this is
> to ensure each certificate is logged to CT and then attach a CSV
> file/spreadsheet of the fingerprints or crt.sh IDs, with one list per
> distinct problem.
#3
Investigation with CABlint:
ISSUE	        |  Expire in 2017  |  Expire in 2018  |  Expire in 2019  | Expire 2020 |
Metadata in OU  |	 0	   |        3         |         62       |     3       |
DNSName is 
not in preferred 
syntax	        |	 0	   |        1         |          3       |     0       |

Own scripts brought 23 further results in different constellations.

Handling of existing certificates: This issue does not impair the security of affected certificates and thus does not necessarily warrant revocation in our opinion.

All findings and our remediation actions will be indicated to our auditors.

Publication of affected certificates
While it has been the position of the Mozilla community for some time that Web PKI certificates are public by nature, German laws, especially data/privacy protection laws, make this situation more complicated. Publication of affected certificates themselves by the PKI is thus subject to prior verification of the lawfulness of such a publication without explicit consent from subscribers.
It is the intention of our PKIs to publish all certificates that we are entitled to publish – if possible all of them. Obviously, some of the certificates are already logged in CT Log because they were used on public web servers and EV certificates. Additionally, we will publish all affected serial numbers and/or fingerprints with validity dates of the affected certificates.
The PKIs are in the progress to establish permission for publication for each server certificate in the course of introducing certificate transparency,  but this work is not finished and does not affect existing certificates.




> 4) Summary of the problematic certificates. For each problem listed below:
> number of certs, date first and last certs with that problem were issued.
> 5) Explanation about how and why the mistakes were made, and not caught and
> fixed earlier.
> 6) List of steps your CA is taking to resolve the situation and ensure such
> issuance will not be repeated in the future, accompanied with a timeline of
> when your CA expects to accomplish these things.
> 7) Regular updates to confirm when those steps have been completed.
> 
> Note Section 4.9.1.1 of the CA/Browser Forum’s Baseline Requirements, which
> states:
> “The CA SHALL revoke a Certificate within 24 hours if one or more of the
> following occurs: …
> 9. The CA is made aware that the Certificate was not issued in accordance
> with these Requirements or the CA’s Certificate Policy or Certification
> Practice Statement; 
> 10. The CA determines that any of the information appearing in the
> Certificate is inaccurate or misleading; …
> 14. Revocation is required by the CA’s Certificate Policy and/or
> Certification Practice Statement; or 
> 15. The technical content or format of the Certificate presents an
> unacceptable risk to Application Software Suppliers or Relying Parties (e.g.
> the CA/Browser Forum might determine that a deprecated
> cryptographic/signature algorithm or key size presents an unacceptable risk
> and that such Certificates should be revoked and replaced by CAs within a
> given period of time).
> 
> However, it is not our intent to introduce additional problems by forcing
> the immediate revocation of certificates that are not BR compliant when they
> do not pose an urgent security concern. Therefore, we request that your CA
> perform careful analysis of the situation. If there is justification to not
> revoke the problematic certificates, then explain those reasons and provide
> a timeline for when the bulks of the certificates will expire or be
> revoked/replaced. 
> 
> We expect that your forthcoming audit statements will indicate the findings
> of these problems. If your CA will not be revoking the certificates within
> 24 hours in accordance with the BRs, then that will also need to be listed
> as a finding in your CA’s BR audit statement.
> 
> We expect that your CA will work with your auditor (and supervisory body, as
> appropriate) and the Root Store(s) that your CA participates in to ensure
> your analysis of the risk and plan of remediation is acceptable. If your CA
> will not be revoking the problematic certificates as required by the BRs,
> then we recommend that you also contact the other root programs that your CA
> participates in to acknowledge this non-compliance and discuss what
> expectations their Root Programs have with respect to these certificates.
> 
> 
> The problems reported for your CA in the mozilla.dev.security.policy forum
> are as follows:
> 
> ** Failure to respond within 24 hours after Problem Report submitted
> https://groups.google.com/d/msg/mozilla.dev.security.policy/PrsDfS8AMEk/
> w2AMK81jAQAJ
> The problems were reported via your CA’s Problem Reporting Mechanism as
> listed here:
> https://ccadb-public.secure.force.com/mozilla/CAInformationReport
> Therefore, if this is the first time you have received notice of the
> problem(s) listed below, please review and fix your CA’s Problem Reporting
> Mechanism to ensure that it will work the next time someone reports a
> problem like this.
> 
> ** Invalid dnsNames (e.g. invalid characters, internal names, and wildcards
> in the wrong position)
> https://groups.google.com/d/msg/mozilla.dev.security.policy/CfyeeybBz9c/
> lmmUT4x2CAAJ
> https://groups.google.com/d/msg/mozilla.dev.security.policy/D0poUHqiYMw/
> Pf5p0kB7CAAJ
> 
> ** Certificates with metadata-only subject fields (at least one subject
> field that only contains ASCII punctuation characters)
> https://groups.google.com/d/msg/mozilla.dev.security.policy/Sae5lpT02Ng/-
> lsC11JnBwAJ
> Prevent further issuance of certs with N/A and other metadata but revocation
> not necessary in this case.

Comment 9

[Lothar Eickholt]

(In reply to Jonathan Rudenberg from comment #3)
> I emailed a problem report about the certificate with an invalid dnsName,
> can you please investigate and explain why it was not received?
> 
> Here are the relevant message headers:
> 
>   From: Jonathan Rudenberg <[e-mail]>
>   Subject: Misissuance - invalid dnsNames
>   Message-Id: <[e-mail]>
>   Date: Sun, 13 Aug 2017 00:48:20 -0400
>   To: [e-mail], [e-mail]

We would like to apologize for the delayed response, but working the initial bug report 1391074 was very time consuming. You will find a reply to your inquiry in comment #6 in this mentioned bug report from Mrs Kathleen Wilson.

Comment 10

[Ryan Sleevi]

(In reply to Lothar Eickholt from comment #8)
> #3
> Investigation with CABlint:
> ISSUE	        |  Expire in 2017  |  Expire in 2018  |  Expire in 2019  |
> Expire 2020 |
> Metadata in OU  |	 0	   |        3         |         62       |     3       |
> DNSName is 
> not in preferred 
> syntax	        |	 0	   |        1         |          3       |     0       |
> 
> Own scripts brought 23 further results in different constellations.
> 
> Handling of existing certificates: This issue does not impair the security
> of affected certificates and thus does not necessarily warrant revocation in
> our opinion.
> 
> All findings and our remediation actions will be indicated to our auditors.
> 
> Publication of affected certificates
> While it has been the position of the Mozilla community for some time that
> Web PKI certificates are public by nature, German laws, especially
> data/privacy protection laws, make this situation more complicated.
> Publication of affected certificates themselves by the PKI is thus subject
> to prior verification of the lawfulness of such a publication without
> explicit consent from subscribers.
> It is the intention of our PKIs to publish all certificates that we are
> entitled to publish – if possible all of them. Obviously, some of the
> certificates are already logged in CT Log because they were used on public
> web servers and EV certificates. Additionally, we will publish all affected
> serial numbers and/or fingerprints with validity dates of the affected
> certificates.
> The PKIs are in the progress to establish permission for publication for
> each server certificate in the course of introducing certificate
> transparency,  but this work is not finished and does not affect existing
> certificates.

Kathleen: I defer to your judgement here as Module Owner. This is another CA who is reporting they are unable to disclose the misissued certificates, which are not presently disclosed via Certificate Transparency.

Comment 11

[Ryan Sleevi]

(In reply to Ryan Sleevi from comment #4)
> For this specific case:
> - You indicate your registration staff have been informed. What were they
> informed of? Is this a manual process of entering domains, or do you have
> systems in place to detect and prevent such human error?
> - You indicated there may be other problems with your issuance. Could you
> expand upon that? Given that the Baseline Requirements have been in place
> since 2012, and RFC 5280 normatively references the specifications such as
> RFC 1034, the issuance processing requirements have been long established by
> the industry. Why were these not integrated into your issuance pipeline?
> - Please provide greater details about the validation rules in place, why
> they failed, and how they're being improved.
> - Please provide greater details about how internal audits are conducted and
> how they're being expanded in a way that would detect and/or mitigate this
> issue.
> - Please provide greater details about how reliable counter measures will be
> reported and monitored.

In addition to the above questions, I do not feel I can find a complete or satisfactory answer as to the root cause or remediations being deployed; Comment #7 indicates partial information (for Metadata OUs) but does not respond to the fullness of issues.

Comment 12

[Kathleen Wilson]

(In reply to Ryan Sleevi from comment #10)
> (In reply to Lothar Eickholt from comment #8)
> > While it has been the position of the Mozilla community for some time that
> > Web PKI certificates are public by nature, German laws, especially
> > data/privacy protection laws, make this situation more complicated.
> > Publication of affected certificates themselves by the PKI is thus subject
> > to prior verification of the lawfulness of such a publication without
> > explicit consent from subscribers.
> > It is the intention of our PKIs to publish all certificates that we are
> > entitled to publish – if possible all of them. Obviously, some of the
> > certificates are already logged in CT Log because they were used on public
> > web servers and EV certificates. Additionally, we will publish all affected
> > serial numbers and/or fingerprints with validity dates of the affected
> > certificates.
> > The PKIs are in the progress to establish permission for publication for
> > each server certificate in the course of introducing certificate
> > transparency,  but this work is not finished and does not affect existing
> > certificates.
> 
> Kathleen: I defer to your judgement here as Module Owner. This is another CA
> who is reporting they are unable to disclose the misissued certificates,
> which are not presently disclosed via Certificate Transparency.


In this particular situation, there appears to be a valid reason why the CA might not be able to disclose some of the certificates, and the CA appears to be working to resolve it.

If T-Systems does find that they are not able to fully disclose some of the problematic certificates, then I request that the CA provide a URL to a webpage regarding that particular law. And for each such cert: Issuer CN/O/OU, serial number, SHA-256 Fingerprint, validity dates.

Comment 13

[Ryan Sleevi]

Greetings, it's been five days since the request in Comment #12. Is there an update with respect to the particular law?

Comment 14

[Lothar Eickholt]

(In reply to Ryan Sleevi from comment #11)
> (In reply to Ryan Sleevi from comment #4)
> > For this specific case:
> > - You indicate your registration staff have been informed. What were they
> > informed of? Is this a manual process of entering domains, or do you have
> > systems in place to detect and prevent such human error?
This relates to metadata in OU, not domain validation. No error had anything to do with validating domains.
There are 3 aspects here:

1) Registration staff let metadata-OUs through => retrain registration staff e.g. concerning constellation of allowed metadata

2) Software allowed simple metadata-only-OU => bugfix is in progress (problem: how to technically enforce the BR-language that also strings like "not applicable" etc are forbidden? Thus training of registration staff is also important)

3) Internal audits of issued certificates did not catch the metadata-in-OU-issue. => Raise awareness of the people who are conducting internal audits.

> > - You indicated there may be other problems with your issuance. Could you
> > expand upon that? Given that the Baseline Requirements have been in place
> > since 2012, and RFC 5280 normatively references the specifications such as
> > RFC 1034, the issuance processing requirements have been long established by
> > the industry. Why were these not integrated into your issuance pipeline?
One could argue that violating the regex from chapter 3.5 of rfc1034 is a rather obscure way of violating the BRs.

=> Improvement: Integration of external-designed tools like cablint into our pki processes is vital to gather external verification of our interpretation of BRs.

> > - Please provide greater details about the validation rules in place, why
> > they failed, and how they're being improved.
Validation of OUs: Manual inspection of registration staff failed as awareness for metadata in OUs was not high enough.  Improvement: See above, 1), 2), 3).

DNSName is not in preferred syntax (trailing dot): This was no validation issue but a technical error. Validated domain names "example.org" could also be entered as "example.org.". This was implemented in this way since pre-BR times because people argued that "example.org." is from a DNS-point of view the technically correct way and all others are just abbreviations. On checking for compliance against BRs we missed this non-compliance with rfc 1034 section 3.5.

Improvement: Integration of external-designed tools like cablint into our pki processes is vital to gather external verification of our interpretation of BRs.

> > - Please provide greater details about how internal audits are conducted and
> > how they're being expanded in a way that would detect and/or mitigate this
> > issue.
1. Deep 3% audit according to BRs: Full certificate details are manually inspected. Validation data is inspected for correctness.

Improvement: Add "not only metadata in OU" to audit checklist.

2. Lighter audit of all issued certificates: Lists with issued certificates are manually inspected.

Improvement: Add "not only metadata in OU" to list of things to look for.

Additional improvement: Use externally developed tools (cablint) on all issued certificates

> > - Please provide greater details about how reliable counter measures will be
> > reported and monitored.
Standard processes (e,g. incident-, problem-, changemanagement according ITIL) control the implementation of effective counter measures. Quality control is an integral component of these processes.

> 
> In addition to the above questions, I do not feel I can find a complete or
> satisfactory answer as to the root cause or remediations being deployed;
> Comment #7 indicates partial information (for Metadata OUs) but does not
> respond to the fullness of issues.
Metadata-in-OU:
Root-cause: Verification of OU-data relied on manual inspection to prevent metadata. Awareness for OU-field too low.

Remediation: Implement software measures against simple types of metadata, raise awareness with registration staff, raise awareness on monitoring/internal audits.

DNSName is not in preferred syntax (trailing dot):
Root-cause: Failure to recognize that trailing dots are a violation of 3.5 rfc1034=> rfc5280 =>BR.

Remediation: Integration of external-designed tools like cablint into our pki processes is vital to gather external verification of our interpretation of BRs.

Comment 15

[Lothar Eickholt]

(In reply to Kathleen Wilson from comment #12)
> (In reply to Ryan Sleevi from comment #10)
> > (In reply to Lothar Eickholt from comment #8)
> > > While it has been the position of the Mozilla community for some time that
> > > Web PKI certificates are public by nature, German laws, especially
> > > data/privacy protection laws, make this situation more complicated.
> > > Publication of affected certificates themselves by the PKI is thus subject
> > > to prior verification of the lawfulness of such a publication without
> > > explicit consent from subscribers.
> > > It is the intention of our PKIs to publish all certificates that we are
> > > entitled to publish – if possible all of them. Obviously, some of the
> > > certificates are already logged in CT Log because they were used on public
> > > web servers and EV certificates. Additionally, we will publish all affected
> > > serial numbers and/or fingerprints with validity dates of the affected
> > > certificates.
> > > The PKIs are in the progress to establish permission for publication for
> > > each server certificate in the course of introducing certificate
> > > transparency,  but this work is not finished and does not affect existing
> > > certificates.
> > 
> > Kathleen: I defer to your judgement here as Module Owner. This is another CA
> > who is reporting they are unable to disclose the misissued certificates,
> > which are not presently disclosed via Certificate Transparency.
> 
> 
> In this particular situation, there appears to be a valid reason why the CA
> might not be able to disclose some of the certificates, and the CA appears
> to be working to resolve it.
> 
> If T-Systems does find that they are not able to fully disclose some of the
> problematic certificates, then I request that the CA provide a URL to a
> webpage regarding that particular law. 

Please find our reference according the publication of issued certificates in consideration of data protection regulations: 

-----
The German Federal Data Protection Act states in Section 4 (1): 

"The collection, processing and use of personal data shall be admissible only if permitted or prescribed by this Act or any other legal provision or if the data subject has consented." [1]

Certificate contents such as IP-Addresses, Domain Names and E-Mail Addresses must (in many cases) be considered to be personal data according to German law. The certificates in question do thus have to be considered to be subject to German data protection and privacy laws (which are, as you might know, among the strictest worldwide). We have not been granted explicit consent for publication upon issuance of the certificates in all cases. If there is neither "consent" nor "legal provision", we cannot publish them.

[1] http://www.gesetze-im-internet.de/englisch_bdsg/englisch_bdsg.html#p0061
-----


And for each such cert: Issuer
> CN/O/OU, serial number, SHA-256 Fingerprint, validity dates.
Now we are preparing the list for disclosing the next days.

Comment 16

[Bernd]

Attachment: T-Systems Non-BR-Compliant certificates

Comment 17

[Bernd]

Hello Kathleen,

I attached the list (DFN-TSI_20170906.csv) to disclose the T-Systems Non-BR-Compliant certificates.  

At the moment, we are implementing the check-tools (CABlint, X509lint, zlint) into our processes to verify all previous and new certificates issued by the T-Systems Roots.

Best regards,
Bernd

Comment 18

[Jürgen Brauckmann]

DFN-PKI cablint findings in Bug 1401486

Comment 19

[Bernd]

Attachment: T-Systems_Findings_20171006.pdf

Comment 20

[Bernd]

Attachment: T-Systems_Certificates_20171006.csv

Comment 21

[Bernd]

After implementing the check-tools (CABlint, X509lint, zlint) we scanned all not expired and not revoked server certificates of the PKI hierarchies under the following CA certificates:
CN=TeleSec ServerPass CA 2 (https://crt.sh/?id=4912414)
CN=Shared Business CA 4 (https://crt.sh/?id=4460355)
CN=TeleSec ServerPass Class 2 CA (https://crt.sh/?id=8733623)


The following issues were found. 
Issue A: Metadata in OU
A set of 2 certificates with empty coded OU was detected. 

Issue B: ST appears to only include metadata
A set of 3 certificates with empty coded ST was detected.

Issue C: Unallowed key usage for EC public key (Key Encipherment)
A set of 3 certificates with the key usage key Encipherment was detected.

Issue D: No Subject alternative name extension
1 certificate without subject alternative name extension was detected.

Issue E: IP address in dns name
A set of 10 certificates with an IP address coded as dns name was detected.

Issue F: Invalid country in countryName
A set of 4 certificates was detected.

Issue G: BR certificates with organizationName must include either localityName or StateOrProvinceName
1 certificate was detected.

All affected certificates were revoked!


1.	How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, via a discussion in mozilla.dev.security.policy, or via a Bugzilla bug), and the date. 
We implemented the check-tools (CABlint, X509lint, zlint) and scanned all not expired and not revoked server certificates. The scan was done on 2017-09-19.

2.	A timeline of the actions your CA took in response. 
See of attached file "T-Systems_Findings_20171006.pdf"

3.	Confirmation that your CA has stopped issuing TLS/SSL certificates with the problem. 
See of attached file "T-Systems_Findings_20171006.pdf"

4.	A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued. 
Issue A: 2, first issued: 10.04.2015 09:37:16 UTC, last issued: 10.04.2015 09:40:24 UTC
Issue B: 3, first issued 06.04.2016 08:34:35 UTC, last issued: 02.02.2017 13:28:24 UTC
Issue C: 3, first issued 22.05.2017 12:25:07 UTC, last issued: 10.08.2017 09:08:40 UTC
Issue D: 1, issued: 15.11.2016 11:11:30 UTC
Issue E: 10, first issued 20.10.2014 09:37:47 UTC, last issued: 20.09.2016 05:30:33 UTC
Issue F: 4, first issued 15.04.2015 09:03:06 UTC, last issued: 29.04.2016 14:11:38 UTC
Issue G: 1, issued: 01.02.2017 14:20:45 UTC

5.	The complete certificate data for the problematic certificates. 
See attached file "T-Systems_Certificates_20171006.csv"

6.	Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now. 
See of attached file "T-Systems_Findings_20171006.pdf"

7.	List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.
See of attached file "T-Systems_Findings_20171006.pdf"

Comment 22

[Gervase Markham [:gerv]]

Bernd: we appreciate the fact that you have proactively scanned your certificate database for additional BR-compliance problems, and taken action. 

Thank you also for your report about the problems. It would be helpful if you could go a little deeper with your root cause analysis and remediation. For example, in one case, you state the problem was a certificate template, and the fix was to update the template to not have the problem. However, nothing was said about how you can prevent a template with an identical problem being created in the future.

This is just one example. Hopefully by analogy you can imagine the same question applied to the other issue types.

In addition, the attached file does not contain complete certificate data for the problematic certificates. I know you have said that there are legal difficulties in providing this in a timely fashion; do you know when you might be able to provide full data for some or all of the certificates?

Thanks,

Gerv

Comment 23

[Bernd]

It is likely, that we expressed ourselves unclear in section 3.3 of the report (T-Systems_Findings_20171006.pdf). The template handling was correct.
The ECC-extension (new feature: certificate requests with EC-keys) caused a system error. As a result of this, the certificates with EC-keys were issued with incorrect KeyUsage. In the subsequent tests this error was not detected.
We will not issue certificates with EC-keys, until the occurred error is fixed – see also remediation in section 3.3.2 of the report.

Regards,
Bernd

Comment 24

[Gervase Markham [:gerv]]

Bernd: comment #23 is not really an adequate response to comment #22. I am asking you to do some root cause analysis on the problems which led to these misissuances, so you can tell us what can be changed so the problems are unlikely to occur again.

We also still want to know when full certificate data will be available.

Gerv

Comment 25

[Bernd]

Hi Gerv,

to fix the problems and to improve our quality, we implemeted a set of measures:
- all measures as indicated in the incident report (T-Systems_Findings_20171006.pdf )
- we use the lint tools within the development and test process of new releases
- we use the lint tools to check all certificates immediately after issuing
- we are working on a feature to check all certificates BEFORE issuing
As a result, we did not issue any nonconforming certificate since this bug was filed. 

The German data protection laws did not change, so we are still not authorized to disclose the full certificate data (see
https://bugzilla.mozilla.org/show_bug.cgi?id=1391074#c15).

Regards,
Bernd

Comment 26

[Wayne Thayer]

(In reply to Bernd from comment #25)
> Hi Gerv,
> 
> to fix the problems and to improve our quality, we implemeted a set of
> measures:
> - all measures as indicated in the incident report
> (T-Systems_Findings_20171006.pdf )
> - we use the lint tools within the development and test process of new
> releases
> - we use the lint tools to check all certificates immediately after issuing
> - we are working on a feature to check all certificates BEFORE issuing
> As a result, we did not issue any nonconforming certificate since this bug
> was filed. 
> 
> The German data protection laws did not change, so we are still not
> authorized to disclose the full certificate data (see
> https://bugzilla.mozilla.org/show_bug.cgi?id=1391074#c15).
> 
> Regards,
> Bernd

Bernd: you have explained the actions you took to prevent the problems from recurring, but still have not shown that any sort of root cause analysis has been performed, or shared the findings from that activity. For your reference, a common technique for root cause analysis is described here: https://en.wikipedia.org/wiki/5_Whys

Comment 27

[Firefox Bug Husbandry Bot]

Changing QA contact per https://bugzilla.mozilla.org/show_bug.cgi?id=1438254

Comment 28

[Bernd]

Wayne: all information in the Incident Report (T-Systems_Findings_20171006.pdf, https://bug1391074.bmoattachments.org/attachment.cgi?id=8915934) are still valid. 
To provide a better understanding why the errors occurred, we have listed the root causes in sequence below:

Issue A: Metadata in OU
-	The PKI system had an error within the asn.1 definition of the OU field
-	A single blank was entered into the OU field
-	The system removed this blank (trimming -> OU field is empty)
-	As a result, the empty OU field was entered into the certificate
-	The error (empty OU field) was not detected by the established monitoring
- 	The system requirements were not sufficient 

Issue B: ST appears to only include metadata
-	The PKI system had an error within the asn.1 definition of the ST field 
-	A single blank was entered into the ST field
-	The system removed this blank (trimming -> ST field is empty)
-	As a result, the empty ST field  was entered into the certificate
-	The error (empty ST field) was not detected by the established monitoring
- 	The system requirements were not sufficient 

Issue C: Unallowed key usage for EC public key (Key Encipherment)
-	Due to a PKI system error the same key usage definition as being defined for RSA keys was used
-	The ECC key was not correctly recognized by the system 
-	As a result, the wrong template and KeyUsage was used
-	The error was not detected by the established monitoring
-	At the time of this error, the monitoring was not yet refined

Issue D: No Subject alternative name extension
-	The error was caused by a database migration of standard certificates
-	Old certificates were erroneously migrated (SAN entry not yet a mandatory field)
-	As a result, the SAN field was not set when they were renewed
-	The error was not detected by the established monitoring
-	At the time of this error, the monitoring was not yet refined

Issue E: IP address in dns name
-	The required format of the SAN field from the BR (iPAddress containing the IP address of a server [7.1.4.2.1. Subject Alternative Name Extension]) was not implemented correctly in Internet Explorer up to version 8
-	As a result, the following warning was displayed for BR-compliant certificates: "Certificate address mismatch"
-	As a workaround, the IP address was entered in the SAN field for the dNSName
-	Issued certificates (old, faulty) were not blocked for these same reasons
-	The implementation of the requirement was only delayed due to the uneven customer/market needs of the German market (high IE8 market share)

Issue F: Invalid country in countryName
-	The country codes in accordance with iso-3166-alpha2 are checked manually
-	Unpermitted country codes were erroneously accepted
-	The knowledge of the country codes were not adequate

Issue G: BR certificates with organizationName must include either localityName or StateOrProvinceName
-	The customer groups are configured manually
-	Due to human error an incorrect template was assigned
-	The training measures were not adequate

Comment 29

[Wayne Thayer]

Bernd: has the linting "feature to check all certificates BEFORE issuing" been completed? If not, when do you expect it to be completed?

Comment 30

[Wayne Thayer]

Bernd: please respond to the question in comment #29.

Comment 31

[Wayne Thayer]

Arnold: please respond to the question in comment #29.

Comment 32

[Ryan Sleevi]

Arnold, Bernd: Please provide an update.

Comment 33

[Arnold Essing]

The Lint service for our Trust Center Services has been developed and we installed and tested it successfully in our test-environment. The first available timeslot for a change to our production environment was planned to the beginning of next week. So next week we will post an update here.

Comment 34

[Arnold Essing]

Our pre-issuance linting tool was configured for the following Intermediate CAs:
-TeleSec ServerPass Extended Validation Class 3 CA
-TeleSec ServerPass Class 2 CA
-TeleSec ServerPass DE-2
An update will be posted latest at the beginning of next week.

Comment 35

[Arnold Essing]

Our pre-issuance linting tool is now also configured for the following Intermediate CAs:
-Shared Business CA 4
-TeleSec Business CA 1
Now all T-Systems Intermediates which are issuing public server certificates use our pre-issuance linting tool.

Comment 36

[Wayne Thayer]

(In reply to Arnold Essing from comment #35)

Our pre-issuance linting tool is now also configured for the following Intermediate CAs:
-Shared Business CA 4
-TeleSec Business CA 1
Now all T-Systems Intermediates which are issuing public server certificates use our pre-issuance linting tool.

Excellent. I believe all remediations have now been completed.