1462797 - E-Tugra: Improper DER results in failure to comply with RFC 5280 - Invalid characters in PrintableString

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Ryan Sleevi]

Example certificate: https://crt.sh/?id=117812066&opt=cablint,x509lint,zlint

This certificate fails to parse, as it contains a Subject attribute with a serialNumber field, and the serialNumber contains an underscore ("_"), which is not a valid character for a PrintableString.

It's further unclear if this represents a potential violation of 7.1.4.2(j), as to whether such an underscore is a "metadata" field, but I believe it reasonably can be argued as such.

Comment 1

[Wayne Thayer]

Davut: As Ryan indicated, this appears to be a misissued certificate. Please provide an incident report, as described here:
https://wiki.mozilla.org/CA/Responding_To_A_Misissuance#Incident_Report
The incident report should be posted to the mozilla.dev.security.policy forum and added to this bug.

Comment 2

[Davut Tokgöz]

Hi,
We continue to review the case. I will provide incident report as soon as possible.

Comment 3

[Davut Tokgöz]

1  How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.
E-Tugra:
We first aware from the problem via BugZilla

2  A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.
E-Tugra:
The following actions took in response after report.
• Certificate is being revoked and a new certificate was issued for the certificate owner. The certificate will be revoked in a week. The certificate owner has a problem to change the system with new certificates. So for not giving a damage to their operations, we are waiting for the replacement for revocation.
• All certificate database is searched for whether the problems is exist more or not. No more certificate is found with the problem.
• The root analysis is applied and we found that why this problem occurs. (the reason of problem is applied in 6)
• The pre-issue and post-issue control libraries in our system is upgraded and test against the problems. We try to create similar problem is we cannot after upgrade.

3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.
E-Tugra:
The related certificate was planned for revocation and replacing with the new certificate getting bug notification from Bugzilla and all problems are fixed in our systems. The problem reasons are fixed and no more certificates will be produced with the problems.

4  A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.
E-Tugra:
There is only one certificate with this problem. There is no more certificate with the same problem.

5  The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.
E-Tugra:
There is only one certificate as reported. https://crt.sh/?id=117812066&opt=cablint,x509lint,zlint

6  Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
E-Tugra:
The certificates owner requested an EV certificate at first. And during EV validation process, some jurisdiction information was entered to the system on our RA systems. Later, we recommended the applicant to convert application to OV based certificates due to customer needs and budgets. The serial number that was original set for jurisdiction number of organization continue to exist after converting application to OV SSL. In fact, We are not being set serialnumber on OV certificates. After final application validation. The certificate is issued with the organization serialnumber.
After getting bug report from Bugzilla, we update our RA system by not allowing extra fields that is not defined in Baseline Requirements.

7  List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.
E-Tugra
All required measures are taken for now. The certificates are controlled by the system. These are enough for preventing the certificates with these measures.

Comment 4

[Ryan Sleevi]

Apologies, this dropped off the radar for reviewing incident reports.

> All required measures are taken for now. The certificates are controlled by the system. These are enough for preventing the certificates with these measures.

This root cause analysis has not addressed how the invalid DER was produced or how it has been resolved/prevented. Can you please explain what steps that was taken? The incident report only seems to respond to serialNumber for OV (in #6 and #7), but that wasn't even the original reported issue.

I highlight this, because even as recently as this past week, it appears that there are still misissued OV certificates being created:

https://crt.sh/?id=1062781171&opt=x509lint
https://crt.sh/?id=1061518954&opt=x509lint
https://crt.sh/?id=1061497011&opt=x509lint

Similarly, certs like https://crt.sh/?id=1062781171&opt=x509lint give me great pause about the design and suitability of this system. Even if it is not strictly an error, the prevalence of that issue with "testing" certificates (such as https://crt.sh/?id=1061295623&opt=x509lint or https://crt.sh/?id=1070751823&opt=x509lint ) suggests there are issues. While I think those I flagged should be reported by the CA as a separate issue, I do want to make sure we get a response to this - which is about invalid DER production for the certificate and how it was resolved.

Comment 5

[Davut Tokgöz]

Hi
Between Dec 27 and 31, we have issued tens of certificates for testing our new Intermediate CA and its’ end entity certificate profiles. All domain names that we used for these certificates belong to E-Tugra (etugratest.com, ssl-turk and more). During these tests, we saw some certificates were invalid because of some settings on new profiles and we took action for these cases. All the certificates that listed in your comments were issued for this test purpose. We did not report these certificates, because all these domains are owned by us, and issued certificates would not be used and would be revoked because they were issued for test, not for subscribers.
We are preparing a detailed report for these certificates with the actions taken and will be taken. we will post it few days.
When we said "all the measures are taken", it means that we have implemented some level of certificate controls in our certificate management systems. We have always continued to improve these controls. Currently, there are 3 levels of control in use.
1: Preparation Level Control: According to type of certificate that is requested, we forced to ensure controlling all the requirements regarding the verification of any data that is subject to certificate.
2. Pre-Issue Control: All subject fields are controlled according to the requirements of CaBForum, If any violation exist, the issue request is paused until the problem is fixed.
3. Post-Issue Control: After issuing a certificate, the issued certificate is controlled programmatically. If any problem exists in the certificate, the system is set to create a correlation and block the sending of the certificate to the applicant. We also used to check the certificates with CaBLint. We began to implement revoking the certificates automatically after failure of checking certificate with CaBLint and/or Zint just after issuing.

Comment 6

[Ryan Sleevi]

Comment #4 suggests that the preparation, pre-issuance, and post-issuance controls are not satisfactory to meet the Baseline Requirements. The goal of this incident report is to understand how those procedures have been concretely improved and what meaningful changes have been made.

Please provide an update with the incident report.

Comment 7

[Davut Tokgöz]

I will provide an indicent report in few days including improvements and actions taken.

Comment 8

[Davut Tokgöz]

1 How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.
E-Tugra:
• We were aware of the problem from both internal monitoring and via BugZilla

2 A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.
E-Tugra:
The following actions are taken after report.
• Dec 26-31 2018 : 28 of certificates were issued between 26 to 31 for testing purposes of new intermediate CA’s. 6 of them were incompatibles with RFC 5280.
• Jan 2: Bugzilla Reported the problem.
• Jan 3: 6 certificates including those which were reported in Bugzilla were revoked.
• Jan 3: All certificates that were issued for test purposes belong to the domain names owned by us were revoked. (etugratest.com, lspro.com.tr, ssl-turk.com, ugur.site and some certificates of e-tugra.com.tr)
• We found the problem on certificate issue process and we immediately decided to rebuild the software to fix the problem.
• Jan 8: We tested all certificates for the same problem and discovered that 13 certificates were issued with “Duplicate DNS names in SAN” problem or “Locality name without Organization Name” problem. Some of them are being replaced and revoked, some of them are being changed and will be revoked after they are changed.
• Jan 31: With system upgrade all bugs that cause problem on certificate are completely fixed.

3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.
   E-Tugra:
   • As of January 31st, the reasons of the problems are fixed and auto control and alert system were introduced. No more certificates will be produced with these problems. We continue to improve the system day by day.

4 A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.
E-Tugra:
For “Certificates without Organization Name must not include Locality Name” problem: totally 30 Certificate: 21 of them are issued for test purpose with own domains and all of them were revoked or is being revoked. Min and Max issue Dates: 2017.04.19 15:16 - 2018.12.28 11:26
For “Duplicate DNS entry in SAN” problem: totally 25 Certificate: 21 of them are issued for test purpose with own domains and all of them were revoked or is being revoked. Min and Max issue Dates: 2017.10.18 12:29 - 2018.12.31 16:19

5 The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.
E-Tugra:
An Excel spreadsheet was prepared with two pages. https://www.e-tugra.com.tr/downloads/reports/CertificatesWithErrorReport2.xlsx

6 Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
E-Tugra:
For “Certificates without Organization Name must not include Locality Name” problem:
We found that our system was checking the requirement of Locality if Organization Name exist. But we found that the reverse checking was not done and post issue controls did not cover these checks in some cases.
For “Duplicate DNS entry in SAN” problem: This issue was not controlled until Jan 3rd. This case occurred when more than one sub domains of same domain are included in a certificate.

7 List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.
E-Tugra
The certificate issue control steps on pre-issue and post-issue steps were rebuilt wholly as follows. We also made some changes on issuing process.

• Pre-issue Controls: We completely rebuilt all controls based on RFC 5280 and CabForum Baseline Requirements. The following controls before issuing a certificate was done successfully tested and were put into use on Jan 31.

• Minimum and Maximum length of fields
• Required fields or unallowed fields according to certificate types DV, OV and EV
• Cross controls between fields such as locality and organization
• Checking unallowed characters
• SAN entry checking and etc.
  We also continue testing and improving the pre-issue processes.

• Post-issue Controls: We also completely rebuilt all controls done after issuing a certificate. The issued certificate is parsed and controlled by own controls and also by Zlint (https://github.com/zmap/zlint) When any error output, the certificate is being revoked and create an incident Report Alert in our internal systems. When warning level output occurs, an incident Report Alert in our internal systems is created to take action. These controls were put into use on Jan 9th. We also continue testing and improving the post-issue also.

In addition to these validation controls, we reviewed all certificate profile in our CA systems and put into account all possible restrictions or requirements on profiles for each type of certificates.

We believe that all controls to prevent a mis issue were put into action. Moreover, we have taken measures to be notified at the time of mis issue. By this way when a certificate is issued which is not compatible with BR and RFC5280, our system gives alert to create incident report and notify to fix problem.

Comment 9

[Wayne Thayer]

Davut: thank you for the detailed response. It appears that remediation is complete.