Please see my comments inline.
(In reply to Ryan Sleevi from comment #2)
Thanks for reporting this! I'm glad to hear this was quickly detected, but I'm quite concerned about this incident.
- This is not the first time Asseco DS / Certum has issued invalid dNSNames. For example, Bug 1524195 details the same issue. That issue was opened by a security researcher, Jonathan Rudenberg, who filed a number of incidents against a number of CAs. Looking holistically at the bugs and discussions related to that, it became clear it was essential to validate that a dNSName SAN was a well-formed domain name, which would have prevented this issuance.
- Did Asseco DS / Certum follow those industry discussions and the other bugs?
Yes, we did. The bug you mentioned concerns the issue with handling iPAddress by older browsers. I wrote in this bug "We were issuing such certificates because placing IP address just in SAN iPAddress make them unusable in some browsers" so, it was not a result of incorrect dNSName validation but intended action we took to comply with browsers implementation. Other CAs did the same.
2. Did Asseco DS / Certum have any systems in place to validate the well-formedness of dNSNames against the relevant RFCs, independent of linting?
Yes, all certification requests are validated at the moment of submission. So, there is no possibility to put in certification request invalid domain or IP address. Of course, all domains are saved as dNSNames and all IP addresses are saved as iPAddress. The pre-issuance liniting is an additional step that protects us if other validations failed.
- The report does not provide detail about what those "certain circumstances" are that create problems. Can you provide more details and explanation about those situations and what their root cause is?
The goal of this exercise is to help understand how bugs like this happen, so that they can be better prevented in the future, by Asseco DS / Certum and by all CAs.
By writing „certain circumstances" I had in mind that this problem occurs only if the certification request was submitted by the web application (and, of course, contains IP address). So, that is less than 5% of all our certification requests. More details about the root cause I described in point 4.
- Asseco DS / Certum was aware of the issue at 2019-10-30, but did not discover the root cause until 2020-01-24. Why is that, and what steps have been taken to ensure that similar delays don't happen in the future?
As a publicly-trusted CA, Asseco DS / Certum is expected to act beyond reproach. Temporary solutions should be just that, or possibly even halting issuance until the root cause is discovered. This is a meaningful gap, and understanding what the process and evaluation was at Asseco DS / Certum that lead to deciding this was not major, and how that process is being changed in light of this, is essential to the confidence and trust in Asseco DS / Certum.
I guess that we have some misunderstanding here. I realized that I omitted unintentionally one important date in the timeline from point 2 of the incident report. On 2019-10-31, so it is one day after we were aware, we discovered the root cause and fixed it. Knowing the root cause and having knowledge about the number of potential certification requests that may be affected by this bug, we decided that this fix will be installed in the normal release cycle. And only then we decided to use a temporary solution.
- What was the cause of the "application error" that prevented issuing an iPAddress SAN? When processing these certificates manually, how were the iPAddresses validated, if the system was not capable of doing so?
The error was in the improper creation of SAN extension at the first stage of the processing of certification requests. As a result of it, the invalid SAN was blocked in the next step when the certification request is creating. The validation of IP addresses was always active at the moment of the certification request submission.
The temporary solution adjusted the SAN extension and the certificate may have been issued. During one of the correcting process additional space was put at the end of iPAddress field and then this field was incorrectly classified as dNSName. These kinds of errors are very rare and are detected by pre-issuance linting. In this case, this error was not detected due to exceptions I listed in point 5.
- You stated this missed pre-issuance lints due to "one exclude rule for checking Subject Alternative Name"
- Can you confirm there are no other lints that are disabled?
We excluded three lints but all three for the same reason (U-Label in Common Name). In details our excluded lints are (based on https://docs.google.com/spreadsheets/d/1ywp0op9mkTaggigpdF2YMTubepowJ50KQBhc_b00e-Y/edit#gid=0):
e_dnsname_bad_character_in_label -> Characters in labels of DNSNames MUST be alphanumeric, - , _ or *
e_subject_common_name_not_from_san -> The common name field in subscriber certificates must include only names from the SAN extension
e_dnsname_not_valid_tld -> DNSNames must have a valid TLD
2. What was the decision making process that lead to disabling this lint? Have you reviewed that process and updated how it will be determined to disable lints in the future?
We launched the pre-issuance liniting in July, 2019. At that time we reviewed Zlint issues using crt.sh database for all our TLS certificates. On that basis, we determined lints that are not errors, in terms of compliance with standards, but they that may cause blocking of issuance many TLS certificates. Having for all three excluded lints good additional validation in the main application we calculated that the risk for misissuance is very low and then decided to do so. The initial configuration contains three disabled lints described above and there have never been other disabled lints. Simultaneously, we determined what we need to do in order to remove disabled lints. We made analysis and works have been scheduled.
3. What was the use case for including U-Labels within the IDN, given the extensive discussions in the CA/Browser Forum?
We have always been doing this probably having in mind that old browsers did not display A-Label from Common Name correctly. We are aware that today this not reasonable and as I mentioned we have a plan to change it.