1611458 - Asseco DS / Certum: Invalid value in SAN dNSName

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Wojciech Trapczyński]

User Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36

Actual results:

During the internal review process, we found that one of the recently issued SSL certificates has an invalid value in Subject Alternative Name. An IP address has been put in dNSName instead of in iPAddress. The misissued certificate has already been revoked: https://crt.sh/?id=2365173299.

We will provide an incident report no later than January 31, 2020.

Comment 1

[Wojciech Trapczyński]

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

During the internal review process, we found that one of the recently issued SSL certificates has an invalid value in Subject Alternative Name. An IP address has been put in dNSName instead of in iPAddress. The misissued certificate has already been revoked: https://crt.sh/?id=2365173299.

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

All times are UTC.

2019-10-24 14:00 - We installed a new version of an application for managing certificates issuance.
2019-10-30 06:15 - We discovered a bug in the application that under certain circumstances makes problems with issuing SSL certificates with IP address in Subject Alternative Name.
2019-10-31 13:00 - Until the new version with the fix will be installed on production we introduced a simple procedure to workaround the problem.
2020-01-24 12:00 - During the internal review process, we found that one of the recently issued SSL certificates has an invalid value in Subject Alternative Name. An IP address has been put in dNSName instead of in iPAddress. We started the investigation.
2020-01-24 12:30 - The misissued certificate was revoked.
2020-01-24 13:00 - We found the direct reason for this misissuance. We immediately stopped using a workaround procedure that leads to this error. We blocked the possibility to repeat this to misissuance with this faulty path.

3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

We issued one certificate on 2020-01-23. The next day we made a procedural change that prevents repeating it.

4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

One certificate issued on 2020-01-23 at 11:47:15 UTC.

5. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

https://crt.sh/?id=2365173299

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

The mistake was made during the manual process of correcting the certification request. In general, we avoid such manual changes but in this case, it was necessary due to an application error which makes it impossible to issue the SSL certificate with IP address in Subject Alternative Name. We classified this bug with normal priority and we planned to implement the fix in the next releases.

We followed the manual procedure many times until the 2020-01-23 when an additional white space at the end of one of the IP address caused that IP address has been treated as a domain. As a result, this IP address was placed in dNSName.

The misissuance was detected the next day.

7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to these things.

We stopped using the manual procedure to correct the certification requests immediately after the incident. Even we knew the direct reason for the error we decided that is too risky to still use this workaround.

We have released version with fix and we are going to install it on production no later than 2020-02-07. After this update, using the temporary procedure will not be necessary.

We have active pre-issuance linting on production (we based on ZLint library) but we had to make one exclude rule for checking Subject Alternative Name. The reason for that exception is the Zlint returns an error for U-label Internationalized Domain Name (IDN) in Common Name. We are going to change IDN representation in Common Name to A-label in the first half of this year. Then we will be able to remove this exclude rule from pre-issuance linting.

Comment 2

[Ryan Sleevi]

Thanks for reporting this! I'm glad to hear this was quickly detected, but I'm quite concerned about this incident.

1. This is not the first time Asseco DS / Certum has issued invalid dNSNames. For example, Bug 1524195 details the same issue. That issue was opened by a security researcher, Jonathan Rudenberg, who filed a number of incidents against a number of CAs. Looking holistically at the bugs and discussions related to that, it became clear it was essential to validate that a dNSName SAN was a well-formed domain name, which would have prevented this issuance.
   Questions:
   1. Did Asseco DS / Certum follow those industry discussions and the other bugs?
   2. Did Asseco DS / Certum have any systems in place to validate the well-formedness of dNSNames against the relevant RFCs, independent of linting?
2. The report does not provide detail about what those "certain circumstances" are that create problems. Can you provide more details and explanation about those situations and what their root cause is?
   The goal of this exercise is to help understand how bugs like this happen, so that they can be better prevented in the future, by Asseco DS / Certum and by all CAs.
3. Asseco DS / Certum was aware of the issue at 2019-10-30, but did not discover the root cause until 2020-01-24. Why is that, and what steps have been taken to ensure that similar delays don't happen in the future?
   As a publicly-trusted CA, Asseco DS / Certum is expected to act beyond reproach. Temporary solutions should be just that, or possibly even halting issuance until the root cause is discovered. This is a meaningful gap, and understanding what the process and evaluation was at Asseco DS / Certum that lead to deciding this was not major, and how that process is being changed in light of this, is essential to the confidence and trust in Asseco DS / Certum.
4. What was the cause of the "application error" that prevented issuing an iPAddress SAN? When processing these certificates manually, how were the iPAddresses validated, if the system was not capable of doing so?
5. You stated this missed pre-issuance lints due to "one exclude rule for checking Subject Alternative Name"
   1. Can you confirm there are no other lints that are disabled?
   2. What was the decision making process that lead to disabling this lint? Have you reviewed that process and updated how it will be determined to disable lints in the future?
   3. What was the use case for including U-Labels within the IDN, given the extensive discussions in the CA/Browser Forum?

Comment 3

[Wojciech Trapczyński]

Please see my comments inline.

(In reply to Ryan Sleevi from comment #2)

Thanks for reporting this! I'm glad to hear this was quickly detected, but I'm quite concerned about this incident.

1. This is not the first time Asseco DS / Certum has issued invalid dNSNames. For example, Bug 1524195 details the same issue. That issue was opened by a security researcher, Jonathan Rudenberg, who filed a number of incidents against a number of CAs. Looking holistically at the bugs and discussions related to that, it became clear it was essential to validate that a dNSName SAN was a well-formed domain name, which would have prevented this issuance.
   Questions:
   1. Did Asseco DS / Certum follow those industry discussions and the other bugs?

Yes, we did. The bug you mentioned concerns the issue with handling iPAddress by older browsers. I wrote in this bug "We were issuing such certificates because placing IP address just in SAN iPAddress make them unusable in some browsers" so, it was not a result of incorrect dNSName validation but intended action we took to comply with browsers implementation. Other CAs did the same.

2. Did Asseco DS / Certum have any systems in place to validate the well-formedness of dNSNames against the relevant RFCs, independent of linting?

Yes, all certification requests are validated at the moment of submission. So, there is no possibility to put in certification request invalid domain or IP address. Of course, all domains are saved as dNSNames and all IP addresses are saved as iPAddress. The pre-issuance liniting is an additional step that protects us if other validations failed.

2. The report does not provide detail about what those "certain circumstances" are that create problems. Can you provide more details and explanation about those situations and what their root cause is?
   The goal of this exercise is to help understand how bugs like this happen, so that they can be better prevented in the future, by Asseco DS / Certum and by all CAs.

By writing „certain circumstances" I had in mind that this problem occurs only if the certification request was submitted by the web application (and, of course, contains IP address). So, that is less than 5% of all our certification requests. More details about the root cause I described in point 4.

3. Asseco DS / Certum was aware of the issue at 2019-10-30, but did not discover the root cause until 2020-01-24. Why is that, and what steps have been taken to ensure that similar delays don't happen in the future?
   As a publicly-trusted CA, Asseco DS / Certum is expected to act beyond reproach. Temporary solutions should be just that, or possibly even halting issuance until the root cause is discovered. This is a meaningful gap, and understanding what the process and evaluation was at Asseco DS / Certum that lead to deciding this was not major, and how that process is being changed in light of this, is essential to the confidence and trust in Asseco DS / Certum.

I guess that we have some misunderstanding here. I realized that I omitted unintentionally one important date in the timeline from point 2 of the incident report. On 2019-10-31, so it is one day after we were aware, we discovered the root cause and fixed it. Knowing the root cause and having knowledge about the number of potential certification requests that may be affected by this bug, we decided that this fix will be installed in the normal release cycle. And only then we decided to use a temporary solution.

4. What was the cause of the "application error" that prevented issuing an iPAddress SAN? When processing these certificates manually, how were the iPAddresses validated, if the system was not capable of doing so?

The error was in the improper creation of SAN extension at the first stage of the processing of certification requests. As a result of it, the invalid SAN was blocked in the next step when the certification request is creating. The validation of IP addresses was always active at the moment of the certification request submission.

The temporary solution adjusted the SAN extension and the certificate may have been issued. During one of the correcting process additional space was put at the end of iPAddress field and then this field was incorrectly classified as dNSName. These kinds of errors are very rare and are detected by pre-issuance linting. In this case, this error was not detected due to exceptions I listed in point 5.

5. You stated this missed pre-issuance lints due to "one exclude rule for checking Subject Alternative Name"
   1. Can you confirm there are no other lints that are disabled?

We excluded three lints but all three for the same reason (U-Label in Common Name). In details our excluded lints are (based on https://docs.google.com/spreadsheets/d/1ywp0op9mkTaggigpdF2YMTubepowJ50KQBhc_b00e-Y/edit#gid=0):

e_dnsname_bad_character_in_label -> Characters in labels of DNSNames MUST be alphanumeric, - , _ or *
e_subject_common_name_not_from_san -> The common name field in subscriber certificates must include only names from the SAN extension
e_dnsname_not_valid_tld -> DNSNames must have a valid TLD

2. What was the decision making process that lead to disabling this lint? Have you reviewed that process and updated how it will be determined to disable lints in the future?

We launched the pre-issuance liniting in July, 2019. At that time we reviewed Zlint issues using crt.sh database for all our TLS certificates. On that basis, we determined lints that are not errors, in terms of compliance with standards, but they that may cause blocking of issuance many TLS certificates. Having for all three excluded lints good additional validation in the main application we calculated that the risk for misissuance is very low and then decided to do so. The initial configuration contains three disabled lints described above and there have never been other disabled lints. Simultaneously, we determined what we need to do in order to remove disabled lints. We made analysis and works have been scheduled.

3. What was the use case for including U-Labels within the IDN, given the extensive discussions in the CA/Browser Forum?

We have always been doing this probably having in mind that old browsers did not display A-Label from Common Name correctly. We are aware that today this not reasonable and as I mentioned we have a plan to change it.

Comment 4

[Ryan Sleevi]

Thanks. These are the sorts of details that we look for in a good incident report up-front, and help address concerns early on.

If I understand the flaw correctly:

• If a certificate was submitted through the web interface (< 5% of Asseco DS / Certum's certificates)
• And it contained an IP Address (what % of overall requests is this?)
  Then
• The IP address would be validated in the first step
• A SAN:dNSName would be constructed and it would be blocked
• An agent would need to manually correct this to a SAN:iPAddress

During this process, an invalid iPAddress character was added (space), which caused it to still be a SAN:dNSName, hence this bug.

Is this correct? If not, it might help to walk through and explain the conditions for this and the process that lead to it.

Two different lints could have caught this issue - e_dnsname_bad_character_in_label and e_dnsname_not_valid_tld - but they had been disabled due to the use of U-Labels in the commonName.

In terms of understanding next steps being taken:

• 2020-01-30 - No new certificates with IPs are being issued
• 2020-02-07 - System fix to guarantee IP addresses are only added to iPAddress fields
• ????-??-?? - Re-enable all disabled lints
• ????-??-?? - Stop issuing U-Labels in commonName

Is that roughly correct? Can you fill in the dates with a more concrete timeline?

Comment 5

[Wojciech Trapczyński]

(In reply to Ryan Sleevi from comment #4)

Thanks. These are the sorts of details that we look for in a good incident report up-front, and help address concerns early on.

If I understand the flaw correctly:

• If a certificate was submitted through the web interface (< 5% of Asseco DS / Certum's certificates)
• And it contained an IP Address (what % of overall requests is this?)
  Then
• The IP address would be validated in the first step
• A SAN:dNSName would be constructed and it would be blocked
• An agent would need to manually correct this to a SAN:iPAddress

During this process, an invalid iPAddress character was added (space), which caused it to still be a SAN:dNSName, hence this bug.

Is this correct? If not, it might help to walk through and explain the conditions for this and the process that lead to it.

Yes, your summary is correct.

We have significantly less than 1% certification requests containing an IP Address.

Two different lints could have caught this issue - e_dnsname_bad_character_in_label and e_dnsname_not_valid_tld - but they had been disabled due to the use of U-Labels in the commonName.

Yes, this is true .

In terms of understanding next steps being taken:

• 2020-01-30 - No new certificates with IPs are being issued
• 2020-02-07 - System fix to guarantee IP addresses are only added to iPAddress fields
• ????-??-?? - Re-enable all disabled lints
• ????-??-?? - Stop issuing U-Labels in commonName

Is that roughly correct? Can you fill in the dates with a more concrete timeline?
Corrected timeline:

2020-01-24 - No new certificates with IPs are being issued. It concerns only these certificates subbmited through the web interface. Others not affected were issued.
2020-02-06 - System fix to guarantee IP addresses are only added to iPAddress fields.
2020-06-30 - Stop issuing U-Labels in commonName and simultaneously re-enable all disabled lints. This date is the deadline.

Comment 6

[Ryan Sleevi]

Could you explain why the delay until June 30? This seems like something that could be done today

Comment 7

[Wojciech Trapczyński]

There are two reasons:

• This change is a part of the bigger changes we are going to implement in our certificate profiles itself and in our certificate profiles management. All of these changes are connected. As these changes will have an impact on the whole validation process we need to test this in a more extended way than usual. We have already an early established process of testing all of them. For this reason, we do not want to do it separately.

• This change itself is connected not only with the process of the process issuing the certificate. It is not enough to change only the certificate profile. We need to adjust the whole validation process also. That why it is not a kind of change we may do immediately.

As I mentioned the end of June is the deadline but we hope that these changes will come sooner. However, we will be able to confirm this only after positive tests results.

Comment 8

[Ryan Sleevi]

I definitely think erring on the side of caution (disabling U-Label issuance) seems a wiser approach, given past issues, and I think any issues going forward, which would have been avoided by doing so, will be seen as rather serious.

That said, we've not historically required a CA to make changes by a particular date, just used the date and reasoning selected as part of understanding the overall approach to security.

As I have no further questions, setting N-I to Wayne for resolution/closure.

Comment 9

[Wayne Thayer]

It appears to me that the move to A-labels in the CN and subsequent enabling if disabled lints is part of the remediation for this issue, so I have set the next update to 1-July. Please update this bug when those changes are complete, or by 1-July if there is a delay.

Comment 10

[Wojciech Trapczyński]

I will update this bug when we made those changes.

Comment 11

[Wojciech Trapczyński]

(In reply to Wayne Thayer from comment #9)

It appears to me that the move to A-labels in the CN and subsequent enabling if disabled lints is part of the remediation for this issue, so I have set the next update to 1-July. Please update this bug when those changes are complete, or by 1-July if there is a delay.

I would like to inform you that we moved to A-labels in the Common Name and on 2020-06-02 at 07:00 (UTC) we removed all ZLint exceptions.

Comment 12

[Wayne Thayer]

It appears that all questions have been answered and remediation is complete.