1744827 - Entrust: SSL Certificates issued with Un-verified IP Addresses

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Bruce Morton]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36

Actual results:

Entrust became aware of the problem on 6 December 2021 at 19:22 UTC, when the post linting check indicated that there we 2 EV SSL certificates issued with IP addresses in the subjectAltName extension. The initial investigation indicated there was a bug which allowed un-verified IP addresses to be added to an OV or EV SSL certificate.

A full incident report is being prepared.

Comment 1

[Bruce Morton]

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

Entrust became aware of the problem on 6 December 2021 at 19:22 UTC, when the post linting check indicated that there we 2 EV SSL certificates issued with IP addresses in the subjectAltName extension. The initial investigation indicated there was a bug which allowed un-verified IP addresses to be added to an OV or an EV SSL certificate.

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

2021-12-06 19:22 UTC – Entrust post-issuance linter indicated that 2 EV SSL certificates were issued with IP addresses
2021-12-06 20:29 UTC – Investigation discovered the Subscriber did not request for the IP addresses to be verified. As such, it was determined that the issue also occurred with other Subscribers who had purchased OV SSL certificates.
2021-12-06 20:54 UTC – Nine (9) miss-issued certificates were documented. All OV SSL certificates had the IP address verified or were revoked with 24 hours. Both EV SSL certificates were revoked.
2021-12-06 21:00 UTC – Compliance team reviewed the incident and provided direction to open cases and advise the Subscribers of the issue. It was determined that either the IP addresses must be verified or the certificates must be revoked within 24 hours.
2021-12-07 15:30 UTC – Three (3) additional certificates were documented. All certificates were revoked on the same day by 19:32 UTC.
2021-12-08 14:35 UTC – Patch deployed to resolve the issue.

3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

Entrust has patched the system and no longer issues SSL certificates with non-verified IP addresses or EV SSL certificates with IP addresses.

4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

Certificates are listed in item 5. The first certificate was issued on 30 November 2021 and the last one was issued on 7 December 2021.

5. The complete certificate data for the problematic certificates.

https://crt.sh/?id=5709042724
https://crt.sh/?id=5715160388
https://crt.sh/?id=5715211065
https://crt.sh/?id=5729482345
https://crt.sh/?id=5730617809
https://crt.sh/?id=5748703029
https://crt.sh/?id=5748873316
https://crt.sh/?id=5748903213
https://crt.sh/?id=5748906482
https://crt.sh/?id=5754252862
https://crt.sh/?id=5754275869
https://crt.sh/?id=5754288527

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

The format of the subjectAltNames (SANs) passed into the validation service was changed during a recent performance improvement initiative. Previously, some paths passed the SANs in simple comma-separated format (value, value, value, …) and other paths used a tag=value format where tag indicates the SAN type and value has the SAN value. To optimize validation caching, all paths were changed to use the tag=value format to give more cache hits. Due to a bug, validation looked only at dNSName SANs so iPAddress SANs were filtered out and not validated; however they remained in the certificate request for inclusion in the certificate.

The issue was not caught because our automated tests do not include a negative test case for unapproved IP addresses. We do have negative test cases for domain names.

Pre-issuance linting did not detect the problem as zlint can not verify that an IP address has been verified.

Post-linting does not verify that the IP address has been verified, but did provide an error when an IP address was found in an EV certificate.

7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

Entrust will add an automated negative test case for IP addresses.

Entrust to develop and recommend a change to zlint to support an error if an EV SSL certificate is issued with an IP address, see https://github.com/zmap/zlint/pull/650.

Entrust will plan to update post-linting to verify the IP address in a certificate to the IP addresses approved for the Subscriber, where an error will occur if there is not a match.

Comment 2

[Ben Wilson]

Hi Bruce,
Do you have time estimates for implementation of the remediation steps submitted in answer to item 7?
Thanks,
Ben

Comment 3

[Bruce Morton]

Hi Ben,

The automated negative test case for IP addresses will be deployed by 28 January 2022. I will provide status on or before that date.
Regarding zlint, we will update after the next release is available per our normal practice.
Regarding post-linting, I believe this is out of scope for remediation as it occurs after a miss-issue has occurred.

Thanks, Bruce.

Comment 4

[Rob Stradling]

(Emphasis mine)

2021-12-06 21:00 UTC – Compliance team reviewed the incident and provided direction to open cases and advise the Subscribers of the issue. It was determined that either the IP addresses must be verified or the certificates must be revoked within 24 hours.

BR 4.9.1.1 (Reasons for Revoking a Subscriber Certificate) says:
"The CA SHALL revoke a Certificate within 24 hours if one or more of the following
occurs: ...
5. The CA obtains evidence that the validation of domain authorization or control for
any Fully‐Qualified Domain Name or IP address in the Certificate should not be
relied upon."

Bruce, can you explain how your Compliance team is able to square BR 4.9.1.1(5) with your idea that revocation can be avoided if post-issuance validation of the IP addresses can be performed successfully?

(ISTM that the 24 hour revocation countdown started as soon as Entrust became aware of the misissued certificates mentioned in comment 1, and that the BRs don't permit retroactive validation to stop a revocation countdown).

Comment 5

[Bruce Morton]

Hi Rob, we treated the detected problem similar to a third party Certificate Problem Report. We contacted the Subscribers and advised of the issue which was detected. Since the IP address was not validated, the solution was to revoke within 24 hours of miss-issue detection. Since the Subscriber had asked for the IP address, we provided the alternative (for OV SSL certificates) to verify control of the IP address. The result would either be to have the certificate revoked or have the problem resolved. We considered this would limit the impact to the Subscribers and resolve the problem. Hopefully we weren't too far off track.

Comment 6

[Ryan Sleevi]

I'm not sure I follow.

At the time of this post, two certificates mentioned in Comment #5 are showing as not revoked:

• https://crt.sh/?id=5715211065&opt=ocsp
• https://crt.sh/?id=5730617809&opt=ocsp

As Rob mentioned, in Comment #4, the Baseline Requirements have clear expectations of revocation if the original certificate was not properly validated. Rob mentions 4.9.1.1 (5), but even if that is disputed by Entrust, there's 4.9.1.1 paragraph 2, Items 2 and 7.

Rob's highlighting that the suggestion by Entrust presently reads as it did not properly validate the certificates prior to issuance, but then retroactively validated, and as a result, decided not to revoke. That seems to be correct, based on the two above certificates, but I'm not sure if I'm reading your Comment #5 correctly Bruce, because it sounds like you're suggesting something different has happened? Could you clarify?

Comment 7

[Ryan Sleevi]

Looking longer-term, your root cause analysis provides useful technical details about the mistake, and the steps to take short-term corrective action.

However, it seems there's equally a process flaw here with how Entrust is designing and testing its changes. That is, I think it's reasonable to ask "Why wasn't there a negative test for IP addresses", and explore the development process factors that lead to that oversight. For any field, or for any change, it's possible for similar issues to be introduced (i.e. only testing positive cases, omitting negative cases), and it seems like it'd be a core part of any development workflow to ensure that you're covering this.

Comment 8

[Bruce Morton]

(In reply to Ryan Sleevi from comment #6)

Rob's highlighting that the suggestion by Entrust presently reads as it did not properly validate the certificates prior to issuance, but then retroactively validated, and as a result, decided not to revoke. That seems to be correct, based on the two above certificates, but I'm not sure if I'm reading your Comment #5 correctly Bruce, because it sounds like you're suggesting something different has happened? Could you clarify?

Hi Ryan, yes I believe that you understood what I was trying to say was that we retroactively validated, and as a result, decided not to revoke.

Comment 9

[Ryan Sleevi]

Right, that's never been permitted by the Baseline Requirements. Notably, this has come up in past incidents, such as Symantec's misissuance. As Rob has highlighted in Comment #4, there are a number of very clear requirements for revocation, ranging from both the failure to validate (at time of issuance) to the failure to abide by the CP/CPS (at time of issuance). It's unclear how Entrust reached the decision it did, so hopefully you can clarify, because this is quite concerning.

Comment 10

[Bruce Morton]

(In reply to Ryan Sleevi from comment #7)

Looking longer-term, your root cause analysis provides useful technical details about the mistake, and the steps to take short-term corrective action.

However, it seems there's equally a process flaw here with how Entrust is designing and testing its changes. That is, I think it's reasonable to ask "Why wasn't there a negative test for IP addresses", and explore the development process factors that lead to that oversight. For any field, or for any change, it's possible for similar issues to be introduced (i.e. only testing positive cases, omitting negative cases), and it seems like it'd be a core part of any development workflow to ensure that you're covering this.

This bug identified a process issue that this negative manual test case for certificates was not automated. To date we have identified one missing negative automated test and are in the process of reviewing all manual negative cases to ensure there is an automated negative test for each negative manual test. The QA process has been enhanced to track the ‘story tickets’ for automated negative tests on a page which is monitored by the QA Lead. This will ensure that all stories for an automated negative test are developed and included for execution in the automated test suite before the next ECS release. The automated negative test case update will be deployed by 28 January 2022. I will provide status on or before that date.

Comment 11

[JR Moir]

https://crt.sh/?id=5715211065&opt=ocsp
https://crt.sh/?id=5730617809&opt=ocsp

Both of these certificate have not been revoked as of now (19:16 CET, 29.12.2021)

Comment 9 says how this is not allowed. Why are the certificates not revoked?

Comment 12

[Paul van Brouwershaven]

We agree that our action was not strictly in line with point 7 of section 4.9.1.1 of the Baseline Requirements:

The CA is made aware that the Certificate was not issued in accordance with these Requirements or the CA's Certificate Policy or Certification Practice Statement;

To minimize the impact on the subscriber and relying parties we wanted to prevent revoking, requesting, and installing a new certificate with the same information, as this doesn't benefit anyone. With that in mind we verified if the certificate could have been 'issued in accordance with these Requirements or the CA's Certificate Policy or Certification Practice Statement'​.

I suggest we discuss this type of rectification with the Server Certificate Working Group in the new year, as I don't think ​it's the intention of the Baseline Requirements to disturb the WebPKI unnecessarily.

Comment 13

[Matthias]

(In reply to Paul van Brouwershaven from comment #12)

We agree that our action was not strictly in line with point 7 of section 4.9.1.1 of the Baseline Requirements:

The CA is made aware that the Certificate was not issued in accordance with these Requirements or the CA's Certificate Policy or Certification Practice Statement;

To minimize the impact on the subscriber and relying parties we wanted to prevent revoking, requesting, and installing a new certificate with the same information, as this doesn't benefit anyone. With that in mind we verified if the certificate could have been 'issued in accordance with these Requirements or the CA's Certificate Policy or Certification Practice Statement'​.

That implies that you only verified information if and when proof of verification is required for certificates, which is very worrying. Relying entities tend to rely on information included in the certificate, which includes the 'not before' date. If the information in the certificate was not verified according to the BR as of the 'not before' date (or in the information reuse period before that), then we can't rely on the certificate.

It is likely that an entity will base their decisions on (amongst others) the existance of certain certificates (e.g. connecting to servers, inputting forms, etc.). As such, it is important that these entities have a signal that shows that these decisions were based on certificates with unverified information; i.e. that those certificates should not have existed at that time; instead of the CA silently sweeping this fact under the rug and the relying parties being none the wiser.

I think this is a worrying precedent, and I don't think that you have provided enough reasons to not revoke, nor have you filed an issue regarding the non-revocation of these certificates.

I suggest we discuss this type of rectification with the Server Certificate Working Group in the new year, as I don't think ​it's the intention of the Baseline Requirements to disturb the WebPKI unnecessarily.

Doing the validation of a certificate only after issuance is effectively taking a loan on trust, and I am definately not OK with CAs unilaterally deciding to do so without prior explanation of the exceptional reasons for doing so and the subsequent agreement of the relevant root stores.

Note that I personally don't agree with post-issuance validation at all, but root stores might differ in opinion when the situation is properly explained. "No explanation" is significantly worse in my book than "some explanation"; but both are on the list of "things a CA should never rely upon".

Also note that I can't agree with any form of post-issuance information validation (but not: re-validation to ensure still-valid certificates ) becoming legal in the BR; it would make the WebPKI significantly less secure as a CA would then be "allowed" in some way or form to do lazy validation of just about anything and start issuing certificates for any subject, only revoking these certificates after the problem of failed validation was pointed out; which moves the job of information validation to the end user instead of the CA; which makes the use of CAs effectively worthless.

Comment 14

[Ryan Sleevi]

I whole heartedly agree with Comment #13.

(In reply to Paul van Brouwershaven from comment #12)

We agree that our action was not strictly in line with point 7 of section 4.9.1.1 of the Baseline Requirements:

The reply here, "not strictly", suggests that it's believed to be partially compliant. Is that a correct understanding of the above statement?

To minimize the impact on the subscriber and relying parties we wanted to prevent revoking, requesting, and installing a new certificate with the same information, as this doesn't benefit anyone.

Thanks for sharing the perspective, although it does seem to overlook a number of substantial (and long-discussed) issues. My hope is that this is accidental, and that comments like Matthias' Comment #13 provide greater insight into why the statement "doesn't benefit anyone" is both without merit and substance.

Relying parties and root stores take upon themselves all of the risk when including particular CAs within their products. They suffer the reputational risk, and the direct security harms to their users, if and when that CA fails to abide by the agreed-upon behaviour, whether it's a failure to format certificates in the correct manner (leading to, e.g. compatibility issues, which can increase the complexity of products due to workarounds or suffer user attrition due to not accepting such invalid certificates) or without proper validation.The CA is included on the basis of both their stated policies, and an assessment of how well those policies align with user needs.

This sort of post-issuance validation completely violates that expectation, and as a consequence, directly undermines trust in the CA. As Matthias points out, allowing this sort of behaviour is to say, in effect, "You can do whatever you want, as long as you don't get caught, and if you do get caught, there are no consequences."

Revocation serves to redress some of these concerns. First, it demonstrates the CAs own awareness of their failure to abide by the agreed-upon practices, and represents their commitment that they shall not benefit from such malfeasance, whether malicious or unintentional. It also serves as an important backstop for server operators, in signalling that a particular CA may not be willing or capable of behaving as it has agreed upon. Although there is some disruption, that only exists if and when the CA has materially failed its duties.

While an argument by analogy is not the strongest, in the hopes of communicating just how serious this is, the parallel here would be saying that it's OK to rob a bank, as long as you only steal as much as is in your account, and leave a note after so that they can debit your account. As a society, we obviously reject such brazen rejection of norms, and as it applies to a CA community, the sort of failure to abide by the expected behaviour is equally serious.

I suggest we discuss this type of rectification with the Server Certificate Working Group in the new year, as I don't think ​it's the intention of the Baseline Requirements to disturb the WebPKI unnecessarily.

I do think this is concerning. Whether or not it was intentional, this would have the direct effect of excluding participation of the broader community. I encourage Entrust to look at the behavior of other CAs when faced with such challenges, which include involving relying parties as stakeholders, by engaging with the Mozilla forum.

I think that, should Entrust look to discuss this in the CA/B Forum first, given the existence of this issue, this should be viewed very negatively with respect to how well Entrust values community feedback and how seriously they take compliance. My hope is that this suggestion just didn't consider this possibility, and that by making it very explicit, a different course of action can be considered.

I remain deeply concerned that, in light of repeated comments from multiple members (Comment #4, Comment #9, Comment #11) pointing out that this is a naked violation of the Baseline Requirements, and Entrust's own acknowledgement that they have violated the Baseline Requirements (Comment #12), Entrust has still not revoked these certificates.

1. Will you be revoking these certificates, now that you've realized the behaviour taken is completely out of line with the Baseline Requirements?
2. If so, when will you be revoking?
3. When can we expect a separate incident report for the failure to revoke?

Comment 15

[Bruce Morton]

Both certificates have been revoked today:

https://crt.sh/?id=5715211065&opt=ocsp
https://crt.sh/?id=5730617809&opt=ocsp

We will create a failure to revoke incident report no later than 7 January 2022.

Comment 16

[Bruce Morton]

We have posted the late revocation incident report, https://bugzilla.mozilla.org/show_bug.cgi?id=1748634.

I will provide a status of the automated negative test case the week of 17 January 2022.

Comment 17

[Bruce Morton]

The automated negative test was deployed today.

Comment 18

[Ben Wilson]

Should the "next update" be re-set?

Comment 19

[Bruce Morton]

(In reply to Ben Wilson from comment #18)

Should the "next update" be re-set?

Hi Ben, I believe that we have addressed all of our proposed actions, have revoked the 2 "unrevoked" certificates, and have created the late revocation report. I am not sure that we have any other actions to close this incident. So if you agree, I would propose the incident be closed.

Thanks, Bruce.

Comment 20

[Ben Wilson]

I'll close this on Friday, 14-Jan-2022.