1748634 - Entrust: Late Revocation for SSL Certificates issued with Un-verified IP Addresses

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Bruce Morton]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36

Steps to reproduce:

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

Entrust file incident https://bugzilla.mozilla.org/show_bug.cgi?id=1744827, where 12 SSL certificates were issued without IP Address verification. Entrust revoked 10 certificates and verified the IP Address for 2 certificates and did not revoke those certificates. Through feedback from the incident, Entrust agreed that the certificates must be revoked.

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

The timeline of the original incident can be viewed here, https://bugzilla.mozilla.org/show_bug.cgi?id=1744827.

2021-12-30 – the subscribers were advised that there certificates would be revoked no later than 4 January 2022.
2022-01-04 – the certificates listed in section 5 were revoked.

3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

Entrust has patched the system and no longer issues SSL certificates with non-verified IP addresses or EV SSL certificates with IP addresses.

4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

Certificates for late revocation are listed in item 5.

5. The complete certificate data for the problematic certificates.

https://crt.sh/?id=5715211065
https://crt.sh/?id=5730617809

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

Entrust incorrectly determined that it was reasonable to retroactively validate the IP addresses, where the goal was not to require the Subscriber to reissue another certificate with the same verified fields. Entrust understands from the original incident discussion, https://bugzilla.mozilla.org/show_bug.cgi?id=1744827#c13, that “Relying entities tend to rely on information included in the certificate, which includes the 'not before' date. If the information in the certificate was not verified according to the BR as of the 'not before' date (or in the information reuse period before that), then we can't rely on the certificate.” As such, Entrust took action to have the certificates revoked and reissued.

7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

Please note that this was our only incident of retroactive verification, so is not our common practice.

Entrust has updated its certification practices to ensure that retroactive verification is not permitted and cannot be used to allow a certificate to not be revoked.

Comment 1

[Matthias]

I don't quite understand why Entrust did not initially consider "retroactive validation" and the subsequent non-revocation as "non-revocation of problematic certificates". Could you explain why you didn't initiate the expected procedure for non-revocation when you decided that retroactive validation would remove the need for revocation/reissuance?

Comment 2

[Bruce Morton]

We did understand that we were not revoking the certificates. We made the incorrect assumption that non-revocation was reasonable, since when we retroactively verified, we considered that we could reissue the certificate with the same IP addresses. We have changed our practices to ensure this action will not happen in the future.

Comment 3

[Matthias]

(In reply to Bruce Morton from comment #2)

We did understand that we were not revoking the certificates. We made the incorrect assumption that non-revocation was reasonable, since when we retroactively verified, we considered that we could reissue the certificate with the same IP addresses. We have changed our practices to ensure this action will not happen in the future.

In the wiki [0] there are expectations that Mozilla places on CAs when they decide not to revoke problematic certificates; but there was no sign that Entrust was following these expectations for the two certificates involved.

Did Entrust stop considering the two certificates 'misissued or otherwise problematic' after they retroactively verified those two certificates, thus not part of the incident? If that was not the case, then why was there seemingly no intention to meet the expectations from the wiki [0]?

[0] https://wiki.mozilla.org/CA/Responding_To_An_Incident#Revocation

Comment 4

[Bruce Morton]

(In reply to Matthias from comment #3)

Did Entrust stop considering the two certificates 'misissued or otherwise problematic' after they retroactively verified those two certificates, thus not part of the incident? If that was not the case, then why was there seemingly no intention to meet the expectations from the wiki [0]?

We did not consider the two certificates to be problematic, since the retroactive verification of the IP addresses confirmed they were controlled by the Subscribers. As such, we did not go through the expectations of wiki [0] to state why we were not revoking. Through the incident discussion, we did understand the certificates are problematic, since there was a period of time when the IP addresses were not validated and the certificates were not issued in accordance with our CPS. We also understand that retroactive verification is not a solution, and that the certificates should be revoked.

Comment 5

[Bruce Morton]

Ben, we have no other updates to this incident. If you agree, we think this incident can be closed.

Comment 6

[Ben Wilson]

I will schedule this for closure on Friday, 21-Jan-2022, unless anyone has questions, concerns, or issues to raise.

Comment 7

[Ryan Sleevi]

Bruce: Thanks for engaging here. I think Matthias is highlighting that there's an understandable question about Entrust's decision making process.

From this issue, the root cause analysis seems deficient, in line with patterns highlighted to other CAs, because it seems to focus on the "We've documented (misunderstanding X) is not correct", but doesn't really take a look at systemic fixes for how to prevent misunderstanding Y, Z, A, B, or C.

I totally understand that to err is human, and mistakes are made, but I think it's reasonable to question how has Entrust changed its decision making process or engaging with the community? For example, on Bug 1744827, multiple commenters pointed out that Entrust's interpretation was not consistent with the BRs or past discussions, but Entrust still persisted.

1. What steps has Entrust taken to ensure it avoids future misunderstandings of requirements?
   • For example, raising questions early in public (e.g. the CA/B Forum and/or m.d.s.p.) to gather feedback, taking maximally conservative interpretations to ensure users don't have reason to question Entrust's decision making process, ensuring critical discussion about requirements and highlighting ambiguities, etc. These are all concrete, long-term steps, and are based on feedback raised to other CAs in other bugs, so it would seem reasonable to expect Entrust to be aware of such suggestions. It's unclear if that's the case, and these were rejected, or if they were things not followed.
2. How has Entrust's process changed for reviewing feedback on bugs and incidents, to prevent future delays or inaction, especially after repeat concerns?
   • I don't think we want to see the approach to handling Bug 1744827 happen again, which contributed to this issue, so what's changed to prevent this?

I don't think we should hold Comment #0 as a positive incident report, mostly because it treats the issue as a one-off, when it's clear there are deeper, systemic issues at play here that would benefit from more attention and consideration.

Comment 8

[Bruce Morton]

(In reply to Ryan Sleevi from comment #7)
Here is some background on the original incident https://bugzilla.mozilla.org/show_bug.cgi?id=1744827:
• We discovered that we issued 2 EV certificates with IP addresses. We considered these need to be revoked within 24 hours as EV cannot have IP addresses.
• The investigation also showed that we issued some OV SSL certificates with unverified IP addresses. We also considered these certificates needed to be revoked with 24 hours as they were issued where we did not validate control of the IP address.
• We did not review the “MUST be 5 days” revocation reasons as the 24 hour revocation was already decided.
• We needed to advise the customers that we needed revoke their certificate and if they wanted it to be reissued, then the IP address would have to be verified. Most customers did not needed the IP address included, so the certificates were revoked and replaced with a new certificate without the IP address.
• For two customers that wanted to reissue their certificates reissued including the IP address, we reverified their IP addresses names within 24 hours from the start of the incident. Since the IP address was retroactively verified and the new certificate would have the same contents as the old certificate, we decided not to revoke.

We believe that our error was that we CANNOT retroactively validate. I do consider this to be a one-of decision as we have had no experience with incidents where retroactive validation could be considered. We did not see in the BRs that retroactive validation was allowed or not allowed. Although we did not discuss with Mozilla or the CA/Browser Forum, we were transparent on our original incident and stated “It was determined that either the IP addresses must be verified or the certificates must be revoked within 24 hours.”

1. What steps has Entrust taken to ensure it avoids future misunderstandings of requirements?
   We will continue to do the following:
   • Monitor all discussions with the CA/Browser Forum for changes to requirements.
   • Monitor the browsers for changes to policies.
   • Monitor the Mozilla dev-security-policy group.
   • Monitor the Mozilla CA Certificate Compliance Incidents to see if incidents impact Entrust’s implementations.
   • Monitor and participate with CA/Browser Forum ballots to ensure we have an understanding of the proposal.
   • Send questions to the CA/Browser Forum or browsers to request clarifications of requirements or policies.

2. How has Entrust's process changed for reviewing feedback on bugs and incidents, to prevent future delays or inaction, especially after repeat concerns?
   We monitor all Mozilla incidents using an internal tracking system. The incidents are reviewed on a continual basis by members of the compliance team to verify if Entrust is impacted.

When responding to https://bugzilla.mozilla.org/show_bug.cgi?id=1744827#c6 we focused revocation reason 2 (Certificate was misused), and did not consider reason 7 (not issued iaw CPS), which does not permit retroactive validation. Going forward we will consider all options provided in the feedback.

Comment 9

[Bruce Morton]

Ben, we have no other updates to this incident. If you agree, we think this incident can be closed.

Comment 10

[Matthias]

2. How has Entrust's process changed for reviewing feedback on bugs and incidents, to prevent future delays or inaction, especially after repeat concerns?

We monitor all Mozilla incidents using an internal tracking system. The incidents are reviewed on a continual basis by members of the compliance team to verify if Entrust is impacted.
When responding to https://bugzilla.mozilla.org/show_bug.cgi?id=1744827#c6 we focused revocation reason 2 (Certificate was misused), and did not consider reason 7 (not issued iaw CPS), which does not permit retroactive validation. Going forward we will consider all options provided in the feedback.

All but the last sentence ( "Going forward we will consider all options provided in the feedback." ) don't seem like a change in process; and that latest sentence seems quite limited in measurable impact. Could you provide some more information to convince us that this kind of issue is not going to repeat again?

---

Apart from that, I can't help but notice that Entrust's track record regarding policy adherance seems to have been quite lax, and has seen quite a few interpretation issues (or situations of misunderstanding / lack of knowledge) over the last few years [0]:

• bug 1535735, filed on 2019-03-15, where the 5-day revocation deadline was misinterpreted as starting the moment the CA's incident report was published;
• bug 1648472, filed on 2020-06-25, where new requirements are not implemented for undetermined reasons, and misissued certificates are initially not revoked because (paraphrased) "this disallowed signature scheme provides the same security strength as one of those allowed by the BR and MRSP", despite this reason (not a security risk) explicitly being called out as not acceptable;
• bug 1651481, filed on 2020-07-08, where the CA explicitly decided to go against the expectations and unilaterally delayed revocation of a batch of 606 misissued certificates (from 1648472 above), some of which by over 30 days;
• bug 1731887, filed on 2021-09-21, where BR non-compliance (not have test web pages for valid, expired, and revoked certificates) was not considered to require an incident report;
• and now bug 1744827, filed on 2021-12-07, where Entrust needed to be reminded two seperate times (comment 4 on 2021-12-14, and 13 and 14 on 2021-12-30) that retroactive validation does not lift the requirement to revoke a certificate that was not validated at time of issuance (and that the implications of retroactive validation make it an unacceptable practise).

To be honest, I'm quite baffled with the gravity of some of these issues. One would hope that a public CA has good institutionalized knowledge about handling security, trust and policies; and although Entrust seems quite proactive in reporting issues that they find, these gaps in policy understanding and compliance show a very worrying pattern of missing both basic and deeper understanding of the policies that Entrust tries to adhere to.

[0] I'll note that these issues are those I could find within a reasonable time, and might thus be incomplete. I have excluded most situations of 'we found this incorrect data in certificates xyz, we updated our certificate profiles/linters/policies to prevent this from happening again', and excluded issues older than 3 years (not many exist on bugzilla).

Comment 11

[Bruce Morton]

(In reply to Matthias from comment #10)

All but the last sentence ( "Going forward we will consider all options provided in the feedback." ) don't seem like a change in process; and that latest sentence seems quite limited in measurable impact. Could you provide some more information to convince us that this kind of issue is not going to repeat again?

Hi Matthias, the root cause of the late revocation was that instead of revoking, we did retroactive validation. This is an issue that we have never done in the past. We have changed our policy to ensure that retractive validation will not occur in the future. We believe that this policy change will prevent this kind of issue from happening in the future.

Comment 12

[Matthias]

(In reply to Bruce Morton from comment #11)

Hi Matthias, the root cause of the late revocation was that instead of revoking, we did retroactive validation. This is an issue that we have never done in the past. We have changed our policy to ensure that retractive validation will not occur in the future. We believe that this policy change will prevent this kind of issue from happening in the future.

Please note that my comment 10 specifically quoted (and formatted(!)) your reply on Ryan's comment 7 question 2: "How has Entrust's process changed for reviewing feedback on bugs and incidents, to prevent future delays or inaction, especially after repeat concerns?". The reply in comment 11 does not provide the requested information to help answer this question 2 from comment 7.

As Ryan said; this issue seems to be caused by older and systemic issues. The issues I mentioned in comment 10 and your latest comment 11 both seem to testify to that deeper underlying issue of repeated misunderstanding, misinterpretation and ignoring of policies, concerns and expectations, often requiring specific (and sometimes repeated) community feedback before Entrust takes corrective action.

Could you further explain your process for reviewing feedback on bugs and incidents, and any recent changes thereof as a response to recent incidents? Could you, in this explanation, also add information how this process (or changes in the process) will prevent future delays or inaction, especially after repeat concerns?

Comment 13

[Bruce Morton]

Our system monitors Bugzilla for any changes to bugs in the component "CA Certificate Compliance" within the "NSS" product and creates a local task for each bug and to review the individual bugs.

The internal tasks are updated when the external bug is updated to reflect the status.
Our team must complete the task by reviewing the bug and update the status of the task including a conclusion, they might decide to follow or escalate the bug by assigning it to themselves or someone in the team.

Comment 14

[Ben Wilson]

Are there any other questions or concerns from the Mozilla community?