Open Bug 1890898 Opened 3 months ago Updated 9 hours ago

Entrust: Failure to revoke OV TLS - CPS typographical (text placement) error

Categories

(CA Program :: CA Certificate Compliance, task)

Product:
CA Program
Component:
CA Certificate Compliance
Type:
task

Tracking

(Not tracked)

Status:
ASSIGNED

People

(Reporter: bruce.morton, Assigned: bruce.morton)

Details

(Whiteboard: [ca-compliance] [policy-failure] [leaf-revocation-delay] Next update 2024-06-17)

Attachments

(3 files, 1 obsolete file)

Bug 1890898 provision comparison chart 1.pdf
123.08 KB, application/pdf
Details
delayed-revocation-OV-cps-typo.csv
513.68 KB, text/csv
Details
Bugzilla Notice for OV Mis-issuance.pdf
139.86 KB, application/pdf
Details

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36

Certificates will be uploaded with 24 hours.

Incident Report

Summary

This incident report covers the certificates reported in bug https://bugzilla.mozilla.org/show_bug.cgi?id=1890896, CPS typographical (text placement) error, where we decided to not revoke due to exceptional conditions listed in this report.

Impact

This incident involves 6,008 OV TLS certificates that were issued between March 22, 2024, 15:02 UTC and March 26, 2024, 17:07 UTC.

These certificates were issued in accordance with the TLS Baseline Requirements and with the certificate profile intended by Entrust. If the certificates were re-issued, they would be re-issued with the identical certificate profile.

Timeline

2024-03-26:

  • 15:28 We self-discovered that there was a typographical error included in the CPS. The CPS was updated to correct the error.
  • 17:07 CPS version 3.20 was posted.

2024-04-11:

Root Cause Analysis

1. Why was there a problem?
A change which was supposed to be made to the “EV SSL certificate” profile was incorrectly added to the “SSL certificate” (OV) profile, see bug https://bugzilla.mozilla.org/show_bug.cgi?id=1890896.

As these certificates are considered mis-issued they require revocation.

2. Why did they require revocation?

Because section 4.9.1.1 of the TLS Baseline Requirements states:

” With the exception of Short-lived Subscriber Certificates, the CA SHOULD revoke a certificate within 24 hours and MUST revoke a Certificate within 5 days and use the corresponding CRLReason (see Section 7.2.2) if one or more of the following occurs:
…
12. The CA is made aware that the Certificate was not issued in accordance with these Requirements or the CA’s Certificate Policy or Certification Practice Statement (CRLReason #4, superseded);"

3. Why did we decide to not revoke these certificates?

We do not believe that revoking certificates already issued as part of our response would benefit the Web PKI. The correction was to the CPS rather than the certificate, so that if the certificates were re-issued, they would be re-issued with the identical certificate profile. We believe this to be an exceptional condition.
.
4. Why is this considered an exceptional condition?

  • These certificates do not form a security risk to relying parties or the ecosystem.
  • Our CDN statistics show that between March 22, 2024, 15:02 UTC and March 26, 2024, 17:07 UTC our CPS versions 3.18 and 3.19 were downloaded 30 times by desktop or mobile browsers, excluding Entrust networks.
    • Of these 30 downloads we have been able to identify 10 downloads to specific subscribers.
    • The downloads are divided over 9 unique IP addresses, one from a known subscriber, one from a known CA.
    • The data likely includes downloads from Entrust employees which we have not been able to identify.
    • The CPS covers the multiple certificate types we offer.
    • This data indicates that the number of subscribers and relying parties that have seen this error is minimal.
  • If the certificates were re-issued, they would be re-issued with the same certificate profile, resulting in the same certificate with a new issuance date.

5. Why do you consider this impact minimal impact on subscribers and relying parties?

As mentioned above, there have only been very few downloads of this CPS from our website. We have updated the CPS documents in question with an errata so that the expectations are clear for anyone checking these CPS versions going forward. .

Lessons Learned

See bug https://bugzilla.mozilla.org/show_bug.cgi?id=1890896

Action Items

In addition to the action items in bug #1890896 we have identified the following:

Action Item Kind Due Date
Update archive CPS with errata to inform subscribers going forward Transparency Done
Inform subscribers of changes to the CPS Transparency 2024-04-12

Appendix

Details of affected certificates

See the attachment to this bug report.

My comment from 1890685

Failure to revoke is another whole incident category, and something a CA will need to work to not happen again. The symptoms of what cause a need to revoke are plenty, and will never be fully preventable. What is Entrust doing so that a Failure To Revoke does not happen again? None of your action items here address this. Right now it seems like the stance is something alone the lines of: well we just wont have incidents that end up necessitating a revocation.

Just like the other incident, none of the action items here address this. I'm sure you could have missed that earlier comment, but can you please clarify if you have any plans to take to prevent the re-occurrence of these?

Assignee: nobody → bruce.morton
Status: UNCONFIRMED → ASSIGNED
Type: defect → task
Ever confirmed: true
Whiteboard: [ca-compliance] [policy-failure]

(In reply to amir from comment #3)

My comment from 1890685

Failure to revoke is another whole incident category, and something a CA will need to work to not happen again. The symptoms of what cause a need to revoke are plenty, and will never be fully preventable. What is Entrust doing so that a Failure To Revoke does not happen again? None of your action items here address this. Right now it seems like the stance is something alone the lines of: well we just wont have incidents that end up necessitating a revocation.

Just like the other incident, none of the action items here address this. I'm sure you could have missed that earlier comment, but can you please clarify if you have any plans to take to prevent the re-occurrence of these?

We have to addressed in https://bugzilla.mozilla.org/show_bug.cgi?id=1890685#c7.

We have to addressed in https://bugzilla.mozilla.org/show_bug.cgi?id=1890685#c7.

I've responded there, but for posterity, you have not addressed this problem.

Regarding the action, today subscribers were informed of the CPS changes.

Comment 2 states:

"Our CDN statistics show that between March 22, 2024, 15:02 UTC and March 26, 2024, 17:07 UTC our CPS versions 3.18 and 3.19 were downloaded 30 times by desktop or mobile browsers, excluding Entrust networks.
- Of these 30 downloads we have been able to identify 10 downloads to specific subscribers.
- The downloads are divided over 9 unique IP addresses, one from a known subscriber, one from a known CA.
- The data likely includes downloads from Entrust employees which we have not been able to identify.
- The CPS covers the multiple certificate types we offer.
- This data indicates that the number of subscribers and relying parties that have seen this error is minimal."

Can you help me better understand this statement and how it relates to the principle expectation of a publicly-trusted CA adhering to its documented policies at all times? Also, can you share more about why this justification was offered?

One way of interpreting this statement is an attempt to justify and/or diminish non-compliance on the basis that very few people downloaded the CPS during the affected time period.

The underlying issue with this type of justification is that it appears to downplay the fundamental expectation that a CA must adhere to its stated policies and procedures at all times. The CA/Browser Forum Baseline Requirements and other related guidelines emphasize that compliance must be continuous and is not conditional on the extent to which documented policies are accessed by external parties.

Fundamentally, a statement claiming that a mistake did not result in significant harm does not diminish the significance of that mistake. The principle at stake is the integrity and trustworthiness of Entrust, which is expected to uphold its commitments irrespective of the level of direct oversight or detection by others in the ecosystem.

Additionally, as stated on https://www.ccadb.org/cas/incident-report, the Action Items section is supposed to contain a list of remediation items that will be undertaken to ensure that a similar incident does not reoccur in the future and that it is not sufficient for the actions to simply stop the incident in question. What actions will prevent this from happening again? Beyond that, “Transparency" is not one of the described Action categories listed on https://www.ccadb.org/cas/incident-report.

Flags: needinfo?(bruce.morton)

We are reviewing the open comments in this bug and will come back with a reply next week.

Flags: needinfo?(bruce.morton)

Why is this considered an exceptional condition?

These certificates do not form a security risk to relying parties or the ecosystem.

So is Entrust's definition of "exceptional condition" anything that isn't a security risk? Because if that is the case, then why does the BR even have the difference between the 24 hour vs 5 day revocation deadline?

This is a ballot that Entrust has voted for: https://cabforum.org/2018/09/14/ballot-sc6-revocation-timeline-extension/

Note that this understanding goes against the actual ballot that was voted for, and that these discussions did happen at the time as well. For example: https://lists.cabforum.org/pipermail/servercert-wg/2018-September/000166.html.

To quote Ryan Sleevi:

I do not believe there is any justifiable or defensible reason to extend
the 5 days requirement, nor do I believe the CA should be expected to have
a 'clean' audit should they chose to do so.

CAs facing challenges of their own creation should not be exploring "How do
I keep these certs working", but "How do I make sure I don't issue
violating certs to begin with". Anything less is gross negligence, and not
the system we should be striving to build.

You were participating in those discussions too. So I'm trying to understand, if there was a disagreement with these rules, why vote for the ballot that mandated these revocations?

Flags: needinfo?(bruce.morton)

Could you also please let us know how the public should reconcile your response here, with the promises you made in 1651481.

Specifically:

In the future our plan will be:

Publish an Incident Report within 1 business day of detection of the incident
Publish a late revocation Incident Report within the required 24 hours or 5 days requirement, as applicable
Advise Subscribers to revoke within 24 hours or 5 days as applicable or provide an explanation why they cannot revoke
Update the late revocation Incident Report with the Subscriber status and the timeline for closure
Ensure that the Incident(s) are listed in the annual audit report

Given that points 1 and 2 have demonstratively not been following in these series of incidents, why should we trust the promises you're giving now?

Apologies for the grammar mistake: s/not been following/not been followed

(In reply to amir from comment #10)

Could you also please let us know how the public should reconcile your response here, with the promises you made in 1651481.

Specifically:

In the future our plan will be:

Publish an Incident Report within 1 business day of detection of the incident
Publish a late revocation Incident Report within the required 24 hours or 5 days requirement, as applicable
Advise Subscribers to revoke within 24 hours or 5 days as applicable or provide an explanation why they cannot revoke
Update the late revocation Incident Report with the Subscriber status and the timeline for closure
Ensure that the Incident(s) are listed in the annual audit report

Given that points 1 and 2 have demonstratively not been following in these series of incidents, why should we trust the promises you're giving now?

This needs an answer from Entrust, quickly.

I did not realize but those statements were for 4 years ago. None of them have been done.

You cannot even do what you told this community you would do several years ago. It's the same person and the same CA making this post.
If in 4 years you have not improved to meet even the MINIMUM standard - Entrust need to be distrusted. I cannot see any other way that trust can be regained now.

(In reply to Ryan Dickson from comment #7)

Comment 2 states:

"Our CDN statistics show that between March 22, 2024, 15:02 UTC and March 26, 2024, 17:07 UTC our CPS versions 3.18 and 3.19 were downloaded 30 times by desktop or mobile browsers, excluding Entrust networks.
- Of these 30 downloads we have been able to identify 10 downloads to specific subscribers.
- The downloads are divided over 9 unique IP addresses, one from a known subscriber, one from a known CA.
- The data likely includes downloads from Entrust employees which we have not been able to identify.
- The CPS covers the multiple certificate types we offer.
- This data indicates that the number of subscribers and relying parties that have seen this error is minimal."

Can you help me better understand this statement and how it relates to the principle expectation of a publicly-trusted CA adhering to its documented policies at all times? Also, can you share more about why this justification was offered?

One way of interpreting this statement is an attempt to justify and/or diminish non-compliance on the basis that very few people downloaded the CPS during the affected time period.

This data demonstrates the minimal impact on subscribers and relying parties contributing to the exceptional circumstances of this incident.

The underlying issue with this type of justification is that it appears to downplay the fundamental expectation that a CA must adhere to its stated policies and procedures at all times. The CA/Browser Forum Baseline Requirements and other related guidelines emphasize that compliance must be continuous and is not conditional on the extent to which documented policies are accessed by external parties.

We agree that we should adhere to the requirements, this is the reason that we have filed an incident report for not following the requirement to revoke all certificates that were not issued to the letter of the CPS within five days.

Fundamentally, a statement claiming that a mistake did not result in significant harm does not diminish the significance of that mistake. The principle at stake is the integrity and trustworthiness of Entrust, which is expected to uphold its commitments irrespective of the level of direct oversight or detection by others in the ecosystem.

We do believe in upholding our commitment to the requirements and our CPS, that we should be detecting these issues ourselves, which we also did for this incident. We also believe, however, that the impact to the ecosystem needs to be taken into consideration.

Additionally, as stated on https://www.ccadb.org/cas/incident-report, the Action Items section is supposed to contain a list of remediation items that will be undertaken to ensure that a similar incident does not reoccur in the future and that it is not sufficient for the actions to simply stop the incident in question. What actions will prevent this from happening again? Beyond that, “Transparency" is not one of the described Action categories listed on https://www.ccadb.org/cas/incident-report.

Please see the action items and our answer to question two in bug #1890896.

Flags: needinfo?(bruce.morton)
Whiteboard: [ca-compliance] [policy-failure] → [ca-compliance] [policy-failure] [leaf-revocation-delay]

(In reply to amir from comment #9)

Why is this considered an exceptional condition?

These certificates do not form a security risk to relying parties or the ecosystem.

So is Entrust's definition of "exceptional condition" anything that isn't a security risk? Because if that is the case, then why does the BR even have the difference between the 24 hour vs 5 day revocation deadline?

This is a ballot that Entrust has voted for: https://cabforum.org/2018/09/14/ballot-sc6-revocation-timeline-extension/

Note that this understanding goes against the actual ballot that was voted for, and that these discussions did happen at the time as well. For example: https://lists.cabforum.org/pipermail/servercert-wg/2018-September/000166.html.

To quote Ryan Sleevi:

I do not believe there is any justifiable or defensible reason to extend
the 5 days requirement, nor do I believe the CA should be expected to have
a 'clean' audit should they chose to do so.

CAs facing challenges of their own creation should not be exploring "How do
I keep these certs working", but "How do I make sure I don't issue
violating certs to begin with". Anything less is gross negligence, and not
the system we should be striving to build.

You were participating in those discussions too. So I'm trying to understand, if there was a disagreement with these rules, why vote for the ballot that mandated these revocations?

Entrust voted in favor of the ballot as we did not have a reason at the time not to provide consensus. But to clarify, as noted above, we made three points on why we believe this to be an exceptional situation: 1) the lack of security risk, 2) the data showing minimal impact, and 3) that If the certificates were re-issued, they would be re-issued with the same certificate profile, resulting in the same certificate with a new issuance date.

It was weighing these three factors together that led us to this conclusion, not one factor by itself.

(In reply to amir from comment #10)

Could you also please let us know how the public should reconcile your response here, with the promises you made in 1651481.

Specifically:

In the future our plan will be:

Publish an Incident Report within 1 business day of detection of the incident
Publish a late revocation Incident Report within the required 24 hours or 5 days requirement, as applicable
Advise Subscribers to revoke within 24 hours or 5 days as applicable or provide an explanation why they cannot revoke
Update the late revocation Incident Report with the Subscriber status and the timeline for closure
Ensure that the Incident(s) are listed in the annual audit report

Given that points 1 and 2 have demonstratively not been following in these series of incidents, why should we trust the promises you're giving now?

Your question here goes to the series of incidents that we have posted in recent weeks. We have provided action items and assurances for each. In the best interests of our customers and the TLS ecosystem, our goal is to avoid mistakes and prepare our customers for future threats and mis-issuances, and in doing so to earn your trust.

(In reply to Bruce Morton from comment #14)

(In reply to amir from comment #9)

Why is this considered an exceptional condition?

These certificates do not form a security risk to relying parties or the ecosystem.

So is Entrust's definition of "exceptional condition" anything that isn't a security risk? Because if that is the case, then why does the BR even have the difference between the 24 hour vs 5 day revocation deadline?

This is a ballot that Entrust has voted for: https://cabforum.org/2018/09/14/ballot-sc6-revocation-timeline-extension/

Note that this understanding goes against the actual ballot that was voted for, and that these discussions did happen at the time as well. For example: https://lists.cabforum.org/pipermail/servercert-wg/2018-September/000166.html.

To quote Ryan Sleevi:

I do not believe there is any justifiable or defensible reason to extend
the 5 days requirement, nor do I believe the CA should be expected to have
a 'clean' audit should they chose to do so.

CAs facing challenges of their own creation should not be exploring "How do
I keep these certs working", but "How do I make sure I don't issue
violating certs to begin with". Anything less is gross negligence, and not
the system we should be striving to build.

You were participating in those discussions too. So I'm trying to understand, if there was a disagreement with these rules, why vote for the ballot that mandated these revocations?

Entrust voted in favor of the ballot as we did not have a reason at the time not to provide consensus.

This statement is deeply concerning. The ballot's been active since October 2018 and was modified after changes proposed by you. Entrust Datacard voted Yes 2018-09-10 following discussion on subjective terms, timeframes and CA obligations.

There hasn't been a hint from Entrust since that vote on 5-days being a barrier. What has changed? Any issues with Subscribers should be going down over time as they adapt to a modern PKI infrastructure.

But to clarify, as noted above, we made three points on why we believe this to be an exceptional situation: 1) the lack of security risk, 2) the data showing minimal impact, and 3) that If the certificates were re-issued, they would be re-issued with the same certificate profile, resulting in the same certificate with a new issuance date.

It was weighing these three factors together that led us to this conclusion, not one factor by itself.

None of these address adherence to the Baseline Requirements and Root Program policies. We have seen no data on security risk beyond a lack of ability to address these certificates within the required timeframe. As has already been stated to you having the 'same certificate profile' is wrong, as they would have the correct issuance date as per your own CPS.

(In reply to Bruce Morton from comment #15)

Your question here goes to the series of incidents that we have posted in recent weeks. We have provided action items and assurances for each. In the best interests of our customers and the TLS ecosystem, our goal is to avoid mistakes and prepare our customers for future threats and mis-issuances, and in doing so to earn your trust.

Wouldn't that be your goal in all of your previous incidents as well? We've had the same statements made year after year, but you're not actually changing at all just providing empty statements. The item that should come first is the public PKI ecosystem, not your customers.

Your question here goes to the series of incidents that we have posted in recent weeks. We have provided action items and assurances for each. In the best interests of our customers and the TLS ecosystem, our goal is to avoid mistakes and prepare our customers for future threats and mis-issuances, and in doing so to earn your trust.

I'm really trying to understand what this means. I'll be clearer here:

In comment 6 on bug 1651481, posted on 2020-07-14, you made a commitment that:

  • We will not the make the decision not to revoke.
  • We will plan to revoke within the 24 hours or 5 days as applicable for the incident.

What I'm trying to understand is:

Is that commitment you made there something we should ignore?

Entrust voted in favor of the ballot as we did not have a reason at the time not to provide consensus. But to clarify, as noted above, we made three points on why we believe this to be an exceptional situation: 1) the lack of security risk, 2) the data showing minimal impact, and 3) that If the certificates were re-issued, they would be re-issued with the same certificate profile, resulting in the same certificate with a new issuance date.

Alright, but can you show me where in the BRs this allowance of going over the 5 day limit is provided? The only place I've seen a reference to this is: https://wiki.mozilla.org/CA/Responding_To_An_Incident#Revocation

That document is not a binding document as far as I can tell. Therefore, whats being said there is really just one root program's opinion and interpretation of the BRs.

However, lets pretend that document is binding and applies to all root programs. In that case, in this incident response you have not provided the full requirements from that same excerpt:

The decision and rationale for delaying revocation will be disclosed in the form of a preliminary incident report immediately; preferably before the BR-mandated revocation deadline. The rationale must include detailed and substantiated explanations for why the situation is exceptional. Responses similar to “we do not deem this non-compliant certificate to be a security risk” are not acceptable. When revocation is delayed at the request of specific Subscribers, the rationale must be provided on a per-Subscriber basis.

Your primary reason for this being an exceptional case is "the lack of security risk". Meanwhile:

Responses similar to “we do not deem this non-compliant certificate to be a security risk” are not acceptable.

Is there another document talking about there being exceptional cases for revocation?

Can we expect Entrust to provide a per-subscriber justification on why this situation is deemed exceptional for that subscriber?

I would like for you to also back up claim of "the lack of security risk". Can you please share what risk assessment methodology you used? What data did you gather to make that determination? Note that CPS download count does not imply anything about the security risk here.

Also, how many of the subscribers impacted by this incident have been made aware of this incident? Are they aware that they will be using misissued certificates?

Entrust voted in favor of the ballot as we did not have a reason at the time not to provide consensus.

Well this sounds like you have a reason to be against it now. I would like to note that, even if you disagree with this rule now - that rule is still in effect here and you are in violation of it.

Based on this incident response, it seems like you have no intention to follow the revocation requirements section of the BRs. Am I correct in my understanding there?

Entrust is currently in complete violation of the baseline requirements. Since this incident doesn't even have a timeline for revocation at all, I'm going to assume that Entrust plans to continue being in complete violation of the baseline requirements.

Are there any other parts of the baseline requirements in which Entrust has decided they won't comply with anymore?

We greatly value everyone’s input because it enables us to engage in informed decision-making. We are considering whether these certificates should be revoked. Are there additional insights and opinions regarding that question? It appears that the certificates complied with industry standards and that the replacement certificates would be nearly identical, except for serial numbers and validity periods. In stating your position, please consider and explain both the potential benefits and drawbacks to the Mozilla root program and the security of the internet. Thanks.

Were this an isolated incident, I wouldn't have a strong opinion. However, Entrust appears to have a history of pushing the limits when it comes to revocation.

Personally, I'm uncomfortable with a CA being allowed to conduct business this way. I have no choice but to delegate my security and trust to these CAs. I rely on the root programs to enforce the BRs on behalf of myself and the rest of the internet.

I see three possible outcomes:

  1. The root programs continue to be lenient with Entrust indefinitely. Nothing changes.
  2. The root programs continue to be lenient with Entrust for a while, but eventually the mistakes pile up enough that one of the root programs pushes for distrust.
  3. The root programs immediately stop being lenient with Entrust. Entrust is forced to make internal changes to remain a CA.

First outcome

The root programs continue to be lenient with Entrust indefinitely. Nothing changes.

Benefits to the Mozilla root program

  1. Avoids angering a lucrative CA with high-profile subscribers

Drawbacks to the Mozilla root program

  1. Sets a precedent that the root program is unwilling to enforce its policies against high-profile CAs
  2. Likely to make enforcement more difficult in the future

Benefits to the security of the internet

None. It would avoid breaking services that use the affected certs and can't handle revocation, but that's not a benefit to the security of the internet.

Drawbacks to the security of the internet

  1. Entrust would never addresses the internal issues that lead to these mostly-harmless mistakes; inevitably, a much more serious mistake will occur

Second outcome

The root programs continue to be lenient with Entrust for a while, but eventually the mistakes pile up enough that one of the root programs pushes for distrust.

Benefits to the Mozilla root program

  1. Should Entrust choose to retaliate upon distrust, Mozilla will have more evidence to back up their decision
  2. Another root program may distrust first

Drawbacks to the Mozilla root program

  1. The onus is on Entrust to make improvements to their infrastructure and internal processes. That seems unlikely to happen. The longer Mozilla waits to enforce, the harsher and more disruptive the enforcement is likely to be.

Benefits to the security of the internet

  1. This probably ends in Entrust being distrusted.

Drawbacks to the security of the internet

  1. Given that Entrust shows signs of systemic internal issues, a lot can go wrong between now and the point at which Mozilla opts to enforce.

Third outcome

The root programs immediately stop being lenient with Entrust. Entrust is forced to make internal changes to remain a CA.

Benefits to the Mozilla root program

  1. The enforcement actions will be less disruptive, meaning a lower chance of retaliation
  2. Mozilla sets a precedent that it will enforce the BRs, even against high-profile CAs; that's a valuable precedent that will make enforcement easier in the future while also sending a strong message to CAs that they should follow the BRs

Drawbacks to the Mozilla root program

  1. There's still a chance of retaliation
  2. If the other root programs don't join in on any enforcement action, Entrust's enterprise customers might just stop supporting certain browsers/platforms that rely on Mozilla's root store

Benefits to the security of the internet

  1. Entrust is gently nudged toward implementing infrastructure and procedure improvements, decreasing the chances of significant mistakes in the future

Drawbacks to the security of the internet

  1. If Entrust is ultimately incapable of meeting the BRs, the path toward distrust might be longer than if Mozilla had opted to take a hands-off approach and let the mistakes pile up further

Conclusion

I'm in favor of enforcing revocation for this incident. It's minimally disruptive, and it nudges Entrust in the right direction while still setting a strong precendent.

Note this is continuing a conversation on from the original bug report by streamlining the conversation into one thread for everyone's convenience.

I'll note for anyone reading this in the future that the bug title is currently "Entrust: Failure to revoke OV TLS - CPS typographical (text placement) error". Despite this there is no 'failure' to revoke here. I put forward that we are in a situation of a CA refusing to revoke and seeing if they have enough inherent authority to stop anyone enforcing policies.

In this bug's own Root Cause Analysis we are provided with succinct explanations for the policies being broken:

2. Why did they require revocation?

Because section 4.9.1.1 of the TLS Baseline Requirements states:

” With the exception of Short-lived Subscriber Certificates, the CA SHOULD revoke a certificate within 24 hours and MUST revoke a Certificate within 5 days and use the corresponding CRLReason (see Section 7.2.2) if one or more of the following occurs:
…
12. The CA is made aware that the Certificate was not issued in accordance with these Requirements or the CA’s Certificate Policy or Certification Practice Statement (CRLReason #4, superseded);"

By the CA's own admission the Baseline Requirements state they MUST revoke. This is a clear cut section of 24 hours, or 5 days for a strict category of issues. But rather than admit further fault to their subscribers in the current environment of their issues recently, the rules must be ignored through sheer policy fabrication.

3. Why did we decide to not revoke these certificates?

We do not believe that revoking certificates already issued as part of our response would benefit the Web PKI. The correction was to the CPS rather than the certificate, so that if the certificates were re-issued, they would be re-issued with the identical certificate profile. We believe this to be an exceptional condition.

No part of BR 4.9 allows for 'exceptional conditions', there is no mechanism to opt-out of compliance when it suits the CA.

A misinterpretation of Mozilla's Responding to an Incident page is instead being used to carve a tiny niche in a single Root Store's with many stipulations that are unaddressed, to instead be broadly applicable to any policy.

To clarify that statement let's run through the six bullet points that'd be required to only account for Mozilla, nevermind any other Root Store (emphasis mine):

  • A separate incident report will be filed in Bugzilla.
  • The decision and rationale for delaying revocation will be disclosed in the form of a preliminary incident report immediately; preferably before the BR-mandated revocation deadline. The rationale must include detailed and substantiated explanations for why the situation is exceptional. Responses similar to “we do not deem this non-compliant certificate to be a security risk” are not acceptable. When revocation is delayed at the request of specific Subscribers, the rationale must be provided on a per-Subscriber basis.
  • Any decision to not comply with the timeline specified in the Baseline Requirements must also be accompanied by a clear timeline describing if and when the problematic certificates will be revoked or expire naturally, and supported by the rationale to delay revocation.
  • The issue will need to be listed as a finding in your CA’s next BR audit statement.
  • Your CA will work with your auditor (and supervisory body, as appropriate) and the Root Store(s) that your CA participates in to ensure your analysis of the risk and plan of remediation is acceptable.
  • You will perform an analysis to determine the factors that prevented timely revocation of the certificates, and include a set of remediation actions in the final incident report that aim to prevent future revocation delays.
    If your CA will not be revoking the problematic certificates as required by the BRs, then we recommend that you also contact the other root programs that your CA participates in to acknowledge this non-compliance and discuss what expectations their Root Programs have with respect to these certificates.

As far as I can tell only the first item has actually occurred. In fairness an attempt at 'detailed and substantial explanations' is included:

4. Why is this considered an exceptional condition?

  • These certificates do not form a security risk to relying parties or the ecosystem.
  • Our CDN statistics show that between March 22, 2024, 15:02 UTC and March 26, 2024, 17:07 UTC our CPS versions 3.18 and 3.19 were downloaded 30 times by desktop or mobile browsers, excluding Entrust networks.
  • Of these 30 downloads we have been able to identify 10 downloads to specific subscribers.
  • The downloads are divided over 9 unique IP addresses, one from a known subscriber, one from a known CA.
  • The data likely includes downloads from Entrust employees which we have not been able to identify.
  • The CPS covers the multiple certificate types we offer.
  • This data indicates that the number of subscribers and relying parties that have seen this error is minimal.
  • If the certificates were re-issued, they would be re-issued with the same certificate profile, resulting in the same certificate with a new issuance date.

However comes far short of meeting the stated expectations, as previously discussed in this bug.

But this is all moot as Mozilla isn't the source of the exceptional revocation policy. To quote from the other bug report:
(In reply to Bruce Morton from comment #13)

We are not aware of any programs providing exceptions for revocation; however Apple and Chrome refer to the Mozilla policy for revocation.
...
We are not relying on any authorities; however, we have opened bug #1890898 to address this incident and to allow the open community to review this issue.

By Entrust's own admission they are not relying on any authorities. We are in a situation where a CA is deciding to act as their own authority above and beyond the Baseline Requirements' 'MUST'.

I will note the above isn't a specific answer to Ben's posed questions. To that end I believe a larger discussion outside of this report is more beneficial as this situation does not come from one issue. Moreover by looking further back into Entrust's issues a pattern is forming of negligence and empty promises to improve.

The shield of subscribers being the victims and Entrust trying their best has to stop at some point. The standards of CAs is only held up by the lowest amongst them, and despite working with their subscribers for years we're still getting the 'expire rather than revoke' story until a Root Program shows up. See #1524876 (2019) for an idea of how long this has been in Entrust's playbook - this is a kind example.

Noting for posterity that Entrust's PR team are now directly involved. Just as a heads up to them their cybersecurity podcast's certificate expired April 28th.

Additional comments outside of this thread relevant to Ben's inquiry are spread over all of the issues too, e.g. Mike Shaver who helped run Mozilla's Root Program during the removal of Diginotar.

If the PR team is attempting to strategize on how best to manage this, I recommend checking through all of Entrust's issues to date and specific promises made.

We took note of comments 22 and 23. We have no updates for this week. We will continue to monitor the bug.

We have no updates for this week and will continue to monitor the bug.

We have no updates for this week. Please move next update to June 17, 2024, which is after we provide the June 7 report to Mozilla.

Whiteboard: [ca-compliance] [policy-failure] [leaf-revocation-delay] → [ca-compliance] [policy-failure] [leaf-revocation-delay] Next update 2024-06-17

Summary

This incident report covers the certificates reported in Bug 1890896 that were issued between March 22, 2024, 15:02 UTC and March 26, 2024, 17:07 UTC. These certificates were issued after we had made an incorrect update to Appendix A of our CPS on March 22, and prior to the publication of a correction to the CPS published on March 26.

In Bug 1890896, and also in earlier statements made in this Bug 1890898, we stated that the certificates issued during this timeframe were mis-issued, but that we were not planning to revoke the affected certificates due to exceptional circumstances.

We now recognize that we did not give due consideration to the question of whether the affected certificates were in fact mis-issued, and whether a revocation requirement applied to these certificates.

As we reviewed our analysis of this incident with additional stakeholders and legal counsel, we were reminded that while the SSL Certificate table of Appendix A of our CPS was inconsistent with the TLS Baseline Requirements, our CPS also states that when there is such an inconsistency, the Baseline Requirements must override. Compliance with the Baseline Requirements certificate profiles was also specifically mandated in another section of the CPS. These other provisions effectively nullify the inconsistent profile phrase in Appendix A and replace it with the applicable profile from BRs. As a result, the issuance of the certificates in question was actually in accordance with our CPS, as well as the BRs.

We reviewed and consulted with independent external experts on this revised analysis, and based on this broader consultation, we now believe there was no mis-issuance and thus no need to revoke the affected certificates. A detailed analysis is below.

The mischaracterization of a revocation event arose from a lack of sufficient due diligence in making such a significant determination. Below we explain how we arrived this revised view, and remedial measures that we will take to avoid making similar mischaracterizations in the future.

Impact

There were 6,008 OV TLS affected certificates that we incorrectly characterized as mis-issued.

Timeline

All times are UTC.

2023-04-11:

  • TLS BR 2.0 released, based on SC-62 certificate profiles ballot, providing that any policyQualifiers (ie. cPSuri) is NOT RECOMMENDED to be present in TLS subscriber certificates.

2023-09-11:

  • 17:40 Entrust OV TLS CP OID (2.16.840.1.114028.10.1.5) and cPSuri removed from OV TLS certificates.

2024-03-22:

  • 15:02 CPS version 3.18 was posted to introduce a new clientAuth only certificate profile and other improvements. In making the change, the policyIdentifier for the SSL Certificate profile in Appendix A was mistakenly updated with the requirement “and must contain policyQualifier with cPSuri.” This requirement was supposed to be added to the EV SSL certificate profile only, and not to the OV SSL certificate profile.

2024-03-26:

  • 15:28 We discovered the text placement error in Appendix A. The CPS was updated to correct the error.
  • 17:07 CPS version 3.20 was posted.

2024-03-27:

  • We began drafting this incident report to declare the exceptional circumstances leading to non-revocation but were delayed due to the concurrent incidents.

2024-04-11:

  • 00:45: We declared our intent to not revoke the affected certificates.

2024-05-21:

  • In discussing this and other concurrent incidents with internal legal counsel, we received advice that we had likely mischaracterized the affected certificates as mis-issued. In the following days, we consulted more widely, including with independent external experts, and confirmed the revised analysis.

Root Cause Analysis

Our incident response process does not include a requirement to obtain legal advice for interpretation.

The initial assessment that a mis-issuance had occurred was therefore carried out without broad consultation and was based on the following facts:

  • at the time of issuance, the BRs (v.2.0.2, s.7.1.2.7.9) stated that the policyQualifiers field was NOT RECOMMENDED to be present.
  • at the time of issuance, Appendix A of the Entrust CPS (v.3.18) contained a misplaced phrase in the OV SSL certificate profile table saying that these certificates must contain the cPSuri policyQualifier.
  • at the time of issuance, the profiles of the affected certificates contained did not contain the CPSuri policyQualifier.
  • section 4.9.1.1 of the BRs requires revocation within 5 days if “the CA is made aware that the Certificate was not issued in accordance with these Requirements or the CA’s Certificate Policy or Certification Practice Statement”.

At the time of our initial assessment, we knew that the misplaced cPSuri requirement in Appendix A of the CPS was inconsistent with this data being NOT RECOMMENDED per s.7.1.2.7.9 of the BRs. This is because the phrase NOT RECOMMENDED means “there may exist valid reasons in particular circumstances when the particular behavior is acceptable or even useful, but the full implications should be understood and the case carefully weighed before implementing any behavior described with this label” (per RFC 2119, incorporated by reference in s.1.6.4 of the BRs). Thus, a unilateral requirement to include the cPSuri in all circumstances, and especially such a requirement being created in error and without any valid reason, was contrary to its inclusion being NOT RECOMMENDED. The certificates themselves, however, were compliant with the BRs.

This led us to declare a mis-issuance but decide not to revoke due to extraordinary circumstances.

Upon further review by more stakeholders and legal counsel, we received an analysis, which we then confirmed with independent outside experts, that helped us recognize that we had mischaracterized the issuance of the affected certificates as a mis-issuance, because our initial assessment did not identify all the relevant sections of our CPS.

In particular, the CPS in effect at the relevant time (v.3.18) contained these other provisions:

  • Section 1.1: “In respect to SSL Certificates, Entrust conforms to the current version of the CA/Browser Forum Baseline Requirements for the Issuance and Management of Publicly-Trusted Certificates published at https://www.cabforum.org. [...] In the event of any inconsistency between this CPS and the Baseline Requirements, the Baseline Requirements take precedence over this CPS.”
  • Section 7.1.2: “All TLS, Code Signing, S/MIME and Time-stamp Certificates are issued in accordance with profiles specified in the Baseline Requirements, Code Signing Baseline Requirements or S/MIME Baseline Requirements. CAs will not include extensions or values unless the CA is aware of a reason for including the data in the Certificate.”
  • Section 7.1.6.1: “Certificates containing a reserved certificate policy identifier indicates compliance with the associated requirements and are issued and managed in accordance with those requirements.” (The affected certificates include the reserved Certificate Policy Identifier for the BRs (2.23.140.1.2.2).)

When the CPS is read as a whole, our intention to comply with the BRs certificate profiles is clearly communicated, and the CPS directs us to prioritize the BRs over any inconsistent provision of the CPS. Not only is this the case generally (s.1.1), but also specifically with respect to extensions and values (s.7.1.2). According to the CPS itself, the erroneous statement in Appendix A must be overridden by and replaced with the relevant profile requirements and recommendations in the
BRs. The affected certificates were issued in accordance with the CPS without the cPSuri since Entrust did not know any good reason to include it, which was in compliance with the BRs and with s.7.1.2 of the CPS.

We have concluded that we had originally mischaracterized the issuance of these certificates as a mis-issuance, because the CPS itself provided guidance as to how we should resolve the inconsistency between Appendix A and the BRs, and the certificates were issued in accordance with this guidance.

Once the inconsistency between Appendix A of Entrust’s CPS and the BRs was corrected, the Entrust CPS language could again be applied to certificate issuance.

The attached chart Attachment #9405816 [details] illustrates the relevant sections of the EV Guidelines, the CPS, and one of the affected certificates.

Our revised view also took into account how the Baseline Requirements address inconsistencies between a CPS and the BRs. They do not make an inconsistency one of the specific events that trigger a requirement to revoke a subscriber certificate, but instead mandate that a CA include the language in s.1.1 (or something similar) in its CPS to resolve potential inconsistencies:

      BRs s.2.2. Commitment to Comply with Recommendations;

      The CA SHALL publicly give effect to these Requirements and represent that it will adhere to the latest published version. The CA MAY fulfill this requirement by incorporating these Requirements directly into its Certificate Policy and/or Certification Practice Statements or by incorporating them by reference using a clause such as the following (which MUST include a link to the official version of these Requirements):

      [Name of CA] conforms to the current version of the Baseline Requirements for the Issuance and Management of Publicly‐Trusted TLS Server Certificates published at http://www.cabforum.org. In the event of any inconsistency between this document and those Requirements, those Requirements take precedence over this document.

This analysis does not support a conclusion that the “In the event of any inconsistency” language will always apply to automatically supersede errors in a CPS. There are many other portions of a CA’s CPS that are not covered by the BRs where this language would not apply.

Further, inadvertent inconsistencies between the CPS and an applicable industry standard are undesirable and there are existing mechanisms (other than revocation of Subscriber certificates) in place to incentivize CAs to avoid and correct such inconsistencies. For example, we expect that inadvertent or unjustified inconsistencies could result in a finding under a WebTrust audit, which is a strong incentive for CAs to prevent such inconsistencies from arising.

The reasons above show how we arrived at the revised conclusion that we had acted in accordance with the CPS and did not need to revoke the affected certificates. Our original analysis and justification for non-revocation were not supported, and process improvements are required to ensure that similar incidents do not reoccur.

Lessons Learned

What went well

  • The decision not to revoke ultimately was consistent with the revised analysis, which minimized impact on subscribers and members of the public.
  • We had access to and eventually consulted with various stakeholders who were able to identify additional relevant factors and provide helpful analysis and advice.

What didn't go well

  • The error in updating Appendix A of the CPS which caused the inadvertent short-term inconsistency between Appendix A and the rest of the CPS, BRs and the certificates.
  • The initial decision not to revoke was not made on the basis of a thorough review of the relevant documents and rules, and instead relied on an overused “exceptional circumstances” justification.
  • It took too long to carry out the full analysis.

Where we got lucky

  • The CPS error was detected and resolved in 4 days
  • The input from additional stakeholders initially arose as an outcome of more general discussions around this and other incidents, and was not specifically built into any process

Action Items In addition to the action items in bug #1890896 we have identified the following:

Action Item Kind Due Date
Update archive CPS with errata to inform subscribers going forward Transparency Done
Inform subscribers of changes to the CPS Transparency Done
Review applicable policy and procedures Prevent 2024-06-28
Improve incident management procedures Prevent 2024-06-30
Reorganize product compliance and verification teams to provide additional organizational resources and oversight Prevent 2024-07-31

but that we were not planning to revoke the affected certificates due to exceptional circumstances.

Let’s not rewrite history here. You decided not to revoke because you made a false assertion that:

The correction was to the CPS rather than the certificate, so that if the certificates were re-issued, they would be re-issued with the identical certificate profile. We believe this to be an exceptional condition.

Despite these certificates being materially different because of a different NotBefore date.

Beyond that, stating something to the effect of “if our document is wrong then ignore it” is not a valid practice in contract law in many parts of the world and certainly is not an acceptable practice in WebPKI.

I’ll hope someone from the root programs do chime in and explicitly say that those types of statements have no meaning in WebPKI documents.

I would also caution Entrust on over-reliance on legal counsel. These documents are not being studied and judged in a court of law, but rather by engineers and researches in the sphere of public trust. Many assumptions that lawyers may have about legal jurisprudence don't apply here, and can definitely dig a deeper hole for Entrust.

Once again, what precisely is Comment #28? Is it the final report? This incident has been open since 2024-04-11 and Entrust's prior statement was that their next update would be 06-17...

(In reply to ngook.kong from comment #28)

We reviewed and consulted with independent external experts on this revised analysis, and based on this broader consultation, we now believe there was no mis-issuance and thus no need to revoke the affected certificates. A detailed analysis is below.

I will not reiterate my questions posed in #1890685. Entrust need to understand that they are now stating that any statement they have provided in the past few months should not be read in good faith, and are subject to being changed retroactively if they consult with their legal counsel.

2024-05-21:

  • In discussing this and other concurrent incidents with internal legal counsel, we received advice that we had likely mischaracterized the affected certificates as mis-issued. In the following days, we consulted more widely, including with independent external experts, and confirmed the revised analysis.

Once again it is over 2 weeks since these clarifications occur within Entrust until any public comment appears.

This analysis does not support a conclusion that the “In the event of any inconsistency” language will always apply to automatically supersede errors in a CPS. There are many other portions of a CA’s CPS that are not covered by the BRs where this language would not apply.

What does it show then? You are again trying to use a specific interpretation that your legal counsel have opined and are trying to remove all responsibility of Entrust in applying their CPS in good faith.

Further, inadvertent inconsistencies between the CPS and an applicable industry standard are undesirable and there are existing mechanisms (other than revocation of Subscriber certificates) in place to incentivize CAs to avoid and correct such inconsistencies. For example, we expect that inadvertent or unjustified inconsistencies could result in a finding under a WebTrust audit, which is a strong incentive for CAs to prevent such inconsistencies from arising.

That Entrust felt the need to state that the existing mechanisms shouldn't include revocation shows where the priorities lie: Find an interpretation that somehow can be read to stop us from revoking. The incentives need enforcement,as we are now in a state of a CA bringing in legal counsel to try and stop themselves being held to their own contracts against their own certificates.

The reasons above show how we arrived at the revised conclusion that we had acted in accordance with the CPS and did not need to revoke the affected certificates. Our original analysis and justification for non-revocation were not supported, and process improvements are required to ensure that similar incidents do not reoccur.

Could Entrust elaborate on what they mean by "Our original analysis and justification for non-revocation were not supported"?

All that these latest statements from Entrust have provided are a greater degree of mistrust in any statements that have provided, and will provide going forward. I have no idea how these comments got approved over weeks for publication, as it seems to have been a narrowly tailored question to their legal counsel and no regard for how trustworthy they appear out of context.

Flags: needinfo?(ngook.kong)

I'm not sure why determining these certificates were in accordance with the CPS after all resolves the issue.

Entrust, even if it mistakenly determines a certificate was misissued, was still bound to revoke it in 5 days. That Entrust months later concludes that it was not misissued after all doesn't change the fact that it violated its obligation to revoke in 5 days, and accompanying obligations to explain why it didn't. Entrust did not and could not have concluded during the relevant timeframe that the revocation was not required for the reasons being expressed here today for the simple fact that four months later is the first time we hear of this explanation.

Mozilla's Incident Response policy includes a set of expectations for delrev incidents.

In https://bugzilla.mozilla.org/show_bug.cgi?id=1886532#c35, Ngook Kong clarifies that Entrust sees that policy as the appropriate standard of response:

Our interpretation is any delayed revocation will comply with the Mozilla revocation policy at https://wiki.mozilla.org/CA/Responding_To_An_Incident#Revocation

One expectation listed there is as follows:

  • Your CA will work with your auditor (and supervisory body, as appropriate) and the Root Store(s) that your CA participates in to ensure your analysis of the risk and plan of remediation is acceptable.

Question: did Entrust consult with their auditor with regard to the analysis of this case as a misissuance, as the Mozilla policy requires?

Question: if so, how does the auditor's analysis differ from the later one that has been referenced here, which holds that this is not a case of misissuance? Will this difference in analysis be represented in the next BR audit?

Question: did Entrust consult with the Root Stores that it participates in with regard to the analysis of this case?

In my opinion, It is absolutely not acceptable for any CA to change their determination for what is, or isn’t, an incident - after they have written up the incident report. This would mean that we can not trust a single “final report” from any CA since it can change at any time.

Beyond that, the reason given here for why this isn’t an incident is absurd.

At this point I will be asking for the root programs to chime in here and set their expectation for the following questions:

  1. Can a CA change their opinion on that an incident that had a final report written up, to say that the incident was in fact not an incident weeks after?

  2. Do the action items that are required to happen for some incidents (such as revocation) still need to happen if a CA changes their mind on an incident like this?

  3. Is the language used in this CPS, like entrust has, acceptable for determining if the CP/S is valid or not?

Flags: needinfo?(ryandickson)
Flags: needinfo?(clintw)
Flags: needinfo?(bwilson)

(In reply to Wayne from comment #30)

Once again, what precisely is Comment #28? Is it the final report? This incident has been open since 2024-04-11 and Entrust's prior statement was that their next update would be 06-17...

(In reply to ngook.kong from comment #28)

We reviewed and consulted with independent external experts on this revised analysis, and based on this broader consultation, we now believe there was no mis-issuance and thus no need to revoke the affected certificates. A detailed analysis is below.

I will not reiterate my questions posed in #1890685. Entrust need to understand that they are now stating that any statement they have provided in the past few months should not be read in good faith, and are subject to being changed retroactively if they consult with their legal counsel.

2024-05-21:

  • In discussing this and other concurrent incidents with internal legal counsel, we received advice that we had likely mischaracterized the affected certificates as mis-issued. In the following days, we consulted more widely, including with independent external experts, and confirmed the revised analysis.

Once again it is over 2 weeks since these clarifications occur within Entrust until any public comment appears.

This analysis does not support a conclusion that the “In the event of any inconsistency” language will always apply to automatically supersede errors in a CPS. There are many other portions of a CA’s CPS that are not covered by the BRs where this language would not apply.

What does it show then? You are again trying to use a specific interpretation that your legal counsel have opined and are trying to remove all responsibility of Entrust in applying their CPS in good faith.

We acknowledge your comments and stand by our analysis. We have concluded that we originally mischaracterized the issuance of these certificates as a mis-issuance, because the CPS itself provided guidance as to how we should resolve the inconsistency between Appendix A and the BRs, and the certificates were issued in accordance with this guidance.

Further, inadvertent inconsistencies between the CPS and an applicable industry standard are undesirable and there are existing mechanisms (other than revocation of Subscriber certificates) in place to incentivize CAs to avoid and correct such inconsistencies. For example, we expect that inadvertent or unjustified inconsistencies could result in a finding under a WebTrust audit, which is a strong incentive for CAs to prevent such inconsistencies from arising.

That Entrust felt the need to state that the existing mechanisms shouldn't include revocation shows where the priorities lie: Find an interpretation that somehow can be read to stop us from revoking. The incentives need enforcement,as we are now in a state of a CA bringing in legal counsel to try and stop themselves being held to their own contracts against their own certificates.

As noted in the paragraph you quoted, mechanisms such as WebTrust audits provide strong incentives for CAs to prevent such inconsistencies from arising.

The reasons above show how we arrived at the revised conclusion that we had acted in accordance with the CPS and did not need to revoke the affected certificates. Our original analysis and justification for non-revocation were not supported, and process improvements are required to ensure that similar incidents do not reoccur.

Could Entrust elaborate on what they mean by "Our original analysis and justification for non-revocation were not supported"?

Please see comment 1 for our original analysis and justification and comment 28 for our updated report.

All that these latest statements from Entrust have provided are a greater degree of mistrust in any statements that have provided, and will provide going forward. I have no idea how these comments got approved over weeks for publication, as it seems to have been a narrowly tailored question to their legal counsel and no regard for how trustworthy they appear out of context.

Flags: needinfo?(ngook.kong)

(In reply to ngook.kong from comment #34)

(In reply to Wayne from comment #30)

Once again, what precisely is Comment #28? Is it the final report? This incident has been open since 2024-04-11 and Entrust's prior statement was that their next update would be 06-17...

And once again I am asking these question as they have still not been answered. What is this new report supposed to be?

(In reply to ngook.kong from comment #28)

We reviewed and consulted with independent external experts on this revised analysis, and based on this broader consultation, we now believe there was no mis-issuance and thus no need to revoke the affected certificates. A detailed analysis is below.

I will not reiterate my questions posed in #1890685. Entrust need to understand that they are now stating that any statement they have provided in the past few months should not be read in good faith, and are subject to being changed retroactively if they consult with their legal counsel.

I will emphasize this point - we can no longer take a statement Entrust's provides at face value in regards to an incident. As it stands all of the statements are preliminary and subject to change months later with no justification.

2024-05-21:

  • In discussing this and other concurrent incidents with internal legal counsel, we received advice that we had likely mischaracterized the affected certificates as mis-issued. In the following days, we consulted more widely, including with independent external experts, and confirmed the revised analysis.

Once again it is over 2 weeks since these clarifications occur within Entrust until any public comment appears.

I would still like to know what caused this delay in what Entrust sees as a vital new viewpoint in this mis-issuance incident. Could this be clarified?

This analysis does not support a conclusion that the “In the event of any inconsistency” language will always apply to automatically supersede errors in a CPS. There are many other portions of a CA’s CPS that are not covered by the BRs where this language would not apply.

What does it show then? You are again trying to use a specific interpretation that your legal counsel have opined and are trying to remove all responsibility of Entrust in applying their CPS in good faith.

We acknowledge your comments and stand by our analysis. We have concluded that we originally mischaracterized the issuance of these certificates as a mis-issuance, because the CPS itself provided guidance as to how we should resolve the inconsistency between Appendix A and the BRs, and the certificates were issued in accordance with this guidance.

It has been Entrust's statement for months that these certificates were mis-issued. All that Entrust are doing to themselves here are creating an environment where we cannot trust future statements, and I see no reason as to why they keep insisting this to be the case?

Further, inadvertent inconsistencies between the CPS and an applicable industry standard are undesirable and there are existing mechanisms (other than revocation of Subscriber certificates) in place to incentivize CAs to avoid and correct such inconsistencies. For example, we expect that inadvertent or unjustified inconsistencies could result in a finding under a WebTrust audit, which is a strong incentive for CAs to prevent such inconsistencies from arising.

That Entrust felt the need to state that the existing mechanisms shouldn't include revocation shows where the priorities lie: Find an interpretation that somehow can be read to stop us from revoking. The incentives need enforcement,as we are now in a state of a CA bringing in legal counsel to try and stop themselves being held to their own contracts against their own certificates.

As noted in the paragraph you quoted, mechanisms such as WebTrust audits provide strong incentives for CAs to prevent such inconsistencies from arising.

In regards to revocation: What actionable items has a finding in a WebTrust audit ever presented to Entrust? Update the risk register, clarify the action items, and tell the auditor that the procedure is there? Are you suggesting that the auditor has any enforcement mechanisms that provide 'strong incentives' for a CA to take action? Given that an auditor in this scenario cannot mention undisclosed mis-issuances to non-parties of the contract I strongly doubt that.

The reasons above show how we arrived at the revised conclusion that we had acted in accordance with the CPS and did not need to revoke the affected certificates. Our original analysis and justification for non-revocation were not supported, and process improvements are required to ensure that similar incidents do not reoccur.

Could Entrust elaborate on what they mean by "Our original analysis and justification for non-revocation were not supported"?

Please see comment 1 for our original analysis and justification and comment 28 for our updated report.

Which specific parts of the original report are not supported? Please clarify as currently that is the final report within this incident as it was presented within 2 weeks of the incident opening, rather than 2 months later.

Flags: needinfo?(ngook.kong)

In bug 1886532 comment 35, on June 1, 2024, Entrust stated in reference to this incident among others:

As we indicated in our response on Bug 1887705 https://bugzilla.mozilla.org/show_bug.cgi?id=1887705#c30, the data we generated for current Bugs “didn’t clarify if these certificates were revoked before they expired or not, we agree this is valuable information and will ensure we include this distinction in our future updates.” That’s our intention for the future.

This is an unambiguous commitment to post regular updates on how many of these certificates have been revoked and expired naturally. More than a week has gone by, and you have not posted these numbers.

Question 1: How many of the certificates affected by this incident have been revoked and how many expired naturally?

Question 2: Why is it that Entrust failed to keep its commitment to report these numbers? Please be detailed and specific.

Question 3: Proper follow-through on this commitment will involve reporting of accurate, current numbers of certificates revoked and expired naturally on at least a weekly cadence for as long as any of these certificates remains active. Can the community expect Entrust to honor its commitment for this incident?

(In reply to amir from comment #33)

In my opinion, It is absolutely not acceptable for any CA to change their determination for what is, or isn’t, an incident - after they have written up the incident report. This would mean that we can not trust a single “final report” from any CA since it can change at any time.

Beyond that, the reason given here for why this isn’t an incident is absurd.

At this point I will be asking for the root programs to chime in here and set their expectation for the following questions:

  1. Can a CA change their opinion on that an incident that had a final report written up, to say that the incident was in fact not an incident weeks after?

As a general principle, a CA operator may review and amend its determination of what constitutes an incident based on new information or a reassessment of the facts, circumstances, and relevant standards. This flexibility is essential for ensuring that actions taken are appropriate and reflective of the most accurate understanding of the situation. However, it is critical for a CA operator to maintain transparency and to document the reasons for any changes in their incident reports to maintain trust and accountability.

  1. Do the action items that are required to happen for some incidents (such as revocation) still need to happen if a CA changes their mind on an incident like this?

If a CA operator changes its determination regarding an incident, it should also review and amend its associated action items. Any decision to change action items must be documented and explained to ensure that changes in action items are traceable and based on the best available information. The CA operator's explanation should also address how its new determination affects previously planned or implemented remedial actions, including whether any planned actions will still take place or be cancelled, or whether past actions will remain in effect, be reversed, or modified. In the latter case, such decision or action/non-action would have to be fully documented, explained, and justified.

  1. Is the language used in this CPS, like entrust has, acceptable for determining if the CP/S is valid or not?

Language used in a CP/CPS should be clear, aligned with Baseline Requirements and provide enough detail to ensure accountability and auditability. Similarly, CPs and CPSes should be reviewed for clarity, appropriateness, and usability as guides for the CA operator to take actions, make decisions, and comply with requirements. When interpreting discrepancies between a CA operator's CP/CPS and the Baseline Requirements, the Baseline Requirements take precedence. (Please let me know if I have answered your third question, because I wasn't sure what you meant by "determining if the CP/S is valid or not".)

Thanks.

Flags: needinfo?(bwilson)

Ben to be specific, Entrust's CPS stated this:

In respect to EV SSL Certificates, Entrust conforms to the current version of the Guidelines for the Issuance
and Management of Extended Validation Certificates published at https://www.cabforum.org. The EV SSL
Guidelines describe certain minimum requirements that a CA must meet in order to issue EV SSL
Certificates. In the event of any inconsistency between this CPS and the EV SSL Guidelines, the EV SSL
Guidelines take precedence over this CPS.

If a CPS contains this language, but also in another point of it, is non-compliant against the BRs/EVGs - does that language allow a CA to argue that their CPS was actually in compliance, because any non-compliance of the CPS can be ignored and the CA can claim that language automatically gets overridden by the the BRs/EVGs?

Does the Mozilla Root Program accept CAs using that language to not have to consistently maintain their CPS anymore?

Flags: needinfo?(bwilson)

(In reply to amir from comment #38)

If a CPS contains this language, but also in another point of it, is non-compliant against the BRs/EVGs - does that language allow a CA to argue that their CPS was actually in compliance, because any non-compliance of the CPS can be ignored and the CA can claim that language automatically gets overridden by the the BRs/EVGs?

This language is quite standard and serves to generalize an assertion of compliance. It essentially means that if there is any conflict between the CPS and the BRs or EVGs, the latter take precedence. However, if the CA operator's assertion of compliance with the CABF guidelines turns out to be false in a specific instance, then that non-compliance would need to be addressed in an audit finding and would constitute a reportable incident in Bugzilla. So, no, their CP/CPS would not be in compliance.

Does the Mozilla Root Program accept CAs using that language to not have to consistently maintain their CPS anymore?

No, the Mozilla Root Program does not accept CA operators using this language as a blanket excuse to avoid consistently reviewing and maintaining their CPs and CPSes.

Flags: needinfo?(bwilson)

In response to Comment 33:

1. Can a CA change their opinion on that an incident that had a final report written up, to say that the incident was in fact not an incident weeks after?

Can a CA change their opinion on an incident and that be OK? Yes, it could - and here’s an example.

Note, in this example, the change of opinion took place hours after creating the incident report - and was well corroborated with opinions from multiple members of the community (and by members of multiple root programs). It’s also worth noting there were no disagreements communicated with this approach. Had the conclusion of the public process been the same, but weeks, rather than hours later, it's still possible that closing the bug as “invalid" would be an acceptable outcome.

Further, had the CA Owner begun, but did not complete revoking certificates in this illustrative scenario, we do not feel it would have been necessary for them to continue with revoking the complete set of initially planned revocations when the community was in agreement that doing so was unnecessary. To that end, it seems reasonable to conclude that both the mis-issuance and presumed revocation delay bugs would have been closed as “invalid" - and the community would have been fine with it.

2. Do the action items that are required to happen for some incidents (such as revocation) still need to happen if a CA changes their mind on an incident like this?

It depends, see above.

3. Is the language used in this CPS, like entrust has, acceptable for determining if the CP/S is valid or not?

We believe a CA’s CPS is a binding, authoritative document that outlines the CA's specific practices and procedures. In many cases, these represent the adoption, strengthening, and application of requirements statements described in a CP and/or the BRs, and are intended to reflect the actual practices in place, rather than establishing what could be guardrails for what’s considered permissible.

We believe mis-issuance is determined by non-compliance with either the BRs, EV Guidelines, or the CA's CP, CPS, or CP/CPS. A certificate that adheres to the BRs but violates the corresponding CPS should still be considered non-compliant because the CPS is a part of the CA’s overall compliance framework. By virtue of issuing a certificate that’s not compliant with the CPS, the CA is violating the expectations it set for itself and its relying parties - and expected of it by the community.

We do not believe the “catch-all" compliance statement required by the BRs in Section 2.2 was intended to serve as a mechanism for absolving a CA Owner from the expectation of adhering to its CPS, as otherwise it calls into question why these documents exist in the first place.

Explicitly, we disagree with the change of position described in this incident report, and still believe these certificates are mis-issued given they do not adhere to the stated practices disclosed within the corresponding CPS at time of issuance. Per BR 4.9.1.1(12) (“The CA is made aware that the Certificate was not issued in accordance with these Requirements or the CA’s Certificate Policy or Certification Practice Statement (CRLReason #4, superseded);”), we believe these certificates should have been revoked within 5 days of the issue’s disclosure.

Flags: needinfo?(ryandickson)

(In reply to Ryan Dickson from comment #40)

In response to Comment 33:

1. Can a CA change their opinion on that an incident that had a final report written up, to say that the incident was in fact not an incident weeks after?

Can a CA change their opinion on an incident and that be OK? Yes, it could - and here’s an example.

Note, in this example, the change of opinion took place hours after creating the incident report - and was well corroborated with opinions from multiple members of the community (and by members of multiple root programs). It’s also worth noting there were no disagreements communicated with this approach. Had the conclusion of the public process been the same, but weeks, rather than hours later, it's still possible that closing the bug as “invalid" would be an acceptable outcome.

Further, had the CA Owner begun, but did not complete revoking certificates in this illustrative scenario, we do not feel it would have been necessary for them to continue with revoking the complete set of initially planned revocations when the community was in agreement that doing so was unnecessary. To that end, it seems reasonable to conclude that both the mis-issuance and presumed revocation delay bugs would have been closed as “invalid" - and the community would have been fine with it.

2. Do the action items that are required to happen for some incidents (such as revocation) still need to happen if a CA changes their mind on an incident like this?

It depends, see above.

3. Is the language used in this CPS, like entrust has, acceptable for determining if the CP/S is valid or not?

We believe a CA’s CPS is a binding, authoritative document that outlines the CA's specific practices and procedures. In many cases, these represent the adoption, strengthening, and application of requirements statements described in a CP and/or the BRs, and are intended to reflect the actual practices in place, rather than establishing what could be guardrails for what’s considered permissible.

We believe mis-issuance is determined by non-compliance with either the BRs, EV Guidelines, or the CA's CP, CPS, or CP/CPS. A certificate that adheres to the BRs but violates the corresponding CPS should still be considered non-compliant because the CPS is a part of the CA’s overall compliance framework. By virtue of issuing a certificate that’s not compliant with the CPS, the CA is violating the expectations it set for itself and its relying parties - and expected of it by the community.

We do not believe the “catch-all" compliance statement required by the BRs in Section 2.2 was intended to serve as a mechanism for absolving a CA Owner from the expectation of adhering to its CPS, as otherwise it calls into question why these documents exist in the first place.

Explicitly, we disagree with the change of position described in this incident report, and still believe these certificates are mis-issued given they do not adhere to the stated practices disclosed within the corresponding CPS at time of issuance. Per BR 4.9.1.1(12) (“The CA is made aware that the Certificate was not issued in accordance with these Requirements or the CA’s Certificate Policy or Certification Practice Statement (CRLReason #4, superseded);”), we believe these certificates should have been revoked within 5 days of the issue’s disclosure.

Ryan, we acknowledge your comment and we are reviewing the points you have raised. We will follow up with confirmation of next steps.

Flags: needinfo?(ngook.kong)

(In reply to Ryan Dickson from comment #40)

In response to Comment 33:

1. Can a CA change their opinion on that an incident that had a final report written up, to say that the incident was in fact not an incident weeks after?

Can a CA change their opinion on an incident and that be OK? Yes, it could - and here’s an example.

Note, in this example, the change of opinion took place hours after creating the incident report - and was well corroborated with opinions from multiple members of the community (and by members of multiple root programs). It’s also worth noting there were no disagreements communicated with this approach. Had the conclusion of the public process been the same, but weeks, rather than hours later, it's still possible that closing the bug as “invalid" would be an acceptable outcome.

Further, had the CA Owner begun, but did not complete revoking certificates in this illustrative scenario, we do not feel it would have been necessary for them to continue with revoking the complete set of initially planned revocations when the community was in agreement that doing so was unnecessary. To that end, it seems reasonable to conclude that both the mis-issuance and presumed revocation delay bugs would have been closed as “invalid" - and the community would have been fine with it.

2. Do the action items that are required to happen for some incidents (such as revocation) still need to happen if a CA changes their mind on an incident like this?

It depends, see above.

3. Is the language used in this CPS, like entrust has, acceptable for determining if the CP/S is valid or not?

We believe a CA’s CPS is a binding, authoritative document that outlines the CA's specific practices and procedures. In many cases, these represent the adoption, strengthening, and application of requirements statements described in a CP and/or the BRs, and are intended to reflect the actual practices in place, rather than establishing what could be guardrails for what’s considered permissible.

We believe mis-issuance is determined by non-compliance with either the BRs, EV Guidelines, or the CA's CP, CPS, or CP/CPS. A certificate that adheres to the BRs but violates the corresponding CPS should still be considered non-compliant because the CPS is a part of the CA’s overall compliance framework. By virtue of issuing a certificate that’s not compliant with the CPS, the CA is violating the expectations it set for itself and its relying parties - and expected of it by the community.

We do not believe the “catch-all" compliance statement required by the BRs in Section 2.2 was intended to serve as a mechanism for absolving a CA Owner from the expectation of adhering to its CPS, as otherwise it calls into question why these documents exist in the first place.

Explicitly, we disagree with the change of position described in this incident report, and still believe these certificates are mis-issued given they do not adhere to the stated practices disclosed within the corresponding CPS at time of issuance. Per BR 4.9.1.1(12) (“The CA is made aware that the Certificate was not issued in accordance with these Requirements or the CA’s Certificate Policy or Certification Practice Statement (CRLReason #4, superseded);”), we believe these certificates should have been revoked within 5 days of the issue’s disclosure.

Ryan, thanks for sharing your position and that of the Chrome Root Program in comment 40.

We believe that we have conducted an appropriate analysis. However, we have fully reviewed and considered your comment, and understand that Chrome disagrees and believes these certificates should be revoked.  On this basis, we will treat this as a mis-issuance, and intend to complete revocation by end of day Saturday, June 22.

Again, thank you for providing your point of view. We will report our certificate burndown progress regularly on this bug.

(In reply to ngook.kong from comment #42)

(In reply to Ryan Dickson from comment #40)

In response to Comment 33:

1. Can a CA change their opinion on that an incident that had a final report written up, to say that the incident was in fact not an incident weeks after?

Can a CA change their opinion on an incident and that be OK? Yes, it could - and here’s an example.

Note, in this example, the change of opinion took place hours after creating the incident report - and was well corroborated with opinions from multiple members of the community (and by members of multiple root programs). It’s also worth noting there were no disagreements communicated with this approach. Had the conclusion of the public process been the same, but weeks, rather than hours later, it's still possible that closing the bug as “invalid" would be an acceptable outcome.

Further, had the CA Owner begun, but did not complete revoking certificates in this illustrative scenario, we do not feel it would have been necessary for them to continue with revoking the complete set of initially planned revocations when the community was in agreement that doing so was unnecessary. To that end, it seems reasonable to conclude that both the mis-issuance and presumed revocation delay bugs would have been closed as “invalid" - and the community would have been fine with it.

2. Do the action items that are required to happen for some incidents (such as revocation) still need to happen if a CA changes their mind on an incident like this?

It depends, see above.

3. Is the language used in this CPS, like entrust has, acceptable for determining if the CP/S is valid or not?

We believe a CA’s CPS is a binding, authoritative document that outlines the CA's specific practices and procedures. In many cases, these represent the adoption, strengthening, and application of requirements statements described in a CP and/or the BRs, and are intended to reflect the actual practices in place, rather than establishing what could be guardrails for what’s considered permissible.

We believe mis-issuance is determined by non-compliance with either the BRs, EV Guidelines, or the CA's CP, CPS, or CP/CPS. A certificate that adheres to the BRs but violates the corresponding CPS should still be considered non-compliant because the CPS is a part of the CA’s overall compliance framework. By virtue of issuing a certificate that’s not compliant with the CPS, the CA is violating the expectations it set for itself and its relying parties - and expected of it by the community.

We do not believe the “catch-all" compliance statement required by the BRs in Section 2.2 was intended to serve as a mechanism for absolving a CA Owner from the expectation of adhering to its CPS, as otherwise it calls into question why these documents exist in the first place.

Explicitly, we disagree with the change of position described in this incident report, and still believe these certificates are mis-issued given they do not adhere to the stated practices disclosed within the corresponding CPS at time of issuance. Per BR 4.9.1.1(12) (“The CA is made aware that the Certificate was not issued in accordance with these Requirements or the CA’s Certificate Policy or Certification Practice Statement (CRLReason #4, superseded);”), we believe these certificates should have been revoked within 5 days of the issue’s disclosure.

Ryan, thanks for sharing your position and that of the Chrome Root Program in comment 40.

We believe that we have conducted an appropriate analysis. However, we have fully reviewed and considered your comment, and understand that Chrome disagrees and believes these certificates should be revoked.  On this basis, we will treat this as a mis-issuance, and intend to complete revocation by end of day Saturday, June 22.

Again, thank you for providing your point of view. We will report our certificate burndown progress regularly on this bug.

For posterity:

Separate from the analysis performed as mentioned, is it still Entrust's opinion that these certificates were not mis-issued?

For root programs:

Does re-characterizing the certificates as mis-issued after originally recharacterizing them as issued properly reset the 24/120h timer? In other words, are these certificates now in delayed revocation (as they were originally mis-issued as of 2024-04-10 per Entrust)? If not, is it the intent of the BRs as written / interpreted to allow for a CA to repeatedly re-characterize certificates as correctly issued and then when questioned by a RP, move the offending certificates back into mis-issuance status, allowing for dragging on the 24/120h time allowed before it's considered delayed revocation?

Flags: needinfo?(ryandickson)
Flags: needinfo?(ngook.kong)
Flags: needinfo?(bwilson)

In response to Comment 43:

Does re-characterizing the certificates as mis-issued after originally recharacterizing them as issued properly reset the 24/120h timer?

No.

In other words, are these certificates now in delayed revocation (as they were originally mis-issued as of 2024-04-10 per Entrust)?

Yes.

If not, is it the intent of the BRs as written / interpreted to allow for a CA to repeatedly re-characterize certificates as correctly issued and then when questioned by a RP, move the offending certificates back into mis-issuance status, allowing for dragging on the 24/120h time allowed before it's considered delayed revocation?

Not applicable, given above opinions.

Flags: needinfo?(ryandickson)

I agree with the answers given in Comment #44.

Flags: needinfo?(bwilson)

(In reply to ngook.kong from comment #42)

(In reply to Ryan Dickson from comment #40)

In response to Comment 33:

1. Can a CA change their opinion on that an incident that had a final report written up, to say that the incident was in fact not an incident weeks after?

Can a CA change their opinion on an incident and that be OK? Yes, it could - and here’s an example.

Note, in this example, the change of opinion took place hours after creating the incident report - and was well corroborated with opinions from multiple members of the community (and by members of multiple root programs). It’s also worth noting there were no disagreements communicated with this approach. Had the conclusion of the public process been the same, but weeks, rather than hours later, it's still possible that closing the bug as “invalid" would be an acceptable outcome.

Further, had the CA Owner begun, but did not complete revoking certificates in this illustrative scenario, we do not feel it would have been necessary for them to continue with revoking the complete set of initially planned revocations when the community was in agreement that doing so was unnecessary. To that end, it seems reasonable to conclude that both the mis-issuance and presumed revocation delay bugs would have been closed as “invalid" - and the community would have been fine with it.

2. Do the action items that are required to happen for some incidents (such as revocation) still need to happen if a CA changes their mind on an incident like this?

It depends, see above.

3. Is the language used in this CPS, like entrust has, acceptable for determining if the CP/S is valid or not?

We believe a CA’s CPS is a binding, authoritative document that outlines the CA's specific practices and procedures. In many cases, these represent the adoption, strengthening, and application of requirements statements described in a CP and/or the BRs, and are intended to reflect the actual practices in place, rather than establishing what could be guardrails for what’s considered permissible.

We believe mis-issuance is determined by non-compliance with either the BRs, EV Guidelines, or the CA's CP, CPS, or CP/CPS. A certificate that adheres to the BRs but violates the corresponding CPS should still be considered non-compliant because the CPS is a part of the CA’s overall compliance framework. By virtue of issuing a certificate that’s not compliant with the CPS, the CA is violating the expectations it set for itself and its relying parties - and expected of it by the community.

We do not believe the “catch-all" compliance statement required by the BRs in Section 2.2 was intended to serve as a mechanism for absolving a CA Owner from the expectation of adhering to its CPS, as otherwise it calls into question why these documents exist in the first place.

Explicitly, we disagree with the change of position described in this incident report, and still believe these certificates are mis-issued given they do not adhere to the stated practices disclosed within the corresponding CPS at time of issuance. Per BR 4.9.1.1(12) (“The CA is made aware that the Certificate was not issued in accordance with these Requirements or the CA’s Certificate Policy or Certification Practice Statement (CRLReason #4, superseded);”), we believe these certificates should have been revoked within 5 days of the issue’s disclosure.

Ryan, thanks for sharing your position and that of the Chrome Root Program in comment 40.

We believe that we have conducted an appropriate analysis. However, we have fully reviewed and considered your comment, and understand that Chrome disagrees and believes these certificates should be revoked.  On this basis, we will treat this as a mis-issuance, and intend to complete revocation by end of day Saturday, June 22.

Again, thank you for providing your point of view. We will report our certificate burndown progress regularly on this bug.

What in Ben (Mozilla Root Program’s) response in an earlier comment: https://bugzilla.mozilla.org/show_bug.cgi?id=1890898#c37 made you interpret your behavior and position as appropriate before Ryan (Chrome Root Program) commented?

I’m wondering because it feels like Entrust seems to be putting more weight behind Google and far less behind Mozilla.

I would also like to ask based on what did you make the call to consider that the initial revocation requirements did not apply until the root programs corrected you? Did you make your decision based on historical data (such as another incident)?

We believe that we have conducted an appropriate analysis. However, we have fully reviewed and considered your comment, and understand that Chrome disagrees and believes these certificates should be revoked.  On this basis, we will treat this as a mis-issuance, and intend to complete revocation by end of day Saturday, June 22.

Can you also please explain this statement a bit more clearly? Specifically:

  • Do you disagree with Mozilla's position here? If so, which points and why?
  • Do you disagree with Chrome's position here? If so, which points and why?

Given the very unusual situation we are in of a CA taking 3 months to start revocation and re-issuance I have a few thoughts:

  1. How many certificates of the originally impacted group have expired?
  2. How many certificate of the originally impacted group have been revoked prior to the statement on 2024-06-18?
  3. Are the total number of impacted certificates still considered to be 6,008?
  4. What time is Entrust considering the original confirmation of the issue? In my prior records I noted 2024-03-26 15:28 (UTC) given the timeline stated in comment 1.
  5. Does a calculation of revocation date from this specified timeframe agree with Entrust's thoughts? If not, what alternative is suggested?
  6. How many subscribers were required to undergo a new verification check for re-issuance? How many subsequently failed?

Normally in delayed revocation incidents these details wouldn't come up, but the gap in time makes this particularly unique and noteworthy for further review. It would be interesting to see data based off of the remaining certificates to be handled in this incident as otherwise the initial figures may be skewed given the initial gap.

(In reply to Mike Shaver (:shaver emeritus) from comment #32)

Question: did Entrust consult with their auditor with regard to the analysis of this case as a misissuance, as the Mozilla policy requires?

Question: if so, how does the auditor's analysis differ from the later one that has been referenced here, which holds that this is not a case of misissuance? Will this difference in analysis be represented in the next BR audit?

Question: did Entrust consult with the Root Stores that it participates in with regard to the analysis of this case?

On the first two questions, Mozilla’s expectation is that the delayed revocation be raised as a finding on the next audit report and discussed with the auditor at that time. We will absolutely do that. On the last question, our position is that there was no mis-issuance—not that there was a mis-issuance and we decided not to revoke which is the situation that recommends discussion with affected root stores.

(In reply to Wayne from comment #35)

And once again I am asking these question as they have still not been answered. What is this new report supposed to be?

The new report is a revised final incident report.

I will emphasize this point - we can no longer take a statement Entrust's provides at face value in regards to an incident. As it stands all of the statements are preliminary and subject to change months later with no justification.

We respectfully disagree. We have provided a full explanation for why we revised our analysis. We feel it is appropriate for a CA to post a revised incident report when it becomes aware of information that shows errors in that incident report.

2024-05-21:

  • In discussing this and other concurrent incidents with internal legal counsel, we received advice that we had likely mischaracterized the affected certificates as mis-issued. In the following days, we consulted more widely, including with independent external experts, and confirmed the revised analysis.

Once again it is over 2 weeks since these clarifications occur within Entrust until any public comment appears.

I would still like to know what caused this delay in what Entrust sees as a vital new viewpoint in this mis-issuance incident. Could this be clarified?

The initial advice given on 05-21-2024 indicated that we had “likely” mischaracterized the certificates as mis-issued. The following sentence explains that we did not immediately change our position on May 21st. Instead, we took additional time to consult more widely, before confirming the revised analysis.

It has been Entrust's statement for months that these certificates were mis-issued. All that Entrust are doing to themselves here are creating an environment where we cannot trust future statements, and I see no reason as to why they keep insisting this to be the case?

We have provided a full explanation for why we no longer believe the certificates were mis-issued. Sometimes new information or perspectives come to light after a final incident report is posted. We feel it benefits the community (and public trust) if CAs have the flexibility to change a position or report if new information or analysis comes to light, as long as that new information or analysis is transparently shared, which is what we have done here.

In regards to revocation: What actionable items has a finding in a WebTrust audit ever presented to Entrust? Update the risk register, clarify the action items, and tell the auditor that the procedure is there? Are you suggesting that the auditor has any enforcement mechanisms that provide 'strong incentives' for a CA to take action? Given that an auditor in this scenario cannot mention undisclosed mis-issuances to non-parties of the contract I strongly doubt that.

Auditors are not enforcers, we did not suggest otherwise. Audit reports are posted publicly in every public CA’s repository and are used by subscribers and root embedding programs to assess CAs. We are simply suggesting that mandating audits of inconsistencies could be an effective alternate to achieving the desired outcomes.

Which specific parts of the original report are not supported? Please clarify as currently that is the final report within this incident as it was presented within 2 weeks of the incident opening, rather than 2 months later.

Acknowledging your comments and question, Wayne. We are reviewing the response from the Chrome Root Program and will follow up on this.

(In reply to Tim Callan from comment #36)

In bug 1886532 comment 35, on June 1, 2024, Entrust stated in reference to this incident among others:

Question 3: Proper follow-through on this commitment will involve reporting of accurate, current numbers of certificates revoked and expired naturally on at least a weekly cadence for as long as any of these certificates remains active. Can the community expect Entrust to honor its commitment for this incident?

Yes.

(In reply to Bruce Morton from comment #49)

On the first two questions, Mozilla’s expectation is that the delayed revocation be raised as a finding on the next audit report and discussed with the auditor at that time. We will absolutely do that. On the last question, our position is that there was no mis-issuance—not that there was a mis-issuance and we decided not to revoke which is the situation that recommends discussion with affected root stores.

Is... is that still Entrust's understanding of the situation? Currently I have doubts as to the time these responses were written and that if they truly reflect Entrust's state of mind at the time they were posted.

(In reply to Bruce Morton from comment #50)

The new report is a revised final incident report.

Thank you for answering.

We respectfully disagree. We have provided a full explanation for why we revised our analysis. We feel it is appropriate for a CA to post a revised incident report when it becomes aware of information that shows errors in that incident report.

I appreciate your disagreement.

The initial advice given on 05-21-2024 indicated that we had “likely” mischaracterized the certificates as mis-issued. The following sentence explains that we did not immediately change our position on May 21st. Instead, we took additional time to consult more widely, before confirming the revised analysis.

Thank you for clarifying.

We have provided a full explanation for why we no longer believe the certificates were mis-issued. Sometimes new information or perspectives come to light after a final incident report is posted. We feel it benefits the community (and public trust) if CAs have the flexibility to change a position or report if new information or analysis comes to light, as long as that new information or analysis is transparently shared, which is what we have done here.

I have raised a question to similar response elsewhere so I won't restate it here.

Auditors are not enforcers, we did not suggest otherwise. Audit reports are posted publicly in every public CA’s repository and are used by subscribers and root embedding programs to assess CAs. We are simply suggesting that mandating audits of inconsistencies could be an effective alternate to achieving the desired outcomes.

My entire point in that statement is that there is little incentive in an ongoing auditing relationship to provide some pushback on an auditor's behalf. I will note this is not a reflection on Entrust, but on the current auditing standards being used and thus will not pose a further question here.

Acknowledging your comments and question, Wayne. We are reviewing the response from the Chrome Root Program and will follow up on this.

Thank you.

(In reply to Bruce Morton from comment #50)

(In reply to Wayne from comment #35)

And once again I am asking these question as they have still not been answered. What is this new report supposed to be?

The new report is a revised final incident report.

Seeing as you have, following clarification from Ryan, stated that you will treat the certificates covered by this incident as mississued again; can we expect a revised revised final incident report from Entrust?

(In reply to Bruce Morton from comment #49)

(In reply to Mike Shaver (:shaver emeritus) from comment #32)

Question: did Entrust consult with their auditor with regard to the analysis of this case as a misissuance, as the Mozilla policy requires?

Question: if so, how does the auditor's analysis differ from the later one that has been referenced here, which holds that this is not a case of misissuance? Will this difference in analysis be represented in the next BR audit?

Question: did Entrust consult with the Root Stores that it participates in with regard to the analysis of this case?

On the first two questions, Mozilla’s expectation is that the delayed revocation be raised as a finding on the next audit report and discussed with the auditor at that time. We will absolutely do that. On the last question, our position is that there was no mis-issuance—not that there was a mis-issuance and we decided not to revoke which is the situation that recommends discussion with affected root stores.

Question: If Entrust does not consider these certificates mis-issued, under what section of the Terms of Use are they being revoked under?

Question: How is this revocation event being characterized to subscribers?

Flags: needinfo?(bruce.morton)

(In reply to Wayne from comment #48)

Given the very unusual situation we are in of a CA taking 3 months to start revocation and re-issuance I have a few thoughts:

  1. How many certificates of the originally impacted group have expired?

As of today, 1095 certificates have expired.

  1. How many certificate of the originally impacted group have been revoked prior to the statement on 2024-06-18?

As of today, 1486 certificates have been revoked.

  1. Are the total number of impacted certificates still considered to be 6,008?

No. The previous list was incorrect as the certificate search was done in EST time instead of UTC time. The updated certificate list has been attached to the bug. The new total is 5910 certificates.

  1. What time is Entrust considering the original confirmation of the issue? In my prior records I noted 2024-03-26 15:28 (UTC) given the timeline stated in comment 1.

We agree that the typographical error was discovered on 2024-03-26 15:28 (UTC).

  1. Does a calculation of revocation date from this specified timeframe agree with Entrust's thoughts? If not, what alternative is suggested?

We understand that it is the position of Google Chrome (Comment 40), Mozilla (Comment 45), and others in community that the certificates should have been revoked within 5 days of on 2024-03-26 15:28 (UTC). We are not proposing an alternative timeframe.

  1. How many subscribers were required to undergo a new verification check for re-issuance? How many subsequently failed?

Normally in delayed revocation incidents these details wouldn't come up, but the gap in time makes this particularly unique and noteworthy for further review. It would be interesting to see data based off of the remaining certificates to be handled in this incident as otherwise the initial figures may be skewed given the initial gap.

For OV certificates, verified information about domains can be re-used for 398 days, and verified information about identity can be re-used for 825 days. Subscriber certificates are generally reissued based on pre-verified data in their account profile, if the reissuance occurs during the validity period of the pre-verified data. Any verified data which has expired would be re-verified. However, we don’t believe that the time gap would inherently create any concerns about verification.

Flags: needinfo?(bruce.morton)

(In reply to Bruce Morton from comment #55)
Then would it be accurate to say that as of "today" there are 3329 certificates to be resolved? I only state this to get a proper scale of actions taken over the next week, and to avoid conflation with actions taken months ago.

We completed the revocation of all affected OV TLS certificates by 23:17 UTC June 22, 2024.

(In reply to Bruce Morton from comment #49)

our position is that there was no mis-issuance—not that there was a mis-issuance and we decided not to revoke…

You just stated in the case of this bug that Entrust did not make the decision not to revoke certificates that it believed at the time to be misissued.

Question 1: How do you reconcile that with this statement from bug 1890896 comment 1

These certificates are considered mis-issued in accordance with the CCADB policy

and this statement from comment 2

  1. Why did we decide to not revoke these certificates?
    We do not believe that revoking certificates already issued as part of our response would benefit the Web PKI. The correction was to the CPS rather than the certificate, so that if the certificates were re-issued, they would be re-issued with the identical certificate profile. We believe this to be an exceptional condition.

and with many other statements on this very thread from April 10 up through at least June 17?

Question 2: Do you agree that between roughly April 10, 2024 and roughly June 5, 2024 Entrust willingly chose not to revoke certificates that it believed at the time to be misissued?

(In reply to Bruce Morton from comment #51)

(In reply to Tim Callan from comment #36)

Question 3: Proper follow-through on this commitment will involve reporting of accurate, current numbers of certificates revoked and expired naturally on at least a weekly cadence for as long as any of these certificates remains active. Can the community expect Entrust to honor its commitment for this incident?

Yes.

Um, sure. Now that in response to pressure from Chrome you have decided you have no choice but to revoke the certificates, providing an ongoing weekly report of unrevoked certificates is a pretty easy task. To have any significance, this commitment should have come prior to comment 40.

In the meantime, I had two other questions in comment 36 that you skipped.

Question 1: How many of the certificates affected by this incident have been revoked and how many expired naturally?
Question 2: Why is it that Entrust failed to keep its commitment to report these numbers? Please be detailed and specific.

Entrust proclaimed on June 17 in comment 42 that it would revoke these certificates. I believe full visibility on this incident and the state of these certificates requires a snapshot of the revocation and expiration status of these certificates from prior to that commitment, let’s say on June 14 when Chrome posted the comment that changed Entrust’s position.

Question 1.1: On June 14 at the time of posting for comment 40, how many of the certificates affected by this incident had been revoked and how many had expired naturally?

I believe that question 2 is still an important question and would like it answered as well.

(In reply to walter.j.marks from comment #43)

For posterity:

Separate from the analysis performed as mentioned, is it still Entrust's opinion that these certificates were not mis-issued?

No, we did our analysis, but acknowledge the community and browser’s clearly expressed disagreement and accepted the direction to revoke for this case. If this bug were to happen again tomorrow, we would report it as a mis-issuance and revoke.

For transparency and because you have asked us to clarify our position, below is the analysis performed, it does not change the outcome.

We also want to clarify that certain questions posed to the Root Programs in Comments 33 and 38 are not accurate representations of our position, in case anyone received that impression.

With reference to the third question in Comment 33, we made no assertion, and it is not our position, that the language in section 1.1 of our CPS (the language recommended by the Baseline Requirements) makes any inaccuracy or error in the CPS somehow “valid” or acceptable. Instead, section 1.1 indicates what rule or practice should take precedence (i.e. should actually be followed) in the event of an inconsistency between a practice described in the CPS and a practice set out in the applicable industry standard.

With reference to the question in Comment 38, we did not argue, nor take the position, that any inconsistency between the CPS and the Baseline Requirements, or any non-compliance of a CA’s actions with the practices stated in the CPS can be ignored. Nor did we claim that any false or inaccurate language in the CPS always gets overridden by the BRs/EVGs. We also never claimed that CAs can use this language to avoid having to consistently maintain their CPS.

For posterity, we agree with Ben’s positions in Comment 37 and Comment 39. In particular, we agree that a CA operator should have the flexibility to review and amend a determination based on new information, and also that it is critical to document the reasons for the change. We sincerely believed that our revised incident report fully documented those reasons. We agree that any changes in action items should be documented, explained and justified—in this case, we had added new action items, the rationale for which we believed was clear. We agree that CPs and CPSes should be reviewed for clarity, appropriateness, and usability. We are not trying to avoid responsibility for reviewing and updating our CPS.

We also agree with many of Ryan’s statements in Comment 40, including that a certificate that adheres to the BRs but violates the corresponding CPS should still be considered non-compliant.

Where we disagree, respectfully, is with the position “these certificates are mis-issued given they do not adhere to the stated practices disclosed within the corresponding CPS at the time of issuance.”

Under our analysis, the affected certificates are compliant with our CPS. We identified that our text placement error created an unintentional internal inconsistency in the CPS, with statements on the one hand that we would not include any extensions in OV certificates without a good reason (s.7.1.2) and that we would issue certificates with the BR policy OID in accordance with the BRs (s.7.1.6.1), and on the other hand, a certificate profile in Appendix A that required a CPSuri qualifier for OV certificates without any good reason. Our position is that in order to issue certificates in accordance with the CPS, it was necessary to resolve which of these contradictory statements should be followed, and the CPS itself provided this resolution, which was to follow the statement that aligns with the BRs, i.e. not include the extension. We believe it is the primary purpose of section 1.1 of the CPS to resolve ambiguities and inconsistencies and provide guidance as to which practice should be followed in such instances. We acted in accordance with the CPS, and therefore the revocation requirement described in the BRs “The CA is made aware that the Certificate was not issued in accordance with…the CA’s Certificate Policy or Certification Practice Statement” was not triggered.

Again, we appreciate the opportunity to clarify our opinion.

For root programs:

Does re-characterizing the certificates as mis-issued after originally recharacterizing them as issued properly reset the 24/120h timer? In other words, are these certificates now in delayed revocation (as they were originally mis-issued as of 2024-04-10 per Entrust)? If not, is it the intent of the BRs as written / interpreted to allow for a CA to repeatedly re-characterize certificates as correctly issued and then when questioned by a RP, move the offending certificates back into mis-issuance status, allowing for dragging on the 24/120h time allowed before it's considered delayed revocation?

We appreciate that this question was directed to the Root Programs, but once again we’d like to clarify that these questions were not based on our position or any assertion we have made. We have not taken the position that anything we have done has “reset the 24/120h timer”. On the basis of the view that the certificates were mis-issued, then the timer for revocation began when the issue was discovered. We agree with Comments 44 and 45 in this matter. We accept that the certificates now revoked for this bug were not revoked within the required timeframe.

(In reply to amir from comment #46)

What in Ben (Mozilla Root Program’s) response in an earlier comment: https://bugzilla.mozilla.org/show_bug.cgi?id=1890898#c37 made you interpret your behavior and position as appropriate before Ryan (Chrome Root Program) commented?

I’m wondering because it feels like Entrust seems to be putting more weight behind Google and far less behind Mozilla.

I would also like to ask based on what did you make the call to consider that the initial revocation requirements did not apply until the root programs corrected you? Did you make your decision based on historical data (such as another incident)?

Ben (Mozilla)’s response to your questions in Comment 37 was clear and cogent, and we agree with it. Nothing in it expressly disagreed with our position. On the other hand, Ryan (Chrome) stated that he explicitly disagreed with our position. While we wish that Ryan had engaged with our actual position instead of expressing his disagreement as a response to a question in Comment 33 that bore little resemblance to our position, we appreciate the clarity of direction.

We value the positions of all browsers equally. In this case, Chrome was the first to clearly reject our position, but it could have been Mozilla and we would have responded the same way.

We disagree with your statement in the form of a question that we made a “call to consider that the initial revocation requirements did not apply until the root programs corrected” us. We transparently disclosed in Comment 28 the full revised analysis. The subsequent decision to revoke was made on the basis of Chrome’s clearly expressed disagreement and belief that the certificates should be revoked, as stated in our Comment 42.

(In reply to Bruce Morton from comment #58)

We completed the revocation of all affected OV TLS certificates by 23:17 UTC June 22, 2024.

I would appreciate is Entrust did what they have actually stated. Here is at least two that they did not properly handle as stated

(In reply to Bruce Morton from comment #2)

This incident involves 6,008 OV TLS certificates that were issued between March 22, 2024, 15:02 UTC and March 26, 2024, 17:07 UTC.

https://crt.sh/?id=12688739474

Not Before: Mar 26 17:07:54 2024 GMT
CRL: Revoked at 2024-06-24 04:01:00 UTC

https://crt.sh/?id=12688739250

Not Before: Mar 26 17:07:20 2024 GMT
CRL: Not Revoked
OCSP: Good

These are both in the original list too... is this the new compliance team in action?

Flags: needinfo?(bruce.morton)

(In reply to amir from comment #47)

We believe that we have conducted an appropriate analysis. However, we have fully reviewed and considered your comment, and understand that Chrome disagrees and believes these certificates should be revoked.  On this basis, we will treat this as a mis-issuance, and intend to complete revocation by end of day Saturday, June 22.

Can you also please explain this statement a bit more clearly? Specifically:

  • Do you disagree with Mozilla's position here? If so, which points and why?
  • Do you disagree with Chrome's position here? If so, which points and why?

Please see https://bugzilla.mozilla.org/show_bug.cgi?id=1890898#c61.

Flags: needinfo?(bruce.morton)

(In reply to Wayne from comment #52)

(In reply to Bruce Morton from comment #49)

On the first two questions, Mozilla’s expectation is that the delayed revocation be raised as a finding on the next audit report and discussed with the auditor at that time. We will absolutely do that. On the last question, our position is that there was no mis-issuance—not that there was a mis-issuance and we decided not to revoke which is the situation that recommends discussion with affected root stores.

Is... is that still Entrust's understanding of the situation? Currently I have doubts as to the time these responses were written and that if they truly reflect Entrust's state of mind at the time they were posted.

Yes, we did our analysis, but acknowledge the community and browser’s clearly expressed disagreement and accepted the direction to revoke for this case. If this bug were to happen again tomorrow, we would report it as a mis-issuance and revoke.

(In reply to Zacharias from comment #53)

(In reply to Bruce Morton from comment #50)

(In reply to Wayne from comment #35)

And once again I am asking these question as they have still not been answered. What is this new report supposed to be?

The new report is a revised final incident report.

Seeing as you have, following clarification from Ryan, stated that you will treat the certificates covered by this incident as mississued again; can we expect a revised revised final incident report from Entrust?

Yes, we will amend the final incident report to reflect that we did our analysis, but that the community and browsers disagreed, and we accepted this and revoked the certificates. If this bug were to happen again tomorrow, we would report it as a mis-issuance and revoke.

(In reply to walter.j.marks from comment #54)

(In reply to Bruce Morton from comment #49)

(In reply to Mike Shaver (:shaver emeritus) from comment #32)

Question: did Entrust consult with their auditor with regard to the analysis of this case as a misissuance, as the Mozilla policy requires?

Question: if so, how does the auditor's analysis differ from the later one that has been referenced here, which holds that this is not a case of misissuance? Will this difference in analysis be represented in the next BR audit?

Question: did Entrust consult with the Root Stores that it participates in with regard to the analysis of this case?

On the first two questions, Mozilla’s expectation is that the delayed revocation be raised as a finding on the next audit report and discussed with the auditor at that time. We will absolutely do that. On the last question, our position is that there was no mis-issuance—not that there was a mis-issuance and we decided not to revoke which is the situation that recommends discussion with affected root stores.

Question: If Entrust does not consider these certificates mis-issued, under what section of the Terms of Use are they being revoked under?

This question is moot since we are now treating these as mis-issued per the community and browser’s clearly expressed direction.

In addition, we do not believe that a discussion about interpretation of contract terms or enforcement is appropriate as communications between corporations and their legal counsel are privileged and confidential, and corporations are not generally free to disclose legal strategy in a public forum.

Question: How is this revocation event being characterized to subscribers?

This event is being explained to subscribers in terms that we believe are appropriate to that audience and designed to impress upon them the urgency of the need to revoke, which we believe is what this community expects, based on the feedback we’ve been receiving. We have used several different methods of communication (e.g. email, phone) and the drafting has been slightly different. If what you want to know is whether we have told the subscribers that the certificates mis-issued, the answer is yes, we have told them that the certificates were mis-issued. We have also transparently disclosed to them, in at least some more detailed communications, the history of this bug, including the fact that it is Chrome’s position that the certificates are mis-issued, and that Entrust’s own analysis came to a different conclusion.

(In reply to Bruce Morton from comment #65)

(In reply to Wayne from comment #52)

(In reply to Bruce Morton from comment #49)

On the first two questions, Mozilla’s expectation is that the delayed revocation be raised as a finding on the next audit report and discussed with the auditor at that time. We will absolutely do that. On the last question, our position is that there was no mis-issuance—not that there was a mis-issuance and we decided not to revoke which is the situation that recommends discussion with affected root stores.

Is... is that still Entrust's understanding of the situation? Currently I have doubts as to the time these responses were written and that if they truly reflect Entrust's state of mind at the time they were posted.

Yes, we did our analysis, but acknowledge the community and browser’s clearly expressed disagreement and accepted the direction to revoke for this case. If this bug were to happen again tomorrow, we would report it as a mis-issuance and revoke.

So it is still Entrust's analysis and position that no mis-issuance occurred, but you are taking the community and browser's positions solely with regard to this precise interpretation? Could you share Entrust's analysis at some point so it can be reviewed?

The lack of reflection on Entrust having a different perspective than any entity commentating is apparent. Going forward there needs to be trust that Entrust are capable of generating and enacting viewpoints in alignment with everyone's expectations outside of this specific case alone.

(In reply to Bruce Morton from comment #66)

Seeing as you have, following clarification from Ryan, stated that you will treat the certificates covered by this incident as mississued again; can we expect a revised revised final incident report from Entrust?

Yes, we will amend the final incident report to reflect that we did our analysis, but that the community and browsers disagreed, and we accepted this and revoked the certificates. If this bug were to happen again tomorrow, we would report it as a mis-issuance and revoke.

I sincerely hope Entrust are not being serious about attempting a second amended incident report to cover more of their shortcomings. This incident does need a port-mortem analysis, but I'm unsure if we can trust Entrust to competently assess it as this rate. The most cursory glance found a cert revoked days past when Entrust claimed it was finished, and another certificate that hasn't even been revoked. I'm dreading even looking deeper into this mess.

(In reply to Bruce Morton from comment #67)

Question: How is this revocation event being characterized to subscribers?

This event is being explained to subscribers in terms that we believe are appropriate to that audience and designed to impress upon them the urgency of the need to revoke, which we believe is what this community expects, based on the feedback we’ve been receiving. We have used several different methods of communication (e.g. email, phone) and the drafting has been slightly different. If what you want to know is whether we have told the subscribers that the certificates mis-issued, the answer is yes, we have told them that the certificates were mis-issued. We have also transparently disclosed to them, in at least some more detailed communications, the history of this bug, including the fact that it is Chrome’s position that the certificates are mis-issued, and that Entrust’s own analysis came to a different conclusion.

Can you share such wording? I haven't seen a statement of Entrust sent "privately" that reflects what you are claiming, or is this conflating phone conversations to specific parties with the upcoming revocation notice? I do know Entrust were grumbling about there being no 'provisions' for delays - so you can feel free to share that email. I hope there are no redactions this time around, we have learned this time?

Flags: needinfo?(bruce.morton)

(In reply to Bruce Morton from comment #67)

(In reply to walter.j.marks from comment #54)

(In reply to Bruce Morton from comment #49)

(In reply to Mike Shaver (:shaver emeritus) from comment #32)

Question: did Entrust consult with their auditor with regard to the analysis of this case as a misissuance, as the Mozilla policy requires?

Question: if so, how does the auditor's analysis differ from the later one that has been referenced here, which holds that this is not a case of misissuance? Will this difference in analysis be represented in the next BR audit?

Question: did Entrust consult with the Root Stores that it participates in with regard to the analysis of this case?

On the first two questions, Mozilla’s expectation is that the delayed revocation be raised as a finding on the next audit report and discussed with the auditor at that time. We will absolutely do that. On the last question, our position is that there was no mis-issuance—not that there was a mis-issuance and we decided not to revoke which is the situation that recommends discussion with affected root stores.

Question: If Entrust does not consider these certificates mis-issued, under what section of the Terms of Use are they being revoked under?

This question is moot since we are now treating these as mis-issued per the community and browser’s clearly expressed direction.

I’m sorry that this is pedantic but I feel that is still necessary because I’m still not seeing alignment of Entrust and the rest of the community. You did not answer the question, which was “If Entrust considers these certificates mis-issued“ (emphasis mine). You answered that Enrust is treating them as mississued.

Can you refer to any other member of the community that share your analysis? Did you at anytime consult with a third party expert? If so, what was their analysis?

In addition, we do not believe that a discussion about interpretation of contract terms or enforcement is appropriate as communications between corporations and their legal counsel are privileged and confidential, and corporations are not generally free to disclose legal strategy in a public forum.

This is not a court of law, but a community about trust. A community which has been disappoints time and time again about your lack of transparency. And I also believe you are wrong about your point, because you were asked about under which Terms of Use the certificates were being revoked under. Are you not sharing that reason with your subscribers? I imagine any lawsuit requiring legal strategy would center around if the stated reason for revocation is valid or not.

Question: How is this revocation event being characterized to subscribers?

This event is being explained to subscribers in terms that we believe are appropriate to that audience and designed to impress upon them the urgency of the need to revoke, which we believe is what this community expects, based on the feedback we’ve been receiving. We have used several different methods of communication (e.g. email, phone) and the drafting has been slightly different. If what you want to know is whether we have told the subscribers that the certificates mis-issued, the answer is yes, we have told them that the certificates were mis-issued. We have also transparently disclosed to them, in at least some more detailed communications, the history of this bug, including the fact that it is Chrome’s position that the certificates are mis-issued, and that Entrust’s own analysis came to a different conclusion.

(In reply to Wayne from comment #63)

(In reply to Bruce Morton from comment #58)

We completed the revocation of all affected OV TLS certificates by 23:17 UTC June 22, 2024.

I would appreciate is Entrust did what they have actually stated. Here is at least two that they did not properly handle as stated

(In reply to Bruce Morton from comment #2)

This incident involves 6,008 OV TLS certificates that were issued between March 22, 2024, 15:02 UTC and March 26, 2024, 17:07 UTC.

https://crt.sh/?id=12688739474

Not Before: Mar 26 17:07:54 2024 GMT
CRL: Revoked at 2024-06-24 04:01:00 UTC

https://crt.sh/?id=12688739250

Not Before: Mar 26 17:07:20 2024 GMT
CRL: Not Revoked
OCSP: Good

These are both in the original list too... is this the new compliance team in action?

Hi Wayne,

The effective period ends March 26, 2024, 17:07 UTC. Your posted certificates are out of this period.

Flags: needinfo?(bruce.morton)

(In reply to Bruce Morton from comment #70)

(In reply to Wayne from comment #63)

(In reply to Bruce Morton from comment #58)

We completed the revocation of all affected OV TLS certificates by 23:17 UTC June 22, 2024.

I would appreciate is Entrust did what they have actually stated. Here is at least two that they did not properly handle as stated

(In reply to Bruce Morton from comment #2)

This incident involves 6,008 OV TLS certificates that were issued between March 22, 2024, 15:02 UTC and March 26, 2024, 17:07 UTC.

https://crt.sh/?id=12688739474

Not Before: Mar 26 17:07:54 2024 GMT
CRL: Revoked at 2024-06-24 04:01:00 UTC

https://crt.sh/?id=12688739250

Not Before: Mar 26 17:07:20 2024 GMT
CRL: Not Revoked
OCSP: Good

These are both in the original list too... is this the new compliance team in action?

Hi Wayne,

The effective period ends March 26, 2024, 17:07 UTC. Your posted certificates are out of this period.

They’re in 20240409-OV-TLS-Certificates-typo-finalcerts.txt, though.

Flags: needinfo?(bruce.morton)

(In reply to Paul Buonopane from comment #71)

Hi Wayne,

The effective period ends March 26, 2024, 17:07 UTC. Your posted certificates are out of this period.

They’re in 20240409-OV-TLS-Certificates-typo-finalcerts.txt, though.

Yup, which is the canonical 6,008 impacted cert figure as this incident report outlines in Comment 2 (04-11), Comment 28 (06-06), June 7th Report, and Updated June 21st Report.

Is the June 21st comment in Comment 55 is now considered authoritative? What went wrong for months to produce an incorrect authoritative list of certificates and certify them as such for multiple audited reports? Notably the times in the certificates I mentioned are minute-inclusive. To my knowledge this is how all time is calculated in BRs - the extra 1-second certificate lifespan incidents are an illustrative example.

I will also note that all discussion for whether revocation would occur was specifically around the list of certificates that Entrust provided and stated as the impacted certificate list for months. Why did a different list get used for revocation purposes and not get noted prior to questions? Not only that all lists to date include expired certificates which is not considered standard practice - but please advise me otherwise on that front.

(In reply to amir from comment #33)

At this point I will be asking for the root programs to chime in here and set their expectation for the following questions:

Hi Amir, I don't know that the below information will provide much additional value at this point, but I wanted to provide a response for reference at least.

  1. Can a CA change their opinion on that an incident that had a final report written up, to say that the incident was in fact not an incident weeks after?

Yes, I believe there are scenarios where this could be done (and has been done). However, when facing a potential incident, CAs should almost always be assuming that there is an incident and taking relevant/appropriate steps on expected timelines while simultaneously raising any areas of confusion with the CA/BF (and other relevant SDOs), Web PKI Communities (such as CCADB Public or MDSP), and Root Programs as necessary. Where a potential incident is believed to have supporting evidence which would lead to a conclusion that it is not an incident, it's similarly expected that the burden of proof and explanation lies upon the CA (i.e. it should not be the responsibility of a reporter acting in good faith to articulate exactly how an incident is a non-compliance to a CA).
With all that in mind, I also wouldn't expect that it should typically take multiple weeks to determine that something is or isn't an incident.. not impossible, just not likely.

  1. Do the action items that are required to happen for some incidents (such as revocation) still need to happen if a CA changes their mind on an incident like this?

Typically, yes.
I haven't identified a hypothetical scenario similar to this incident where the action items wouldn't be required, but if (for example) an initial incident report was created with sufficient detail that it was identified very quickly (e.g. in under ~20 hours) that the incident was INVALID, then I would expect some significant portion of certificate revocations would be avoided. At the same time, replacing a subscriber certificate should -- in an ideal world -- be a complete non-event, so a CA that pursues revocation even when an incident turns out not to be would be acting appropriately in my opinion.

This also highlights why it's so important and valuable for CAs to report potential incidents quickly as the later disclosure occurs, the less likely an incident could be identified as invalid in time for expected actions to have not already been required.

  1. Is the language used in this CPS, like entrust has, acceptable for determining if the CP/S is valid or not?

From my perspective, this language primarily describes how the CA's practices are expected to map to and inherit requirements (e.g. constraints regarding what practices are allowed) from the Certificate Policy(ies) in CA/BF Guidelines. It does not obviate the need for the CA to accurately describe those practices in its CPS and it certainly doesn't mean that 4.9.1.1, #12 is irrelevant in a scenario like this incident. On that last point, 4.9.1.1, #12 clearly (in my opinion) indicates that a certificate may be issued that does not comply with only one of:

  • the TBRs;
  • the CA's CP; or
  • the CA's CPS,

and non-compliance with any one of these is sufficient to require revocation, regardless of whether the CA is simultaneously compliant with either or both of the other documents.

Flags: needinfo?(clintw)

That last part makes sense and goes back to default deny on any potential misinterpretation as articulated over the years, thanks for the response.

Thanks for the clarifications from all three root programs.

(In reply to Clint Wilson from comment #73)

(In reply to amir from comment #33)

At this point I will be asking for the root programs to chime in here and set their expectation for the following questions:

Hi Amir, I don't know that the below information will provide much additional value at this point, but I wanted to provide a response for reference at least.

  1. Can a CA change their opinion on that an incident that had a final report written up, to say that the incident was in fact not an incident weeks after?

Yes, I believe there are scenarios where this could be done (and has been done). However, when facing a potential incident, CAs should almost always be assuming that there is an incident and taking relevant/appropriate steps on expected timelines while simultaneously raising any areas of confusion with the CA/BF (and other relevant SDOs), Web PKI Communities (such as CCADB Public or MDSP), and Root Programs as necessary. Where a potential incident is believed to have supporting evidence which would lead to a conclusion that it is not an incident, it's similarly expected that the burden of proof and explanation lies upon the CA (i.e. it should not be the responsibility of a reporter acting in good faith to articulate exactly how an incident is a non-compliance to a CA).
With all that in mind, I also wouldn't expect that it should typically take multiple weeks to determine that something is or isn't an incident.. not impossible, just not likely.

  1. Do the action items that are required to happen for some incidents (such as revocation) still need to happen if a CA changes their mind on an incident like this?

Typically, yes.
I haven't identified a hypothetical scenario similar to this incident where the action items wouldn't be required, but if (for example) an initial incident report was created with sufficient detail that it was identified very quickly (e.g. in under ~20 hours) that the incident was INVALID, then I would expect some significant portion of certificate revocations would be avoided. At the same time, replacing a subscriber certificate should -- in an ideal world -- be a complete non-event, so a CA that pursues revocation even when an incident turns out not to be would be acting appropriately in my opinion.

This also highlights why it's so important and valuable for CAs to report potential incidents quickly as the later disclosure occurs, the less likely an incident could be identified as invalid in time for expected actions to have not already been required.

  1. Is the language used in this CPS, like entrust has, acceptable for determining if the CP/S is valid or not?

From my perspective, this language primarily describes how the CA's practices are expected to map to and inherit requirements (e.g. constraints regarding what practices are allowed) from the Certificate Policy(ies) in CA/BF Guidelines. It does not obviate the need for the CA to accurately describe those practices in its CPS and it certainly doesn't mean that 4.9.1.1, #12 is irrelevant in a scenario like this incident. On that last point, 4.9.1.1, #12 clearly (in my opinion) indicates that a certificate may be issued that does not comply with only one of:

  • the TBRs;
  • the CA's CP; or
  • the CA's CPS,

and non-compliance with any one of these is sufficient to require revocation, regardless of whether the CA is simultaneously compliant with either or both of the other documents.

Thanks Clint. The guidance from the root programs is clear. All certificates related to bug 1890898 were revoked on June 22. This guidance will be incorporated into Entrust standard operating practice.

From your analysis, although this is posted in the incident for bug 1890898, we believe you would consider that it applies to bug 1890685 (Late CPS Update to EV Certificates).

We will plan to start revocation for bug 1890685 and intend to complete revocation by end of day Sunday, June 30, and will report certificate status in the incident update.

Flags: needinfo?(bruce.morton)

(In reply to Bruce Morton from comment #76)

Thanks Clint. The guidance from the root programs is clear. All certificates related to bug 1890898 were revoked on June 22. This guidance will be incorporated into Entrust standard operating practice.

For what it's worth, it was not my intent to discourage or shut down a discussion related to this topic. Especially if this is an area of confusion or disagreement by other CAs, I hope that, as an industry, we can maintain an open dialogue around how and why we've all arrived at the differing conclusions we have. A major part of what these incidents hope to accomplish is enabling better decisions, more consistently across the entire Web PKI.

Evidence of a CA consistently arriving at the same conclusions as their Relying Parties -- including, but not limited to, Root Programs -- with regards to what's expected of them in their CA operations, is a critical component of the "public" in "public trust". However, it's at least equally vital to ensure any interpretations and conclusions that don't ~perfectly align with community and Root Program expectations are well understood so that we can collectively question our assumptions and re-validate our conclusions -- there are numerous (or perhaps closer to innumerable) examples where some interpretation of or proposal for a requirement by a small percentage of stakeholders has resulted in better outcomes and greater security for the internet (or, for that matter, any number of other groups).

From your analysis, although this is posted in the incident for bug 1890898, we believe you would consider that it applies to bug 1890685 (Late CPS Update to EV Certificates).

That's correct. I hope any disagreement (by anyone) with what I shared will be highlighted; I understand that our analyses differed in their conclusions, but I don't yet fully understand why.

We will plan to start revocation for bug 1890685 and intend to complete revocation by end of day Sunday, June 30, and will report certificate status in the incident update.

(In reply to Bruce Morton from comment #76)

All certificates related to bug 1890898 were revoked on June 22.

No, they weren’t: https://bugzilla.mozilla.org/show_bug.cgi?id=1890898#c71

Flags: needinfo?(bruce.morton)
Attachment #9396137 - Attachment is obsolete: true
Flags: needinfo?(bruce.morton)

(In reply to Paul Buonopane from comment #71)

(In reply to Bruce Morton from comment #70)

(In reply to Wayne from comment #63)

(In reply to Bruce Morton from comment #58)

We completed the revocation of all affected OV TLS certificates by 23:17 UTC June 22, 2024.

I would appreciate is Entrust did what they have actually stated. Here is at least two that they did not properly handle as stated

(In reply to Bruce Morton from comment #2)

This incident involves 6,008 OV TLS certificates that were issued between March 22, 2024, 15:02 UTC and March 26, 2024, 17:07 UTC.

https://crt.sh/?id=12688739474

Not Before: Mar 26 17:07:54 2024 GMT
CRL: Revoked at 2024-06-24 04:01:00 UTC

https://crt.sh/?id=12688739250

Not Before: Mar 26 17:07:20 2024 GMT
CRL: Not Revoked
OCSP: Good

These are both in the original list too... is this the new compliance team in action?

Hi Wayne,

The effective period ends March 26, 2024, 17:07 UTC. Your posted certificates are out of this period.

They’re in 20240409-OV-TLS-Certificates-typo-finalcerts.txt, though.

Apologize for any confusion. The original file was obsolete as the search was done using EDT instead time. The certificate list was updated to delayed-revocation-OV-cps-typo.csv.

(In reply to Bruce Morton from comment #79)

(In reply to Paul Buonopane from comment #71)

(In reply to Bruce Morton from comment #70)

(In reply to Wayne from comment #63)

(In reply to Bruce Morton from comment #58)

We completed the revocation of all affected OV TLS certificates by 23:17 UTC June 22, 2024.

I would appreciate is Entrust did what they have actually stated. Here is at least two that they did not properly handle as stated

(In reply to Bruce Morton from comment #2)

This incident involves 6,008 OV TLS certificates that were issued between March 22, 2024, 15:02 UTC and March 26, 2024, 17:07 UTC.

https://crt.sh/?id=12688739474

Not Before: Mar 26 17:07:54 2024 GMT
CRL: Revoked at 2024-06-24 04:01:00 UTC

https://crt.sh/?id=12688739250

Not Before: Mar 26 17:07:20 2024 GMT
CRL: Not Revoked
OCSP: Good

These are both in the original list too... is this the new compliance team in action?

Hi Wayne,

The effective period ends March 26, 2024, 17:07 UTC. Your posted certificates are out of this period.

They’re in 20240409-OV-TLS-Certificates-typo-finalcerts.txt, though.

Apologize for any confusion. The original file was obsolete as the search was done using EDT instead time. The certificate list was updated to delayed-revocation-OV-cps-typo.csv.

Should have said, "The original file was obsolete as the search was done using EDT instead UTC time."

(In reply to Paul Buonopane from comment #78)

(In reply to Bruce Morton from comment #76)

All certificates related to bug 1890898 were revoked on June 22.

No, they weren’t: https://bugzilla.mozilla.org/show_bug.cgi?id=1890898#c71

All we revoked based on the updated list "delayed-revocation-OV-cps-typo.csv."

(In reply to Bruce Morton from comment #80)

(In reply to Bruce Morton from comment #79)

(In reply to Paul Buonopane from comment #71)

(In reply to Bruce Morton from comment #70)

(In reply to Wayne from comment #63)

(In reply to Bruce Morton from comment #58)

We completed the revocation of all affected OV TLS certificates by 23:17 UTC June 22, 2024.

I would appreciate is Entrust did what they have actually stated. Here is at least two that they did not properly handle as stated

(In reply to Bruce Morton from comment #2)

This incident involves 6,008 OV TLS certificates that were issued between March 22, 2024, 15:02 UTC and March 26, 2024, 17:07 UTC.

https://crt.sh/?id=12688739474

Not Before: Mar 26 17:07:54 2024 GMT
CRL: Revoked at 2024-06-24 04:01:00 UTC

https://crt.sh/?id=12688739250

Not Before: Mar 26 17:07:20 2024 GMT
CRL: Not Revoked
OCSP: Good

These are both in the original list too... is this the new compliance team in action?

Hi Wayne,

The effective period ends March 26, 2024, 17:07 UTC. Your posted certificates are out of this period.

They’re in 20240409-OV-TLS-Certificates-typo-finalcerts.txt, though.

Apologize for any confusion. The original file was obsolete as the search was done using EDT instead time. The certificate list was updated to delayed-revocation-OV-cps-typo.csv.

Should have said, "The original file was obsolete as the search was done using EDT instead UTC time."

That doesn’t align with your prior comments stating UTC, the dates of the two certificates in question, or the fact that they have both since been revoked, albeit after June 22.

Flags: needinfo?(bruce.morton)

(In reply to Wayne from comment #57)

(In reply to Bruce Morton from comment #55)
Then would it be accurate to say that as of "today" there are 3329 certificates to be resolved? I only state this to get a proper scale of actions taken over the next week, and to avoid conflation with actions taken months ago.

All certificates have been revoked.

Flags: needinfo?(ngook.kong)

Action Items

Action Item Kind Due Date
Update archive CPS with errata to inform subscribers going forward Transparency Done
Inform subscribers of changes to the CPS Transparency Done
Update archive CPS with errata to inform subscribers going forward Transparency Done
Inform subscribers of changes to the CPS Transparency Done
Review applicable policy and procedures Prevent Done
Improve incident management procedures Prevent 2024-06-30
Reorganize product compliance and verification teams to provide additional organizational resources and oversight Prevent Done
Flags: needinfo?(bruce.morton)

(In reply to Tim Callan from comment #59)

(In reply to Bruce Morton from comment #49)

our position is that there was no mis-issuance—not that there was a mis-issuance and we decided not to revoke…

You just stated in the case of this bug that Entrust did not make the decision not to revoke certificates that it believed at the time to be misissued.

Question 1: How do you reconcile that with this statement from bug 1890896 comment 1

These certificates are considered mis-issued in accordance with the CCADB policy

Our views and position have changed over the time since Comment 1 was posted, so if you are trying to reconcile different statements, you will absolutely find an inconsistency.

At the beginning of this incident, we sincerely believed that the community would agree that the circumstances were extremely unusual and revocation wouldn’t be required, since the problematic text in our CPS was simply accidently pasted into the wrong section for a few days and never reflected an actual practice, there was no likelihood that any relying party would have had any confusion or misunderstanding about our practices. Yes - we decided not to revoke, despite also believing the certificates to be mis-issued:

  • It was not a security matter nor did it impact public trust.
  • There was a lack of impact on relying parties;
  • Revocation on a mass scale is known to be challenging and disruptive to subscribers, and doing so carries the risk of disruption to the very relying parties that the PKIweb ecosystem wishes to protect.

We understand now that no-one else in the community believed those were valid reasons not to revoke, and we agree that none of these reasons were based on a strict, reading of the written rules or requirements of the CA/B rules or root program policies.

We now fully understand and appreciate that the community places priority on strict adherence to the rules, expects a presumption of mis-issuance in the face of any ambiguity (although simultaneous investigation/questioning is ok, as long as it doesn’t delay acting as if a mis-issuance had actually happened).

In keeping with this, going forward we have agreed to apply the rules as stated.

and this statement from comment 2

  1. Why did we decide to not revoke these certificates?
    We do not believe that revoking certificates already issued as part of our response would benefit the Web PKI. The correction was to the CPS rather than the certificate, so that if the certificates were re-issued, they would be re-issued with the identical certificate profile. We believe this to be an exceptional condition.

Same answer as above.

and with many other statements on this very thread from April 10 up through at least June 17?

Same answer as above

Question 2: Do you agree that between roughly April 10, 2024 and roughly June 5, 2024 Entrust willingly chose not to revoke certificates that it believed at the time to be misissued?

This is accurate for the time period between April 10, 2024 and the date on which we issued our updated incident report; however, we also solicited and were awaiting browser feedback during this same time period.

(In reply to Tim Callan from comment #60)

(In reply to Bruce Morton from comment #51)

(In reply to Tim Callan from comment #36)

Question 3: Proper follow-through on this commitment will involve reporting of accurate, current numbers of certificates revoked and expired naturally on at least a weekly cadence for as long as any of these certificates remains active. Can the community expect Entrust to honor its commitment for this incident?

Yes.

Um, sure. Now that in response to pressure from Chrome you have decided you have no choice but to revoke the certificates, providing an ongoing weekly report of unrevoked certificates is a pretty easy task. To have any significance, this commitment should have come prior to comment 40.

In the meantime, I had two other questions in comment 36 that you skipped.

Question 1: How many of the certificates affected by this incident have been revoked and how many expired naturally?

As of 22 June 2024, there were 4809 revoked and 1098 expired.

Question 2: Why is it that Entrust failed to keep its commitment to report these numbers? Please be detailed and specific.

Unfortunately, this was an oversight, and it has been incorporated into our updated process.

Entrust proclaimed on June 17 in comment 42 that it would revoke these certificates. I believe full visibility on this incident and the state of these certificates requires a snapshot of the revocation and expiration status of these certificates from prior to that commitment, let’s say on June 14 when Chrome posted the comment that changed Entrust’s position.

Question 1.1: On June 14 at the time of posting for comment 40, how many of the certificates affected by this incident had been revoked and how many had expired naturally?

On 14 June 2024 there were 942 revoked and 1130 expired.

I believe that question 2 is still an important question and would like it answered as well.

(In reply to Wayne from comment #68)

(In reply to Bruce Morton from comment #65)

(In reply to Wayne from comment #52)

(In reply to Bruce Morton from comment #49)

On the first two questions, Mozilla’s expectation is that the delayed revocation be raised as a finding on the next audit report and discussed with the auditor at that time. We will absolutely do that. On the last question, our position is that there was no mis-issuance—not that there was a mis-issuance and we decided not to revoke which is the situation that recommends discussion with affected root stores.

Is... is that still Entrust's understanding of the situation? Currently I have doubts as to the time these responses were written and that if they truly reflect Entrust's state of mind at the time they were posted.

Yes, we did our analysis, but acknowledge the community and browser’s clearly expressed disagreement and accepted the direction to revoke for this case. If this bug were to happen again tomorrow, we would report it as a mis-issuance and revoke.

So it is still Entrust's analysis and position that no mis-issuance occurred, but you are taking the community and browser's positions solely with regard to this precise interpretation? Could you share Entrust's analysis at some point so it can be reviewed?

Although we still believe the analysis was appropriate, the rationale and guidance from the browsers has clarified our understanding of why it is a mis-issuance and if the same issue was to happen tomorrow, we will declare mis issuance and revoke per rules. The. analysis is reflected in the updated incident report.

The lack of reflection on Entrust having a different perspective than any entity commentating is apparent. Going forward there needs to be trust that Entrust are capable of generating and enacting viewpoints in alignment with everyone's expectations outside of this specific case alone.

(In reply to Bruce Morton from comment #66)

Seeing as you have, following clarification from Ryan, stated that you will treat the certificates covered by this incident as mississued again; can we expect a revised revised final incident report from Entrust?

Yes, we will amend the final incident report to reflect that we did our analysis, but that the community and browsers disagreed, and we accepted this and revoked the certificates. If this bug were to happen again tomorrow, we would report it as a mis-issuance and revoke.

I sincerely hope Entrust are not being serious about attempting a second amended incident report to cover more of their shortcomings. This incident does need a port-mortem analysis, but I'm unsure if we can trust Entrust to competently assess it as this rate. The most cursory glance found a cert revoked days past when Entrust claimed it was finished, and another certificate that hasn't even been revoked. I'm dreading even looking deeper into this mess.

(In reply to Bruce Morton from comment #67)

Question: How is this revocation event being characterized to subscribers?

This event is being explained to subscribers in terms that we believe are appropriate to that audience and designed to impress upon them the urgency of the need to revoke, which we believe is what this community expects, based on the feedback we’ve been receiving. We have used several different methods of communication (e.g. email, phone) and the drafting has been slightly different. If what you want to know is whether we have told the subscribers that the certificates mis-issued, the answer is yes, we have told them that the certificates were mis-issued. We have also transparently disclosed to them, in at least some more detailed communications, the history of this bug, including the fact that it is Chrome’s position that the certificates are mis-issued, and that Entrust’s own analysis came to a different conclusion.

Can you share such wording? I haven't seen a statement of Entrust sent "privately" that reflects what you are claiming, or is this conflating phone conversations to specific parties with the upcoming revocation notice? I do know Entrust were grumbling about there being no 'provisions' for delays - so you can feel free to share that email. I hope there are no redactions this time around, we have learned this time?

Attachment 9410365 [details] Initial notification to customers

(In reply to Zacharias from comment #69)

(In reply to Bruce Morton from comment #67)

(In reply to walter.j.marks from comment #54)

(In reply to Bruce Morton from comment #49)

(In reply to Mike Shaver (:shaver emeritus) from comment #32)

Question: did Entrust consult with their auditor with regard to the analysis of this case as a misissuance, as the Mozilla policy requires?

Question: if so, how does the auditor's analysis differ from the later one that has been referenced here, which holds that this is not a case of misissuance? Will this difference in analysis be represented in the next BR audit?

Question: did Entrust consult with the Root Stores that it participates in with regard to the analysis of this case?

On the first two questions, Mozilla’s expectation is that the delayed revocation be raised as a finding on the next audit report and discussed with the auditor at that time. We will absolutely do that. On the last question, our position is that there was no mis-issuance—not that there was a mis-issuance and we decided not to revoke which is the situation that recommends discussion with affected root stores.

Question: If Entrust does not consider these certificates mis-issued, under what section of the Terms of Use are they being revoked under?

This question is moot since we are now treating these as mis-issued per the community and browser’s clearly expressed direction.

I’m sorry that this is pedantic but I feel that is still necessary because I’m still not seeing alignment of Entrust and the rest of the community. You did not answer the question, which was “If Entrust considers these certificates mis-issued“ (emphasis mine). You answered that Enrust is treating them as mississued.

Since this was first posted, we have aligned our policy and practice with the community, and we will do so going forward. We have declared these certificates mis-issued and revoked them as well as the certificates in bug 1890685.

In the future, in a similar situation and under today’s requirements, we also would declare them mis-issued and revoke them in alignment with the requirements of the CA/Browser Forum and root programs.

Can you refer to any other member of the community that share your analysis? Did you at anytime consult with a third party expert? If so, what was their analysis?

Without calling out specific names, we solicited feedback from within the web community and received feedback that this was an interesting rationale and had a sound basis, but that the community might reject it nonetheless—in the end we decided to present our conclusion and did so sincerely.

There has been no lack of reflection within our organization on these differing viewpoints, and moving forward we are aligned as an organization around the requirements of the CA/Browser Forum and root programs.

In addition, we do not believe that a discussion about interpretation of contract terms or enforcement is appropriate as communications between corporations and their legal counsel are privileged and confidential, and corporations are not generally free to disclose legal strategy in a public forum.

This is not a court of law, but a community about trust. A community which has been disappoints time and time again about your lack of transparency. And I also believe you are wrong about your point, because you were asked about under which Terms of Use the certificates were being revoked under. Are you not sharing that reason with your subscribers? I imagine any lawsuit requiring legal strategy would center around if the stated reason for revocation is valid or not.

We told our subscribers that the certificates are mis-issued and must be revoked as required by the CA/Browser Forum’s TLS Baseline Requirements. We did not review our Terms of Use with subscribers in this message.

Question: How is this revocation event being characterized to subscribers?

This event is being explained to subscribers in terms that we believe are appropriate to that audience and designed to impress upon them the urgency of the need to revoke, which we believe is what this community expects, based on the feedback we’ve been receiving. We have used several different methods of communication (e.g. email, phone) and the drafting has been slightly different. If what you want to know is whether we have told the subscribers that the certificates mis-issued, the answer is yes, we have told them that the certificates were mis-issued. We have also transparently disclosed to them, in at least some more detailed communications, the history of this bug, including the fact that it is Chrome’s position that the certificates are mis-issued, and that Entrust’s own analysis came to a different conclusion.

(In reply to ngook.kong from comment #86)

As of 22 June 2024, there were 4809 revoked and 1098 expired.

On 14 June 2024 there were 942 revoked and 1130 expired.

So the number of expired certs went down between June 14 and June 22? Are you saying that at least 32 certificates unexpired?

Flags: needinfo?(ngook.kong)

(In reply to Wayne from comment #72)

(In reply to Paul Buonopane from comment #71)

Hi Wayne,

The effective period ends March 26, 2024, 17:07 UTC. Your posted certificates are out of this period.

They’re in 20240409-OV-TLS-Certificates-typo-finalcerts.txt, though.

Yup, which is the canonical 6,008 impacted cert figure as this incident report outlines in Comment 2 (04-11), Comment 28 (06-06), June 7th Report, and Updated June 21st Report.

Is the June 21st comment in Comment 55 is now considered authoritative? What went wrong for months to produce an incorrect authoritative list of certificates and certify them as such for multiple audited reports? Notably the times in the certificates I mentioned are minute-inclusive. To my knowledge this is how all time is calculated in BRs - the extra 1-second certificate lifespan incidents are an illustrative example.

I will also note that all discussion for whether revocation would occur was specifically around the list of certificates that Entrust provided and stated as the impacted certificate list for months. Why did a different list get used for revocation purposes and not get noted prior to questions? Not only that all lists to date include expired certificates which is not considered standard practice - but please advise me otherwise on that front.

We acknowledge and will provide responses back to the questions.

Flags: needinfo?(ngook.kong)

(In reply to Clint Wilson from comment #77)

(In reply to Bruce Morton from comment #76)

Thanks Clint. The guidance from the root programs is clear. All certificates related to bug 1890898 were revoked on June 22. This guidance will be incorporated into Entrust standard operating practice.

For what it's worth, it was not my intent to discourage or shut down a discussion related to this topic. Especially if this is an area of confusion or disagreement by other CAs, I hope that, as an industry, we can maintain an open dialogue around how and why we've all arrived at the differing conclusions we have. A major part of what these incidents hope to accomplish is enabling better decisions, more consistently across the entire Web PKI.

Evidence of a CA consistently arriving at the same conclusions as their Relying Parties -- including, but not limited to, Root Programs -- with regards to what's expected of them in their CA operations, is a critical component of the "public" in "public trust". However, it's at least equally vital to ensure any interpretations and conclusions that don't ~perfectly align with community and Root Program expectations are well understood so that we can collectively question our assumptions and re-validate our conclusions -- there are numerous (or perhaps closer to innumerable) examples where some interpretation of or proposal for a requirement by a small percentage of stakeholders has resulted in better outcomes and greater security for the internet (or, for that matter, any number of other groups).

From your analysis, although this is posted in the incident for bug 1890898, we believe you would consider that it applies to bug 1890685 (Late CPS Update to EV Certificates).

That's correct. I hope any disagreement (by anyone) with what I shared will be highlighted; I understand that our analyses differed in their conclusions, but I don't yet fully understand why.

We will plan to start revocation for bug 1890685 and intend to complete revocation by end of day Sunday, June 30, and will report certificate status in the incident update.

Clint, we appreciate your openness to discuss our thinking in this case, and we are happy to continue discussions in the appropriate forums on where and how CPS errors should be interpreted as mis-issuances. That said, it is clear to us that the root programs and community today see the kinds of CPS errors in this case and in bug 1890685 as mis-issuances, and we will treat them as such going forward. Any discussion around changing or clarifying this interpretation will be outside the context of how we address these or similar issues in the future.

(In reply to Wayne from comment #72)

(In reply to Paul Buonopane from comment #71)

Hi Wayne,

The effective period ends March 26, 2024, 17:07 UTC. Your posted certificates are out of this period.

They’re in 20240409-OV-TLS-Certificates-typo-finalcerts.txt, though.

Yup, which is the canonical 6,008 impacted cert figure as this incident report outlines in Comment 2 (04-11), Comment 28 (06-06), June 7th Report, and Updated June 21st Report.

Is the June 21st comment in Comment 55 is now considered authoritative? What went wrong for months to produce an incorrect authoritative list of certificates and certify them as such for multiple audited reports? Notably the times in the certificates I mentioned are minute-inclusive. To my knowledge this is how all time is calculated in BRs - the extra 1-second certificate lifespan incidents are an illustrative example.

I will also note that all discussion for whether revocation would occur was specifically around the list of certificates that Entrust provided and stated as the impacted certificate list for months. Why did a different list get used for revocation purposes and not get noted prior to questions? Not only that all lists to date include expired certificates which is not considered standard practice - but please advise me otherwise on that front.

The original list was incorrect as the certificate search was done in EDT time instead of UTC time. We noticed the problem with the time zone when we began creating reports with certificates that our subscribers had to revoke, these reports showed different outcomes than the original list provided.

(In reply to Tim Callan from comment #90)

(In reply to ngook.kong from comment #86)

As of 22 June 2024, there were 4809 revoked and 1098 expired.

On 14 June 2024 there were 942 revoked and 1130 expired.

So the number of expired certs went down between June 14 and June 22? Are you saying that at least 32 certificates unexpired?

The reason that the June 22 report had fewer expired certs was that on June 18th, the query used to generate the report was changed to use UTC times to select impacted certificates. The updated certificate list was reported in Comment 55 and attached in Comment 56.

You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: