Closed Bug 1885754 Opened 7 months ago Closed 18 days ago

Entrust: CPR was not responded to in 24 hours

Categories

(CA Program :: CA Certificate Compliance, task)

Product:
CA Program
Component:
CA Certificate Compliance
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: amir, Assigned: paul.vanbrouwershaven)

Details

(Whiteboard: [ca-compliance] [external] [policy-failure])

Steps to reproduce:

Please note I reached out to their contact on their CPS document on abuse@affirmtrust.com over 24 hours ago with the following example of a mis-issued certificate: https://crt.sh/?q=a6745632c7400c8b9ab84e6b6ec7c560fdec36534ffdc664b311d5f917af72ab

Unfortunately, I also did not get any responses or acknowledgements in the 24 hour period.

Hi Amir. FYI, https://crt.sh/?id=1318444845&opt=problemreporting uses publicly available CCADB reports to show that Entrust is the CA Owner and that CPRs for leaf certs issued under "AffirmTrust Extended Validation CA - EVEC1" can be submitted to any of several different Entrust and Affirmtrust contact points. Since Affirmtrust is part of Entrust, I think the "EV Certificate with cps uri missing" part of this bug report should probably be regarded as a duplicate of bug 1883843.

Thank you! I considered them a different entity since they had a different CPR reporting endpoint, and were a different line item under the root stores.

I agree that part of this bug is a duplicate, however I do think that the lack of response or acknowledgement on the abuse@affirmtrust.com endpoint is probably another bug.

From the CPS:

Certificate Problem Reports, such as Certificate misuse, vulnerability reports or external reports
of key compromise, must be emailed to abuse@affirmtrust.com. Contact details are also
provided and maintained in the CCADB.

And that broke the requirement here: https://github.com/cabforum/servercert/blob/8e7fc7d5cac0cc27c44fe2aa88cf45f5606f4b94/docs/BR.md#495-time-within-which-ca-must-process-the-revocation-request

I agree that part of this bug is a duplicate, however I do think that the lack of response or acknowledgement on the abuse@affirmtrust.com endpoint is probably another bug.

In that case, may I suggest changing this bug's Summary to "Entrust: CPR was not responded to in time" or something along those lines?

Summary: Affirmtrust: EV Certificate with cps uri missing && CPR was not responded to in time → Affirmtrust/Entrust: CPR was not responded to in 24 hours

Sounds good! Thank you for the suggestion!

Note, I just received a response from Entrust to my original CPR.

Time of original request (per their ticket management software): 2024/03/15 12:07:37 PM CDT

Time of their response (per my email client): 2024/03/18 10:59:01 AM CDT

I confirm that we have seen this bug report, we have requested more information and will come back with a response to this bug soon.

Would be possible to change the title so that it starts with "Entrust: "? (this would help our systems to highlight this bug)

Summary: Affirmtrust/Entrust: CPR was not responded to in 24 hours → Entrust: CPR was not responded to in 24 hours

CCADB lists "Entrust" as the CA Owner (rather than "Affirmtrust" or "Entrust/Affirmtrust"), so this Summary change seems correct to me.

Assignee: nobody → paul.vanbrouwershaven
Status: UNCONFIRMED → ASSIGNED
Type: defect → task
Ever confirmed: true
Whiteboard: [ca-compliance] [external] [policy-failure]

We are preparing an incident report for this bug.

Incident Report

Summary

A CPR reporter discovered an AffirmTrust (an Entrust company) EV SSL certificate which did not include the certificatePolicies:policyQualifiers:qualifier:cPSuri. The Technical Support Specialist team did not provide a preliminary report to the CPR reporter within 24 hours as required by the TLS BRs https://cabforum.org/working-groups/server/baseline-requirements/requirements/#495-time-within-which-ca-must-process-the-revocation-request.

Entrust was already aware of the issue reporter per this incident, https://bugzilla.mozilla.org/show_bug.cgi?id=1883843.

Impact

The CPR reporter was not aware that the CA had received the CPR and was actioning the issue within the expected timeline requirements.

Timeline

All times are UTC.
2024-03-15:

  • 15:07:37 CPR reporter sent an email to AffirmTrust opening a case in our case management system advising of potentially mis-issued certificates.

2024-03-16:

  • 14:07:37 CPR reporter sent a second email to check on response.
  • 22:14 CPR reporter opened this Bugzilla case to address the issue that the CPR was not responded to within 24 hours.

2024-03-18:

  • 13:58:55 Entrust provided acknowledgement of the CPR to the reporter.
  • 14:41:55 Entrust provided an update to the CPR reporter that this was a known incident that was already being addressed.

2024-03-20:

  • 08:50 Entrust confirmed acknowledgment of the issue through this Bugzilla incident.

2024-03-22:

  • 09:02 Entrust provided an update to the CPR reporter that the issue has been reported to Bugzilla.

Root Cause Analysis

The Technical Support Specialist did not understand that the email received was a CPR, and so did not immediately activate CPS process as stated in incident https://bugzilla.mozilla.org/show_bug.cgi?id=1667690.

Lessons Learned

We have opportunity to improve our CPR reporting process and provide automation.

What went well

What didn't go well

  • The technical support specialist did not interpret the email as a CPR notification.

Where we got lucky

  • The issue reported by the CPR was already a known issue and is currently being addressed.

Action Items

Action Item Kind Due Date
Re-Training for all Technical Support Specialists and Customer Advocates. Prevent 2024-05-31
Research and implement automation opportunities for flagging cases coming into our case management system whether via the form or blind email . Prevent 2024-08-31

The root cause analysis here seems to be "person made a mistake"?

Historically, human made mistake as a root cause has been frowned upon as its not really a root cause, but usually rather a symptom of something else, or lack of other supporting systems.

This is particularly confusing since the CPR was:

Hello,

I've found that you've misissued certificates such as: https://crt.sh/?q=a6745632c7400c8b9ab84e6b6ec7c560fdec36534ffdc664b311d5f917af72ab

Per the BR rules for EVs. This is similar to the following incident: https://bugzilla.mozilla.org/show_bug.cgi?id=1883843

Since this email included the word misissued, I'd suspect that any training would flag that as a reasonable CPR?

(In reply to amir from comment #10)

The root cause analysis here seems to be "person made a mistake"?

Historically, human made mistake as a root cause has been frowned upon as its not really a root cause, but usually rather a symptom of something else, or lack of other supporting systems.

Hi Amir, we agree that we will need to do some work with the supporting systems. We have assigned an action to flag CPR cases, provide automation and alerts. I am thinking that an immediate response would be worth while, so the CPR reporter knows a case has been open. The automation would also want to start a 24 hour clock with alerts to ensure there is a proper CPR response. Personally, I am thinking the best way to gather CPR data is to provide a "CPR form" which the CPR reporter would complete. The information provided may help for the CPR to be triaged in a much shorter period of time. The URL for the form would be provided in Section 1.5.2 of the CPS.

If there are no other comments, it is requested set the next update to 3 May 2024.

Whiteboard: [ca-compliance] [external] [policy-failure] → [ca-compliance] [external] [policy-failure] Next update 2024-05-03

In regards to Entrust's CCADB Entry the Problem Reporting Mechanism field shows a misalignment with their CPS and outdated information.

Problem Reporting Mechanism
ecs[dot]support[at]entrustdatacard[dot]com
abuse[at]affirmtrust[dot]com
https://www.entrust.net/ev/misuse.cfm
https://www.affirmtrust.com/ssl/

Q1) Entrust's CPS 3.20 notes an email address of the domain entrust.com instead of entrustdatacard.com, so for consistency would it be better to update that record?

Q2) The misuse form was last active in 2019, according to the wayback machine, would this be better off removed?

Q3) The Affirmtrust link points at generic ssl sales page with an abuse email address already noted above, is it useful at all?

I trust that this is not an issue that reflects on Entrust alone, but reflects a need for the CA Issuers to be more thorough in their contact methods. To that end I'm keeping this as a comment in a relevant bug report rather than opening an additional one. As a general question is there a practice in the CA community of reviewing these records annually?

(In reply to Wayne from comment #13)

In regards to Entrust's CCADB Entry the Problem Reporting Mechanism field shows a misalignment with their CPS and outdated information.

Problem Reporting Mechanism
ecs[dot]support[at]entrustdatacard[dot]com
abuse[at]affirmtrust[dot]com
https://www.entrust.net/ev/misuse.cfm
https://www.affirmtrust.com/ssl/

Q1) Entrust's CPS 3.20 notes an email address of the domain entrust.com instead of entrustdatacard.com, so for consistency would it be better to update that record?

Per the Baseline Requirements, CPS section 1.5.2 is where the CA must provide the method to submit a certificate problem report. We also need to provide this information to the CCADB and apologize as this is out of date. We have opened a case in CCADB to update the information.

Please note that we have migrated away from @entrustdatacard,com, so email addresses ending with @entrust.com are correct.

Q2) The misuse form was last active in 2019, according to the wayback machine, would this be better off removed?

We have removed with our CCADB case.

Q3) The Affirmtrust link points at generic ssl sales page with an abuse email address already noted above, is it useful at all?

Agreed, not useful. The address will be removed with the CCADB case.

I trust that this is not an issue that reflects on Entrust alone, but reflects a need for the CA Issuers to be more thorough in their contact methods. To that end I'm keeping this as a comment in a relevant bug report rather than opening an additional one. As a general question is there a practice in the CA community of reviewing these records annually?

I am not aware if this is a practice. We will consider creating an annual review in our process to ensure the CCADB CA OWNER page is correct.

As a general question is there a practice in the CA community of reviewing these records annually?

I am not aware if this is a practice. We will consider creating an annual review in our process to ensure the CCADB CA OWNER page is correct.

Chrome Root Program Policy v1.1 (published 1st June 2022) said (emphasis mine):
"Minimally, CA operators must ensure information stored in the CCADB is reviewed monthly and updated as needed."

That "reviewed monthly and updated" phrase persisted through to Chrome Root Program Policy v1.4, but was removed in v1.5 (the current version) because it had become redundant. The current requirement is even stricter than "monthly"...

The CCADB policy says (emphasis mine):
"Regardless of more specific provisions in these requirements, CA Owners have an overarching responsibility to keep the information in the CCADB about themselves, their operations and their certificates accurate, and to make updates in a timely fashion. Minimally, CA Owners with certificates included in a participating Store must ensure their information stored in the CCADB is kept up to date as changes occur."

Chrome Root Program Policy v1.5 (and v1.4) says (emphasis mine):
"Chrome Root Program Participants must...Follow the requirements defined in the CCADB policy...In instances where the CCADB policy conflicts with this policy, this policy must take precedence...When a timeline is not defined for a requirement specified within the CCADB policy, updates must be submitted to the CCADB within 14 calendar days of being completed."

I don't see any timeline defined in the CCADB policy regarding how quickly a CA must update its Problem Reporting Mechanism information in CCADB when any change occurs, which means that the effective requirement (for any CA that is a Chrome Root Program Participant) is 14 calendar days.

I will note that in addition to the above the Mozilla Root Program Policy 2.9 states:
"For this policy and the CCADB policies, "a timely manner" means within 30 days of when the appropriate data or documentation becomes available to the CA operator, unless a Mozilla policy document specifies a different rule."

The Mozilla Communication and Survey August 2023 notes that you have both read version 2.9 of Mozilla's Rot Store Policy, will comply with it and noted no concerns.

We have read version 2.9 of Mozilla’s Root Store Policy.

Yes

We understand and intend to comply with version 2.9 of Mozilla’s Root Store Policy.

Yes

Additional questions or concerns with MRSP v. 2.9

None

To quote Section 2.4 of Mozilla's Root Store Policy v2.9:
"When a CA operator fails to comply with any requirement of this policy - whether it be a misissuance, a procedural or operational issue, or any other variety of non-compliance - the event is classified as an incident and MUST be reported to Mozilla as soon as the CA operator is made aware. At a minimum, CA operators MUST promptly report all incidents to Mozilla in the form of an Incident Report that follows guidance provided on the CCADB website."

I am very interested in the root cause of over 5 year old information being in the Problem Reporting Mechanism. Every policy update from each Root Program policy, CCADB policy, Baseline Requirements, and your own CPS itself would at a minimum cause a recheck of this vital information, no? Not to mention audits, and followups from prior incidents for checking other compliance faults.

When will there be an incident raised?

(In reply to Rob Stradling from comment #15)

As a general question is there a practice in the CA community of reviewing these records annually?

I am not aware if this is a practice. We will consider creating an annual review in our process to ensure the CCADB CA OWNER page is correct.

Chrome Root Program Policy v1.1 (published 1st June 2022) said (emphasis mine):
"Minimally, CA operators must ensure information stored in the CCADB is reviewed monthly and updated as needed."

That "reviewed monthly and updated" phrase persisted through to Chrome Root Program Policy v1.4, but was removed in v1.5 (the current version) because it had become redundant. The current requirement is even stricter than "monthly"...

The CCADB policy says (emphasis mine):
"Regardless of more specific provisions in these requirements, CA Owners have an overarching responsibility to keep the information in the CCADB about themselves, their operations and their certificates accurate, and to make updates in a timely fashion. Minimally, CA Owners with certificates included in a participating Store must ensure their information stored in the CCADB is kept up to date as changes occur."

Chrome Root Program Policy v1.5 (and v1.4) says (emphasis mine):
"Chrome Root Program Participants must...Follow the requirements defined in the CCADB policy...In instances where the CCADB policy conflicts with this policy, this policy must take precedence...When a timeline is not defined for a requirement specified within the CCADB policy, updates must be submitted to the CCADB within 14 calendar days of being completed."

I don't see any timeline defined in the CCADB policy regarding how quickly a CA must update its Problem Reporting Mechanism information in CCADB when any change occurs, which means that the effective requirement (for any CA that is a Chrome Root Program Participant) is 14 calendar days.

Hi Rob, thanks for the clarification.

(In reply to Wayne from comment #16)

When will there be an incident raised?

Hi Wayne, we have opened a bug #1894111 and will complete an incident report this week.

We have no updates for this week and will continue to monitor the bug.

We have no updates for this week and will continue to monitor the bug.

Whiteboard: [ca-compliance] [external] [policy-failure] Next update 2024-05-03 → [ca-compliance] [external] [policy-failure]
Whiteboard: [ca-compliance] [external] [policy-failure] → [ca-compliance] [external] [policy-failure] 2024-05-31
Whiteboard: [ca-compliance] [external] [policy-failure] 2024-05-31 → [ca-compliance] [external] [policy-failure] Next update 2024-05-31

(In reply to Bruce Morton from comment #9)

Action Item Kind Due Date
Re-Training for all Technical Support Specialists and Customer Advocates. Prevent 2024-05-31

Has this action item been completed by 2024-05-31?

Flags: needinfo?(paul.vanbrouwershaven)

(In reply to Wayne from comment #21)

(In reply to Bruce Morton from comment #9)

Action Item Kind Due Date
Re-Training for all Technical Support Specialists and Customer Advocates. Prevent 2024-05-31

Has this action item been completed by 2024-05-31?

Hi Wayne, thanks for pointing this out. The re-training was actually completed ahead of schedule on 2024-04-02, the day before posting the incident report. We should have updated the report accordingly.

We have no updates for this week and will continue to monitor the bug.

Whiteboard: [ca-compliance] [external] [policy-failure] Next update 2024-05-31 → [ca-compliance] [external] [policy-failure]

We have no updates for this week, but still have one action item remaining due 31 August 2024. Request next update to be 31 July 2024.

Action item refresh.

Action Item Kind Due Date
Re-Training for all Technical Support Specialists and Customer Advocates. Prevent Done
Research and implement automation opportunities for flagging cases coming into our case management system whether via the form or blind email . Prevent 2024-08-31
Whiteboard: [ca-compliance] [external] [policy-failure] → [ca-compliance] [external] [policy-failure] Next update 2024-07-31

We have no updates for this week, but still have one action item remaining due 31 August 2024. Request next update to be 4 September 2024.

We have no updates for this week, but still have one action item remaining due 2024-08-31.

May we please set the next update at 2024-08-31?

Flags: needinfo?(paul.vanbrouwershaven) → needinfo?(bwilson)
Flags: needinfo?(bwilson)
Whiteboard: [ca-compliance] [external] [policy-failure] Next update 2024-07-31 → [ca-compliance] [external] [policy-failure] Next update 2024-08-31

Action item refresh.

Action Item Kind Due Date
Re-Training for all Technical Support Specialists and Customer Advocates Prevent Done
Research and implement automation opportunities for flagging cases coming into our case management system whether via the form or blind email Prevent Done

We have updated our CPR system and have launched our new problem reporting page, https://www.entrust.com/support/certificate-solutions/report-a-problem. The input to the new page will update and alert in our case management system. This page will be added to our CPS and to CCADB.

I'm assuming that Entrust would like us to close this bug, or should we wait until the CPR page is added to your CPS and the CCADB?

Flags: needinfo?(bwilson)
Flags: needinfo?(bruce.morton)
Whiteboard: [ca-compliance] [external] [policy-failure] Next update 2024-08-31 → [ca-compliance] [external] [policy-failure]

(In reply to Ben Wilson from comment #29)

I'm assuming that Entrust would like us to close this bug, or should we wait until the CPR page is added to your CPS and the CCADB?

Let's wait for a week or so. I will post an update next week and may request for the bug to be closed.

Flags: needinfo?(bruce.morton)

(In reply to Ben Wilson from comment #29)

I'm assuming that Entrust would like us to close this bug, or should we wait until the CPR page is added to your CPS and the CCADB?

There are no mor comments, so yes, we request this bug be closed. Thanks.

I will close this on Wed. 11-Sept-2024.

Status: ASSIGNED → RESOLVED
Closed: 18 days ago
Flags: needinfo?(bwilson)
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.