1667690 - Entrust: Failure to provide a preliminary report within 24 hours.

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[George [:fozzie]]

I sent a problem report to [e-mail] and have yet to receive a preliminary report:

Saturday 26 September 10:04 UTC - I sent a report concerning an invalid stateOrProvinceName field of "North Sydney" in 1C89E9AADAC03098DB9EB6AA26A374A6E99E66F0ACE20A6E2D9C4349E0349D54
Saturday 26 September 11:35 UTC - I received a response asking for "The common/domain name or other details on the certificate"
Saturday 26 September 11:37 UTC - I responded and provided the censys link for this certificate (https://censys.io/certificates/1C89E9AADAC03098DB9EB6AA26A374A6E99E66F0ACE20A6E2D9C4349E0349D54)

As of Sunday 27 September 20:27 UTC (32 hours and 50 minutes after my response) I have yet to receive a preliminary response after providing additional information for this certificate.

Comment 1

[Dathan Demone]

George - this is an unfortunate miss on our end, as we did receive, review, and confirm action for this incident yesterday within a couple of hours receiving your first email. Our customer support team did respond to you, but they did not provide the preliminary report details to you directly. BR 4.9.5 clearly states that we should have notified the original reporter (you) with a preliminary report within 24 hours. We will investigate this failure to provide you with a preliminary report and post a summary in this thread.

Comment 2

[Dathan Demone]

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

On September 27th at approximately 21:27 UCT, an incident was assigned to Entrust stating that we had failed to provide a preliminary report to the original reporter within 24 hours.

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

26 September 2020, 10:06 UTC: The reporter sends us an alert via our incident report email address to inform us of a certificate was issued with an invalid ST value.

26 September 2020, 11:37 UTC: Our support team receives the initial message as part of our weekend on-call checks. They respond to the reporter asking for more information.

26 September 2020, 11:39 UTC: Reporter responds with a link to the certificate

26 September 2020, 12:11 UTC: The certificate problem report is escalated internally at Entrust for further review. Customer Support and Verification Management are alerted and a review process is started.

26 September 2020, 14:58 UTC: A Verification on call agent is paged to review the incident and quickly confirms the issue. To avoid any further mis-issuance based on the approved certificate information, the invalid state data is fixed in the verification system.

26 September 2020,15:26 UTC: A Verification secondary approver reviews and approves the new state/locality values for the subscriber so that a new certificate can be issued to replace the invalid one.

26 September 2020, 15:30 UTC: A preliminary report is sent to the certificate subscriber explaining that the certificate included an invalid ST value and told that the certificate must be replaced and that it will be revoked within the 5 day revocation period

27 September 2020, 14:30 UTC: Entrust posts a report on the certificate that has been mis-issued in bug 1658792 to disclose the issue, explain what happened, and commit to revoking the certificate on time.

27 September 2020 21:27 UTC: This Bugzilla incident is assigned to Entrust to inform us that we did not properly alert the original reporter with a preliminary report within 24 hours.

27 September 2020, 19:53 UTC: A preliminary report is sent to the original reporter confirming that the issue was confirmed on our end and that the certificate would be revoked.

28 September 2020, 16:30 UTC: After numerous email discussions, an incident review meeting is held to walk through the timeline and discuss next steps.

29 September 2020, 15:00 UTC: Compliance team meets to discuss the incident and potential changes to the certificate problem report process and specifically our process around providing preliminary reports.

30 September 2020, 13:00 UTC: An incident review meeting is held to confirm our plan of action to address our failure to provide a preliminary report within 24 hours.

3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

We are making changes to our incident response process and systems to make sure we provide the original reporter with a preliminary report within 24 hours.

4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

This incident is related to our failure to send a preliminary report to the original reporter within 24 hours.

5. The complete certificate data for the problematic certificates.

https://crt.sh/?id=2663086686

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

In this case, we were able to follow all of the steps in our incident response procedure, including receiving the request using our on-call process on the weekend, internal escalation and review process, and notification to the subscriber of mandatory revocation. Unfortunately, we missed the step to provide a preliminary report to the original reporter within 24 hours.
After further analysis, we did not provide a preliminary report due to a missed step by our on-call agent when responding and escalating the incident. Our on-call agent did respond to the reporter with an acknowledgment and question but did not respond with proper preliminary report information.

There are a number of steps that must be completed when a certificate problem report is received based on section 4.9.3 of our CPS.

7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

There are 3 things that we will implement to improve our incident response process and our requirement to provide a preliminary report:

(1) Create a workflow in our CRM tool to enforce the incident response process steps and send out periodic email alerts to the compliance team if the ticket does not proceed to the next step within a certain amount of time. The workflow would allow the agent to transition the issue throughout the various milestones:
1. Certificate problem report received and tagged in our system as a compliance issue (this step already exists)
2. Certificate problem confirmed internally as mis-issuance or key compromise
3. Provide preliminary report to subscriber
4. Provide a preliminary report to the reporter
5. Revoke certificates within the deadline
6. Case closed

(2) As part of this workflow, implement a preliminary report template to make sure that the agent always sends out the right information.

(3) Training on the CRM updates and template for the agents who will be handling intake of certificate problem reports

Comment 3

[Dathan Demone]

We held meetings this week to review the planned CRM workflow changes that I described in section 7 of our last comment. No definative date has been set yet for implementation of the new workflow but I wanted to provide an update that we are continuing to work through our design process with the key stakeholders.

Comment 4

[Dathan Demone]

We met with our Director of IT today who manages our CRM systems and kicked off our formal requirements review for the workflow that was proposed in our initial report. We received a committment today from IT that they will start the design and implementation process in their next sprint that starts next Thursday, November 5th. We are targeting a release date of January 2021 to complete these CRM updates.

Comment 5

[Dathan Demone]

The CRM changes that are described above are currently being worked on and should be ready for review in the near future. I will provide another update in 2 weeks to re-confirm our planned target release date of January 20201.

Comment 6

[Dathan Demone]

We received confirmation today that the planned changes to our CRM that are described above will go live on 13 January 2021. I will provide a further update once the new feature is in Production next week. We are also going to provide training to our agents who will be managing the intake of our Certificate Problem Reports to make sure they are familiar with the new workflow feature.

Comment 7

[Dathan Demone]

I can confirm that the CRM changes are live and are behaving as expected after running some test cases. The UI clearly shows the agent what the next steps are in the Certificate Reporting Process and will send email escalations to service managers before any deadlines (providing preliminary reports and revocation) are exceeded. Agent training on the new system update was also provided last week prior to launch.

Comment 8

[Ben Wilson]

It looks like the remediation steps set forth under 7. in Comment#2 have been completed. I intend to close this matter on or about 27-Jan-2021 unless there are other items to review or discuss.