1783272 - Google Trust Services: Failure to send preliminary report to subscriber within 24h

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[kylepomalley]

1. How your CA first became aware of the problem

During a routine review of Certificate Problem Reports (CPR) from the previous week, a Google Trust Services Engineer noticed that one of the required responses was sent to the incorrect recipient. No sensitive information was included in the response.

After having discovered the issue, an Engineer sent the associated response to the correct recipient. This incident introduced a delay which resulted in the response being beyond the 24 hour response window required by 4.9.5 of the BRs:

"Within 24 hours after receiving a Certificate Problem Report, the CA SHALL investigate the facts and circumstances related to a Certificate Problem Report and provide a preliminary report on its findings to both the Subscriber and the entity who filed the Certificate Problem Report."

2. A timeline of the actions your CA took in response.

YYYY-MM-DD (UTC)	Description
2022-07-31 10:23	The certificate in question was issued.
2022-07-31 14:54	A CPR containing an inquiry was received via contact form at https://pki.goog/faq/ requesting we investigate why the certificate was issued.
2022-07-31 21:05	CA Engineer 1 triaged the CPR and made the determination the certificate was issued to a third-party service provider on behalf of the reporter.
2022-07-31 21:09	CA Engineer 1 responds to the reporter and the incorrect subscriber with the outcome of our initial investigation.
2022-08-02 15:10	During a review of CPRs from the previous week, CA Engineer 2 discovered that the correct subscriber was not sent a findings report. CA Engineer 2 sent a response to the correct subscriber.

3. Whether your CA has stopped, or has not yet stopped, certificate issuance or the process giving rise to the problem or incident.

Google Trust Services has not stopped issuing certificates as this incident did not produce misissued certificates.

4. In a case involving certificates, a summary of the problematic certificates. For each problem: the number of certificates, and the date the first and last certificates with that problem were issued. In other incidents that do not involve enumerating the affected certificates (e.g. OCSP failures, audit findings, delayed responses, etc.), please provide other similar statistics, aggregates, and a summary for each type of problem identified. This will help us measure the severity of each problem.)

Google Trust Services did not find any issues with the certificate mentioned in the CPR, nor is it relevant to this particular incident.

5. In a case involving certificates, the complete certificate data for the problematic certificates.

N/A.

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

In recent weeks, Google Trust Services has made our services available to more users and partners, and as a result have received an increasing number of Certificate Problem Reports. These increases have been driven by the delegation of SSL lifecycle management to service providers combined with the increased usage of CT monitoring by domain registrants.

As we described in Bug 1770510 Comment 2, issuance of certificates to third party service providers has surprised some users when they are notified of new certificates provisioned on their behalf.

On 2022-07-31, we received a CPR from a person who was concerned about a multi-SAN certificate containing one of their subdomains that was issued to a service provider on their behalf. The CPR was correctly triaged by our handling pipeline and a CA Engineer responded with preliminary reports sent separately to both the entity who filed the CPR and the incorrect subscriber; this separation is done to protect potentially sensitive information. However, one of the emails was erroneously sent to the incorrect email address, which belongs to another partner service provider.

Responding to correspondence requires some manual work to investigate the report and can be error-prone. Because of this, we have routine reviews of CPRs and their responses. On 2022-08-03, during a review of the CPRs for the previous week it was discovered that the subscriber was never notified and an email was promptly sent to correct this. Investigations into the CPR resulted in no revocation being required.

In Bug 1770510 we described several changes we introduced to reduce the likelihood of mistakes being made. While those changes have been impactful, there is still more work we are doing to improve this process. To that end we are working on a support solution that we believe will both reduce the spam in the associated system and in general improve individual CPR handling, which we mentioned in Bug 1770510.

In this particular case, a mistake was made when manually copying fields into the recipient field, which resulted in this incident. We believe the aforementioned planned tooling improvements will help reduce such mistakes.

7. List of steps your CA is taking to resolve the situation and ensure that such situation or incident will not be repeated in the future, accompanied with a binding timeline of when your CA expects to accomplish each of these remediation steps.

Google Trust Services has been working on the design of a support solution to improve the end-to-end CPR process for both the reporter and those triaging CPRs as part of our continual improvement plan for support incident handling.

Although the full support solution will not be implemented until the end of 2022, we are prioritizing creation of a guided tool to lookup information provided in a CPR and output email contents with the correct information. We plan to implement this tool by 2022-09-02. Until this tool is complete and in use, we will implement two-person review of CPR responses when sent during business hours.

Comment 1

[kylepomalley]

Google Trust Services is actively working on the guided tool described in Comment 0. We will continue to monitor this bug for comments or questions. We will post another update on our progress on or before the promised date of September 2nd, 2022 (which is prior to the “Next update” assigned).

Comment 2

[kylepomalley]

We have completed implementation of the initial version of the guided CPR processing tool described in Comment 0 and it is now in active use. The tool works by looking up information provided in a CPR and then guides the person handling the CPR through a workflow to ensure the correct information is sent to the correct parties.

We plan to continue making improvements to this tool and closely integrate it with the solution mentioned in Bug 1770510 Comment 3 once it is ready later this year.

While this tool was being developed, we implemented a two-person review of CPR responses sent during business hours as a preventative measure, as promised. No further issues have occurred since this incident.

If there are no comments or questions, we request consideration to close this bug.

Comment 3

[Ben Wilson]

I will close this on or about Friday, 9-9-2022, unless there are questions or issues still to be discussed.