1. How your CA first became aware of the problem
During public discussion about bug 1675821, a member of the community identified a delayed publication of updates to our CPS to align it with behavior changes that were deployed to ensure compliance with CABF Ballot SC46.
2. A timeline of the actions your CA took in response.
|2017-03-08 00:00 UTC
||Ballot 187 enters into effect mandating CAA checking with the exception of DNS Operators.
|2021-04-29 17:54 UTC
||Change to remove the DNS Operator Exception for CAA is flagged by the Policy Authority for review by Engineering.
|2021-05-05 12:31 UTC
||Work on a full review and update of our CPS begins, which includes removal of the DNS Operator Exception.
|2021-05-13 20:58 UTC
||Change submitted to remove DNS Operator Exception from our systems.
|2021-06-15 23:59 UTC
||Delivery of automation items related to Bug 1708516
|2021-06-22 10:32 UTC
||New CPS version is sent for review including the DNS Operator Exception removal.
|2021-07-12 00:00 UTC
||Ballot SC46 on subsetting the CAA Exception for DNS Operators enters into effect.
|2021-08-11 00:00 UTC
||New CPS gets final approval and is published.
3. Whether your CA has stopped, or has not yet stopped, certificate issuance or the process giving rise to the problem or incident.
We did not stop issuance as the issuance practices in use were covered by the CPS.
4. In a case involving certificates, a summary of the problematic certificates. For each problem: the number of certificates, and the date the first and last certificates with that problem were issued. In other incidents that do not involve enumerating the affected certificates (e.g. OCSP failures, audit findings, delayed responses, etc.)
We have reviewed all issuance that took place since the code changes supporting SC46 entered into effect (2021-05-13) through the publication of the CPS (2021-08-11) and in 100% of those cases SC46-compliant CAA checks were successfully performed.
5. In a case involving certificates, the complete certificate data for the problematic certificates.
6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
On 2017-03-08, ballot 187 entered into effect making CAA checking mandatory for all CAs. The ballot included an optional allowance for skipping CAA checking if the CA or an affiliate of the CA is the domain's DNS Operator. We historically used this provision to skip CAA checking for a small subset of certificates that we issued where Google was the DNS operator.
A discussion to remove allowance for this method was initiated in the CA/B Forum, and as a result ballot SC46 was proposed. Our change review process flagged this for engineering review on 2021-04-29, and we subsequently removed the option to use the exception. The method was last used 14 days later on 2021-05-13, which was 60 days before the ballot changes went into effect.
On 2021-05-05 we began our annual CPS update, which bundled an update to Section 4.2.4 to remove the reference to the CAA checking exception for the DNS Operator, along with several other updates.
We also had multiple incidents open at that time (Bug 1708516, Bug 1706967, Bug 1709223, Bug 1715421, Bug 1652581) and were making process improvements detailed in Bug 1708516 Comment 44. These updates all involved a number of additional partner teams and stakeholders for reviews with the goal of ensuring we covered all CPS updates and improvements in one pass. These factors combined resulted in delays reviewing and publishing an updated version of our CPS.
When SC46 was published on 2021-06-02, it triggered another review, the conclusion of which was that the change had been addressed on 2021-05-13. This determination was made based on the records captured in the bug tracking the associated code changes and deployment which had been marked fixed, and because the CPS was in the process of being approved for release.
The aforementioned manual process to approve updates to our CPS resulted in further delays in its publication, and the impending deadline to publish an update for SC46 was not flagged because we lacked a control to alert us of the CPS change deadline, and insufficient controls in place to coordinate related code changes to documentation changes. This was the root cause of this issue.
On review of past incidents this may appear to be related to Bug 1706967, however that incident was the result of a scheduling issue that prevented a full review of BR changes from being completed. In the case of this issue, the review was completed in a timely manner but the CPS changes had not yet been approved due to factors mentioned earlier.
7. List of steps your CA is taking to resolve the situation and ensure that such situation or incident will not be repeated in the future, accompanied with a binding timeline of when your CA expects to accomplish each of these remediation steps.
To prevent this issue from occurring again in the future, we plan to introduce changes so updates to our CPS are published by the time new requirements become effective.
To that end:
- We will further update our processes to require that the tracking bug for compliance documentation revisions includes a machine parseable publication deadline of the effective date of any requirements changes, and require that this bug be linked to the tracking bug for any software change. This process change will be implemented immediately.
- We will leverage our ticketing system to proactively alert of impending deadlines for documentation updates. This change will be implemented by 2021-09-17
- We will conduct another internal review of all CAB/F ballots since SC3 to double check that all required changes have been made and are accurately reflected in our CPS. This will be completed by 2021-09-24.