1734265 - GoDaddy: Root CRLs exceed maximum validity period by 1 second

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Brittany Randall]

User Agent: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.71 Safari/537.36

Comment 1

[Brittany Randall]

Incident Report

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

Problem Summary:
On 09/01/2021, our root CRLs for GoDaddy and Starfield were issued with a validity period of 365 days and 1 second as follows.

• gdroot-g2.crl
  lastUpdate=Sep 1 17:57:11 2021 GMT
  nextUpdate=Sep 1 17:57:11 2022 GMT
• gdroot.crl
  lastUpdate=Sep 1 17:33:28 2021 GMT
  nextUpdate=Sep 1 17:33:28 2022 GMT
• sfroot-g2.crl
  lastUpdate=Sep 1 18:52:17 2021 GMT
  nextUpdate=Sep 1 18:52:17 2022 GMT
• sfroot.crl
  lastUpdate=Sep 1 18:27:54 2021 GMT
  nextUpdate=Sep 1 18:27:54 2022 GMT
• sfsroot.crl
  lastUpdate=Sep 1 19:08:20 2021 GMT
  nextUpdate=Sep 1 19:08:20 2022 GMT

Within the Baseline Requirements (BRs) for Publicly Trusted SSL certificates, Section 4.9.7 CRL issuance frequency states that “The value of the nextUpdate field MUST NOT be more than twelve months beyond the value of the thisUpdate field.” While not specifically mentioned as inclusive within this section, other definitions and sections within the BRs define validity periods as inclusive.

Discovery Details:
On 09/23/2021 at 09:39 MST, PKI Compliance team reviewed Bug 1731164 and noted information about CRLs validity period that could be applicable to Starfield PKI. PKI Compliance team reached out to PKI Development to review and determine if root and/or issuing CRLs are impacted.

On 09/23/2021 at 10:41 MST, PKI Development confirms that all (five) root CRLs are impacted.

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

DD/MM/YYYY HH:MM (Times are all MST)

• 09/16/2021 16:57 Google Trust Services files Bug 1731164 noting that “CRL validity period [is] set to expected value plus one second”.
• 09/23/2021 09:39 PKI Compliance reviews Bug 1731164 and reaches out to PKI Development to determine if root and/or issuing CRLs are impacted.
• 09/23/2021 10:30 PKI Engineering and PKI Compliance meet to discuss possibility of needing a ceremony to address potential root CRL validity period issue.
• 09/23/2021 10:41 PKI Development confirms that the validity period for root CRLs are impacted and issuing CRLs are not impacted.
• 09/24/2021 11:29 PKI Compliance team meets to discuss findings of the root CRL which included confirming preliminary understanding of the issue, assessing impact against requirements, collecting timeline details, and next steps.
• 09/24/2021 12:06 PKI Engineering and PKI Compliance meet and confirm that only root CRLs are impacted.
• 09/27/2021 09:30 Certainly files Bug 1732745 noting that “Root CRL validity period exceeds maximum by one second”
• 09/27/2021 14:00 A stakeholder meeting was held which included review of the previous week’s findings, collection of additional timeline details, decision to move forward with ceremony to address validity period for root CRLs, decision to issue root CRLs with 48-hour buffer from 1 year policy maximum, and determination to target ceremony completion by the end of the week (by 10/1/2021).
• 09/27/2021 14:36 PKI Engineering creates CRL ceremony slack channel to engage participants and prepare for ceremony.
• 09/27/2021 14:40 PKI Engineering team creates internal Jira ticket to track ceremony
• 09/28/2021 PKI Engineering continues drafting ceremony script, finalizing role participation, and submitting documentation for review and approval.
• 09/28/2021 14:02 Ceremony is approved by a member of the Governance and Policy Committee for execution.
• 09/29/2021 PKI Engineering finalizes timing for ceremony execution.
• 09/30/2021 9:00 - 17:15 PKI engineering generates new root CRLs during a ceremony. Note: CRLs are issued between 13:48 to 14:20.
• 10/01/2021 12:45 PKI Engineering deploys updated root CRLs to production.
• 10/01/2021 13:10 PKI Engineering verifies that the root CRLs are updated on repository and synced across CRL Responders as expected.
• 10/04/2021 11:03 Internal Stakeholder meeting held to review issue, share additional details, and begin finalizing this incident report.
• 10/05/2021 10:30 Bi-weekly compliance touch point meeting held to review this incident report. Also reviewed open bugs, and their updates including full incident report for Bug 1732745.

3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

As of 10/1/2021, the root CRL validity period has been addressed. The validity period issue was limited to the Root CRLs which only contain data for revoked intermediate and root certificates, of which we have none. Subscriber certificates were not impacted.

4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

Impact was limited to five (5) root CRLs. Subscriber certificates were not impacted.

5. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

Impact was limited to five (5) root CRLs. Subscriber certificates were not impacted.

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now. See Google's guidance on root cause analysis for ideas of what to include.

Root CRL generation is an annual event completed under PKI ceremony. The 2021 annual ceremony occurred on 9/1/2021.

Prior to July 2021, the change monitoring process (ballot monitoring, industry trends, and Bugzilla reviews), included a process which sent changes to specific stakeholders who had oversight and accountability into PKI functional areas. Ballot SC31 was communicated to a stakeholder, who at the time, had oversight over both PKI Development and Engineering. In response to the internal ballot change task, the implementation plan that was developed was limited to development efforts specific to subscriber certificates and did not consider potential updates to the PKI Engineering ceremony scripts (those which would affect the root CRL generation).

As of July 2021, updates to the change monitoring process have been made which direct change reviews to each functional/tactical area where an individual stakeholder may receive more than one ticket for sign-off in the event that they have oversight and accountability over multiple functional areas.

7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

Mitigation Strategy

• (M1) Impact Verification (Completed): Review OCSP responses to verify that timing is not impacted in a similar way.
• (M2) System Update (Completed): Update the scripts used to issue root CRLs.
• (M3) Tactical Fix (Completed): Issue updated root CRLs with the correct validity period and deploy to production.
• (M4) Policy Update (On Deck): Update the Starfield CP/CPS to reflect the current validity periods in use.

Implementation and Future Actions:

• (M1) On 09/24/2021 at 13:20 MST, confirmed OCSP responses are compliant. Completed.
• (M2) On 09/27/2021, ceremony scripts were updated from 365 to 363 days for root CRLs. Completed.
• (M3) On 09/31/2021, a ceremony was executed which resulted in new root CRLs being issued for the updated validity period. On 10/1/2021 at 12:45 MST, new root CRLs were deployed into production. Completed.
• (M4) Publish the updated CP/CPS which includes the updated time validity details and profiles associated with the new root CRLs. Target completion Date: 10/22/2021.

In addition to the aforementioned items above, we continue to iterate and improve on our change (ballot, industry trends, and Bugzilla) monitoring process. In July 2021, we completed a review of the process which resulted in shifts to how changes were being communicated (change per functional area rather than by specific stakeholder). Additionally, outstanding change reviews and Bugzilla items were more formally added as a review line items to the bi-weekly compliance touch point meetings. Our next quality review over the change monitoring process is scheduled for January 2022, which will include reviewing the current process with stakeholders and identifying areas for improvement in functionality, tooling, and quality.

We noted within bugs #1731164 and #1732745, both GTS and Certainly have referenced an update to the BRs. We would be happy to assist and/or endorse these referenced updates.

Comment 2

[Brittany Randall]

CP/CPS updates related to M4 are underway and still targeting 10/22/2021 for publication.

Comment 3

[Brittany Randall]

Update related to (M4) above. CP/CPS has obtained internal policy authority approval and should be deployed this week to the repository. Still targeting 10/22/2021 (or sooner) for publication.

Comment 4

[Brittany Randall]

Summarized update of item 7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things. Note specific changes to original post bolded.

Mitigation Strategy

• (M1) Impact Verification (Completed): Review OCSP responses to verify that timing is not impacted in a similar way.
• (M2) System Update (Completed): Update the scripts used to issue root CRLs.
• (M3) Tactical Fix (Completed): Issue updated root CRLs with the correct validity period and deploy to production.
• (M4) Policy Update (Completed): Update the Starfield CP/CPS to reflect the current validity periods in use.

Implementation and Future Actions:

• (M1) On 09/24/2021 at 13:20 MST, confirmed OCSP responses are compliant. Completed.
• (M2) On 09/27/2021, ceremony scripts were updated from 365 to 363 days for root CRLs. Completed.
• (M3) On 09/31/2021, a ceremony was executed which resulted in new root CRLs being issued for the updated validity period. On 10/1/2021 at 12:45 MST, new root CRLs were deployed into production. Completed.
• (M4) On 10/20/2021, version 4.14 of the CP/CPS was published to https://certs.godaddy.com/repository. Note that updates were made to the following sections 4.9.7, 4.9.10, 7.2.2.1, and 7.2.2.2 related to OCSP and CRL timeframes. Completed.

Comment 5

[Brittany Randall]

All mitigation strategy items have been completed as described in comments above. We are continuing to track this bug for any questions/comments from the community.

Comment 6

[Brittany Randall]

If there isn't any objection, we would like to request closing this bug on 11/1/2021. We will continue to monitor for questions/comments from the community.

Comment 7

[Ben Wilson]

I'll schedule this for closure on or about Friday, 30-Oct-2021.