Closed Bug 1793642 Opened 2 years ago Closed 2 years ago

GoDaddy: CRLs are version 1 and lack CRL Number extension

Categories

(CA Program :: CA Certificate Compliance, task)

Product:
CA Program
Component:
CA Certificate Compliance
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: agwa-bugs, Assigned: brittany)

Details

(Whiteboard: [ca-compliance] [crl-failure])

Attachments

(1 file)

godaddy_crls.zip
2.05 KB, application/zip
Details
Attached file godaddy_crls.zip —

GoDaddy has disclosed these CRLs in the CCADB:

These CRLs are version 1 CRLs and contain no extensions. This is a violation of GoDaddy's CP/CPS, which specifies a CRL profile which includes the CRL Number extension (section 7.2.2).

It is also a violation of RFC 5280, which requires that CRLs be version 2 and contain the CRL Number extension (section 5.3.3).

Assignee: bwilson → brittany
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true
Whiteboard: [ca-compliance]

Thank you for bringing this to our attention. We are reviewing internally and should have an incident report in the coming week.

Incident Report

Problem Summary:

Four GoDaddy root CRLs are version 1 and do not contain CRL numbers and two GoDaddy rool CRLs are version 2 but do not contain CRL numbers which is a deviation from RFC 5280 section 5.3.3. This deviation from RFC 5280 is a violation of the Baseline Requirements for Publicly Trusted SSL certificates (BRs), section 7.1.2.4 All Certificates which states "All other fields and extensions MUST be set in accordance with RFC 5280"

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

On 10/04/2022 at 10:55 MST, Andrew Ayer submitted Mozilla bug, https://bugzilla.mozilla.org/show_bug.cgi?id=1793642.

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

MM/DD/YYYY HH:MM (Times are all MST)

  • 10/04/2022 10:55 Andrew Ayer submits bug detailing CRL version and extension issue (this bug, https://bugzilla.mozilla.org/show_bug.cgi?id=1793642)
  • 10/04/2022 14:00 PKI Development, Engineering, and Compliance teams meet to review bug details, confirm reported issue, and begin research to confirm impact to other CRLs.
  • 10/04/2022 15:52 PKI Engineering confirms that 6 CRLs are impacted (including the 4 from the original bug report)

3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

This bug is limited to root CRLs and subscriber certificate issuance was not impacted.

4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

6 root CRLs total

  • 4 root CRLs are version 1 without the CRL number extension
  • 2 root CRLs are version 2 without the CRL number extension

5. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now. See Google's guidance on root cause analysis for ideas of what to include.

The ceremony process used to generate CRLs is a manual process. During the ceremony event for these CRLs, it was noted that these were generated incorrectly, while the remaining were generated as expected which was likely a result of rolling forward old manual ceremony scripts to complete the most current event.

7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

Updated, compliant root CRLs will be generated under ceremony in November 2022. Additionally, to prevent errors in future ceremonies, as committed in bug, https://bugzilla.mozilla.org/show_bug.cgi?id=1777128, the team recently completed a review of the ceremony process and added steps to help prevent issues as a result of manual ceremony events.

Thank you for this report. Can you please make sure all relevant events are added to the event timeline? It is currently missing the CRL generation activities, timelines related to the instantiation of the existing generation script, etc.

In the explanation about how and why the mistakes were made it was stated:

During the ceremony event for these CRLs, it was noted that these were generated incorrectly, while the remaining were generated as expected which was likely a result of rolling forward old manual ceremony scripts to complete the most current event.

  • As stated, do you mean to say these CRLs were known to be generated incorrectly during the signing ceremony and were still hosted?
  • As stated, will the root cause be confirmed before the next ceremony? The use of “was likely a result of” reads as if the root cause has not been confirmed.

In the proposed resolution:

  • Can you elaborate on the steps added to the ceremony process that will help prevent these issues in the future?
  • Is there an opportunity for GoDaddy to contribute to open-source linting projects, for example ZLint, to help detect or otherwise prevent this issue in the future?

Just wanted to acknowledge that we have received these question and should be able to provide answers next week.

(In reply to Chris Clements from comment #3)

Thank you for this report. Can you please make sure all relevant events are added to the event timeline? It is currently missing the CRL generation activities, timelines related to the instantiation of the existing generation script, etc.

MM/DD/YYYY HH:MM (Times are all MST) 
Timeline additions 

  • 09/25/2014 09:43 MST - PKI Engineering creates the ceremony script for root CRL generation. Note: This script is specific to root CRLs and does not impact intermediate CRLs files.  
  • 06/22/2022 16:49 MST – PKI Engineering generates most recent root CRLs under ceremony

In the explanation about how and why the mistakes were made it was stated:

During the ceremony event for these CRLs, it was noted that these were generated incorrectly, while the remaining were generated as expected which was likely a result of rolling forward old manual ceremony scripts to complete the most current event.

  • As stated, do you mean to say these CRLs were known to be generated incorrectly during the signing ceremony and were still hosted?

No – Apologies for confusion with the wording. Following the posting of this bug, we reviewed the last ceremony document from 06/22/2022 16:49 MST (i.e. when these CRLs would have been generated) and confirmed that they had been generated as version 1 without the CRL numbers.  

  • As stated, will the root cause be confirmed before the next ceremony? The use of “was likely a result of” reads as if the root cause has not been confirmed.

The root cause is in a preexisting manual ceremony process and associated scripts which have been reused resulting in these CRL issues since 2014. To that end, the PKI Engineering team is working to develop updated ceremony documentation, processes, and scripts to correct this issue going forward. If any additional details surface a more specific cause, we will be working to address those in process/automation improvements where feasible.    

In the proposed resolution:

  • Can you elaborate on the steps added to the ceremony process that will help prevent these issues in the future?

Yes. PKI Engineering has been moving toward use of Boulder’s open-source ceremony tool which has built in linting functionality. While that process is still in flight for some of our older roots (or ceremonies which support our older roots), the PKI Engineering team has added a step to manually lint and review any certificates generated under the legacy ceremony process. For this particular event, a solution has been developed and a mock ceremony is scheduled for the week of 10/24. Following the mock ceremony, we are targeting to schedule the production ceremony prior to the end of November based on schedules of multiple needed individuals. 

  • Is there an opportunity for GoDaddy to contribute to open-source linting projects, for example ZLint, to help detect or otherwise prevent this issue in the future?

This is a great call out. While we go through the ceremony process and through generating the updated CRLs, we will work to understand what is covered by the existing linters in Boulder’s tool. If we identify any gaps (potential contribution opportunities), we would work to contribute back into one of the public linters. 

Thanks for these questions. This has been a learning opportunity for us and as we work through the remediation process, if we discover additional information that warrants an update to the incident report or additional areas of opportunity for improvement, we will be sure to report back. In the meantime, we will continue to monitor this bug for any additional comments and questions.

Product: NSS → CA Program

Progress update: On 11/2/2022, we held a mock ceremony to generate updated root CRLs. As a result of the mock ceremony, we discovered adjustments needed within our generation process and have been making updates in preparation for the production ceremony. We have tentatively scheduled our production ceremony for the week of 11/28 to generate and publish the updated CRLs. We will report back when the ceremony is completed and the new CRLs are published. Thanks.

Progress Update: On 11/28/2022, root CRLs were generated under ceremony and on 11/29/2022 the PKI Engineering team verified that those CRLs were deployed as expected. With this change, all remediation activities have been completed. We will continue to monitor for any questions from the community.

This bug has several references to CRL Number extension section 5.3.3, but I believe they should refer to section 5.2.3

(In reply to Matthew McPherrin from comment #8)

This bug has several references to CRL Number extension section 5.3.3, but I believe they should refer to section 5.2.3

Yup, I typoed that in Comment 0.

I'll close this out next Wed. 11-Jan-2023 unless there are additional questions or issues to discuss.

Flags: needinfo?(bwilson)
Whiteboard: [ca-compliance] → [ca-compliance] [crl-failure]
Status: ASSIGNED → RESOLVED
Closed: 2 years ago
Flags: needinfo?(bwilson)
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: