(In reply to Chris Clements from comment #3)
Thank you for this report. Can you please make sure all relevant events are added to the event timeline? It is currently missing the CRL generation activities, timelines related to the instantiation of the existing generation script, etc.
MM/DD/YYYY HH:MM (Times are all MST)
- 09/25/2014 09:43 MST - PKI Engineering creates the ceremony script for root CRL generation. Note: This script is specific to root CRLs and does not impact intermediate CRLs files.
- 06/22/2022 16:49 MST – PKI Engineering generates most recent root CRLs under ceremony
In the explanation about how and why the mistakes were made it was stated:
During the ceremony event for these CRLs, it was noted that these were generated incorrectly, while the remaining were generated as expected which was likely a result of rolling forward old manual ceremony scripts to complete the most current event.
- As stated, do you mean to say these CRLs were known to be generated incorrectly during the signing ceremony and were still hosted?
No – Apologies for confusion with the wording. Following the posting of this bug, we reviewed the last ceremony document from 06/22/2022 16:49 MST (i.e. when these CRLs would have been generated) and confirmed that they had been generated as version 1 without the CRL numbers.
- As stated, will the root cause be confirmed before the next ceremony? The use of “was likely a result of” reads as if the root cause has not been confirmed.
The root cause is in a preexisting manual ceremony process and associated scripts which have been reused resulting in these CRL issues since 2014. To that end, the PKI Engineering team is working to develop updated ceremony documentation, processes, and scripts to correct this issue going forward. If any additional details surface a more specific cause, we will be working to address those in process/automation improvements where feasible.
In the proposed resolution:
- Can you elaborate on the steps added to the ceremony process that will help prevent these issues in the future?
Yes. PKI Engineering has been moving toward use of Boulder’s open-source ceremony tool which has built in linting functionality. While that process is still in flight for some of our older roots (or ceremonies which support our older roots), the PKI Engineering team has added a step to manually lint and review any certificates generated under the legacy ceremony process. For this particular event, a solution has been developed and a mock ceremony is scheduled for the week of 10/24. Following the mock ceremony, we are targeting to schedule the production ceremony prior to the end of November based on schedules of multiple needed individuals.
- Is there an opportunity for GoDaddy to contribute to open-source linting projects, for example ZLint, to help detect or otherwise prevent this issue in the future?
This is a great call out. While we go through the ceremony process and through generating the updated CRLs, we will work to understand what is covered by the existing linters in Boulder’s tool. If we identify any gaps (potential contribution opportunities), we would work to contribute back into one of the public linters.
Thanks for these questions. This has been a learning opportunity for us and as we work through the remediation process, if we discover additional information that warrants an update to the incident report or additional areas of opportunity for improvement, we will be sure to report back. In the meantime, we will continue to monitor this bug for any additional comments and questions.