DigiCert: Outdated CPS for 13 Roots in CCADB
Categories
(CA Program :: CA Certificate Compliance, task)
Tracking
(Not tracked)
People
(Reporter: dcbugzillaresponse, Assigned: dcbugzillaresponse)
Details
(Whiteboard: [ca-compliance] [policy-failure] [disclosure-failure])
Attachments
(2 files)
1945536c4 - screenshot.png
Steps to reproduce:
Preliminary report coming.
Incident Report
Summary
This is a preliminary report and we will provide updates.
On January 10th, a Sectigo employee emailed DigiCert to let us know that 13 Root Certificate records had been flagged on https://crt.sh/mozilla-disclosures due to non-compliance with Mozilla Policy 3.3, which requires that a CA update their CP/CPS at least once every 365 days. DigiCert released seven versions in 2024 and uploaded each of them. However, the CCADB entry was not updated with the last submission due to ALV errors. We re-submitted the CP/CPS update and it was approved.
Impact
An outdated CP/CPS entry was displayed in CCADB for 13 Roots. Anyone reviewing the “AllCertificateRecordsCSVFormatv2” file from CCADB would see the outdated CP/CPS v6.04 numbering rather than version v7.04 The link still pointed to the correct CPS as we specify http://www.digicert.com/CPS, which points to our general repository and lists the current version v7.04 of our CP/CPS.
Timeline
All times are UTC.
2024-11-19: DigiCert released Public Trust CP/CPS v7.04
2024-11-28 01:09: DigiCert created case 00002121 to update DigiCert annual WebTrust audits and CP/CPS
2024-12-02: ALV scans cause case 00002121 to revert back to DigiCert to resolve our seal URLs. The ALV failure notice was sent to the uploader, who went on leave before receiving the notice. Notice was not sent to a general DigiCert alias or other CCADB users.
2025-01-10 16:06: Sectigo notified DigiCert that they were seeing an outdated CP/CPS for 13 Root Certificates. DigiCert examines and finds that the CP/CPS was uploaded on 2024-11-28.
2025-01-10 21:18: DigiCert created case #00002192 in CCADB to submit CP/CPS v7.04 separate from the ALV failure.
2025-01-14 17:57: Case #00002192 was approved.
2025-01-31 16:46: Mozilla notified DigiCert that there was no bug in CCADB limiting the number of entries in the “Non-Audit documents” tab.
2025-01-31 19:45: We modified our weekly scans to ensure our latest CP/CPS is reflected in CCADB for our Root Certificates.
Root Cause Analysis
Sectigo reported that we were still showing CP/CPS version 6.04 from December 14, 2023, despite having released 8 different updates since version 6.04. Our initial investigation concluded that Sectigo was pulling an old Non-Audit Document from CCADB as we saw the records as being uploaded promptly after a CPS update. To the best of our understanding, the error was caused by was due to DigiCert not marking older versions as “Superseded” in our Non-Audit Documents tab. DigiCert has an entry for each trust bit case, which means there are a lot of audits.
We are still investigating the root cause and need two additional questions answered before we can determine what happened.
1.) If you look up the QuoVadis roots in the “AllCertificateRecordsCSVFormatv2” file from https://www.ccadb.org/resources, there are two different CP/CPS versions v6-2 and v6-01 in the same field. When we marked the previous CP/CPS as “Superseded” in case # 00002174 on our last update.
2.) [see screenshot attached] From the same “AllCertificateRecordsCSVFormatv2” file, for our SubCAs, we have 'CP Same As Parent' selected, but in that file, there does not appear to be any pattern or logic in what the data displays compared to what is listed in the CCADB web portal.
Understanding the display on those two items will help us determine:
[1] why was Sectigo showing such an outdated version of our CP/CPS.
[2] why is there inconsistency between the “AllCertificateRecordsCSVFormatv2” file and the CCADB web portal data.
We think this might be related to CCADB release v1.3.1 (Oct 29, 2024, 6:14:53 AM) as the issues with CPS numbering seem to start there. At that point, our older CPS started being shown instead of the newer version.
Lessons Learned
We need to run the ALV scan and resolve any errors before submitting a case. Employees using CCADB must access the system using a shared email to ensure that multiple team members receive communication back from the system.
We need to mark older versions as “Superseded” in our Non-Audit Docs tab.
What went well
The CCADB web portal displayed our second-latest version, 7.03, which was released on September 25, 2024, just two months prior to the latest version. Instead of the >365 days document that Sectigo was reporting.
What didn't go well
DigiCert missed a CCADB case that was sent back for action for ALV reasons, which delayed an update of our CP/CPS and audits.
Where we got lucky
Because the internal tools at DigiCert, as well as https://crt.sh/mozilla-disclosures, use “AllCertificateRecordsCSVFormatv2” to automate checks and reports, getting extra attention on this issue will help the community and others using the report.
Action Items
| Action Item | Kind | date | Description |
|---|---|---|---|
| Update the records in CCADB | Remediation | 2025-01-10 | Update the Roots in CCADB to the latest CP/CPS. |
| Run weekly reports | Prevent | 2025-01-31 | Changed our scans to ensure that our latest CP/CPS is reflected in CCADB for our Root Certificates. |
| Role-based CCADB | Prevent | 2025-02-10 | Turn CCADB access into a role-based access instead of individual access so multiple individuals receive reports of CCADB issues. |
| CCADB investigation | Remediation | TBD | We need to understand why the data in the “AllCertificateRecordsCSVFormatv2” file doesn’t match the information entered and displayed in the CCADB web portal. |
|
||
Updated•1 year ago
|
|
||
Comment 2•1 year ago
|
||
Thank you for providing this report. We have several questions and clarifications for consideration.
In the Impact Section:
An outdated CP/CPS entry was displayed in CCADB for 13 Roots. Anyone reviewing the “AllCertificateRecordsCSVFormatv2” file from CCADB would see the outdated CP/CPS v6.04 numbering rather than version v7.04
Describing which Roots were impacted and for what amount of time someone relying on the CCADB report might have seen outdated information would be a helpful update.
The link still pointed to the correct CPS as we specify http://www.digicert.com/CPS, which points to our general repository and lists the current version v7.04 of our CP/CPS.
Can you specify which “link still pointed to the correct CPS”? It’s not clear, since the CCADB tracks both (1) CA Document Repository URLs and (2) URLs to CP, CPS, and CP/CPS documents separately.
In the Timeline Section:
2024-11-28 01:09: DigiCert created case 00002121 to update DigiCert annual WebTrust audits and CP/CPS
2024-12-02: ALV scans cause case 00002121 to revert back to DigiCert to resolve our seal URLs. The ALV failure notice was sent to the uploader, who went on leave before receiving the notice. Notice was not sent to a general DigiCert alias or other CCADB users.
To clarify, ALV cannot revert a CCADB Case. ALV parses Audit Reports to identify key information such as SHA256 thumbprints and audit period dates.
Case 00002121 was created on 2024-11-28 and reviewed by a Root Store Operator on 2024-12-02, where it was identified in a Case Comment that the correct WebTrust URLs were not provided by DigiCert. Case Comments are automatically sent to the Case creator via email. However, DigiCert can, and should, run ALV at any point after adding audit information to their Case and per Section 5.1.3 of the CCADB Policy: “If the audit statement fails to meet any of the following requirements, the CA Owner MUST work with their auditor to provide an audit statement that passes ALV.” DigiCert could have seen and resolved the ALV errors before submitting the Case for Root Store Operator review (Guidance for resolving common findings is posted on ccadb.org). The Case status was manually changed from “Verification by Root Store” back to “CA Providing Data” so that the correct information could be provided for further review.
In the Root Cause Analysis Section:
Our initial investigation concluded that Sectigo was pulling an old Non-Audit Document from CCADB as we saw the records as being uploaded promptly after a CPS update. To the best of our understanding, the error was caused by was due to DigiCert not marking older versions as “Superseded” in our Non-Audit Documents tab. DigiCert has an entry for each trust bit case, which means there are a lot of audits.
It’s not clear what is meant by “DigiCert has an entry for each trust bit case, which means there are a lot of audits.” or how it relates to Non-Audit Documents.
However, to confirm your understanding, identifying older versions as “Superseded” does remove those documents from the AllCertificateRecordsCSVFormatv2 file.
1.) If you look up the QuoVadis roots in the “AllCertificateRecordsCSVFormatv2” file from https://www.ccadb.org/resources, there are two different CP/CPS versions v6-2 and v6-01 in the same field. When we marked the previous CP/CPS as “Superseded” in case # 00002174 on our last update.
It would be helpful in the future if you could reference a specific Root as an example. CCADB Case #00002174 was created on 2024-12-30 and was closed on 2025-01-03. The Case identified the following policy documents as superseded:
- CP/CPS (https://www.quovadisglobal.com/wp-content/uploads/2023/12/digicert-europe-cpcps-v5-1-en.pdf)
- CP/CPS (https://www.quovadisglobal.com/wp-content/uploads/2023/11/digicert-europe-cpcps-v5-0-en.pdf)
- CP/CPS (https://www.quovadisglobal.com/wp-content/uploads/2022/09/QV_RCA1_RCA3_CPCPS_V4_40_Approved.pdf)
- CP/CPS (https://www.quovadisglobal.com/wp-content/uploads/2023/03/QV_RCA2_CPCPS_v2.20_Approved.pdf)
- CP/CPS (https://www.quovadisglobal.com/wp-content/uploads/2023/03/QV_RCA1_RCA3_CPCPS_V4_41_Approved.pdf)
- CP/CPS (https://www.quovadisglobal.com/wp-content/uploads/2023/03/QV_RCA2_CPCPS_v2.20_Approved.pdf)
- CP/CPS (https://www.quovadisglobal.com/wp-content/uploads/2022/09/QV_RCA2_CPCPS_v2.19_Approved.pdf)
It does not appear that the Case referenced made any changes to CP/CPS v6-2 or v6-01.
2.) [see screenshot attached] From the same “AllCertificateRecordsCSVFormatv2” file, for our SubCAs, we have 'CP Same As Parent' selected, but in that file, there does not appear to be any pattern or logic in what the data displays compared to what is listed in the CCADB web portal.
Can you please attach the screenshot?
We think this might be related to CCADB release v1.3.1 (Oct 29, 2024, 6:14:53 AM) as the issues with CPS numbering seem to start there. At that point, our older CPS started being shown instead of the newer version.
The ability to supersede a policy document originated in conversation on public@ccadb.org back in August. The subsequent CCADB Enhancement Request landed in production on October 28, 2024 as described here. We confirmed with the CCADB development team that before deploying the October Enhancement Request, the AllCertificateRecordsCSVFormatv2 file did not exclude superseded (or what was referred to as “deleted” at the time) policy documents.
In the Lessons Learned Section:
Employees using CCADB must access the system using a shared email to ensure that multiple team members receive communication back from the system.
The ‘Add/Update Contacts’ Case type in the CCADB includes a tooltip for ‘Email’ which reads: “A unique email address for a single individual who is associated with this CA. This MUST NOT be an email alias/list.” This tooltip exists to enhance accountability and reduce the risk of unauthorized access in the system. If DigiCert is sharing an account across multiple users, we ask that they stop doing so. Further, we believe this behavior is unnecessary because any DigiCert user with a CCADB account can view/edit existing DigiCert Cases. If DigiCert disagrees with this position, or feels functionality is otherwise lacking, an Enhancement Request describing more would be appreciated.
To clarify CCADB communications, here's what happens today:
- when a Case is created or closed, or when a Comment is added, the Case creator receives an email notification.
- monthly, via automation, CA Owners are notified of outstanding tasks/issues (e.g., Root Certificates with Outdated Audits, Root Certificates with Outdated Non-Audit Documents, Intermediate Certificates with Failed ALV Results, etc.). These emails are sent to all CA Owner Primary POCs with the CA email alias address(es) copied.
The CCADB Steering Committee could consider changing the Case notifications to include all CA Owner POCs and the CA Email Alias, or anything in between. This could be another opportunity for an Enhancement Request describing desirable functionality.
DigiCert missed a CCADB case that was sent back for action for ALV reasons, which delayed an update of our CP/CPS and audits.
This could be restated because it appears it’s not so much that a CCADB Case was missed, as it was that the ALV presented a failure at the time the Case was in a status of “CA Providing Data” and that error was not acted upon at the time of presentation.
With the clarifications we’ve provided, we would encourage some additional exploration of Root Cause Analysis and possibly some revisions to the proposed Action Items.
Lastly, as a general note, the Subject of this report (“DigiCert: CCADB showing wrong CPS for 13 roots”) could be (mis)interpreted to suggest a CCADB bug or misconfiguration caused this incident. At this time, there is no indication that the CCADB Salesforce application, itself, caused this issue.
Thank you for the quick and detailed response, that will assist with us figuring out what's going on here.
In response to comment 2
Thank you for providing this report. We have several questions and clarifications for consideration.
In the Impact Section:
Describing which Roots were impacted and for what amount of time someone relying on the CCADB report might have seen outdated information would be a helpful update.
Can you specify which “link still pointed to the correct CPS”? It’s not clear, since the CCADB tracks both (1) CA Document Repository URLs and (2) URLs to CP, CPS, and CP/CPS documents separately.
Child CAs and leaf certificates include the Certificate Policies extension, which contains the DigiCert’s legal repository URI (CPS Pointer). [2] CCADB share a redirect URL to the same page listed under the “CA Document Repository URLs”.
In the Timeline Section:
To clarify, ALV cannot revert a CCADB Case. ALV parses Audit Reports to identify key information such as SHA256 thumbprints and audit period dates.
A DigiCert employee uploaded the audits to a Bugzilla bug following instructions here https://www.ccadb.org/cas/fields#uploading-documents complying with CCADB Policy 5.1.2. The attachments were then linked in the “Audit Statement URL” fields for each different audit. But the ALV scan still returned errors indicating that it required the Seal URLs we obtained from CPA Canada instead. A different DigiCert employee updated the “Statement URLs” on 2024-01-10, ran the ALV scan before submitting the case back to the Root Store for review.
Case 00002121 was created on 2024-11-28 and reviewed by a Root Store Operator on 2024-12-02, where it was identified in a Case Comment that the correct WebTrust URLs were not provided by DigiCert. Case Comments are automatically sent to the Case creator via email. However, DigiCert can, and should, run ALV at any point after adding audit information to their Case and per Section 5.1.3 of the CCADB Policy: “If the audit statement fails to meet any of the following requirements, the CA Owner MUST work with their auditor to provide an audit statement that passes ALV.” DigiCert could have seen and resolved the ALV errors before submitting the Case for Root Store Operator review (Guidance for resolving common findings is posted on ccadb.org). The Case status was manually changed from “Verification by Root Store” back to “CA Providing Data” so that the correct information could be provided for further review.
Noted. Thank you for this. Sounds like the primary issue was a failure of our team to run ALV before submitting, fixing the ALV errors, and then resubmitting. Would you like an updated incident report that includes this information?
It’s not clear what is meant by “DigiCert has an entry for each trust bit case, which means there are a lot of audits.” or how it relates to Non-Audit Documents.
Just that there are lot of entries. We were unaware that we needed to check superseded. That would have cleaned out the file and made the information more manageable.
However, to confirm your understanding, identifying older versions as “Superseded” does remove those documents from the AllCertificateRecordsCSVFormatv2 file.
Please help understand why doesn’t marking documents as “Superseded” remove them from the AllCertificateRecordsCSVFormatv2? This makes it difficult to track the latest documents, or match what’s in CCADB. As you know, CCADB's API doesn’t support GET calls, leaving AllCertificateRecordsCSVFormatv2 as our only resource for identifying CCADB contents and building automation and reports around the available information.
We’ve made this our practice going forward.
1.) If you look up the QuoVadis roots in the “AllCertificateRecordsCSVFormatv2” file from https://www.ccadb.org/resources, there are two different CP/CPS versions v6-2 and v6-01 in the same field. When we marked the previous CP/CPS as “Superseded” in case # 00002174 on our last update.
It would be helpful in the future if you could reference a specific Root as an example. CCADB Case #00002174 was created on 2024-12-30 and was closed on 2025-01-03.
Thank you! We will make sure we do this.
Can you please attach the screenshot?
Attached this time.
The ability to supersede a policy document originated in conversation on public@ccadb.org back in August. The subsequent CCADB Enhancement Request landed in production on October 28, 2024 as described here. We confirmed with the CCADB development team that before deploying the October Enhancement Request, the AllCertificateRecordsCSVFormatv2 file did not exclude superseded (or what was referred to as “deleted” at the time) policy documents.
Thank you for the clarification.
We’re trying to understand the benefit in keeping policy documents that are marked as “Superseded” or “Deleted” in AllCertificateRecordsCSVFormatv2 as they’re no longer active. Displaying only the latest policy documents to ensure accurate details seems like a better solution. This might explain why Sectigo was referencing such an outdated DigiCert CP/CPS in the initial CPR.
If an Intermediate is marked as “CP Same As Parent”, “CPS Same As Parent”, or “CP/CPS Same As Parent” then the corresponding fields “certificate_policy_cp_url”, “certificate_practice_statement_cps_url”, and “certificate_practice_and_policy_statement” in AllCertificateRecordsCSVFormatv2 should be either empty or, at the very least, inherit the parent record's values.
Here are 3 examples (among others) where the data in AllCertificateRecordsCSVFormatv2 doesn’t match what’s in the CCADB’s web portal. Each of these records has “Same As CP” with the other CP fields empty. However, in AllCertificateRecordsCSVFormatv2, the data appears as follows
| certificate_name | cp_same_as_parent | certificate_policy_cp_url | cp_last_update_date |
|---|---|---|---|
| Apple Public EV Server RSA CA 3 - G1 | true | https://www.digicert.com/content/dam/digicert/pdfs/legal/digicert-public-trust-cpcps-v7.04.pdf | 2024-11-19 |
| Microsoft Azure ECC TLS Issuing CA 03 | true | https://www.microsoft.com/pkiops/Docs/Content/policy/Microsoft_PKI_Services_CP_v3.1.8.pdf | 2024-07-21 |
| Apple Public Server ECC CA 1 - G1 | true | None | 1970-01-01 |
The top record seems to be inheriting data from the parent record.
The middle record is showing an outdated URL.
The bottom record matches what we would expect to see, as it accurately reflects the web portal.
Apologies for not participating in the public@ccadb.org conversation, but these seem like important details to consider given our current Bugzilla bug.
The ‘Add/Update Contacts’ Case type in the CCADB includes a tooltip for ‘Email’ which reads: “A unique email address for a single individual who is associated with this CA. This MUST NOT be an email alias/list.” This tooltip exists to enhance accountability and reduce the risk of unauthorized access in the system. If DigiCert is sharing an account across multiple users, we ask that they stop doing so. Further, we believe this behavior is unnecessary because any DigiCert user with a CCADB account can view/edit existing DigiCert Cases. If DigiCert disagrees with this position, or feels functionality is otherwise lacking, an Enhancement Request describing more would be appreciated.
We aren’t currently sharing account information. However, we thought this might be a good improvement to ensure this type of information isn’t missed. We filed an Enhancement Request with CCADB so there can be a better discussion on the impact of account sharing https://bugzilla.mozilla.org/show_bug.cgi?id=1946849.
The CCADB Steering Committee could consider changing the Case notifications to include all CA Owner POCs and the CA Email Alias, or anything in between. This could be another opportunity for an Enhancement Request describing desirable functionality.
DigiCert missed a CCADB case that had ALV issues and was not uploaded. This ALV issue an update of our CP/CPS and audits.
This could be restated because it appears it’s not so much that a CCADB Case was missed, as it was that the ALV presented a failure at the time the Case was in a status of “CA Providing Data” and that error was not acted upon at the time of presentation.
This is correct. There was an ALV error presented when the information was uploaded and the person uploading it did not realize the information was not saved.
With the clarifications we’ve provided, we would encourage some additional exploration of Root Cause Analysis and possibly some revisions to the proposed Action Items.
We will investigate and figure out some additional action items.
Lastly, as a general note, the Subject of this report (“DigiCert: CCADB showing wrong CPS for 13 roots”) could be (mis)interpreted to suggest a CCADB bug or misconfiguration caused this incident. At this time, there is no indication that the CCADB Salesforce application, itself, caused this issue.
The change in CCADB behavior led to the incorrect information being displayed, not because of an issue in CCADB, but because DigiCert needed to change its process to account for the change in CCADB. Our processes were not updated to account for suspension of previous documents. We also had a weakness in our process for ALV failures that you pointed out needs to be addressed.
|
|
||
Comment 6•1 year ago
|
||
Comment #5 includes several references that do not appear to be directly relevant to the root cause of DigiCert's delayed update of the CP/CPS information in the CCADB. To maintain focus on the key issues, the following clarifications may be helpful:
CCADB Complexity and Process Changes:
While the CCADB can be complex at times, the process for updating policy documents (CPs, CPSes, and CP/CPSes) is generally straightforward. The changes to how the CCADB handles superseding non-audit documents may have introduced some confusion, but there is no clear indication that these changes directly caused this incident. Similarly, audits, ALV, and the way the AllCertificateRecordsCSVFormatv2 file is structured do not appear to be directly related to the failure to update the CP/CPS. (ALV is not run on non-audit/policy documents. If ALV errors occurred when reviewing newly submitted audits, those errors should have been addressed separately. If the CP/CPS update was intended to be included in an "Add/Update Root Request" case, tracking the necessary updates more carefully could have helped prevent the disclosure delays. Removing these factors from the explanation may help clarify core issues.
Staff Turnover and CA Responsibility for Updating Information in the CCADB:
DigiCert's CCADB enhancement request to notify multiple individuals when a CCADB case comment is posted is a reasonable suggestion and could be beneficial. However, ensuring that CCADB cases are properly tracked and managed remains the responsibility of the CA. If staff turnover played a role in this incident, acknowledging that factor would provide valuable insight. Additionally, if a documented process exists for updating policy documents in the CCADB, clarification on whether that process could have been followed would help identify areas for improvement. Understanding what tabs and fields are being updated in an “Add/Update Root Request” case is essential. Again, CA operators are responsible for tracking changes made in the CCADB and should not rely solely on automated notifications when comments are added to the CCADB case.
Refining the response to focus on the core issues and provide a clearer explanation of the root cause and associated action items would be beneficial.
|
|
||
Comment 7•1 year ago
|
||
Thank you for the updates. To further address the questions asked in Comment 5:
Noted. Thank you for this. Sounds like the primary issue was a failure of our team to run ALV before submitting, fixing the ALV errors, and then resubmitting. Would you like an updated incident report that includes this information?
We would encourage an update that aligns with Comment 6.
Please help understand why doesn’t marking documents as “Superseded” remove them from the AllCertificateRecordsCSVFormatv2? This makes it difficult to track the latest documents, or match what’s in CCADB. As you know, CCADB's API doesn’t support GET calls, leaving AllCertificateRecordsCSVFormatv2 as our only resource for identifying CCADB contents and building automation and reports around the available information.
Marking Non-Audit Documents as ‘Superseded’ in the CCADB does remove them from the AllCertificateRecordsCSVFormatv2 file. In Comment 1 DigiCert stated: “To the best of our understanding, the error was caused by was due to DigiCert not marking older versions as “Superseded” in our Non-Audit Documents tab.” and in Comment 2, we were attempting to confirm this understanding.
Also, the AllCertificateRecordsCSVFormatv2 file should not be considered the only resource for identifying CCADB contents, as the CCADB has options for CA Owners to review and export their own data (e.g., the “CA REPORTS” tab under MyCA, the overarching Reports function for Community reports, etc.).
1.) If you look up the QuoVadis roots in the “AllCertificateRecordsCSVFormatv2” file from https://www.ccadb.org/resources, there are two different CP/CPS versions v6-2 and v6-01 in the same field. When we marked the previous CP/CPS as “Superseded” in case # 00002174 on our last update.
In Comment 2 we provided a review of Case #00002174 and stated It did not appear that it included any changes to CP/CPS v6-2 or v6-01.
We’re trying to understand the benefit in keeping policy documents that are marked as “Superseded” or “Deleted” in AllCertificateRecordsCSVFormatv2 as they’re no longer active. Displaying only the latest policy documents to ensure accurate details seems like a better solution. This might explain why Sectigo was referencing such an outdated DigiCert CP/CPS in the initial CPR.
To clarify, the AllCertificateRecordsCSVFormatv2 file does not include ‘Superseded’ Non-Audit Documents.
If an Intermediate is marked as “CP Same As Parent”, “CPS Same As Parent”, or “CP/CPS Same As Parent” then the corresponding fields “certificate_policy_cp_url”, “certificate_practice_statement_cps_url”, and “certificate_practice_and_policy_statement” in AllCertificateRecordsCSVFormatv2 should be either empty or, at the very least, inherit the parent record's values.
Thank you for providing the attachment. If ‘$Same as Parent’ is selected, then the corresponding fields will be empty, unless a Non-Audit Document already exists for that certificate record and is not currently marked as ‘Superseded’. These documents can be viewed on the ‘Non-Audit Documents’ tab of the certificate record. The CCADB is reporting the collection of Non-Audit Documents associated with a specific certificate record.
The top record seems to be inheriting data from the parent record.
In the CCADB, on the Non-Audit Documents tab of Apple Public EV Server RSA CA 3 - G1 we see an active CP (https://www.digicert.com/content/dam/digicert/pdfs/legal/digicert-public-trust-cpcps-v7.04.pdf) with a Last Update Date of
11/19/2024. This document is being reflected in the AllCertificateRecordsCSVFormatv2 file. If this document has been superseded by a Parent document, it should be marked as such in the CCADB.
The middle record is showing an outdated URL.
In the CCADB, on the Non-Audit Documents tab of Microsoft Azure ECC TLS Issuing CA 03 we see an active CP (https://www.microsoft.com/pkiops/Docs/Content/policy/Microsoft_PKI_Services_CP_v3.1.8.pdf) with a Last Update Date of 7/21/2024. Again, this document is being reflected in the AllCertificateRecordsCSVFormatv2 file and if it’s been superseded by a Parent document, it should be marked as such in the CCADB.
In response to comment 6 and comment 7
We would encourage an update that aligns with Comment 6.
Root Cause Analysis
A DigiCert employee did not run the ALV scan or act on a case notification to resolve an ALV error. The error went to their employee’s email while they were on sabbatical. Because the email only went to the one individual, our other PoC CCADB users were not aware that the ALV failure occurred which resulted in a delay in updating our Policy Documents in CCADB.
We have an internal SOP in place, but that SOP did not require two people to review a case. The SOP also needed to be updated to include the procedure for marking outdated documents as "superseded." This functionality was not available when the SOP was created and, thus, not part of the process. Since Feb 7th, we have updated the SOP and implemented a new approach to improve oversight. We have a dual control process that technically requires at least two individuals review and complete a case.
Marking Non-Audit Documents as ‘Superseded’ in the CCADB does remove them from the AllCertificateRecordsCSVFormatv2 file. In Comment 1 DigiCert stated: “To the best of our understanding, the error was caused by was due to DigiCert not marking older versions as “Superseded” in our Non-Audit Documents tab.” and in Comment 2, we were attempting to confirm this understanding.
Also, the AllCertificateRecordsCSVFormatv2 file should not be considered the only resource for identifying CCADB contents, as the CCADB has options for CA Owners to review and export their own data (e.g., the “CA REPORTS” tab under MyCA, the overarching Reports function for Community reports, etc.).
Understood. Thank you for the reminder that AllCertificateRecordsCSVFormatv2 shouldn't be the sole data source. Until there's a GET call for the CCADB API, we will run manual reports in "CA REPORTS," as these reports provide more details than AllCertificateRecordsCSVFormatv2.
In Comment 2 we provided a review of Case #00002174 and stated It did not appear that it included any changes to CP/CPS v6-2 or v6-01.
We appreciate you looking into it. We willl submit a new case to mark v6-01 as "Superseded. " CPS v6-01 seems to be causing internal issues with our new automation alerts.
To clarify, the AllCertificateRecordsCSVFormatv2 file does not include ‘Superseded’ Non-Audit Documents.
Perfect. That makes sense and is what we would expect to see.
Thank you for providing the attachment. If ‘$Same as Parent’ is selected, then the corresponding fields will be empty, unless a Non-Audit Document already exists for that certificate record and is not currently marked as ‘Superseded’. These documents can be viewed on the ‘Non-Audit Documents’ tab of the certificate record. The CCADB is reporting the collection of Non-Audit Documents associated with a specific certificate record.
Appreciate the reiteration. We'll give this a try, and, if we encounter any issues, we'll address them separately as suggested in Comment 6 [https://bugzilla.mozilla.org/show_bug.cgi?id=1945536#c6]
In the CCADB, on the Non-Audit Documents tab of Apple Public EV Server RSA CA 3 - G1 we see an active CP (https://www.digicert.com/content/dam/digicert/pdfs/legal/digicert-public-trust-cpcps-v7.04.pdf) with a Last Update Date of
11/19/2024. This document is being reflected in the AllCertificateRecordsCSVFormatv2 file. If this document has been superseded by a Parent document, it should be marked as such in the CCADB.
In the CCADB, on the Non-Audit Documents tab of Microsoft Azure ECC TLS Issuing CA 03 we see an active CP (https://www.microsoft.com/pkiops/Docs/Content/policy/Microsoft_PKI_Services_CP_v3.1.8.pdf) with a Last Update Date of 7/21/2024. Again, this document is being reflected in the AllCertificateRecordsCSVFormatv2 file and if it’s been superseded by a Parent document, it should be marked as such in the CCADB.
Thanks again. Same as above. If we encounter any issues or have further questions, we'll address them separately.
NOTE: As per request from Chrome we have changed the subject of the report to “DigiCert: Outdated CPS for 13 Roots in CCADB”
Incident Report Closure Summary
-
Incident Description: An outdated CP/CPS entry was displayed in CCADB for 13 Roots. Anyone reviewing the “AllCertificateRecordsCSVFormatv2” file from CCADB would see the outdated CP/CPS v6.04 numbering rather than version v7.04
-
Incident Root Cause(s): A DigiCert employee did not run the ALV scan or act on a case notification to resolve an ALV error. We have an internal SOP in place, but that SOP did not require two people to review a case. The SOP also needed to be updated to include the procedure for marking outdated documents as "superseded."
-
Remediation Description: We’ve updated our SOPs to require two people to review a case, and to be sure to mark outdated documents as “superceded’. In addition we’ve added CP/CPS checks to our automated CCADB checking to ensure that we detect and correct inaccuracies. We also submitted an enhancement request to have error alerts sent to all CCADB contacts rather than just the person who submitted the case.
-
Commitment Summary: Our commitment is to a) continue automating against CCADB to the maximum extent possible and b) include the correct information going forward. We also commit to automating the CCADB upload process when f CCADB APIs are available in the future. We look forward to working with the community to improve CCADB to make submission, updating and review of CCADB data as automatable as possible.
All Action Items disclosed in this Incident Report have been completed as described, and we request its closure.
|
||
Comment 10•11 months ago
|
||
(In reply to DigiCert from comment #9)
All Action Items disclosed in this Incident Report have been completed as described, and we request its closure.
I sent the Certificate Problem Report to DigiCert that led to the creation of this incident bug. I have some areas of concern regarding DigiCert's response to this incident that I would like to bring to the community's attention.
Ben, please leave this bug open for now. I will post a comment within the next week.
|
Assignee | |
Comment 11•11 months ago
|
||
All action items are still complete, and we are still requesting closure.
|
|
||
Comment 12•11 months ago
|
||
(In reply to DigiCert from comment #8)
NOTE: As per request from Chrome we have changed the subject of the report to “DigiCert: Outdated CPS for 13 Roots in CCADB”
This updated bug summary still doesn't fully capture the scope of the incident. In the CPR I sent to DigiCert, I noted that due to "same as parent" inheritance there were also hundreds of impacted Intermediate Certificate CCADB records.
For the details, see these snapshots of the mozilla-disclosures (2025-01-19) and AllCertificateRecordsCSVFormatv2 (2025-01-10) reports.
(In reply to DigiCert from comment #1)
What went well
The CCADB web portal displayed our second-latest version, 7.03, which was released on September 25, 2024, just two months prior to the latest version.
I agree with this statement. Case 00002052 (opened 2024-10-02, closed 2024-10-07) did associate DigiCert's CP/CPS v7.03 with a large number of DigiCert's Root Certificate CCADB records. However...
Instead of the >365 days document that Sectigo was reporting.
"Instead of" is just plain wrong. At that time (2025-01-10), the CCADB web portal also displayed the v6.x DigiCert CP and CPS, because these non-audit documents had not been Deleted (aka Superseded).
Simply disclosing a new version of a non-audit document to CCADB is not enough. In a discussion thread on the CCADB Public list in August 2024, Apple and Chrome root program representatives both stated that all non-audit documents disclosed to CCADB are considered authoritative until they are deleted/superseded. This is also stated explicitly in Apple Root Certificate Program Requirement 1.3.1 (emphasis mine):
'CA providers must strictly adhere to their Certificate Policy (CP) and/or Certification Practices Statement (CPS) document(s) as disclosed within the CCADB (and not marked as "Deleted").'
The ">365 days document" was correctly included in the AllCertificateRecordsCSVFormatv2 CCADB report because it was still an authoritative non-audit document to which DigiCert was required to 'strictly adhere', and what "Sectigo was reporting" at https://crt.sh/mozilla-disclosures was a faithful representation of that accurate CCADB report.
(In reply to DigiCert from comment #9)
Incident Root Cause(s): A DigiCert employee did not run the ALV scan or act on a case notification to resolve an ALV error.
I can accept this as the explanation for why processing of Case 00002121 was delayed, but I don't accept that it's the root cause of this incident.
Firstly, it appears that Case 00002121 did not delete the outdated non-audit documents or mark them as superseded; in which case, even if the ALV error had been resolved and that Case had been successfully processed soon after it was opened (2024-11-28), those outdated documents would still have been considered authoritative and so the underlying problem would have remained. (I believe that Case 00002191, entitled "Archive old CPCPS docs" and created about 2 hours after I sent the CPR, was responsible for deleting/superseding the outdated DigiCert CP and CPS documents; however, the CCADB Case UI doesn't seem to clearly show which details were changed by a Closed Case, so I'm partly relying on my memory of what I observed on Case 00002121 when I reviewed it on 2025-01-10 whilst it was still Open).
(In reply to DigiCert from comment #1)
Summary
...
We re-submitted the CP/CPS update and it was approved.
...
Timeline
All times are UTC.
2024-11-19: DigiCert released Public Trust CP/CPS v7.04
2024-11-28 01:09: DigiCert created case 00002121 to update DigiCert annual WebTrust audits and CP/CPS
...
2025-01-10 16:06: Sectigo notified DigiCert that they were seeing an outdated CP/CPS for 13 Root Certificates. DigiCert examines and finds that the CP/CPS was uploaded on 2024-11-28.
2025-01-10 21:18: DigiCert created case #00002192 in CCADB to submit CP/CPS v7.04 separate from the ALV failure.
Secondly, in the portions of the incident report quoted above and in the response I received to my CPR, DigiCert claims that it submitted details of CP/CPS v7.04 in Case 00002121 and then later verified that it had indeed done so. I dispute this claim.
Here is my own timeline of relevant events that led to the creation of this bug:
All times are UTC.
2024-11-19: DigiCert publishes its CP/CPS v7.04.
2025-01-10 16:06: I send the CPR to DigiCert.
2025-01-10 16:35: DigiCert responds 'We will investigate and revert with a reply'.
2025-01-10 18:16: DigiCert opens CCADB Case 00002191 to "archive old CPCPS docs".
2025-01-10 21:18: DigiCert opens CCADB Case 00002192 to disclose (late, >30 days) CP/CPS v7.04.
2025-01-10 22:30: DigiCert responds again, blaming the continued presence in CCADB of outdated non-audit documents and the failure to disclose CP/CPS v7.04 in a timely fashion on 'limitations in CCADB. To be more specific, it appears that whenever the “Non-Audit Documents” section of a new case is heavily populated with too many entries, not all the Doc-IDs are displayed. I suspect there is a limit to how many documents can be shown on the page’ and claiming 'Hence, the CAs you referred to did indeed contain our most recent CP/CPS but the link was not properly shown.' DigiCert does not inform me that it has created Case 00002191 and/or 00002192 to correct the problems identified by my CPR.
2025-01-13 22:29: Noting that 72 hours has elapsed without the creation of any new incident bug in Bugzilla, I ask DigiCert 'am I to understand that DigiCert's position is that there are no 'incident(s)' in this CPR for DigiCert to publicly disclose?'
2025-01-14 18:04: DigiCert replies 'yes you are correct, we don't believe an actionable compliance issue was identified in the CPR.'
2025-01-31 16:46: (From the incident report) Mozilla notifies DigiCert that there was no bug in CCADB limiting the number of entries in the “Non-Audit documents” tab.
2025-02-03 15:51: DigiCert opens this incident bug.
On every occasion I've examined Case 00002121 (opened 2024-11-28, closed 2025-01-24) over the past ~7 weeks, it has contained no mention of CP/CPS v7.04; and in light of the confirmed non-existence of the alleged CCADB bug, several of DigiCert's actions in response to my CPR start to look suspicious.
The evidence shows that DigiCert's CP/CPS v7.04 was eventually disclosed to CCADB for the first time in Case 00002192 (opened 2025-01-10, closed 2025-01-14), 52 days after that document was published (2024-11-19), in contravention of:
-
CRPP 6.1: 'When a timeline is not defined for a requirement specified within the CCADB Policy, updates MUST be submitted to the CCADB within 14 calendar days of being completed.'
-
MRSP 7.3: 'A failure to provide notifications or updates in the CCADB or as otherwise required in a timely manner SHALL also be grounds for disabling a CA operator's root certificates or removing them from Mozilla's root store. For this policy and the CCADB policies, "a timely manner" means within 30 days of when the appropriate data or documentation becomes available to the CA operator, unless a Mozilla policy document specifies a different rule.'
Q1: Does DigiCert accept that the ALV issues that delayed processing of Case 00002121 are actually not the root cause of this incident?
Q2: Can DigiCert explain how its repeated claim that CP/CPS v7.04 was disclosed in a CCADB Case in a timely fashion is consistent with the evidence I've presented to the contrary?
Wherever there is evidence of a CA Owner making inconsistent or conflicting statements, or of a CA Owner making claims that are at odds with the available evidence, it is reasonable to wonder if that CA Owner is being entirely truthful. Since the WebPKI relies on the trustworthiness of CA Owners, in my view it is important to follow such evidence wherever it leads.
I can certainly imagine a CA Owner being tempted to conceal an incident that is "grounds for disabling a CA operator's root certificates or removing them from Mozilla's root store", and I hope that you are able to reassure me that this isn't what has happened. I know you've said previously that you "don't think implying that people are being deliberately disingenuous is necessary or appropriate", so please help me understand the explanation here.
|
|
Assignee | |
Comment 13•11 months ago
|
||
In response to Rob Stradling in comment 12:
The ">365 days document" was correctly included in the AllCertificateRecordsCSVFormatv2 CCADB report because it was still an authoritative non-audit document to which DigiCert was required to 'strictly adhere', and what "Sectigo was reporting" at https://crt.sh/mozilla-disclosures was a faithful representation of that accurate CCADB report.
Correct. If you referenced AllCertificateRecordsCSVFormatv2 at the time, our 13 Roots were showing CP/CPS documents older than 365 days. We’d already uploaded the latest CP/CPS to CCADB. The CPR provided did not make sense as the documentation was present on CCADB’s interface. We did not realize that CAs needed to mark older documents as 'superseded'. This incident makes that requirement clear.
Firstly, it appears that Case 00002121 did not delete the outdated non-audit documents or mark them as superseded; in which case, even if the ALV error had been resolved and that Case had been successfully processed soon after it was opened (2024-11-28), those outdated documents would still have been considered authoritative and so the underlying problem would have remained. (I believe that Case 00002191, entitled "Archive old CPCPS docs" and created about 2 hours after I sent the CPR, was responsible for deleting/superseding the outdated DigiCert CP and CPS documents; however, the CCADB Case UI doesn't seem to clearly show which details were changed by a Closed Case, so I'm partly relying on my memory of what I observed on Case 00002121 when I reviewed it on 2025-01-10 whilst it was still Open).
While investigating the CPR, we created both cases 00002191 and 00002190. Neither of these resulted in changes and were closed without CCADB edits. The two cases were created as research to ensure we did not overlook any information in case 00002121. These two cases can be deleted as unnecessary,
Q1: Does DigiCert accept that the ALV issues that delayed processing of Case 00002121 are actually not the root cause of this incident?
Yes. Upon further reflection, even if case 00002121 had passed ALV, your query would still have showed the old information as the documents weren't marked as 'superseded'. We updated our SOP to align with the requirement to mark these documents as 'superseded' based on the discussion here: https://groups.google.com/a/ccadb.org/g/public/c/CIR6vB52Z-g/m/MBJpUz80BgAJ.
Q2: Can DigiCert explain how its repeated claim that CP/CPS v7.04 was disclosed in a CCADB Case in a timely fashion is consistent with the evidence I've presented to the contrary?
After receiving the CPR, we researched your claim and reached out to CCADB for assistance. Mozilla informed us on 2025-01-31 at 16:46 that we had an issue with the upload.
The CP/CPS was uploaded to CCADB and correctly reflected on the CCADB web portal. However but not, AllCertificateRecordsCSVFormatv2. Because we could see the document on the CCADB webportal, we thought the info at https://crt.sh/mozilla-disclosures was wrong.
We appreciate you reaching out and bringing the issue between the Web portal and AllCertificateRecordsCSVFormatv2 to our attention.
|
|
Assignee | |
Comment 14•11 months ago
|
||
As there have been no additional comments or questions, all action items are complete, and we've posted our Closing Summary in comment 9, @Ben, we again request closure of this bug.
|
|
||
Comment 15•11 months ago
|
||
(In reply to DigiCert from comment #13)
We’d already uploaded the latest CP/CPS to CCADB. The CPR provided did not make sense as the documentation was present on CCADB’s interface.
The timeline in comment 1 seems to imply that Case 00002121 is where DigiCert would have seen details for CP/CPS v7.04. However, as I have already laid out in comment 12, Case 00002121 has not shown any details of CP/CPS v7.04 on any occasion that I have examined it. So I must ask:
Q3: Where "on CCADB's interface" is DigiCert claiming that "the documentation" for "the latest CP/CPS" (v7.04) "was present" at the time you received my CPR (2025-01-10)?
While investigating the CPR, we created both cases 00002191 and 00002190. Neither of these resulted in changes and were closed without CCADB edits. The two cases were created as research to ensure we did not overlook any information in case 00002121. These two cases can be deleted as unnecessary,
That's true for Case 00002190 ('Adhoc request to adjust records'), which shows a note that 'This Case is Closed. Reason for Close without Sync: Superseded by Case #2121'. However, I did not mention Case 00002190 in my rationale in comment 12 that you were responding to.
In contrast, the 'Case Progress' tab of Case 00002191 ('Archive old CPCPS docs') shows the 'Non-Audit Documents' Status as 'Data Synced'. Therefore, I don't agree that Case 00002191 was "closed without CCADB edits" and I don't believe it "can be deleted as unnecessary". I still believe that Case 00002191 was responsible for deleting/superseding the outdated DigiCert CP and CPS documents.
Q1: Does DigiCert accept that the ALV issues that delayed processing of Case 00002121 are actually not the root cause of this incident?
Yes. Upon further reflection, even if case 00002121 had passed ALV, your query would still have showed the old information as the documents weren't marked as 'superseded'. We updated our SOP to align with the requirement to mark these documents as 'superseded' based on the discussion here: https://groups.google.com/a/ccadb.org/g/public/c/CIR6vB52Z-g/m/MBJpUz80BgAJ.
In comment 5 you concluded your initial investigation into the root cause by writing "Sounds like the primary issue was a failure of our team to run ALV before submitting, fixing the ALV errors, and then resubmitting. Would you like an updated incident report that includes this information?". You then provided a corresponding Root Cause Analysis (comment 8), and you stood behind that Root Cause Analysis in your Closure Summary (comment 9).
After then agreeing in comment 13 that this Root Cause Analysis was flawed, it's surprising that DigiCert's very next move (comment 14) was to request closure of this incident bug based on the flawed "Closing Summary in comment 9".
Q4: Will DigiCert now provide an updated Root Cause Analysis?
Q2: Can DigiCert explain how its repeated claim that CP/CPS v7.04 was disclosed in a CCADB Case in a timely fashion is consistent with the evidence I've presented to the contrary?
After receiving the CPR, we researched your claim and reached out to CCADB for assistance. Mozilla informed us on 2025-01-31 at 16:46 that we had an issue with the upload.
It's not yet clear to me if/how this information is relevant to my question Q2.
Comment 9 notes the current lack of automation for "the CCADB upload process", so I presume you are referring here to "an issue" with a manually created CCADB Case.
Q5: Which CCADB Case are you referring to and what was the "issue" with it?
The CP/CPS was uploaded to CCADB and correctly reflected on the CCADB web portal.
This occurred in Case 00002192 (opened 2025-01-10, closed 2025-01-14), so yes, on 2025-01-31 at 16:46 you would have observed CP/CPS v7.04 "correctly reflected on the CCADB web portal". However, as I already mentioned in comment 12, 2025-01-10 was 52 days after CP/CPS v7.04 was published, which was far too late for "in a timely fashion".
If you are still telling us that CP/CPS v7.04 was disclosed to CCADB in a timely fashion (i.e., within 14 days of that document’s publication), then I can only hope that by answering my question Q3 (above) you will show us how that is the case. At present, I don’t believe you've meaningfully answered my question Q2.
However but not, AllCertificateRecordsCSVFormatv2. Because we could see the document on the CCADB webportal, we thought the info at https://crt.sh/mozilla-disclosures was wrong.
Q6: Are you referring here to your observations on 2025-01-31 at 16:46, or to your observations some weeks earlier when your v6.x documents had not yet been marked as Superseded?
I am trying to understand what you saw, where you saw it, and (crucially, because deadlines matter) when you saw it.
We appreciate you reaching out and bringing the issue between the Web portal and AllCertificateRecordsCSVFormatv2 to our attention.
Prior to both Mozilla’s rebuttal of DigiCert's mistaken claim of a 'bug in CCADB limiting the number of entries in the "Non-Audit documents" tab' and DigiCert's realisation of the need to mark old "Non-Audit documents" as Superseded, I can understand why DigiCert might have believed that there was an "issue between the Web portal and AllCertificateRecordsCSVFormatv2".
But for the record, I did not bring any "issue between the Web portal and AllCertificateRecordsCSVFormatv2" to DigiCert's attention, and I am not aware of any such issue ever having existed.
|
|
Assignee | |
Comment 16•11 months ago
|
||
Rob, we are reviewing your questions in comment 15 and will post answers tomorrow.
|
|
Assignee | |
Comment 17•11 months ago
|
||
In response to Rob Stradling in comment 15
Q3: Where "on CCADB's interface" is DigiCert claiming that "the documentation" for "the latest CP/CPS" (v7.04) "was present" at the time you received my CPR (2025-01-10)?
CPS v7.04 was present in our audit statement case. The CPS was moved to a separate case for processing to separate the CPS upload from the audit statements. We did not create new records. Instead, we edited the existing Non-Audit ID record with the latest CPS.
In CCADB’s interface, the 13 Roots listed the CPS v7.03 and not the > 365 day old CP/CPS that you referenced in your CPR. We reached out to CCADB to understand why you were seeing a different CPS than we were.
In contrast, the 'Case Progress' tab of Case 00002191 ('Archive old CPCPS docs') shows the 'Non-Audit Documents' Status as 'Data Synced'. Therefore, I don't agree that Case 00002191 was "closed without CCADB edits" and I don't believe it "can be deleted as unnecessary". I still believe that Case 00002191 was responsible for deleting/superseding the outdated DigiCert CP and CPS documents.
We didn’t mean to apply changes using with Case “00002191". Per CCADB’s recommendation, we used case 0002192 to combine updating the CPS (removing it from its previous case) and archive non-audit documents.
Case "00002192" is relevant over “0002191” as it solved the problems encountered when filing “00002121”
Q4: Will DigiCert now provide an updated Root Cause Analysis?
If the RCA needs more information, then we can update it.
Specifically, we can add the following: A DigiCert employee did not perform the ALV scan, did not update the CP/CPS correctly, did not mark the previous non-audit documents as "Superseded" and did not respond to the case notification to resolve the errors properly. The error went to the individual employee’s email while they were on sabbatical. Because the email only went to the one individual, our other PoC CCADB users were unaware of the ALV failures resulting in a delay in updating the policy documents in CCADB.
On Feb 7th, we updated our SOP to mark outdated documents as "superseded." This is newer functionality in CCADB (compared to the SOP) so the step of marking documents as superseded was not included. We also updated the SOP to require two manual reviews of each case. We will automate the upload and outdating process when such functionality is permitted in CCADB.
Q5: Which CCADB Case are you referring to and what was the "issue" with it?
We do not understand your question. Please clarify.
If you are still telling us that CP/CPS v7.04 was disclosed to CCADB in a timely fashion (i.e., within 14 days of that document’s publication), then I can only hope that by answering my question Q3 (above) you will show us how that is the case. At present, I don’t believe you've meaningfully answered my question Q2.
Do you mean that our non-audit documents were not approved and did not mark old non-audits documents superseded within 14 days? If so, then we agree.
Q6: Are you referring here to your observations on 2025-01-31 at 16:46, or to your observations some weeks earlier when your v6.x documents had not yet been marked as Superseded?
We were notified about the old CPS documents needing to be marked superseded via your CPR.
|
|
Assignee | |
Comment 18•10 months ago
|
||
All questions were answered and a closing summary was already provided.
|
|
||
Comment 19•10 months ago
|
||
I'll close this on or about Wed. 2-Apr-2025.
|
|
||
Updated•10 months ago
|
|
|
||
Comment 20•10 months ago
|
||
(In reply to DigiCert from comment #17)
In response to Rob Stradling in comment 15
The timeline in comment 1 seems to imply that Case 00002121 is where DigiCert would have seen details for CP/CPS v7.04. However, as I have already laid out in comment 12, Case 00002121 has not shown any details of CP/CPS v7.04 on any occasion that I have examined it. So I must ask:
Q3: Where "on CCADB's interface" is DigiCert claiming that "the documentation" for "the latest CP/CPS" (v7.04) "was present" at the time you received my CPR (2025-01-10)?CPS v7.04 was present in our audit statement case.
This can only be a reference to Case 00002121 ("DigiCert Annual Webtrust audit"). It seems we're going around in circles.
As I have already said, and will say again, CP/CPS v7.04 was not present on Case 00002121 on any occasion that I have examined it. The first time I examined Case 00002121 was on 2025-01-10 whilst I was putting together the CPR that I sent to DigiCert later that day. Therefore, I do not accept that CP/CPS v7.04 was "present in (y)our audit statement case" at the time you received my CPR.
The CPS was moved to a separate case for processing to separate the CPS upload from the audit statements. We did not create new records. Instead, we edited the existing Non-Audit ID record with the latest CPS.
DigiCert originally tried to attribute the late disclosure of CP/CPS v7.04 on its Root Certificate records to (irrelevant) ALV failures that delayed the processing of Case 00002121 along with a (bogus) claim about 'CCADB limiting the number of entries in the "Non-Audit documents" tab'. By making that claim in response to my CPR, DigiCert clearly accepted that CP/CPS v7.04 was not visible at that time in the "Non-Audit Documents" tab of that Case. Coupled with Mozilla's confirmation that the alleged CCADB bug does not and did not exist, we can be confident that CP/CPS v7.04 had in fact not been submitted in that Case by the time I sent the CPR.
So I find it incredible that DigiCert now expects us to believe that CP/CPS v7.04 was "present in our audit statement case" and then "moved to a separate case" (Case 00002192) that was created in response to my CPR. Whilst this story might sound plausible at face value, and perhaps DigiCert really has convinced itself that it's true, it simply is not what I observed. For CP/CPS v7.04 to have been "moved", it would first have needed to have actually been present in the place from where it was allegedly moved!
It's clear to me that Case 00002192 is the earliest CCADB Case that disclosed CP/CPS v7.04. As I pointed out in comment 15, Case 00002192 was opened 52 days after publication of CP/CPS v7.04, which (as I pointed out in comment 12) exceeds the CRPP and MRSP timelines for CCADB disclosures.
I first drew DigiCert's attention to the absence in CCADB of CP/CPS v7.04 when I sent the CPR on 2025-01-10. Nearly three months later, DigiCert is still inventing excuses rather than accepting that this represents a late disclosure incident.
I am reopening this bug because I consider this matter to be far from closed. Whilst I have no new questions for DigiCert, I am not satisfied with the answers DigiCert has provided to several of the questions I've asked so far.
Q5: Which CCADB Case are you referring to and what was the "issue" with it?
We do not understand your question. Please clarify.
You wrote "Mozilla informed us on 2025-01-31 at 16:46 that we had an issue with the upload". Q5 simply asked you to explain what you meant by that, since it was not at all clear.
If you are still telling us that CP/CPS v7.04 was disclosed to CCADB in a timely fashion (i.e., within 14 days of that document’s publication), then I can only hope that by answering my question Q3 (above) you will show us how that is the case. At present, I don’t believe you've meaningfully answered my question Q2.
Do you mean that our non-audit documents were not approved and did not mark old non-audits documents superseded within 14 days? If so, then we agree.
Both Q2 and Q3 sought to establish whether CP/CPS v7.04 really was, or conceivably could have been, submitted in a timely fashion. If it had been submitted in a timely fashion but "not approved" (i.e., submitted via a still-open Case), then that would have been compliant. However, the observations I've made and the evidence I've presented show that it was not submitted in a timely fashion. Your answers to Q2 and Q3 have not persuaded me to rethink my conclusion.
Q6: Are you referring here to your observations on 2025-01-31 at 16:46, or to your observations some weeks earlier when your v6.x documents had not yet been marked as Superseded?
We were notified about the old CPS documents needing to be marked superseded via your CPR.
That doesn't answer the question. Q6 asked you for a timestamp corresponding to your statement "Because we could see the document on the CCADB webportal, we thought the info at https://crt.sh/mozilla-disclosures was wrong".
(In reply to DigiCert from comment #18)
All questions were answered and a closing summary was already provided.
In comment 15, I wrote 'After then agreeing in comment 13 that this Root Cause Analysis was flawed, it's surprising that DigiCert's very next move (comment 14) was to request closure of this incident bug based on the flawed "Closing Summary in comment 9".’ In reply to my Q4, DigiCert offered some additional RCA text in comment 17 (albeit without a Markdown heading to clearly identify it as such), but DigiCert has not updated the closing summary.
What are the expectations of the root programs and the community in circumstances such as this?
When the RCA in an incident report turns out to be flawed, is it expected that a full, corrected incident report should be posted? Or that a full, corrected RCA should be posted? Or is a partial RCA that just contains new information enough? Or is there no expectation that a flawed RCA should be corrected?
When any part of a closing summary turns out to be flawed, is it expected that a full, corrected closing summary should be posted? Or not?
|
|
||
Comment 21•10 months ago
|
||
Based on Rob’s comments and the timeline of events, I agree that CP/CPS v7.04 was not disclosed promptly in the CCADB by DigiCert. This appears to have been due to either a misunderstanding of how the CCADB is intended to be used to update non-audit documents (such as CPs and CPSes) or a lack of diligence in ensuring timely disclosure. DigiCert’s belief that the update was prevented by the failure to mark older documents as “Superseded” was, in hindsight, incorrect. DigiCert noted in its analysis that “to the best of our understanding, the error was caused by DigiCert not marking older versions as ‘Superseded’ in our Non-Audit Documents tab.” While this may have reflected DigiCert’s perspective at the time, it now seems clear that this explanation was not the true root cause. The focus on the “Superseded” issue unfortunately diverted attention from a more accurate understanding of the disclosure failure.
Although I believe DigiCert has acknowledged aspects of this mistake in other parts of the discussion, it would be helpful—and in the interest of closing this bug—to update the Root Cause Analysis accordingly. I recommend that DigiCert restate its Root Cause Analysis to clearly reflect the actual cause of the delayed disclosure, revise its Action Items based on that updated understanding, and when ready, file an updated Closure Summary that includes a revised section for “Incident Root Causes.”
Also, the revised Root Cause Analysis that DigiCert provides should align with the current guidance in version 3.0 of the CCADB Incident Reporting Guidelines, which uses the following template:
Contributing Factor #: title
- Description:
- Timeline:
- Detection:
- Interaction with other factors:
- Root Cause Analysis methodology used:
Thanks all,
Ben
|
|
Assignee | |
Comment 22•10 months ago
|
||
In response to Rob Stradling in comment 20:
This can only be a reference to Case 00002121 ("DigiCert Annual Webtrust audit"). It seems we're going around in circles.
As I have already said, and will say again, CP/CPS v7.04 was not present on Case 00002121 on any occasion that I have examined it. The first time I examined Case 00002121 was on 2025-01-10 whilst I was putting together the CPR that I sent to DigiCert later that day. Therefore, I do not accept that CP/CPS v7.04 was "present in (y)our audit statement case" at the time you received my CPR.
It does seem that we’re going around in circles. In comment 12 you stated that “…the CCADB Case UI doesn't seem to clearly show which details were changed by a Closed Case, so I'm partly relying on my memory of what I observed on Case 00002121 when I reviewed it on 2025-01-10 whilst it was still Open).”
When the bug was opened, we confirmed that CP/CPS v7.04 was present on Case 00002121. We recall it being there because we also recall removing it shortly thereafter in order to get it processed on its own (cases 00002190-2192). In the absence of clear contemporaneous evidence we can only conclude that we remember this differently. We believe your recollection based on what we believe CCADB showed at the time, but it’s possible you missed it the first time you looked, or you looked after we’d already removed it. We should have taken a screenshot of the listing.
DigiCert originally tried to attribute the late disclosure of CP/CPS v7.04 on its Root Certificate records to (irrelevant) ALV failures that delayed the processing of Case 00002121 along with a (bogus) claim about 'CCADB limiting the number of entries in the "Non-Audit documents" tab'. By making that claim in response to my CPR, DigiCert clearly accepted that CP/CPS v7.04 was not visible at that time in the "Non-Audit Documents" tab of that Case. Coupled with Mozilla's confirmation that the alleged CCADB bug does not and did not exist, we can be confident that CP/CPS v7.04 had in fact not been submitted in that Case by the time I sent the CPR.
We take exception to your characterization of our statement as “bogus.” Conclusory and dismissive language like this is against the values of this forum and does not help foster collaboration to resolve issues raised in this bug report. We are providing the information we have, which is that the CP/CPS was uploaded when we created case 00002121. We cannot comment on what CCADB is doing because we do not have access to CCADB or the source code. We accepted that you could not see our CP/CPS. Do you have screenshots of Case 0002121 before we separated into the other cases? If so, could you please provide these so we can investigate the difference in what we saw vs. what you saw?
We are not contesting that a mistake was made in getting the case properly processed in CCADB; in fact, we believe two mistakes led to this bug. First, ALV failures went unnoticed by the person who uploaded the audit and CPS. The person then left on vacation and failed to rectify the ALV failure. Second, we did not mark documents as superseded which likely led you to believe that more than a year had passed since we updated the CP/CPS. We update our CP/CPS much more frequently than that. Your comment suggests that we are not being fully transparent, but we assure you that all of the information we have about this issue is provided in this bug.
Both Q2 and Q3 sought to establish whether CP/CPS v7.04 really was, or conceivably could have been, submitted in a timely fashion. If it had been submitted in a timely fashion but "not approved" (i.e., submitted via a still-open Case), then that would have been compliant.
Your comment is again based on our difference in recollection. In the absence of clear evidence to set the record straight we cannot be absolutely sure one way or the other, but our recollection was that CP/CPS v7.04 was attached to Case 00002121. Our original RCA, which reflected our understanding at the time, mis-states the reasons that case 00002121 was not processed. Our DBE team was tasked with figuring out what happened when we first understood there was an issue. That same team removed the CPS from 00002121 on January 10th in order to try to resolve your issue in viewing the correct CPS version. Regarding the three cases, 00002190, 00002191 and 00002192, the first two were created to both mark old documents as superseded and upload the v7.04 CP/CPS as separate cases. As our DBE was working on those cases, he received feedback that he could perform both those actions in a single case and created 00002192. The understanding we had was that 00002190 and 00002191 would be closed w/out processing. However, this did not happen. Our belief (again, with limited visibility into how CCADB works behind the scenes) is that because 00002192 was processed after 00002191, the part of 00002192 that overlapped 00002191 would not have been processed a second time and resulted in the view you believe you saw.
That doesn't answer the question. Q6 asked you for a timestamp corresponding to your statement "Because we could see the document on the CCADB webportal, we thought the info at https://crt.sh/mozilla-disclosures was wrong".
We did not capture screenshots of what we saw in the web portal. The screenshot we have is the one provided in the original Incident Report. There were two things we saw in the web portal:
- v7.04 was attached to Case 00002121 when reviewed on January 10th.
- We saw v7.03 (as well as the older versions previously uploaded to CCADB at various times prior to all of this) in CCADB. CPS v7.03 was correctly reflected in the CPS field of the web interface when viewing our root certificates.
This fact that CPS v7.03 was listed as the CPS for our root certificates in the web interface is cause for confusion in trying to figure out the CPR you provided. The web interface did not show version 6.x.x.
In comment 15, I wrote 'After then agreeing in comment 13 that this Root Cause Analysis was flawed, it's surprising that DigiCert's very next move (comment 14) was to request closure of this incident bug based on the flawed "Closing Summary in comment 9".’ In reply to my Q4, DigiCert offered some additional RCA text in comment 17 (albeit without a Markdown heading to clearly identify it as such), but DigiCert has not updated the closing summary.
What are the expectations of the root programs and the community in circumstances such as this?
When the RCA in an incident report turns out to be flawed, is it expected that a full, corrected incident report should be posted? Or that a full, corrected RCA should be posted? Or is a partial RCA that just contains new information enough? Or is there no expectation that a flawed RCA should be corrected?
When any part of a closing summary turns out to be flawed, is it expected that a full, corrected closing summary should be posted? Or not?
We believed the discussion throughout the bug clarified the issue and that the root cause was addressed. We have no problem providing an updated RCA and Incident Closure Summary. We will begin w/the revised RCA and post it early next week. Once the community has had a chance to review the revised RCA, we’ll draft the Closure Summary.
|
|
Assignee | |
Comment 23•10 months ago
|
||
As per request from Ben Wilson in comment 21 please see below our revised RCA for this bug:
Contributing Factor 1: Delayed Submission of CP/CPS v7.04 Due to Unresolved ALV Errors
- Description: DigiCert included CP/CPS v7.04 in Case 00002121, created on 2024-11-28, to update audits and CPS. However, the case was not properly submitted due to ALV errors, which were only known to the employee who created the case. This employee went on vacation without seeing or resolving the ALV errors, resulting in CP/CPS v7.04 not being disclosed in the CCADB within the required 14-day timeline per CRPP 6.1. On 2025-01-10, upon realizing the issue via Sectigo’s CPR, DigiCert removed v7.04 from Case 00002121 and submitted it separately in Case 00002192 to ensure proper processing.
- Timeline:
- 2024-11-19: CP/CPS v7.04 published by DigiCert.
- 2024-11-28: Case 00002121 created to update audits and include CP/CPS v7.04.
- 2024-12-02: ALV errors identified; case reverted to "CA Providing Data" for resolution, but notification was only sent to the employee who was on vacation.
- 2025-01-10 16:06: Sectigo notified DigiCert via CPR about outdated CPS entries.
- 2025-01-10 21:18: DigiCert removed CP/CPS v7.04 from Case 00002121 and created Case 00002192 to submit v7.04 separately.
- 2025-01-14: Case 00002192 approved, disclosing CP/CPS v7.04.
- Detection: The issue was detected on 2025-01-10 when Sectigo’s CPR highlighted that the "AllCertificateRecordsCSVFormatv2" file showed an outdated CP/CPS (v6.04) for 13 Roots. Investigation revealed that ALV errors had prevented Case 00002121 from being processed, and these errors were unknown to other team members due to the employee’s absence.
- Interaction with other factors: The delay was exacerbated by inadequate internal procedures (Contributing Factor 2), which failed to ensure notifications reached multiple team members. Additionally, a lack of awareness about marking older documents as "Superseded" (Contributing Factor 3) contributed to the outdated CPS being displayed in the "AllCertificateRecordsCSVFormatv2" file.
- Root Cause Analysis methodology used: Analysis of CCADB case logs, review of Bugzilla comments, and comparison with CRPP 6.1 and MRSP 7.3 requirements.
Contributing Factor 2: Inadequate Internal Procedures for CCADB Case Management
- Description: CCADB did not allow case notifications, such as ALV error alerts, to be sent to multiple team members or a shared alias. This allowed the ALV errors in Case 00002121 to go unnoticed when the employee responsible went on vacation. Additionally, the SOPs did not require dual review of cases to ensure errors were addressed, contributing to the delayed disclosure of CP/CPS v7.04.
- Timeline:
- Pre-incident: SOPs lacked requirements for shared notifications or dual case reviews.
- 2024-12-02: ALV errors in Case 00002121 went unnoticed due to the employee’s absence.
- 2025-01-10: Issue identified during CPR investigation, highlighting procedural gaps.
- 2025-02-07: SOPs updated to require dual review and shared notification processes.
- Detection: Recognized on 2025-01-10 during the CPR investigation, with procedural gaps confirmed through internal process review.
- Interaction with other factors: The lack of robust procedures directly enabled the delayed submission (Contributing Factor 1) by failing to ensure ALV errors were addressed. It also interacted with the lack of awareness about "Superseded" marking (Contributing Factor 3), as there were no checks to verify the accuracy of displayed documents.
- Root Cause Analysis methodology used: Internal audit of SOPs, examination of case handling records, and assessment against CCADB Policy requirements.
Contributing Factor 3: Lack of Awareness of CCADB "Superseded" Document Requirement
- Description: DigiCert was unaware of the need to mark older non-audit documents as "Superseded" in the CCADB to ensure only the latest CPS was reflected in the "AllCertificateRecordsCSVFormatv2" file. This led to outdated CP/CPS versions (e.g., v6.04) being displayed, contributing to the perception of non-compliance reported in the CPR.
- Timeline:
- Pre-2024-11-19: DigiCert unaware of "Superseded" marking requirement introduced in the October 2024 CCADB update.
- 2025-01-10: CPR investigation highlighted outdated CPS versions in "AllCertificateRecordsCSVFormatv2."
- 2025-01-31: Mozilla clarified no CCADB bug existed, emphasizing the need to mark documents as "Superseded."
- 2025-02-07: SOPs updated to include "Superseded" marking.
- Detection: Identified on 2025-01-10 during the CPR investigation, further clarified by Mozilla on 2025-01-31.
- Interaction with other factors: This lack of awareness exacerbated the impact of the delayed submission (Contributing Factor 1) by allowing outdated documents to remain authoritative. It was compounded by inadequate procedures (Contributing Factor 2), which did not include steps to verify document status in the CCADB.
- Root Cause Analysis methodology used: Review of CCADB documentation, public@ccadb.org discussions (e.g., August 2024 thread), and community feedback in Bugzilla.
|
|
Assignee | |
Comment 24•9 months ago
|
||
Report Closure Summary
-
Incident description: DigiCert included CP/CPS v7.04 in Case 00002121, created on 2024-11-28, to update audits and CPS. However, the case was not properly submitted due to ALV errors, which were unknown to the employee who created the case. This employee went on vacation without seeing or resolving the ALV errors, resulting in CP/CPS v7.04 not being disclosed in the CCADB within the required 14-day timeline per CRPP 6.1. On 2025-01-10, upon realizing the issue via Sectigo’s CPR, DigiCert removed v7.04 from Case 00002121 and submitted it separately in Case 00002192 to ensure proper processing.
-
Incident Root Cause(s): The new CPS failed to upload properly to CCADB due to three issues. 1) The case created in CCADB had ALV scan errors. However, the employee uploading the CPS did not notice the errors before leaving on vacation. 2) Staff lacked knowledge that older CPS versions needed to be marked as “superseded”. 3) DigiCert had a gap in the SOP for CCADB management. Only one person reviewed CPS uploads to CCADB. This should have been multi-party. The SOP did not contain an instruction to mark old CPS documents as “superseded” which led to the old CPS showing up in Rob’s report but available in the web interface.
-
Remediation description: We’ve updated our SOPs to require multiple people to review a case and to mark outdated documents as “superseded’. In addition, we’ve enhanced our automated CCADB checks to verify the CPS version number and the CPS URL. Previous checks verified that the timeframe for a new CPS did not exceed 365 days. We also submitted an enhancement request to CCADB to have notifications about a case sent to all CA’s POCs rather than just the person who submitted the case.
-
Commitment summary: Our commitment is to a) continue automating against CCADB to the maximum extent possible, b) have multiple people reviewing uploads to CCADB, and c) mark old CPS documents as “superseded”. We also commit to automating the CCADB upload process when the CCADB APIs are available for updating roots (creating cases).
All Action Items disclosed in this report have been completed as described, and we request its closure.
|
|
||
Updated•9 months ago
|
|
|
Assignee | |
Comment 25•9 months ago
|
||
We have posted our closing summary. As we have no additional comments at this time and there have been no additional comments or questions from the community we request that this bug be closed, or please set an appropriate nextUpdate date.
|
|
||
Updated•9 months ago
|
|
||
Comment 26•9 months ago
|
||
This is a final call for comments or questions on this Incident Report.
Otherwise, this bug will be closed on approximately 2025-05-08.
|
||
Updated•9 months ago
|
|
|
||
Comment 27•9 months ago
|
||
(In reply to DigiCert from comment #22)
Do you have screenshots of Case 0002121 before we separated into the other cases? If so, could you please provide these so we can investigate the difference in what we saw vs. what you saw?
No, sadly, I don't have screenshots.
(In reply to Rob Stradling from comment #12)
For the details, see these snapshots of the mozilla-disclosures (2025-01-19) and AllCertificateRecordsCSVFormatv2 (2025-01-10) reports.
I took these snapshots to document the non-compliant CCADB disclosures that I reported in the CPR, but of course neither of these sources cover CCADB Cases.
Nevertheless, I have been able to unearth first-hand evidence of what DigiCert saw at the time. Here is DigiCert's initial response to my CPR, reproduced in full, with emphasis mine:
Subject: RE: Problem Report(s) - Outdated DigiCert CP and CPS information in CCADB records
Date: Fri, 10 Jan 2025 22:29:59 +0000 (GMT)Hello Rob,
We genuinely appreciate your email reporting this to us. It emphasizes one of the limitations in CCADB. To be more specific, it appears that whenever the “Non-Audit Documents” section of a new case is heavily populated with too many entries, not all the Doc-IDs are displayed. I suspect there is a limit to how many documents can be shown on the page.
This considerable number of Docs (Non-Audited Document IDs) on our end is because we had an entry for each trust bit case, and it resulted in an excess of documents. Hence, the CAs you referred to did indeed contain our most recent CP/CPS but the link was not properly shown. The system takes the documents to be in an ascending order, and then it truncates after a certain number of characters in the CP/CPS URL field, the list is cut-off, instead of being in a descending order which would make the most recent CP/CPS appear first. It would have made more sense to use descending order sorting and eliminate some of the problems.
As you are aware, we do not only release CP/CPS updates once every year. After 6.05 we released 7 versions. However, for the 13 Roots you referenced this truncation results in version 6.05 remaining at the top.
We are in touch with Mozilla, and they are assisting us in making sure all the old Doc-IDs are marked with “Superseded” tags to hide them. This will ensure that only the newest version of the CP/CPS document remains available in the future.
Thank you again for raising this matter. We are actively working to resolve it. You will know this issue is resolved when you can see v7.04 reflected across all our roots.
Kind Regards,
Technical Support
This shows that DigiCert's initial observation when handling my CPR was that its “most recent CP/CPS” (CP/CPS v7.04) was ”not properly shown” (on Case 00002121, or indeed on any other Case). It also shows DigiCert's initial belief that this was due to "limitations in CCADB" - a belief that Mozilla subsequently refuted.
Given that we now know that no such CCADB bug existed, DigiCert's confirmation that the link was “not properly shown” makes it hard to conclude anything other than that CP/CPS v7.04 simply had not yet been submitted to CCADB (in Case 00002121 or in any other Case).
Therefore, this initial CPR response from DigiCert confirms my recollection and aligns with my claims and evidence in comment 12, comment 15, and comment 20.
So let's rewind. At the risk of sounding like a broken record, I'm going to ask this question one more time:
Q2: Can DigiCert explain how its repeated claim that CP/CPS v7.04 was disclosed in a CCADB Case in a timely fashion is consistent with the evidence I've presented to the contrary?
|
|
Assignee | |
Comment 28•9 months ago
|
||
In response to comment 27
Thank you Rob for your interest in this subject.
So let's rewind. At the risk of sounding like a broken record, I'm going to ask this question one more time:
We reiterate that neither of us had any screenshots of the CCADB at the time the CPR was sent. DigiCert’s focus is to prevent a recurrence of the issue. In doing so, we now capture screenshots and attach them to our tracking tickets for evidence of any changes made in CCADB. We believe this addresses the issue and any questions pertaining to it.
Q2: Can DigiCert explain how its repeated claim that CP/CPS v7.04 was disclosed in a CCADB Case in a timely fashion is consistent with the evidence I've presented to the contrary?
Our explanation is an accurate recounting of what occurred, and we don’t believe any new evidence has been introduced to the contrary. The Action Steps described in the Report Closure Summary submitted on 2025-04-23 are designed to avoid the recurrence of the issue.
|
|
||
Updated•9 months ago
|
|
|
Assignee | |
Comment 29•9 months ago
|
||
We have answered all open questions. Our answers to the previous set of questions did not result in any changes to the commitments made in our Report Closure Summary that was posted on 2025-04-23.
All remediation items have been addressed, and the Report Closure Summary has already been posted, so we request that this bug be closed.
|
|
||
Comment 30•9 months ago
|
||
I'll close this on 16-May-2025, unless there are any open discussion items.
|
|
||
Updated•9 months ago
|
|
|
||
Updated•9 months ago
|
|
|
||
Comment 31•9 months ago
|
||
Ben,
I intend to write a (hopefully) final comment on this incident, which I expect to post next week. Please don't close this bug yet. Thanks.
|
|
Assignee | |
Comment 32•9 months ago
|
||
We have posted our closing summary. As we have no additional comments at this time and there have been no additional comments or questions from the community we request that this bug be closed, or please set an appropriate nextUpdate date.
|
||
Comment 33•9 months ago
|
||
(In reply to DigiCert from comment #32)
We have posted our closing summary. As we have no additional comments at this time and there have been no additional comments or questions from the community we request that this bug be closed, or please set an appropriate nextUpdate date.
While the community knows you struggle to reply properly to most bugs - the comment made by Rob Stradling 4 (four) complet days ago has an intent to reply. That is not good for Digicert, but I hope Mozilla will keep the incident open until a respected community member has time to post the comment.
|
|
Assignee | |
Comment 34•8 months ago
|
||
Hi JR - The response provided was our standard response for situations where closing summary was provided and there aren't additional comments or questions to address. This response was provided to meet the CCADB requirement to post every seven days. Community members are (obviously) welcome to ask additional questions, which we will happily address.
|
|
||
Comment 35•8 months ago
|
||
"Thank you, Rob - we will await your questions before requesting this bug be closed. This post meets the requirement for us to post every 7 days."
I wrote a better reply for you. It did not even need the second sentence.
Posting comment 32 AFTER Rob Stradling's post 4 day earlier just shows once again how little respect Digicert has for this community.
|
|
||
Comment 36•8 months ago
|
||
|
|
||
Comment 37•8 months ago
|
||
[In response to Comment 35]
We welcome and deeply appreciate community participation in the Web PKI Incident Reporting process. However, we've noted the tone of this particular comment. While impassioned opinions are sometimes a part of these discussions, we urge everyone to communicate in a way that fosters constructive dialogue, as outlined in the CCADB Code of Conduct and Bugzilla Etiquette guidance. Accusatory language can make it harder to reach productive outcomes.
|
|
||
Comment 38•8 months ago
|
||
Apologies. I had not mean to violate any Code-Of-Conduct with my comment. I simply have frustration that, again, Digicert do not have respect for this forum and community. More when commenters are 'known' or from other CAs.
The intenzion of my post was to show how a simple and respecftul post could have been made.
I will make comments with more care now.
I hope Rob Stradling makes his comment soon so Digicert can respond.
|
|
||
Updated•8 months ago
|
|
|
||
Comment 39•8 months ago
|
||
|
|
||
Comment 40•8 months ago
|
||
I have attached my CPR (attachment 9490523 [details]) that led to this incident bug. Apart from that, I have no further evidence to present or questions to ask. However, I would like to present my own closing statement:
(In reply to DigiCert from comment #28)
In response to comment 27
...
We reiterate that neither of us had any screenshots of the CCADB at the time the CPR was sent. DigiCert’s focus is to prevent a recurrence of the issue.
My focus has been to try to persuade DigiCert to accept and own its mistakes that I reported in my CPR, but sadly I have only been partially successful.
The CPR notified DigiCert that its CCADB records included outdated (>1yr old) non-audit documents (for 13 Roots and at least 413 Intermediates) and did not include the then-current version of DigiCert's CP/CPS (v7.04). Although DigiCert eventually accepted that the outdated documents needed to be marked as Superseded in CCADB, it has consistently denied my claim that CP/CPS v7.04 was not disclosed to CCADB in a timely manner.
Our explanation is an accurate recounting of what occurred, and we don’t believe any new evidence has been introduced to the contrary.
Over the course of this incident, DigiCert has made the following two claims about the disclosure status of CP/CPS v7.04 at the time it received my CPR:
-
CP/CPS v7.04 was present on Case 00002121, but it was not visible due to an alleged CCADB bug.
-
CP/CPS v7.04 was present and visible on Case 00002121.
DigiCert insists that these two claims are not in conflict, whereas I insist that a non-audit document cannot simultaneously be both not visible and visible on the same CCADB Case.
I reviewed Case 00002121 prior to sending the CPR, and on several occasions since then. CP/CPS v7.04 was not visible on that Case on any of those occasions. This was confirmed by DigiCert's initial response to my CPR. Furthermore, Mozilla's rebuttal of the alleged CCADB bug confirmed that if a non-audit document is not visible, then it is in fact not present in the Case!
Only DigiCert knows the true motivations behind its false initial claim of a CCADB bug and its incompatible subsequent claim that CP/CPS v7.04 was present and visible on Case 00002121. I can only speculate, based on what DigiCert stood to gain if it could convince the community to believe either one of these two claims.
|
|
Assignee | |
Comment 41•8 months ago
|
||
In response to comment 36:
We appreciate the work that the Chrome team, along with the other root store operators, are doing to clarify and improve the CCADB Incident Reporting Guidelines and look forward to the final version.
|
|
||
Comment 42•8 months ago
|
||
This is a final call for comments or questions on this Incident Report.
Otherwise, it will be closed on approximately 2025-05-30.
|
|
Assignee | |
Comment 43•8 months ago
|
||
Thank you for providing a closing date. We have no additional comments at this time, so please close at your convenience.
|
|
||
Updated•8 months ago
|
|
|
||
Updated•8 months ago
|
Description
•