1942651 - Sectigo / SSL.com: Late disclosure of updated SSL.com CP/CPS to CCADB

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Rob Stradling]

Initial Incident Report

Summary

Sectigo has issued sixteen Cross-Certified Subordinate CA Certificates to SSL.com and disclosed them to CCADB. Sectigo is responsible for the entire PKI hierarchy under its Root Certificates, which includes an ongoing duty to keep the information in these sixteen CCADB records up to date.

SSL.com did not notify Sectigo promptly upon the publication of either v1.21 or v1.22 of the SSL.com CP/CPS. By the time Sectigo became aware, in both cases it was already too late for Sectigo to disclose the updated CP/CPS information to CCADB within the tightest disclosure deadline required by the root store policies.

Sectigo became aware that this needed to be treated as a compliance incident on 2025-01-17.

Sectigo will provide a full incident report no later than 2025-01-31.

Comment 1

[Rob Stradling]

Incident Report

Summary

Sectigo has issued sixteen Cross-Certified Subordinate CA Certificates to SSL.com and disclosed them to CCADB. Sectigo is responsible for the entire PKI hierarchy under its Root Certificates, which includes an ongoing duty to keep the information in these sixteen CCADB records up to date.

SSL.com published two CP/CPS versions and updated its own CCADB records within required timelines but failed to notify Sectigo promptly. By the time Sectigo became aware, in both cases it was already too late for Sectigo to disclose the updated CP/CPS information to CCADB within the tightest disclosure deadline required by the root store policies. Prior to this incident, Sectigo did not anticipate the need for automated systems to combat the potential for errors like these.

Impact

Sixteen CCADB records for Cross-Certified Subordinate CA Certificates issued by Sectigo to SSL.com contained outdated CP/CPS information for longer than permitted by the Chrome Root Program Policy. SSL.com, the CA Owner of the affected CAs, did not cease issuance during the incident.

Timeline

All times are UTC.

2024-04-18:

• Publication date of SSL.com's CP/CPS v1.20.

2024-04-19:

• SSL.com creates an Add/Update Root Request to disclose its CP/CPS v1.20 to CCADB.

2024-05-31:

• We enter into a contract with SSL.com to provide Cross-Certified Subordinate CA Certificates. This contract requires SSL.com to send timely notifications of audit and CP/CPS updates to us, so that we are always able to keep the corresponding CCADB records up-to-date and avoid any compliance incidents.

2024-06-21:

• 16:23 We issue sixteen Cross-Certified Subordinate CA Certificates to SSL.com.

2024-09-13:

• Publication date of SSL.com's CP/CPS v1.21. Sectigo is not notified.

2024-09-18:

• SSL.com creates an Add/Update Root Request to disclose its CP/CPS v1.21 to CCADB. Sectigo is not notified.

2024-09-30:

• 10:25 Whilst updating the audit information in the CCADB records for the sixteen Cross-Certified Subordinate CA Certificates, we take the opportunity to compare those records' CP/CPS information against SSL.com's own CCADB records. We consequently discover SSL.com's new CP/CPS v1.21.
• 10:41 We resolve to discuss and reiterate our expectations of SSL.com when representatives from both companies meet in person at the upcoming CABForum F2F in Seattle.
• 11:16 We update the CCADB records for the sixteen Cross-Certified Subordinate CA Certificates to reference SSL.com's CP/CPS v1.21. This is 17 days after publication of that document.

2024-10-08 - 2024-10-10:

• During the Seattle F2F, representatives from Sectigo and SSL.com meet in person. In informal discussion, we reiterate that for us to maintain compliance with root store requirements we depend on timely notification of CP/CPS updates.

2024-12-13:

• Publication date of SSL.com's CP/CPS v1.22. Sectigo is not notified.

2024-12-20:

• SSL.com creates an Add/Update Root Request to disclose its CP/CPS v1.22 to CCADB. Sectigo is not notified.

2025-01-02:

• 17:12 We happen to notice that SSL.com has updated its CP/CPS (to v1.22).
• 17:37 We begin to discuss alternative approaches to ensuring that we are informed of future changes to SSL.com's CP/CPS.

2025-01-07:

• 14:19 We update the CCADB records for the sixteen Cross-Certified Subordinate CA Certificates to reference SSL.com's CP/CPS v1.22. This is 25 days after publication of that document.

2025-01-08:

• 11:25 We contact SSL.com to explore better coordination between the two CAs and set up a regular call between the Sectigo and SSL.com teams to keep lines of communication open.

2025-01-17:

• 16:51 Whilst reviewing the various root store policies, we become aware that the Chrome Root Program Policy's "When a timeline is not defined for a requirement specified within the CCADB Policy, updates MUST be submitted to the CCADB within 14 calendar days of being completed" requirement applies to CP/CPS updates. We determine that the disclosures of SSL.com's CP/CPS v1.21 and v1.22 on the sixteen CCADB records controlled by Sectigo were therefore late, and that this qualifies as an incident.

2025-01-20:

• 15:43 We create this bug and post an Initial Incident Report.

2025-01-21:

• 13:08 We complete and deploy scripts that automatically monitor and alert on changes to the CP/CPS repository, audit seal website, and CCADB records of SSL.com's Root CAs.

2025-01-24:

• 16:30 The two CAs hold the first, regularly scheduled, coordination meeting.

Root Cause Analysis

On two occasions over the past 6 months, the SSL.com CP/CPS was updated without notification to Sectigo. By the time we independently discovered the existence of SSL.com's CP/CPS v1.21, 17 days had elapsed since that document's publication; and by the time we independently discovered the existence of SSL.com's CP/CPS v1.22, 25 days had elapsed since that document's publication.

Our initial review of the various root store policies and the CCADB Policy found that the Chrome Root Program Policy requires that "Chrome Root Program Participants...SHOULD ensure the updated versions of a CA's policy documents are submitted to the CCADB within 7 days of the policy documents being uploaded to their own publicly accessible repository". Since the CP/CPS in question belongs to SSL.com and is therefore not uploaded to Sectigo's publicly accessible repository, and since this is a "SHOULD" requirement rather than a "MUST", we did not feel that this requirement triggered an incident for Sectigo.

We also found that the MRSP explains and requires the following: 'A failure to provide notifications or updates in the CCADB or as otherwise required in a timely manner SHALL also be grounds for disabling a CA operator's root certificates or removing them from Mozilla's root store. For this policy and the CCADB policies, "a timely manner" means within 30 days of when the appropriate data or documentation becomes available to the CA operator, unless a Mozilla policy document specifies a different rule.' It's not clear to us whether the "CA operator" here is Sectigo or SSL.com, but either way it is clear that we did not exceed 30 days and that this requirement did not trigger an incident for Sectigo.

It was only in a more recent policy review that we noticed that the Chrome Root Program Policy (CRPP) also requires that "When a timeline is not defined for a requirement specified within the CCADB Policy, updates MUST be submitted to the CCADB within 14 calendar days of being completed." Whereas previously we'd concluded that the "SHOULD...7 days" requirement was all the CRPP had to say regarding timelines for disclosing CP/CPS updates to the CCADB, we now understand that "MUST...within 14 calendar days of being completed" also applies. We interpret "of being completed" to apply to the publication dates of each version of the SSL.com CP/CPS. Therefore, we believe that this CRPP requirement was not met, and so an incident has occurred for which Sectigo is ultimately responsible.

Sectigo contractually relies on SSL.com to provide timely notification of every update to SSL.com's CP/CPS and audits, to enable us to satisfy all root store requirements related to keeping information about Cross-Certified Subordinate CA Certificates up to date in the CCADB.

Relying on timely notifications through a manual process was a single point of failure. We were aware that this single point of failure was fragile, but only in the rigour of responding to this incident did we take the opportunity to supplement it with automation.

Lessons Learned

What went well

• We initially communicated to SSL.com the need to provide us with timely information on updates to CP/CPS and audit information.

• Upon determining that an incident had occurred, we were able to rapidly remove the single point of failure through automation.

What didn't go well

• The single point of failure failed, on more than one occasion.

• We are big believers in removing the possibility of human error wherever we can, but we did not identify the opportunity to shore up our process through automation until we determined that an incident had occurred.

Where we got lucky

• We independently discovered both updates to the relevant CP/CPS within 30 days of publication, meaning that this incident is not "grounds for disabling a CA operator's root certificates or removing them from Mozilla's root store".

Action Items

Action Item	Kind	Due Date
Setup a standing monthly call with SSL.com	Prevent	Done
Setup a cron job to monitor for any changes to the contents of the file available at the versionless SSL.com CP/CPS URL	Prevent & Detect	Done
Setup a GitHub Action to monitor for any changes to the contents of the file available at the versionless SSL.com CP/CPS URL and WebTrust Seal URLs	Prevent & Detect	Done
Setup a GitHub Action to monitor the CCADB's AllCertificateRecordsCSVFormatv2 report for any changes to the records for SSL.com's Root Certificates	Prevent & Detect	Done

Appendix

Details of affected certificates

None. This late CCADB disclosure incident did not directly impact any certificates.

Details of indirectly impacted certificates

The sixteen Cross-Certified Subordinate CA Certificates can be found here.

Details of affected CCADB records

These are the IDs/URLs of the sixteen CCADB records involved in this incident:

001TO00000BXYNxYAP
001TO00000BXXuvYAH
001TO00000BXTUzYAP
001TO00000BXRfpYAH
001TO00000BXJLiYAP
001TO00000BXEh0YAH
001TO00000BXPPWYA5
001TO00000BXX25YAH
001TO00000BXZ5WYAX
001TO00000BXAn4YAH
001TO00000BXMWUYA5
001TO00000BXH8fYAH
001TO00000BXGPRYA5
001TO00000BXOJlYAP
001TO00000BXMMnYAP
001TO00000BXLKIYA5

The links above require CCADB access. Interested parties that don't have access to the CCADB can find details of these CCADB records in the public AllCertificateRecordsCSVFormatv2 report. The CCADB record IDs are in the "Salesforce Record ID" column.

Comment 2

[Martijn Katerbarg]

Sectigo continues to monitor this bug.

We plan to post an Incident Report Closure Summary by this time next week, unless any questions are raised before then.

Comment 3

[Rebecca Kelley]

SSL.com would like to take this opportunity to address the community about our contributions leading up to this incident.

While we did open an add/update CCADB case to update our CP/CPS within the required time frame set by the root stores, we failed to notify Sectigo directly on two separate occasions about these updates.

After an internal investigation, we determined that we had not updated our processes to include notifying Sectigo of all CP/CPS updates. As a corrective action, we have updated our CP/CPS Change Management procedure along with scheduling regular coordination meetings between the two CAs.

Comment 4

[Martijn Katerbarg]

Incident Report Closure Summary

Incident Description:
Sixteen CCADB records for Cross-Certified Subordinate CA Certificates issued by Sectigo to SSL.com contained outdated CP/CPS information for longer than permitted by the Chrome Root Program Policy.

Incident Root Cause(s):
Sectigo contractually relies on SSL.com to provide timely notification of every update to SSL.com's CP/CPS and audits. Relying on timely notifications through a manual process was a single point of failure. We were aware that this single point of failure was fragile, but only in the rigour of responding to this incident did we take the opportunity to supplement it with automation.

Remediation Description:
We have setup a standing monthly call with SSL.com to keep in close contact and implemented 2 distinct methods of automated monitoring for updates to CCADB records and the SSL.com website.

Commitment Summary:
We intend to keep the standing monthly call until such time as the Cross-Certified Subordinate CA Certificates naturally expire or are revoked. Likewise we intend to keep automated checks in place for the monitoring of CP, CPS, CP/CPS and audit details for all current and potential future Cross-Certified Subordinate CA Certificates and Externally Operated CA Certificates issued by Sectigo, as outlined in our Action Items.

All Action Items disclosed in this Incident Report have been completed as described, and we request closure of this bug.

Comment 5

[Ben Wilson]

I intend to close this on Friday, 14-February-2025.