1945197 - Sectigo: Late receipt and disclosure to CCADB of ETSI audit letters

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Martijn Katerbarg]

Initial Incident Report

On 2025-01-30 at 15:11 while submitting details of our ETSI audit letters for our QWAC intermediate CAs, we were surprised to discover that our audit letters listed an audit period of less than 365 days. As a result of that unexpected audit period truncation, at the time of submission to CCADB more than 3 months / 92 calendar days had already elapsed since the audit period end date specified in the audit letter.

As instructed by the ALV tool, we have requested that our auditor reach out to the CCADB Root Store Members to explain why the final audit statement was not provided within 90 days of the Audit Period End date.

Since the late disclosure additionally violates both the Mozilla and Chrome Root Program Policies, we are filing this incident report.

We are currently working on the complete incident report, which will be posted no later than 2025-02-13.

Comment 1

[Martijn Katerbarg]

Incident Report

Summary

On 2025-01-30 at 15:11, while submitting details of our ETSI audit letters for our QWAC Subordinate CAs, we were surprised to discover that our audit letters listed an audit period of less than 365 days. As a result of that unexpected audit period truncation, at the time of submission to CCADB more than 3 months / 92 calendar days had already elapsed since the Audit Period End Date specified in the audit letter.

As instructed by the ALV tool, we have requested that our auditor reach out to the CCADB Root Store Members to explain why the final audit letters were not provided within 90 days of the Audit Period End Date. However, the late disclosure of audit letters violates both the Mozilla and Chrome Root Program Policies.

Impact

The CCADB records for 8 Subordinate CA Certificates did not receive details of updated audit letters within the required time.

Timeline

All times are UTC.

2024-09-18:

• 07:08 We receive the first version of the audit plan from our ETSI auditor. The Audit Period Start and End Dates are not stated in this plan.
• 07:22 We reply that the first version looks alright at first sight. We specify the need to include 4 new Subordinate CA Certificates intended for the issuance of QWAC TLS certificates within this year's audit.
• 11:30 We issue 4 new Subordinate CA Certificates intended for the issuance of QWAC TLS certificates.

2024-09-26:

• During an in-person event, we have an informal talk with our auditor during which we reiterate the need to include 4 new Subordinate CA Certificates within the scope of our ETSI audit.

2024-10-17:

• 11:42 We receive the final audit plan from our auditor. The audit plan shows documentation requirements including but not limited to our CA key ceremony protocol. The plan incorporates all expected elements for a regular audit, without specifying any specific CA certificates, new or old. The Audit Period Start and End Dates are not stated in this plan.

2024-10-21:

• 09:00 Our three-day on-site audit commences with three auditors present. After these three days, we provide further evidence on an ongoing basis as requested.

2024-11-15:

• 15:00 During our regularly scheduled WebPKI Incident Response (WIR) team call we discuss the ongoing ETSI audit and its progress. As we understand the audit period to have very recently ended, we do not identify any schedule risk at this time.

2024-12-13:

• 15:00 During our regularly scheduled WIR call we again discuss the ongoing ETSI audit and its progress. Based on our previous Audit Period End Date and the presumed agreed Audit Period End Date for our current audit, we calculate the latest date by which we will be required to submit new audit details to CCADB, in order to start tracking to ensure timely submission. The final date for submission to CCADB is set and tracked as February 7th.

2025-01-13:

• While preparing the audit letters, our auditor notices that the on-site audit visit was performed earlier than the presumed agreed Audit Period End Date, and that therefore, in their judgment, the actual Audit Period End Date needed to be brought forward by several weeks. Sectigo is not notified of this decision.

2025-01-15:

• 10:25 We receive the draft audit letters.

2025-01-16:

• 18:22 After a brief initial scan, we respond that the draft audit letters appear correct.

2025-01-17:

• 10:49 We receive version 1.0 of our audit letters. Team members receiving the letters are travelling, and based on our tracked submission deadline we do not deem it necessary to perform the CCADB submissions before the travelers return.

2025-01-21:

• 11:05 We share V1.0 of our audit letters internally for review and processing.
• 11:07 We discover our 4 new QWAC Subordinate CA Certificates are missing. We do not yet notice the shortened audit period.
• 11:54 We notify our auditor of our discovery. At this moment, since we have not noticed nor been informed of the shortened audit period, we believe we still have at least 2 weeks left before our CCADB disclosure deadline.

2025-01-30:

• 08:27 We receive an updated draft of our audit letters, version 1.1.
• 11:20 We respond that the updated draft look correct by a single person review, but that we need to perform a double check internally.
• 12:55 We receive the finalized version 1.1 audit letters.
• 15:11 We update the relevant CCADB records with the new audit letter details. CCADB’s ALV facility returns errors indicating that “The StatementDate ‘01/30/2025’ is not within 93 days after AuditPeriodEnd ‘10/20/2024’”, with the recommendation to “Have your auditor send email to the root store manager to explain why the audit statement was not provided within 90 days of the Audit Period End Date”.
• 15:17 We notice the audit letters state an audit period lasting from 2023-11-12 to 2024-10-20, rather than the expected 2023-11-12 to 2024-11-11. We calculate that the CCADB disclosure deadline for this shorter audit period has already passed.
• 15:26 We ask our auditor to explain why the audit period has been unexpectedly shortened.
• 16:22 Our auditor states that as the audit was performed prior to 2024-11-11, they cannot extend the audit period for a full year, despite some of the evidence having been provided after the on-site event.

2025-01-30:

• 08:44 We review the previous year’s audit letters and note that that audit period ended after the on-site audit, with a single date added as a remote audit date after the end of the audit period.
• 10:20 We have a call with our auditors. We point out that shortening that audit period was not deemed necessary, even though the circumstances were the same. We are notified that despite having provided evidence during a 3-month period, they are not able to class any such date as a remote audit date after the fact and will not be able to update the Audit Period End Date to reflect the presumed agreed 1 year audit period. Our auditor agrees to notify the CCADB Root Store Managers as instructed by CCADB.

2025-01-31:

• 15:55 We open this incident report.

2025-02-03:

• 12:07 We receive a copy of the email sent by our auditor to the CCADB Root Store Managers.

Root Cause Analysis

Based on our historical experience with audits, we believed the audit period would always be for a complete year, unless a different period had specifically been requested. Such an event happened last year, when we underwent our first WebTrust for S/MIME audit, for which we agreed to a 7 month audit period in order to align that audit with our regular WebTrust audit schedule going forward. As a full year’s audit has always been the standard, we have never before required written confirmation from our auditors of the target audit period.

When we received the audit plan, we were aware that the on-site visit was scheduled before the end of the presumed agreed Audit Period End Date. This was not unusual and so did not raise any flags. On-site audits frequently kick off the largest part of our audit activity, and performing this early on enables both us and our auditors to have ample time to work through all the required evidence. Questions and follow-ups are usually raised on remote calls in the weeks and months following the on-site visit.

Our auditor confirmed that they themselves did not realize that they had to cut the audit period short until they were preparing the audit attestations on 2025-01-13. This was 25 days before the CCADB disclosure deadline that Sectigo was expecting. However, due to the shortened audit period, this was in fact 85 days after the brought-forward Audit Period End Date, and therefore only a few days before the actual disclosure deadline. Our auditor did not notify Sectigo of the shortened audit period or the brought-forward Audit Period End Date. Had they done so, we would have escalated the matter and done everything within our power to ensure timely disclosure to CCADB.

While we did receive the first draft reports 2 days later, on 2025-01-15, we did not notice the less than one year audit period at this time.

Another two days later, on 2025-01-17, we received the final audit reports. This was at that point 89 days after the end of the updated audit period. This was a Friday, with team members receiving the reports whilst travelling. As we were under the impression that we still had more than two weeks before our CCADB disclosure deadline, we did not have other personnel on standby to perform a prompt CCADB disclosure. We have dealt with audit reports coming in during the last few days before the disclosure deadline in the past, in which cases we have always had multiple people on standby to make sure the reports are disclosed to CCADB in a timely fashion. We did not believe this was one of those circumstances.

Early the next week, we reviewed the final reports and noticed the missing Subordinate CA Certificates. After this moment we started a dialog with our auditor to make sure these Subordinate CA Certificates were added. At this point however, we were already too late for a timely disclosure to CCADB.

Lessons Learned

What went well

• While the first version of the audit reports was missing a number of Subordinate CA Certificates, no further audit work was required before our auditor was able to provide an updated report that incorporated those Subordinate CA Certificates.
• After becoming aware of this incident, communication with our auditors occurred in a good and timely fashion from both sides, allowing us to resolve the matter swiftly.

What didn't go well

• Due to ongoing travel, it was not possible to expose the first draft version of our audit letters to our usual peer review process and to also provide a timely response to our auditors. Had we followed this process, it’s likely that we would have noticed the shortened audit period sooner.
• We did not take advantage of the optional CCADB "Test Preliminary Audit Statements" facility, which probably would have alerted us sooner to the problems we subsequently found in the audit letters.
• The first official versions of the audit letters were provided to us on Friday, 2025-01-17. Monday, 2025-01-20 would have been the deadline to submit these audit reports to CCADB. As we did not notice and were not made aware of the change in Audit Period End Date, we were unaware that we needed to use our proven escalation paths to ensure that the CCADB disclosures would occur on time.
• We believed, without independently confirming it, that the audit period would be 1 year by default, as had always been the case in the past. Even our auditor was of this understanding until they made a very late discovery that led them to decide to shorten the audit period.
• Prior to this incident we didn’t anticipate the need to explicitly confirm that calendaring of audit dates would not affect our ability to maintain our expected audit timeline, nor to explicitly monitor if ongoing audit decisions might alter required timelines. We believe this is a pitfall that most or all public CAs could fall into, and we strongly urge all CAs to proactively monitor for scheduling risk of this nature.

Where we got lucky

• N/A

Action Items

Action Item	Kind	Due Date
Update internal practices to have draft audit letters always be reviewed through our peer review process	Prevent	Completed
Update internal practices to always utilize CCADB’s Test Preliminary Audit Statements facility	Prevent	Completed
Update internal policies to request and require written confirmation of the targeted audit period during the audit planning phase	Prevent	Completed
Setup a biweekly update call with our ETSI auditors and multiple Sectigo staff, just as we have been doing for some years with our WebTrust auditors	Prevent and Detect	Completed

Appendix

Details of indirectly impacted CA certificates

The 8 Subordinate CA Certificates can be found here.

Details of affected CCADB records

These are the IDs/URLs of the 8 CCADB records involved in this incident:

0014o00001l043LAAQ
0011J00001kXXzYQAW
0014o00001l043aAAA
0011J00001kXY0CQAW
001TO00000G7fExYAJ
001TO00000G7dWWYAZ
001TO00000G7dOQYAZ
001TO00000G7gUOYAZ

The links above require CCADB access. Interested parties that don't have access to the CCADB can find details of these CCADB records in the public AllCertificateRecordsCSVFormatv2 report. The CCADB record IDs are in the "Salesforce Record ID" column.

Comment 2

[Martijn Katerbarg]

Sectigo continues to monitor this bug for any questions or comments.

We plan to post an Incident Report Closure Summary by this time next week, unless any questions are raised before then.

Comment 3

[Martijn Katerbarg]

Incident Report Closure Summary

Incident Description:

During the submission of our ETSI audit letters for our QWAC Subordinate CAs, we were made aware that our audit period had been shortened to less than one year without notification by our auditor. Due to this we disclosed the audit letters after the allowed timeframe.

Incident Root Cause(s):

The initial audit letters were provided 5 days prior to the disclosure deadline. We discovered that 4 newly issued and in-scope Subordinate CA certificates were not included in the audit letters and thus requested these to be added, as was previously communicated with our auditor.

Unbeknownst to us, the auditor had also shortened our annual audit to less than 365 days, meaning our calculated submission deadline was incorrect.

Remediation Description:

We have updated internal practices and policies to include reviews of draft audit letters in our usual peer-review process, utilize CCADBs Test Preliminary Audit Statements option, request written confirmation of the targeted audit period during the planning phase and attend biweekly calls with our ETSI auditors when audits are on-going.

Commitment Summary:

We are committed to following our internal practices and policies.

All Action Items disclosed in this Incident Report have been completed as described, and we request closure of this bug.

Comment 4

[Ben Wilson]

I will close this on Friday, 28-Feb-2025, unless there are remaining issues or questions to discuss.