Closed Bug 2052377 Opened 3 months ago Closed 2 months ago

Apple: Audit Documents missing 2 Sub-CAs

Categories

(CA Program :: CA Certificate Compliance, task)

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: certification_authority, Assigned: certification_authority)

Details

(Whiteboard: [ca-compliance] [policy-failure])

Attachments

(1 file)

Steps to reproduce:

Preliminary Incident Report

Summary

  • Incident description: The Apple 2026 WebTrust Audit reports did not include 2 Sub-CAs created on 2/4/2026. They should have been on the WTCA, WTBR, and WTNS reports.
    • Apple Public Server ECC CA 13 - G1
      • SHA-256: 01682DA2E6A878468D0A4777162AEBE3233DC6935B066F71F486D5672375E1F1
    • Apple Public Server RSA CA 13 - G1
      • SHA-256: EF7147B7031ACF7942DD749549DD3FD9A564B916B325DFA9E255EBB9E16AEEEB
  • Relevant policies:
    • CCADB 5.1
    • CA/Browser Forum Baseline Requirements 8.1
  • Source of incident disclosure: Our root vendor, Sectigo, notified us that ALV came back with errors when uploading our audit documents regarding the 2 missing Sub-CAs.

We will post a Full Incident Report on or before July 16, 2026.

Component: CA Documents → CA Certificate Compliance
Whiteboard: [ca-compliance]
Assignee: nobody → certification_authority
Status: UNCONFIRMED → ASSIGNED
Type: defect → task
Ever confirmed: true

Full Incident Report

Summary

  • CA Owner CCADB unique ID: A005667
  • Incident description: The Apple 2026 WebTrust Audit reports did not include 2 Subordinate CA Certificates created on 2/4/2026. The two impacted Subordinate CAs should have been included in the WTCA, WTBR, and WTNS reports. The CA certificates were included in the audit, and controls were sampled against them, but when the final WebTrust audit report document was created, they were not included in the list of Subordinate CAs covered by the report.
  • Timeline summary:
    • Non-compliance start date: June 29, 2026
    • Non-compliance identified date: June 30, 2026
    • Non-compliance end date: August 5, 2026
  • Relevant policies: CCADB 5.1 and CA/Browser Forum Baseline Requirements 8.1 and 8.4
  • Source of incident disclosure: Our root vendor, Sectigo, notified us that ALV came back with errors when uploading audit documents to CCADB which should have included the 2 missing Subordinate CA Certificates.

Impact

  • Total number of certificates: 2
  • Total number of "remaining valid" certificates: 2
  • Affected certificate types: This incident affects Subordinate CA Certificates (Intermediate CAs).
  • Incident heuristic: All certs listed in Appendix
  • Was issuance stopped in response to this incident, and why or why not?: Issuance from these CAs had not started and no issuance has occurred under these CAs.
  • Analysis: No issuance from these certs occurred.
  • Additional considerations: Apple has chosen to revoke both Subordinate CA certificates rather than seek a revised audit report. Given that no subscriber certificates have been issued under either CA, revocation is the most straightforward path to resolving the non-compliance. Requesting revised audit documents would require re-engagement with CPA Canada and re-delivery to all root vendors, with no material security benefit since the Subordinate CAs were never operational.

Timeline

All times are in UTC

  • 12/12/2025 20:24: Statement of Work signed with Auditor for scope of audit
  • 1/29/2026 20:25: Final review of audit scope was performed by Apple
  • 2/4/2026 21:32: The 2 Subordinate CA Certificates were created
  • 3/31/2026 23:59: Final day of Audit period
  • 6/10/2026 19:02: Audit reports were signed
  • 6/23/2026 16:01: Apple was notified Audit Seals were live from CPA Canada
  • 6/25/2026 15:52: Audit reports were sent to Sectigo to implement into CCADB
  • 6/29/2026 11:59: This is the final day to post the Audit reports, 90 days after the period end date. The point at which this became an incident.
  • 6/30/2026 15:47: Sectigo notified Apple of errors in ALV when uploading reports to CCADB.
  • 7/2/2026 21:56: Preliminary Incident report filed.

Related Incidents

Bug Date Description
1952635 3/8/2025 Entrust roots not included in audit report.
1952639 3/8/2025 TWCA Missing Roots in BR Audits.
1904494 6/25/2024 Asseco audit statement missing certificate.
1724528 8/6/2021 Apple: Intermediate CA certificates omitted from audit statement

Root Cause Analysis

Contributing Factor #1: Reliance on a freeze period

  • Description: We have a general freeze period of new controls, CAs, or other program level changes 60 days before the end of the audit period. It ensures anything included in the audit has at minimum 60 days of provenance. However, we relied upon the freeze period, and did not perform an additional check at the end of the audit period that would have caught the missing Subordinate CA Certificates.
  • Timeline: This requirement has been in place for years at the recommendation from our auditor.
  • Detection: Normally the freeze period was successful in pushing planning of CA additions to well before the audit period. But using this as a reason not to perform additional validation at period close was not a good decision.
  • Interaction with other factors: The reliance on a freeze period compounded with cause #3 to weaken communication between compliance and engineering during the freeze period.
  • Root Cause Analysis methodology used: 5-Whys

Contributing Factor #2: We did not run ALV on the draft reports, only the finals.

  • Description: We did not run ALV on our draft reports, only the finals. We had intended to get reports signed early enough (June 10) that we could run ALV on the final reports and make corrections if necessary. This intent was born out of the similar incident 5 years ago.
  • The seals took 23 days to get, mostly due to problems with some of the seals needing to be sent back and fixed. We do not own our Roots, meaning we are unable to update the root records ourselves and we are unable to run ALV on the draft audit reports. We can, and will in the future, schedule time with our root vendors and ask them to run ALV on our draft audit reports, even before sending them to CPA Canada for seals.
  • This is an update to the lesson learned 5 years ago, where we will run ALV as early as possible so that mistakes can be caught and remediated.
  • Timeline: This has been an issue for Subordinate CA Certificates since ALV was implemented.
  • Detection: Not taking advantage of the ability to run ALV on drafts prevented early detection.
  • Interaction with other factors: Not doing this prevented early detection.
  • Root Cause Analysis methodology used: 5-whys.

Contributing Factor #3: We added 2 Subordinate CA Certificates during our freeze period

  • Description: On February 4, 2026 we got 2 new Subordinate CA Certificates. This was 55 days before the end of the audit period. Their creation was approved in December, with the intention to create in January. Engineering moved the creation date, and compliance did not record the new date.
  • Timeline: On February 4, 2026 we got 2 new Subordinate CA Certificates.
  • Detection: The problem wasn’t getting the Sub-CAs or putting them in the audit, but documenting them in the audit document review.
  • Interaction with other factors: This was the event coupled with the first contributing factor that resulted in an incident.
  • Root Cause Analysis methodology used: 5-whys

Lessons Learned

  • What went well: The Subordinate CA certificates were correctly included in the audit scope, controls were sampled against them. The audit itself was accurate; the gap was in the report document compilation. The previous similar incident spawned quality review procedures that operated correctly and prevented similar issues for 5 years.
  • What didn’t go well: Two Subordinate CAs were left out of the audit report. We didn’t know they were missing until ALV came back after the 90 day window. The quality review procedures that were created 5 years ago left some holes that allowed this to happen, and we thought leaving the ALV review for the final reports would be enough.
  • Where we got lucky: We haven’t started using the certs, and nothing has been issued from them.
  • Additional: We moved out document signing to earlier in the 90 day window this year with the intention of uploading the documents sooner, but the seals process took longer than normal, pushing the upload to CCADB close to the end of the period. We will continue to aim to sign documents 60-70 days after close of the audit period.

Action Items

Action Item Kind Corresponding Root Cause(s) Evaluation Criteria Due Date Status
Add additional review at close of audit period Prevent Root Cause # 1 Updated Audit Process. 7/10/2026 Complete
Add step of having root vendors run ALV on draft audit documents Prevent Root Cause # 2 Updated Audit Process. 7/10/2026 Complete
Schedule communication reminders about the freeze period to all CA employees Prevent Root Cause # 3 Updated Annual Process. 7/10/2026 Complete
We already notify auditors of all CAs at audit kickoff, but will add a required notification of all Sub-CAs at creation Prevent Root Cause # 3 Updated Auditor communication process 7/10/2026 Complete
Revoke both Sub CAs Correct Root Cause # 3 Both SubCAs confirmed revoked with CRL/OCSP responses updated 7/20/2026 In Progress

Appendix

We have discussed the revocation with the root vendor Sectigo and opted to schedule the revocation for August 5, 2026. We have updated the Action item list below with a new due date of August 6, 2026, the day after, when a proper update can be made.

We will continue to monitor this bug for questions and comments, but if there are none, our next expected update is August 6, 2026, please update the “Next update” field to this date.

Thank you.

Action Items

Action Item Kind Corresponding Root Cause(s) Evaluation Criteria Due Date Status
Add additional review at close of audit period Prevent Root Cause # 1 Updated Audit Process. 7/10/2026 Complete
Add step of having root vendors run ALV on draft audit documents Prevent Root Cause # 2 Updated Audit Process. 7/10/2026 Complete
Schedule communication reminders about the freeze period to all CA employees Prevent Root Cause # 3 Updated Annual Process. 7/10/2026 Complete
We already notify auditors of all CAs at audit kickoff, but will add a required notification of all Sub-CAs at creation Prevent Root Cause # 3 Updated Auditor communication process 7/10/2026 Complete
Revoke both Sub CAs Correct Root Cause # 3 Both SubCAs confirmed revoked with CRL/OCSP responses updated 8/6/2026 In Progress
Whiteboard: [ca-compliance] → [ca-compliance] Next update 2026-08-06

Report Closure Summary

  • Incident description: The Apple 2026 WebTrust Audit reports did not include 2 Subordinate CA Certificates created on 2/4/2026. The two impacted Subordinate CAs should have been included in the WTCA, WTBR, and WTNS reports. The CA certificates were included in the audit, and controls were sampled against them, but when the final WebTrust audit report document was created, they were not included in the list of Subordinate CAs covered by the report.
  • Incident Root Cause(s): The Apple CA relied on a freeze period, added 2 Sub-CAs during the freeze period, and did not run ALV on the draft reports.
  • Remediation description: Apple requested Sectigo revoke the certificates, and updated our processes to prevent this type of issue in the future, and catch it earlier if it occurrs.
  • Commitment summary: We commit to working with our Root vendors to run ALV on draft reports, and not waiting for finals.

All Action Items disclosed in this report have been completed as described, and we request its closure.

Action Items

Action Item Kind Corresponding Root Cause(s) Evaluation Criteria Due Date Status
Add additional review at close of audit period Prevent Root Cause # 1 Updated Audit Process. 7/10/2026 Complete
Add step of having root vendors run ALV on draft audit documents Prevent Root Cause # 2 Updated Audit Process. 7/10/2026 Complete
Schedule communication reminders about the freeze period to all CA employees Prevent Root Cause # 3 Updated Annual Process. 7/10/2026 Complete
We already notify auditors of all CAs at audit kickoff, but will add a required notification of all Sub-CAs at creation Prevent Root Cause # 3 Updated Auditor communication process 7/10/2026 Complete
Revoke both Sub CAs Correct Root Cause # 3 Both SubCAs confirmed revoked with CRL/OCSP responses updated 8/5/2026 Complete

This is a final call for comments or questions on this Incident Report.

Otherwise, it will be closed on approximately 2026-08-17.

Whiteboard: [ca-compliance] Next update 2026-08-06 → [close on 2026-08-17] [ca-compliance]
Status: ASSIGNED → RESOLVED
Closed: 2 months ago
Resolution: --- → FIXED
Whiteboard: [close on 2026-08-17] [ca-compliance] → [ca-compliance] [policy-failure]
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: