Closed Bug 1724528 Opened 2 years ago Closed 2 years ago

Apple: CAs omitted from audit statement


(CA Program :: CA Certificate Compliance, task)


(Not tracked)



(Reporter: certification_authority, Assigned: certification_authority)


(Whiteboard: [ca-compliance])

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.1.2 Safari/605.1.15

Summary: CAs Left out of WTBR Report in Clerical Error → Apple: CAs omitted from audit statement

Incident Report

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in, a Bugzilla bug, or internal self-audit), and the time and date.

  • Apple received a notification from our root vendor (DigiCert) of intermediate certificate records in the Common CA Database (CCADB) that have outdated audit statements on August 03, 2021 at 10:23 PT

  • The notification prompted a review of Apple’s intermediate CA records, which resulted in identifying 3 EV Sub-CAs (below) that were not listed on the recently issued “WebTrust for Certification Authorities – SSL Baseline with Network Security” (WTBR) audit statement, though they were listed on the “WebTrust Principles and Criteria for Certification Authorities – Extended Validation SSL” (WTEV) and the “WebTrust Principles and Criteria for Certification Authorities” (WTCA) audit statements:

    • Certificate Name: Apple Public EV Server ECC CA 1 - G1
      SHA-256 Fingerprint: 2585928D2C5BFD952E025BD12E27C6776224CF752EC362D3031CDD49351844D4
    • Certificate Name: Apple Public EV Server RSA CA 1 - G1
      SHA-256 Fingerprint: 340CA5BA402D140B65A2C976E7AE8128A1505C29D190E0E034F59CCAE7A92BC2
    • Certificate Name: Apple Public EV Server RSA CA 2 - G1
      SHA-256 Fingerprint: D6EF3E09EBE0D9370E51F5C09A532B3AC70D3CE822253F9FC84C28E9BFA550D5
  • Apple verified with our external WebTrust auditors that this was an unintended error of omission on the WTBR audit statement, that the completed audits correctly covered these omitted CAs, and that a corrected audit statement would be issued

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

  • 2020-04-16: Audit Period Start Date
  • 2021-04-15: Audit Period End Date
  • 2021-06-09: Audit Reports finalized and signed
  • 2021-06-09: Completed Audit Reports sent to CPA Canada
  • 2021-06-15: Seal ID URLs received from CPA Canada
  • 2021-07-01: Updated Seals and Audit Reports published to
  • 2021-08-03 10:23 PT: Apple received notification of intermediate certificate records in the Common CA Database (CCADB) that have outdated audit statements
  • 2021-08-03 11:03 PT: Apple communicated with the root vendor to perform analysis of the discovered issue
  • 2021-08-03 13:43 PT: Apple verified that 3 EV Sub-CAs were not listed in the published WTBR audit statement, resulting in the above notification
  • 2021-08-03 13:50 PT: Apple notified the external auditors of the discrepancy in the WTBR audit statement
  • 2021-08-03 15:45 PT: Apple received verification from the external auditors that this omission was unintended and did not accurately represent the work completed in the audit; the 3 EV Sub-CAs were correctly scoped in to the WTBR audit and the audit statement was incorrect in not having these 3 EV Sub-CAs listed
  • 2021-08-06: Received amended audit statements and distributed them to Mozilla and root vendor

3. Whether your CA has stopped, or has not yet stopped, certificate issuance or the process giving rise to the problem or incident. A statement that you have stopped will be considered a pledge to the community; a statement that you have not stopped requires an explanation.

Certificate issuance was not affected in this incident

4. In a case involving certificates, a summary of the problematic certificates. For each problem: the number of certificates, and the date the first and last certificates with that problem were issued. In other incidents that do not involve enumerating the affected certificates (e.g. OCSP failures, audit findings, delayed responses, etc.), please provide other similar statistics, aggregates, and a summary for each type of problem identified. This will help us measure the severity of each problem.

Three (3) EV Sub-CAs were erroneously omitted from our recently issued “WebTrust for Certification Authorities – SSL Baseline with Network Security” (WTBR) audit statement

5. In a case involving certificates, the complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or IDs, either in the report or as an attached spreadsheet, with one list per distinct problem. In other cases not involving a review of affected certificates, please provide other similar, relevant specifics, if any.

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

  • The WTBR audit was completed in the expected timeline with the correct scope of included Sub-CAs and its audit statement was published in a timely manner. The discovery of the erroneously omitted EV Sub-CAs was made from the CCADB report generated by Mozilla and shared with the root vendor
  • Reviews of the published audit statements were manual and relied on visual inspection and comparison of the included Sub-CAs. This approach was prone to human error which is what occurred in this instance; notably, despite multiple independent people performing that review, this error was not caught

7. List of steps your CA is taking to resolve the situation and ensure that such situation or incident will not be repeated in the future, accompanied with a binding timeline of when your CA expects to accomplish each of these remediation steps.

Assignee: bwilson → certification_authority
Type: defect → task
Ever confirmed: true
Whiteboard: [ca-compliance]

The CA, root vendor, and external auditors are reviewing the quality review procedures to ensure this type of clerical error does not occur again

What are/were the current procedures? The description just state:

relied on visual inspection and comparison of the included Sub-CAs

And it's unclear whether or not formal procedures existed, written guidance, whether multiple parties were involved, etc. Given the nature of CA incidents, as well as Apple's own role in operating a root program, it's equally important to understand why there weren't pre-existing controls here, or if there were, why this failure mode wasn't accounted for already.

Flags: needinfo?(certification_authority)

Thank you for your inquiry.

Formal procedures do exist and were followed, including written guidance of what information must be included in each audit report.

These procedures include a review of the list of Sub-CAs in each audit report by two separate people against our internal documentation showing which CAs should be included in each report.

Our investigation revealed that while the three (3), newly created, EV Sub-CAs were listed in our internal documentation and were labeled for inclusion in the WTBR report, they were listed in their own subsection separate from the subsection intended to list all Sub-CA’s to be included in the WTBR report.

This is why both reviewers missed them when reviewing the WTBR report. To remediate this issue and avoid further issues, we improved our documentation to remove the opportunity for misunderstanding which now makes it clear which Sub-CAs should be included in each report.

We have also updated our annual procedures to request a list of the Sub-CAs independently generated by our root vendors in order to ensure our list of Sub-CAs is complete and accurate. This listing will be used as part of the quality review procedures performed on the audit reports.

Flags: needinfo?(certification_authority)

We have completed the following items for this incident:

There are no outstanding tasks for this incident report.

There are no outstanding tasks for this incident report. Since no questions or concerns have been raised, can this incident be closed?

I'll close this next Wednesday, 1-Sept-2021.

Flags: needinfo?(bwilson)
Closed: 2 years ago
Flags: needinfo?(bwilson)
Resolution: --- → FIXED
Product: NSS → CA Program
You need to log in before you can comment on or make changes to this bug.