1724528 - Apple: Intermediate CA certificates omitted from audit statement

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[certification_authority]

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.1.2 Safari/605.1.15

Comment 1

[certification_authority]

Incident Report

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

• Apple received a notification from our root vendor (DigiCert) of intermediate certificate records in the Common CA Database (CCADB) that have outdated audit statements on August 03, 2021 at 10:23 PT

• The notification prompted a review of Apple’s intermediate CA records, which resulted in identifying 3 EV Sub-CAs (below) that were not listed on the recently issued “WebTrust for Certification Authorities – SSL Baseline with Network Security” (WTBR) audit statement, though they were listed on the “WebTrust Principles and Criteria for Certification Authorities – Extended Validation SSL” (WTEV) and the “WebTrust Principles and Criteria for Certification Authorities” (WTCA) audit statements:

  • Certificate Name: Apple Public EV Server ECC CA 1 - G1
    SHA-256 Fingerprint: 2585928D2C5BFD952E025BD12E27C6776224CF752EC362D3031CDD49351844D4
  • Certificate Name: Apple Public EV Server RSA CA 1 - G1
    SHA-256 Fingerprint: 340CA5BA402D140B65A2C976E7AE8128A1505C29D190E0E034F59CCAE7A92BC2
  • Certificate Name: Apple Public EV Server RSA CA 2 - G1
    SHA-256 Fingerprint: D6EF3E09EBE0D9370E51F5C09A532B3AC70D3CE822253F9FC84C28E9BFA550D5

• Apple verified with our external WebTrust auditors that this was an unintended error of omission on the WTBR audit statement, that the completed audits correctly covered these omitted CAs, and that a corrected audit statement would be issued

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

• 2020-04-16: Audit Period Start Date
• 2021-04-15: Audit Period End Date
• 2021-06-09: Audit Reports finalized and signed
• 2021-06-09: Completed Audit Reports sent to CPA Canada
• 2021-06-15: Seal ID URLs received from CPA Canada
• 2021-07-01: Updated Seals and Audit Reports published to https://www.apple.com/certificateauthority/
• 2021-08-03 10:23 PT: Apple received notification of intermediate certificate records in the Common CA Database (CCADB) that have outdated audit statements
• 2021-08-03 11:03 PT: Apple communicated with the root vendor to perform analysis of the discovered issue
• 2021-08-03 13:43 PT: Apple verified that 3 EV Sub-CAs were not listed in the published WTBR audit statement, resulting in the above notification
• 2021-08-03 13:50 PT: Apple notified the external auditors of the discrepancy in the WTBR audit statement
• 2021-08-03 15:45 PT: Apple received verification from the external auditors that this omission was unintended and did not accurately represent the work completed in the audit; the 3 EV Sub-CAs were correctly scoped in to the WTBR audit and the audit statement was incorrect in not having these 3 EV Sub-CAs listed
• 2021-08-06: Received amended audit statements and distributed them to Mozilla and root vendor

3. Whether your CA has stopped, or has not yet stopped, certificate issuance or the process giving rise to the problem or incident. A statement that you have stopped will be considered a pledge to the community; a statement that you have not stopped requires an explanation.

Certificate issuance was not affected in this incident

4. In a case involving certificates, a summary of the problematic certificates. For each problem: the number of certificates, and the date the first and last certificates with that problem were issued. In other incidents that do not involve enumerating the affected certificates (e.g. OCSP failures, audit findings, delayed responses, etc.), please provide other similar statistics, aggregates, and a summary for each type of problem identified. This will help us measure the severity of each problem.

Three (3) EV Sub-CAs were erroneously omitted from our recently issued “WebTrust for Certification Authorities – SSL Baseline with Network Security” (WTBR) audit statement

5. In a case involving certificates, the complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem. In other cases not involving a review of affected certificates, please provide other similar, relevant specifics, if any.

• Apple Public EV Server ECC CA 1 - G1: https://crt.sh/?id=2768157552 / https://crt.sh/?caid=168226
• Apple Public EV Server RSA CA 1 - G1: https://crt.sh/?id=2768157551 / https://crt.sh/?caid=168227
• Apple Public EV Server RSA CA 2 - G1: https://crt.sh/?id=2768161933 / https://crt.sh/?caid=168228

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

• The WTBR audit was completed in the expected timeline with the correct scope of included Sub-CAs and its audit statement was published in a timely manner. The discovery of the erroneously omitted EV Sub-CAs was made from the CCADB report generated by Mozilla and shared with the root vendor
• Reviews of the published audit statements were manual and relied on visual inspection and comparison of the included Sub-CAs. This approach was prone to human error which is what occurred in this instance; notably, despite multiple independent people performing that review, this error was not caught

7. List of steps your CA is taking to resolve the situation and ensure that such situation or incident will not be repeated in the future, accompanied with a binding timeline of when your CA expects to accomplish each of these remediation steps.

• An amended audit statement has been finalized and is being submitted to Mozilla via upload to https://bugzilla.mozilla.org/attachment.cgi?id=9235216 and the root vendor
• The CA, root vendor, and external auditors are reviewing the quality review procedures to ensure this type of clerical error does not occur again
• In addition to improving our manual review process, we will work with our root vendor to leverage the process for testing preliminary audit statements, as outlined at https://www.ccadb.org/cas/updates#test-preliminary-audit-statements, for future audit statements

Comment 2

[Ryan Sleevi]

The CA, root vendor, and external auditors are reviewing the quality review procedures to ensure this type of clerical error does not occur again

What are/were the current procedures? The description just state:

relied on visual inspection and comparison of the included Sub-CAs

And it's unclear whether or not formal procedures existed, written guidance, whether multiple parties were involved, etc. Given the nature of CA incidents, as well as Apple's own role in operating a root program, it's equally important to understand why there weren't pre-existing controls here, or if there were, why this failure mode wasn't accounted for already.

Comment 3

[certification_authority]

Thank you for your inquiry.

Formal procedures do exist and were followed, including written guidance of what information must be included in each audit report.

These procedures include a review of the list of Sub-CAs in each audit report by two separate people against our internal documentation showing which CAs should be included in each report.

Our investigation revealed that while the three (3), newly created, EV Sub-CAs were listed in our internal documentation and were labeled for inclusion in the WTBR report, they were listed in their own subsection separate from the subsection intended to list all Sub-CA’s to be included in the WTBR report.

This is why both reviewers missed them when reviewing the WTBR report. To remediate this issue and avoid further issues, we improved our documentation to remove the opportunity for misunderstanding which now makes it clear which Sub-CAs should be included in each report.

We have also updated our annual procedures to request a list of the Sub-CAs independently generated by our root vendors in order to ensure our list of Sub-CAs is complete and accurate. This listing will be used as part of the quality review procedures performed on the audit reports.

Comment 4

[certification_authority]

We have completed the following items for this incident:

• Received and published an amended WTBR audit report including the omitted EV CAs
  • Report attached to Bugzilla: https://bugzilla.mozilla.org/attachment.cgi?id=9235216
  • Report published to Apple website: https://www.apple.com/certificateauthority/public/
• Updated the quality review procedures to include:
  • New CA inventory format for clearer documentation
  • New automated documentation quality review step (outlined at https://www.ccadb.org/cas/updates#test-preliminary-audit-statements)
  • Additional Sub-CA inventory procedures incorporating Root Vendors

There are no outstanding tasks for this incident report.

Comment 5

[certification_authority]

There are no outstanding tasks for this incident report. Since no questions or concerns have been raised, can this incident be closed?

Comment 6

[Ben Wilson]

I'll close this next Wednesday, 1-Sept-2021.