1841534 - Apple: TLS certificates issued outside the TTL of the CAA record

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[certification_authority]

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.5.1 Safari/605.1.15

Comment 1

[certification_authority]

On June 29, 2023, we discovered that in some cases, where additional issuance approvals were required, the requirements of BRs 3.2.2.8 were not met as greater than 8 hours passed between the CAA lookup and certificate issuance. BR section 3.2.2.8 states “If the CA issues, they MUST do so within the TTL of the CAA record, or 8 hours, whichever is greater.” Our initial investigation indicates there are 1,726 valid impacted certificates all issued to domains that Apple owns. We deployed a fix to our production environment on June 30, 2023, but are continuing investigation in support of a full incident report which we expect to deliver no later than July 17, 2023.

Comment 2

[certification_authority]

1 ) How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

On 2023-06-29 at 14:47 PT, while completing the annual CCADB Self Assessment, we discovered that in some cases, when additional issuance approvals were required, the requirements of BR section 3.2.2.8 were not met as greater than 8 hours passed between the CAA check and certificate issuance. The TTL for the CAA records present in DNS for the affected certificates was less than 8 hours, so the maximum time permitted under the BRs was 8 hours.

This incident has highlighted the value of the Self Assessment process for CAs participating in the Web PKI.

2) A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

• 2017-09-07: We implemented CAA checking.
• 2017-09-08: Effective date of BR section 3.2.2.8 (CAA Records), which states:

“As part of the issuance process, the CA must check for a CAA record for each dNSName in the subjectAltName extension of the certificate to be issued, according to the procedure in RFC 6844, following the processing instructions set down in RFC 6844 for any records found. If the CA issues, they must do so within the TTL of the CAA record, or 8 hours, whichever is greater.”

• 2017-09-08 at 03:59 PT: The first affected certificate was issued in which greater than 8 hours passed between the CAA check and certificate issuance.
• 2023-06-29 at 09:30 PT: The last affected certificate was issued.
• 2023-06-29 at 13:22 PT: While completing the annual CCADB Self Assessment, the compliance team began investigating compliance with the following requirement: “If the CA issues, they must do so within the TTL of the CAA record, or 8 hours, whichever is greater.”.
• 2023-06-29 at 14:26 PT: An initial review of the CAA checking code indicated that we could be violating the requirement in instances in which additional approvals were required that took longer than 8 hours to obtain, resulting in the certificate being issued more than 8 hours after the CAA check was performed.
• 2023-06-29 at 14:47 PT: We confirmed that we had issued at least one public TLS certificate in which greater than 8 hours passed between the CAA check and certificate issuance.
• 2023-06-29 at 16:12 PT: We notified the certificate approvers of the issue and that a fix was coming. Certificate issuance ceased.
• 2023-06-29 at 16:22 PT: We began working on a software fix so that the CAA check is performed immediately before issuance.
• 2023-06-29 at 17:53 PT: We began investigating the scope of affected certificates issued since 2017-09-08.
• 2023-06-30 at 11:24 PT: We began initial outreach to some affected teams to notify them that affected certificates must be replaced and revoked.
• 2023-06-30 at 11:30 PT: We began formalizing the communication plan to notify all Apple teams that affected certificates must be replaced and revoked. We also performed a risk assessment focused on the impact revocation would have, if it occurred prior to a replacement certificate having been configured.
• 2023-06-30 at 12:20 PT: We deployed the software fix that ensures that the CAA check is performed, in all cases, immediately prior to issuance.
• 2023-06-30 at 12:28 PT: We notified the certificate approvers that the fix had been applied. Certificate issuance resumed.
• 2023-07-03 at 14:00 PT: We updated our pre-issuance control to address that the CAA check must happen within the TTL of the CAA record, or 8 hours, whichever is greater.
• 2023-07-03 at 16:52 PT: In continuation of the communication plan, we sent a notification to all Apple teams with affected certificates informing them that their certificates would need to be replaced and revoked.
• 2023-07-03 at 20:57 PT: We filed the Bugzilla and posted our initial issue report.

3) Whether your CA has stopped, or has not yet stopped, certificate issuance or the process giving rise to the problem or incident. A statement that you have stopped will be considered a pledge to the community; a statement that you have not stopped requires an explanation.

We have stopped issuing certificates with this problem.

4) In a case involving certificates, a summary of the problematic certificates. For each problem: the number of certificates, and the date the first and last certificates with that problem were issued. In other incidents that do not involve enumerating the affected certificates (e.g. OCSP failures, audit findings, delayed responses, etc.), please provide other similar statistics, aggregates, and a summary for each type of problem identified. This will help us measure the severity of each problem.

The set of affected certificates were issued between 2017-09-08 at 03:59 PT and 2023-06-29 at 09:30 PT.

• All affected certificates: 9,568
• Valid affected certificates: 1,717

At the time of issuance, all affected certificates were issued to Apple-owned domains for which the Apple Public CA is authorized to issue. Now that we have corrected the problem, our priority is replacing and revoking the valid affected certificates. We will be opening a second incident report on non-compliance with BR section 4.9.1.1 item #12.

5) In a case involving certificates, the complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem. In other cases not involving a review of affected certificates, please provide other similar, relevant specifics, if any.

Two files have been attached with the following data:

• A list of all affected certificates: all_affected_certificates.csv (9,568 certificates)
• A list of affected certificates still valid (not expired and not revoked): valid_affected_certificates.csv (1,717 certificates)

6) Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

This problem affected issuance of public TLS certificates where additional issuance approvals were required that took greater than 8 hours to obtain.

• How and why the mistakes were made or bugs introduced

  • We had a flaw in our original implementation. The CAA check was performed at certificate enrollment rather than immediately before certificate issuance. In some cases where additional issuance approvals were required, more than 8 hours passed between the CAA check and certificate issuance. The TTL for the CAA records present in DNS for the affected certificates was less than 8 hours, so the maximum time permitted under the BRs was 8 hours.

• How the mistakes avoided detection until now

  • Our pre-issuance control included checking CAA records but did not address that if the CA issues, they must do so within the TTL of the CAA record (if present), or 8 hours, whichever is greater.
  • We did not have post-issuance monitoring and alerting to check that the maximum permitted time under BR section 3.2.2.8 was not exceeded.

7) List of steps your CA is taking to resolve the situation and ensure that such situation or incident will not be repeated in the future, accompanied with a binding timeline of when your CA expects to accomplish each of these remediation steps. The incident report should explain how the CA Owner’s systems or processes will be made more robust, and how other CAs may learn from the incident.

• We deployed a fix on 2023-06-30 at 12:20 PT that ensures that the CAA check is performed immediately before issuance.
• We updated our pre-issuance control on 2023-07-03 at 14:00 PT to address that the CAA check must happen within the TTL of the CAA record, or 8 hours, whichever is greater.
• In addition to the pre-issuance controls to detect and prevent any issuance, we will be adding additional post-issuance monitoring to verify those controls are functioning and alert if the controls fail. We expect to complete this work by 2023-08-14.

Comment 3

[certification_authority]

Attachment: all_affected_certificates.csv

Comment 4

[certification_authority]

Attachment: valid_affected_certificates.csv

Comment 5

[certification_authority]

We have added the additional post-issuance monitoring to verify the pre-issuance controls are functioning and alert if the controls fail.

We have no additional remediation items and consider this issue resolved unless there are further questions.

Comment 6

[certification_authority]

There are no outstanding tasks for this incident report. Since no other questions or concerns have been raised, can this incident be closed?

Comment 7

[Ben Wilson]

I'll close this on or about next Wed. 23-Aug-2023, unless there are additional concerns or questions to discuss.

Comment 8

[certification_authority]

We continue to monitor this bug for comments and questions.