1777757 - Apple: EV TLS pre-certificates issued without EKU extension

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[certification_authority]

On July 1, 2022, Apple Public CA issued two EV TLS pre-certificates without an EKU extension (https://crt.sh/?id=7042225360 and https://crt.sh/?id=7042228702), which do not conform to the Baseline Requirements Section 7.1.2.3 and have subsequently been revoked. The final certificates were not issued.

A full report will be provided no later than July 15, 2022.

Comment 1

[certification_authority]

1 ) How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

On July 1, 2022, Apple Public CA received an internal alert that issuance of a final EV TLS certificate failed due to an insufficient number of SCTs collected. Multiple CT log servers had returned an error, instead of an SCT, rejecting the precertificate due to no EKU extension.

2) A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

• 2022-06-01: Effective date of the Mozilla Root Store Policy version 2.8, which says in section 5.4 that precertificates are in-scope for compliance with the policy and if a precertificate does not comply with the policy, it is considered a misissuance of the final certificate, even if the certificate does not actually exist.
• 2022-05-11: We made a change to increase automation of our deployment processes to leverage the EJBCA ConfigDump tool.
• 2022-05-13 at 09:08 PT: We identified a bug in the ConfigDump import functionality in our non-production environment. If the default setting Extended Key Usage Used: true is not included in the YAML file, then any EKUs listed under Extended Key Usage are not imported. We identified a workaround to enable the EKU extension directly in the GUI.
• 2022-06-22 at 7:36 PT: We opened a support ticket with our software vendor.
• 2022-06-22 at 18:47 PT: The affected EV TLS certificate profile was deployed, along with several other certificate profiles, without the default setting Extended Key Usage Used: true in the YAML file.
• 2022-06-23 at 08:08 PT: The workaround was implemented for all certificate profiles except the affected EV TLS certificate profile. This resulted in the EKU extension being excluded from the affected certificate profile.
• 2022-06-23 at 2:40 PT : We enabled issuance of the affected certificate request type in our certificate management service.
• 2022-07-01 at 06:45 PT: Apple Public CA issued the first EV TLS precertificate without an Extended Key Usage (EKU) extension (https://crt.sh/?id=7042225360) which did not conform to the Baseline Requirements Section 7.1.2.3.
• 2022-07-01 at 06:45 PT: Apple Public CA issued the second EV TLS precertificate without an EKU extension (https://crt.sh/?id=7042228702) which did not conform to the Baseline Requirements Section 7.1.2.3.
• 2022-07-01 at 06:46 PT: Apple Public CA received the internal alert that issuance of a final EV TLS certificate failed due to an insufficient number of SCTs collected. Multiple CT log servers had returned an error, instead of an SCT, rejecting the precertificate due to no EKU extension.
• 2022-07-01 at 09:17 PT: We began investigating the alert.
• 2022-07-01 at 09:58 PT: We identified that the two EV TLS precertificates were issued from the certificate profile that was deployed on 2022-06-22 at 18:47 PT.
• 2022-07-01 at 10:49 PT: We communicated with the internal team that had requested the 2 EV TLS certificates that the final certificate requests would be rejected due to a misconfiguration and communicated our plan to re-enable the affected certificate request type the following week during our regular deployment cycle. Following our standard weekly deploy cycle would allow us time to ensure we understood the root cause so that we did not risk any additional mis-issuances.
• 2022-07-01 at 11:01 PT: The certificate requests for both affected precertificates were rejected in our certificate management service preventing issuance of the final EV TLS certificates.
• 2022-07-01 at 12:05 PT: We disabled the affected certificate request type to prevent further issuance while we completed our investigation.
• 2022-07-01 at 12:23 PT: The first affected precertificate (https://crt.sh/?id=7042225360) was revoked.
• 2022-07-01 at 12:24 PT: The second affected precertificate (https://crt.sh/?id=7042228702) was revoked.
• 2022-07-07 at 12:13 PT: We fixed the issue by applying the workaround which restored the EKU extension.
• 2022-07-07 at 14:49 PT: We reenabled issuance of the affected EV TLS certificate type and informed the impacted team.

3) Whether your CA has stopped, or has not yet stopped, certificate issuance or the process giving rise to the problem or incident. A statement that you have stopped will be considered a pledge to the community; a statement that you have not stopped requires an explanation.

Apple Public CA stopped certificate issuance of the affected certificate type on 2022-07-01 at 12:05 PT after identifying that two EV TLS precertificates had been issued without EKU extensions.

4) In a case involving certificates, a summary of the problematic certificates. For each problem: the number of certificates, and the date the first and last certificates with that problem were issued. In other incidents that do not involve enumerating the affected certificates (e.g. OCSP failures, audit findings, delayed responses, etc.), please provide other similar statistics, aggregates, and a summary for each type of problem identified. This will help us measure the severity of each problem.

• first affected precertificate issued at 2020-07-01 at 6:45 PT (serial number: 1d:40:46:26:14:2c:0f:d4:48:dd:22:48:0c:a9:5b:e4)
• second affected precertificate issued at 2022-07-01 at 06:45 PT (serial number: 2c:a8:61:d5:9c:68:b0:79:58:8f:65:58:e9:97:a6:7f)

5) In a case involving certificates, the complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem. In other cases not involving a review of affected certificates, please provide other similar, relevant specifics, if any.

• first affected precertificate: https://crt.sh/?id=7042225360
• second affected precertificate: https://crt.sh/?id=7042228702

6) Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

How and why the mistakes were made or bugs introduced:

• We made a change to increase automation of our deployment processes on 2022-05-11 to leverage the EJBCA ConfigDump tool. We identified a bug in the ConfigDump import functionality in our non-production environment but chose to proceed because we had a workaround for that bug.
• We utilized the workaround for all certificate profiles except the affected EV TLS certificate profile. This resulted in the EKU extension being excluded from the affected certificate profile. The workaround involved enabling the EKU extension directly in the GUI, rather than relying on the configuration in the YAML file.

How the mistakes avoided detection until now:

• We were not using the EJBCA pre-sign certificate or CT pre-certificate Validators in the issuance process.

7) List of steps your CA is taking to resolve the situation and ensure that such situation or incident will not be repeated in the future, accompanied with a binding timeline of when your CA expects to accomplish each of these remediation steps.

• We opened a support ticket with our software vendor on 2022-06-22 to fix the bug in the ConfigDump import functionality. Once the fix is available, we will test it and update our deployment processes accordingly. In the interim, instead of relying on a GUI-based workaround, all default values will be explicitly included in the YAML file. Additionally, as part of our post deployment verification processes, we will verify that the configurations in the YAML file match the GUI configuration.
• In addition to existing post certificate issuance linting, we will begin using the EJBCA pre-sign certificate and CT pre-certificate Validators in our issuance process. We expect to complete this work by 2022-07-29.

Comment 2

[certification_authority]

Apple Public CA continues to monitor this bug for comments or questions.

Comment 3

[BugBot [:suhaib / :marco/ :calixte]]

The bug assignee is inactive on Bugzilla, so the assignee is being reset.

Comment 4

[certification_authority]

We are now using the the EJBCA pre-sign certificate and CT pre-certificate Validators in our issuance process.

The fix from our software vendor for the bug in the ConfigDump import functionality is not yet available. When it is, we will test it and update our deployment processes.

Apple Public CA continues to monitor this bug for comments or questions.

Comment 5

[certification_authority]

The fix from our software vendor for the bug in the ConfigDump import functionality is not yet available. As stated last week, we will test it and update our deployment processes once it is available. We continue to monitor this bug for comments or questions.

Comment 6

[Ben Wilson]

Once the bug fix from your vendor is installed, configured, and tested, what, if any, additional items need to be completed? In other words, what is your timeline for when you think this incident can be closed?

Comment 7

[certification_authority]

We expect to receive the bug fix from our vendor in their September release. After we install, configure, and test it, there will be no additional items that need to be completed. Can you please set the Next Update to Sept 30, 2022?

Comment 8

[certification_authority]

We have received the bug fix from our vendor to address the ConfigDump import functionality. We are in the process of testing and implementing this fix. Can you please set the Next Update to October 14, 2022?

Comment 9

[certification_authority]

We are still in the process of testing and implementing this fix. We continue to monitor this bug for comments or questions.

Comment 10

[certification_authority]

While testing this fix, we identified the need for an unrelated performance fix which must be addressed before we upgrade our production environment. We do not currently have an ETA for receiving this unrelated fix from our vendor, but based on past experience, this could take up to a month. Can you please set the Next Update to November 18, 2022? In the interim, we continue to use our short-term remediation.

We continue to monitor this bug for comments or questions.

Comment 11

[certification_authority]

The unrelated performance fix has been addressed and we have upgraded our production environment to include the fix for the bug in the ConfigDump import functionality.

We have no additional remediation items and consider this issue resolved unless there are further questions.

Comment 12

[Ben Wilson]

Are there any further questions? If not, I will look at closing this on or about next Wednesday, 23-Nov-2022.