1659316 - Apple: EV Certificate Approver Authorization

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Apple CA]

Incident Report

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.
   • In the course of a self-audit for internal EV Certificate requests, a member of the CA team identified on Thursday August 13, 2020 at 09:13PT that a recently issued EV Certificate had been approved by a Certificate Approver who had not been properly authorized for EV Authority.
   • This prompted additional investigation, which resulted in identifying 5 additional internal EV Certificates exhibiting the same pattern.
   • Apple only issues EV Certificates to itself for domains it owns.
2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.
   • 2020-07-08: CA received request to add new Certificate Approvers with authority to approve EV Certificates.
   • 2020-08-01: CA completed the verification process to verify name, title and agency and denial list of the new Certificate Approvers.
   • 2020-08-01: Apple’s process to determine EV Authority of Certificate Approvers is based on a Contract Signer’s authorization. The Certificate Approvers who approved the impacted certificates were configured in the system to approve EV Certificates before a Contract Signer’s authorization had been confirmed.
   • 2020-08-08 11:33:17 PT: First impacted certificate issued.
   • 2020-08-12 09:38:14 PT: Final impacted certificate issued.
   • 2020-08-13 09:13:00 PT: In the course of a self-audit for internal EV Certificate requests, it was identified that multiple Certificate Approvers were configured to approve EV Certificates before documented authorization from a Contract Signer was confirmed.
   • 2020-08-13 09:22:00 PT: All pending requests for EV Certificates were blocked from issuance.
   • 2020-08-13 09:33:00 PT: Requests to authorize the Certificate Approvers in question with EV Authority were sent to a Contract Signer.
   • 2020-08-13 10:26:00 PT: Reports were run to determine the full scope of impact and 6 certificates were identified.
   • 2020-08-13 10:52:00 PT: EV Authority for all Certificate Approvers in question was authorized by a Contract Signer.
   • 2020-08-13 11:05:00 PT: The compliance team met to review the details of the issue and determine next steps.
   • 2020-08-13 17:37:00 PT: After EV Authority for all Certificate Approvers in question was authorized by a Contract Signer, certificate issuance was unblocked.
3. Whether your CA has stopped, or has not yet stopped, certificate issuance or the process giving rise to the problem or incident. A statement that you have stopped will be considered a pledge to the community; a statement that you have not stopped requires an explanation.
   • Steps were taken to prevent further issuance of all EV Certificates from the time the issue was noted until after it was resolved. No EV Certificates were issued during that time.
4. In a case involving certificates, a summary of the problematic certificates. For each problem: the number of certificates, and the date the first and last certificates with that problem were issued. In other incidents that do not involve enumerating the affected certificates (e.g. OCSP failures, audit findings, delayed responses, etc.), please provide other similar statistics, aggregates, and a summary for each type of problem identified. This will help us measure the severity of each problem.
   • Number of affected certificates: 6
   • Date of first certificate issuance: 2020-08-08 11:33:17 PT
   • Date of final certificate issuance: 2020-08-12 09:38:14 PT
5. In a case involving certificates, the complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem. In other cases not involving a review of affected certificates, please provide other similar, relevant specifics, if any.
   • The following 6 certificates were impacted. 3 certificates have already been revoked. We expect the remaining 3 certificates to be revoked within 5 days of discovering the issue.
     • https://crt.sh/?id=3207359723
     • https://crt.sh/?id=3227943189
     • https://crt.sh/?id=3227943129
     • https://crt.sh/?id=3227943151
     • https://crt.sh/?id=3227947830
     • https://crt.sh/?id=3227947774
6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
   • Apple’s process to determine EV Authority of Certificate Approvers is based on a Contract Signer’s authorization. Confirming a Contract Signer’s authorization is a manual process dependent on one person’s review. The Certificate Approvers who approved the impacted certificates were configured in the system to approve EV Certificates before a Contract Signer’s authorization had been confirmed. The manual step to confirm the authorization was missed.
   • The final cross-correlations and due diligence did verify that the Certificate Approvers approved the EV requests, but it assumed a Contract Signer’s authorization being correctly confirmed in a preceding step.
7. List of steps your CA is taking to resolve the situation and ensure that such situation or incident will not be repeated in the future, accompanied with a binding timeline of when your CA expects to accomplish each of these remediation steps.
   • An examination of all historical issuance was performed to ensure that this report represents the full problem set.
   • The process to determine EV Authority of Certificate Approvers has been updated to require that a minimum of two people verify a Contract Signer’s authorization before a Certificate Approver is configured in the system to approve EV Certificates. Based on a review of all manual processes related to issuance of EV Certificates, we’ve confirmed that a minimum of two-person confirmation is in place. We are exploring options for systematically enforcement of all such workflows.

We expect to provide an update next week.

Comment 1

[Apple CA]

As of 2020-08-17 08:49:25 PT all impacted certificates have been revoked.

Comment 2

[Apple CA]

We’ve identified a solution to systematically enforce a Contract Signer’s approval before a Certificate Approver is configured for EV Authority. We expect the solution to be implemented in production next week. We will provide an update by the end of next week.

Comment 3

[Apple CA]

We have implemented a solution to systematically enforce a Contract Signer’s approval before a Certificate Approver is configured for EV Authority.

Additionally, we took a closer look at our other manual processes. We use ticketing systems for tracking manual processes. In cases where we felt strengthening a process was needed, we are now leveraging additional features in our ticketing systems to reinforce the steps in these manual processes, and ensure all steps are completed and reviewed by two people.

We believe this completes our remediation for this incident.

Comment 4

[Ryan Sleevi]

we are now leveraging additional features in our ticketing systems to reinforce the steps in these manual processes

Can you share more detail about what is meant by "additional features"? In particular, trying to look at how this can generalize into good practice for the industry, by understanding what are some of the additional features CAs should look for in a ticketing system to help prevent similar mistakes in manual processes.

Comment 5

[Apple CA]

We use an in-house ticketing system that offers the following features, which we now use more extensively:

• Predefined workflow states
• Ownership of individual states
• Automated transition to the next owner

We reviewed our manual processes end to end, and broke them into distinct states with defined outcomes and ownership. We then 1) aligned the steps in the manual process with the workflow states of our ticketing system, 2) assigned ownership for each state, and 3) configured the system to automatically transition the ticket to the next owner in the workflow when the state is changed.

The requirement that all steps are completed and reviewed by two people is incorporated into the ticketing workflow states based on the assigned ownership of each state.

Comment 6

[Ryan Sleevi]

Thanks! This is useful information, and seems to track how other CAs have configured their ticket management flow.

One thing that stands out here is the use of owner singular; we've seen several CAs struggle with situations such as a responsible person being OOO, and deadlines or other requirements being missed. Is there a chance of that happening here? If not, could you describe a bit more about how that flow is managed, particularly for time-sensitive events (e.g. escalations and/or pages)

Comment 7

[Apple CA]

For each manual process, there are multiple people qualified to perform each step in the workflow, and we have documentation that indicates which people are qualified to perform each step. The ticketing system workflow configuration is designed to reinforce the process, but if the primary owner of a particular step is unavailable the ticketing system has enough flexibility (e.g., proxy approver designation) for another qualified person to perform the step.

Since each process is well-defined and documented, requires a minimum of two-person confirmation, and multiple qualified people are available to perform each step in a workflow, we believe the risk that a requirement or deadline would be missed is extremely low.

Additionally, since we only issue EV certificates for domains owned by Apple, the initial validation functions (e.g., domain validation and certificate approver authorization) are not typically time sensitive, not high volume, and are managed by a small group of individuals with easy access to each other. For the final correlation and approval pursuant to certificate issuance, which can be more time-sensitive and higher volume, a larger, highly redundant team with 24x7 coverage is leveraged.

Comment 8

[Ben Wilson]

I think this bug can be closed, unless there are additional questions or concerns to be addressed. I will schedule it for closure on or about 2-Oct-2020.