1533655 - DigiCert: Apple: Non-compliant Serial Numbers

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[certification_authority]

On 2019-03-06 we determined that we were issuing certificates with non-compliant serial numbers because of the EJBCA issue [1]. We fixed the problem within 24 hours and stopped issuing non-compliant certificates as of the afternoon of 2019-03-07 and are working to address previously issued certificates. All impacted certificates were issued to Apple entities.

To minimize impact to our users, we do not expect to revoke all impacted certificates within the 5-day requirement. We expect to provide a timeline for revoking all impacted certificates in a forthcoming update.

Certificates issued by the following CA's were impacted:

• Apple IST CA 2 - G1 (https://crt.sh/?id=5250464): TLS Server
• Apple IST CA 8 - G1 (https://crt.sh/?id=21760447): TLS Server
• Apple IST CA 5 - G1 (https://crt.sh/?id=12716200): S/MIME

Based on our initial analysis, the number of certificates impacted are as follows:

• TLS Server Certificates (from Apple IST CA 2 - G1, Apple IST CA 8 - G1)

  • Total number of impacted certificates (issued since 2016-09-30): ~878,000
  • Total number of impacted certificates that are still valid (not expired and not revoked) as of 2019-03-07 3:20 PST: ~558,000

• S/MIME Certificates (from Apple IST CA 5 - G1)

  • Total number of impacted certificates (issued since 2016-09-30): ~2,400
  • Total number of impacted certificates that are still valid (not expired and not revoked) as of 2019-03-07 3:20 PST: ~2,000

We expect to reply back with more details and a full incident report in a forthcoming update.

[1] Configurable SN Entropy, Default Value Raised to 20 Octets (https://www.ejbca.org/docs/EJBCA_7.0.1_Release_Notes.html)

Comment 1

[Ryan Sleevi]

Apple folks,

Can you please provide the details of the 558,000 TLS certificates and 2,000 S/MIME certificates?

Further, given that this is not (yet) a preliminary incident report in https://wiki.mozilla.org/CA/Responding_To_An_Incident , can you please provide one in the next 24 hours?

Similarly, when deferring matters to forthcoming updates, can you provide clear timelines about when the next update is expected? It's unclear whether you expect to provide an update one day, one week, or one month from now, and providing clear timelines helps manage expectations about that.

Comment 2

[certification_authority]

Incident Report

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

   Apple first became aware that there might be an issue while reviewing the release notes for an updated version of the CA Software used for issuing publicly trusted certificates [1]. After further investigation, including a review of related discussions on the Mozilla Dev Security Policy forum, it was determined that certificates with non-compliant serial numbers were being issued.

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

   • 2016-09-30 - CA/Browser Forum requires the use of 64 bit entropy when generating serial numbers [2].
   • 2017-02-28 - The Mozilla Root Store Policy requires that certificates must have a serial number greater than zero (0) containing at least 64 bits of output from a CSPRNG [3].
   • 2017-04-10 3:45 PM - Mistakenly suppressed alerts detecting serial numbers suspected to be insufficient in length.
   • 2019-03-05 8:00 AM - We became aware of a potential issue with our understanding and configuration of the CA Software used for issuing publicly trusted certificates, and began investigating.
   • 2019-03-06 5:19 PM - Determined that certificates were being issued with non-compliant serial numbers.
   • 2019-03-06 7:43 PM - Leadership engaged the team to begin gathering data to assess risk and determine next steps.
   • 2019-03-06 9:21 PM - Performed an initial risk assessment and discussed next steps for remediating the issue including revocation of impacted certificates.
   • 2019-03-07 12:13 AM - Generated an initial report on the volume and nature of the impacted certificates.
   • 2019-03-07 8:19 AM - Notified DigiCert (the Root CA) of the incident.
   • 2019-03-07 10:00-11:00 AM - Finalized the action plan to stop issuing non-compliant TLS Server certificates and initial steps taken.
   • 2019-03-07 2:10 PM - Stopped issuance of TLS Server certificates with non-compliant serial numbers.
   • 2019-03-07 2:17 PM - Re-instated alerts for detecting serial numbers suspected to be insufficient in length.
   • 2019-03-07 3:00 PM - Met with DigiCert to further discuss the issue and remediation steps.
   • 2019-03-07 3:55 PM - Determined that S/MIME certificates were also impacted and developed a plan to stop issuance.
   • 2019-03-07 4:32 PM - Stopped issuance of S/MIME certificates with non-compliant serial numbers.
   • 2019-03-07 5:00 PM - Began revocation impact assessment (see below for more details).
   • 2019-03-07 11:19 PM - Posted preliminary incident report to Bugzilla [4].
   • 2019-03-08 10:00 AM - Notified Ernst & Young (WebTrust assessors).
   • 2019-03-08 5:02 PM - Commenced revocation of impacted certificates.

3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

   Issuance of non-compliant TLS Server certificates stopped on 2019-03-07 at 2:10 PM.
   Issuance of non-compliant S/MIME certificates stopped on 2019-03-07 at 4:32 PM.

4. summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

   The number of certificates impacted are as follows:

   • TLS Server Certificates (from Apple IST CA 2 - G1 (https://crt.sh/?id=5250464), Apple IST CA 8 - G1 (https://crt.sh/?id=21760447))

     • Total number of impacted certificates (issued since 2016-09-30): 877,253
     • Date the first certificate with the problem was issued: 2016-09-30
     • Date the last certificate with the problem was issued: 2019-03-07

   • S/MIME Certificates (from Apple IST CA 5 - G1 (https://crt.sh/?id=12716200))

     • Total number of impacted certificates (issued since 2017-02-28[3]): 2,224
     • Date the first certificate with the problem was issued: 2017-02-28
     • Date the last certificate with the problem was issued: 2019-03-07

5. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

   Files have been attached with the following data:

   • List of all impacted certificates issued.
   • List of impacted certificates still valid (not expired and not revoked) as of 5 days from when the incident was identified.

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

   When Ballot 164 [2] was passed, we confirmed that the CA Software was already configured for 64-bit serial numbers, but did not realize the serial number generation algorithm resulted in 63 bits of entropy.

   In addition to the CA Software, a validator (similar to linting tools) checks each issued certificate for compliance with SSL Baseline Requirements. Alerts were generated that the serial numbers were insufficient, however; because the checks were performed against individual certificates and not collections of certificates, the alerts were mistakenly determined to be incorrect and were suppressed.

7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

   • Apple stopped issuance of certificates with non-compliant serial numbers, and is continuing to work with users to revoke impacted certificates.
   • CA Software has been configured to generate serial numbers with 16 octets, ensuring entropy greater than 64 bits.
   • Re-instated alerts for detecting serial numbers suspected to be insufficient in length.
   • The validator software that checks certificates for SSL Baseline compliance will be enhanced to evaluate collections of certificates instead of individual certificates. These enhancements are expected to be implemented by April 30, 2019.
   • Subsequent to suppressing the serial number check alert, and prior to identifying the current issue, a process was implemented to provide more oversight for changes to alerts. This enhanced process may have been sufficient to prevent the incorrect suppression of the serial number alert, but the process will be reviewed again to identify if further enhancements are required. This will be completed by March 31, 2019.

Revocation Status

More than 355,000 certificates have been revoked since the incident was detected.

23% of total, impacted certificates issued were still valid (not expired and not revoked) as of 5 days from when the incident was identified. Further analysis is being performed to determine an appropriate schedule for revoking these certificates.

While the CA had the capability to revoke all impacted certificates within 5 days and provides APIs enabling automated management of certificates, it was determined that for some certificates the operational risk outweighed the security risk and those certificates have not yet been revoked.

In conjunction with teams responsible for assessing risk to users, risk assessments were performed. The operational risk was determined to be high because immediate revocation of certain impacted certificates would cause significant interruption to users and services. The security risk was determined to be low because the certificates are SHA256 signed, serial numbers were generated with 63 bits of entropy, and all certificates were issued to Apple entities.

We are working with the remaining teams across Apple to determine an appropriate timeline for revoking the remaining certificates that balances the non-compliance risk against the operational impact to users and the ecosystem at large.

We expect to provide more details in an update as early as March 21, 2019.

[1] Configurable SN Entropy, Default Value Raised to 20 Octets (https://www.ejbca.org/docs/EJBCA_7.0.1_Release_Notes.html)
[2] CA/Browser Forum Ballot 164 (https://cabforum.org/pipermail/public/2016-June/007861.html)
[3] Mozilla Root Store Policy, version 2.4, section Maintaining Confidence in Included Root Certificates, number 7 (https://wiki.mozilla.org/CA/Root_Store_Policy_Archive)
[4] DigiCert: Apple: Non-compliant Serial Numbers Mozilla Bug Report (https://bugzilla.mozilla.org/show_bug.cgi?id=1533655)

Comment 3

[certification_authority]

Attachment: List of all impacted certificates issued.

Comment 4

[certification_authority]

Attachment: Apple_204705_remaining_impacted_certificates.txt.zip

Comment 5

[certification_authority]

Over 115,000 additional certificates have been revoked since our last update leaving less than 10% of the total population of impacted certificates remaining (file attached with remaining certificates).

We continue to work with teams across Apple to revoke the remaining certificates on a timeline that balances the non-compliance risk against the operational impact to users and the ecosystem at large.

Another update will be provided next week.

Comment 6

[certification_authority]

Attachment: Apple_84207_remaining_impacted_certificates.txt.zip

Apple_84207_remaining_impacted_certificates.txt.zip

Comment 7

[Ryan Sleevi]

Another update will be provided next week.

Apple folks: In general, this approach makes it difficult for the community to understand the effectiveness of Apple's incident response process and to have confidence that Apple is tackling this matter with all urgency. In particular, the lack of clear timelines as to when Apple expects to resolve this matter holistically - if it does - makes it difficult to measure Apple's progress and effectiveness in achieving those timelines.

When considering incident response reports, a key benefit for the community is having clear timelines, both to the holistic resolution and to individual milestones. As can be seen from past incidents on the https://wiki.mozilla.org/CA/Incident_Dashboard , this is a common expectation of CAs. While such deadlines are fundamentally non-binding, as they rely upon the CA's actions, they provide insight into how well a CA is able to scope the problem and achieve results. If matters arise that otherwise cause deviation from these timelines, sharing such information as it becomes available helps the broader community understand and assess impact.

In your future update, it would be useful to understand what the timelines are for remaining actions. If further revocations are expected, when will they be performed? If Apple does not plan to revoke some certificates, what are the scope of those certificates, etc? Including such in your next update would best align with industry standard practice for CA incidents.

Comment 8

[certification_authority]

We've been working our plan to revoke impacted certificates. Thus far over 500,000 certificates have been revoked since the issue was identified and 54,853 remain (file attached with remaining certificates). Our plan will result in all impacted certificates being revoked.

Our approach to resolving this incident has been to strike a balance between a compliance incident with low associated security risk and impact to critical services. We have established a timeline to address the remaining certificates that minimizes service impact and allows standard QA and change control processes to ensure uptime is not affected.

As part of the remediation plan, a number of certificates will be migrated to an internal, enterprise PKI, which will take more time.

Based on these factors, it is expected that most certificates will be revoked by April 30 with less than 2% extending until July 15.

Another update will be provided next week.

Comment 9

[certification_authority]

Attachment: Apple_54853_remaining_impacted_certificates.txt.zip

Comment 10

[certification_authority]

Over 10,000 additional certificates have been revoked since our last update.

In a previous update, we committed to doing the following:

“Subsequent to suppressing the serial number check alert, and prior to identifying the current issue, a process was implemented to provide more oversight for changes to alerts. This enhanced process may have been sufficient to prevent the incorrect suppression of the serial number alert, but the process will be reviewed again to identify if further enhancements are required. This will be completed by March 31, 2019.”

This review was completed. As a result, additional approval is required before alerts are modified and alerts will be tested on a quarterly basis to ensure they continue to function as intended.

We expect to provide the next update soon after the April 30 milestone, on or about May 3.

Comment 11

[certification_authority]

Most certificates have been revoked and less than 1% of the total population of impacted certificates remains. As previously noted, it is expected that all impacted certificates will be revoked by July 15.

In a previous update, we committed to doing the following:

“The validator software that checks certificates for SSL Baseline compliance will be enhanced to evaluate collections of certificates instead of individual certificates. These enhancements are expected to be implemented by April 30, 2019.”

The referenced software enhancements were implemented as planned prior to April 30.

We expect to provide the next update soon after the July 15 milestone, on or about July 19.

Comment 12

[certification_authority]

In a previous update, we committed to addressing all impacted certificates by July 15. As of July 10, all valid, impacted certificates had been revoked.

Additionally, we would like to supplement two of our responses in the incident report:

In response number 4, the final number of impacted S/MIME certificates is 2,225, one more than originally reported. Its serial number is: 4119314310222888743. The window of time used to generate the original report of impacted S/MIME certificates was off by a few minutes.

In response number 7, one additional step was executed, and we believe it will simplify future massive revocation events. Close to 80% of certificates were transitioned to an enterprise PKI.

No additional updates to this incident report are planned.

Comment 13

[Wayne Thayer]

Thank you for the update.

It appears that all questions have been answered and remediation is complete.