1771398 - Apple: OCSP responders return ‘unknown’ for valid S/MIME and TLS certificates

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[certification_authority]

On May 23, 2022, it came to our attention that our OCSP validation service was responding ‘unknown’ instead of ‘good’ for valid S/MIME certificates, which does not meet our stated practice for OCSP responses in the Apple Public CPS. This has since been resolved.

A full report will be provided no later than June 9, 2022.

Comment 1

[certification_authority]

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

On May 23, 2022, an internal team brought to our attention that our OCSP validation service was responding ‘unknown’ instead of ‘good’ for valid S/MIME certificates.

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

• 2020-04-29: Effective date of the Apple Public CPS version 5.0, which first stated that OCSP responses for valid S/MIME certificates should be ‘good’ and not ‘unknown’.
• 2021-10-07 at 15:24 PT: We added our OCSP linter to our monitoring solution to check OCSP responses for TLS certificates, for conformance with RFC 6960. See Comment 21 of Apple: OCSP responders return responses with incorrect issuer for our OCSP lints and test cases during that time. We did not configure our OCSP linter for our S/MIME CAs at that time.
• 2022-01-14 at 08:21 PT: Our operations team changed the OCSP publisher configuration for S/MIME certificates to only publish when a certificate is revoked. The resulting behavior of that change is that the OCSP responder began responding as 'unknown' for issued non-revoked non-expired certificates instead of 'good'.
• 2022-05-23 at 10:56 PT: An internal team notified our support team that the OCSP response for their valid S/MIME certificate was ‘unknown’ instead of ‘good’.
• 2022-05-23 at 11:27 PT: Our operations team confirmed the behavior and identified the cause was the change made to the OCSP publisher for S/MIME certificates on 2022-01-14 at 08:21 PT.
• 2022-05-23 at 12:57 PT: Our compliance team confirmed that this was an incident because our OCSP validation service was responding ‘unknown’ instead of ‘good’ for valid S/MIME certificates, which did not meet our stated practice for OCSP responses in section 7.3.1 of the Apple Public CPS.
• 2022-05-23 at 15:22 PT: Our operations team applied a fix to update the OCSP publisher configuration that caused our OCSP responders to respond ‘good’ for valid S/MIME certificates.
• 2022-05-23 at 16:48 PT: The internal team was notified that the issue was fixed.
• 2022-05-26 at 13:05 PT: We held a Root Cause Analysis Review.
• 2022-05-26 at 15:38 PT: We filed the Bugzilla and posted our initial issue report.
• 2022-06-02 at 15:05 PT: We updated our OCSP linter to add a lint to indicate a response of ‘good’ should be returned for non-revoked certificates.
• 2022-06-08 at 21:21 PT: We added our OCSP linter to our S/MIME CAs so we can detect if our OCSP responses violate industry requirements/standards.

3. Whether your CA has stopped, or has not yet stopped, certificate issuance or the process giving rise to the problem or incident. A statement that you have stopped will be considered a pledge to the community; a statement that you have not stopped requires an explanation.

This did not affect certificate issuance. Our operations team applied a fix to update the OCSP publisher configuration that caused our OCSP responders to respond ‘good’ for valid certificates on 2022-05-23 at 15:22 PT.

4. In a case involving certificates, a summary of the problematic certificates. For each problem: the number of certificates, and the date the first and last certificates with that problem were issued. In other incidents that do not involve enumerating the affected certificates (e.g. OCSP failures, audit findings, delayed responses, etc.), please provide other similar statistics, aggregates, and a summary for each type of problem identified. This will help us measure the severity of each problem.

There were no problematic certificates issued in error, hence no certificates needed to be revoked and reissued. However, 230 S/MIME certificates were issued between the start of the incident and remediation of the incident that had incorrect OCSP responses:

• first affected cert issued at 2022-01-14 22:28:54 GMT
• last affected cert issued at 2022-05-20 18:12:00 GMT

5. In a case involving certificates, the complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem. In other cases not involving a review of affected certificates, please provide other similar, relevant specifics, if any.

There were no problematic certificates issued in error, hence no certificates needed to be revoked and reissued.

Because these are S/MIME certificates and are not logged to CT servers, they do not exist in crt.sh. In lieu of that, a file has been attached listing all certificates issued between the date ranges provided in response to question 4: affected_sha256.txt

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

How and why the mistakes were made or bugs introduced:

• Our operations team made an intentional change to the OCSP publisher configuration for S/MIME certificates and did not realize it violated our stated practice for OCSP responses in the Apple Public CPS. The change was operationally driven to manage database size for our validation service.

How the mistakes avoided detection until now:

• Our OCSP linter, while configured for our TLS CAs, was not configured to monitor OCSP responses for our S/MIME CAs.
• Had our OCSP linter been monitoring OCSP responses for our S/MIME CAs, it would not have caught this issue because it was designed to check for statements like "MUST", "MUST NOT", "REQUIRED", "SHALL", and “SHALL NOT” in the standards and there is nothing in RFC 6960 that explicitly states an OCSP response MUST respond with the status ‘good’ for an issued non-revoked non-expired certificate.

7) List of steps your CA is taking to resolve the situation and ensure that such situation or incident will not be repeated in the future, accompanied with a binding timeline of when your CA expects to accomplish each of these remediation steps.

• Future changes to OCSP publishers will be reviewed by members of the team familiar with the practice for OCSP responses in the Apple Public CPS.
• Our OCSP linter has been added to our S/MIME CAs so we can proactively detect if our OCSP responses violate industry requirements/standards.
• Our OCSP linter was updated to add a lint to indicate a response of ‘good’ should be returned for issued non-revoked non-expired certificates.

Comment 2

[certification_authority]

Attachment: affected_sha256.txt

Comment 3

[certification_authority]

Apple CA is continuing to monitor this bug for comments or questions.

Comment 4

[certification_authority]

Further investigation has found an additional case of the same issue for one TLS certificate (https://crt.sh/?id=6927218896). This has since been resolved. We will provide additional details in next week’s update.

Comment 5

[certification_authority]

On 2022-01-14 at 08:21 PT, when our operations team changed the OCSP publisher configuration for S/MIME certificates to only publish when a certificate is revoked, they applied the same change to the certificate profile applicable to the one affected TLS certificate. Our investigation revealed that this was the only TLS certificate issued from the affected TLS certificate profile between the start of the incident and the ultimate remediation of the incident on 2022-06-14 at 15:10 PT. We have re-verified the OCSP publisher configurations for all public S/MIME and TLS certificates to ensure that the OCSP publisher configurations are now correct.

Amended timeline:

• 2022-01-14 at 08:21 PT: Our operations team changed the OCSP publisher configuration for S/MIME certificates and the affected TLS certificate profile to only publish when a certificate is revoked. The resulting behavior of that change is that the OCSP responder began responding as 'unknown' for issued non-revoked non-expired certificates instead of 'good'.
• 2022-06-14 at 14:03 PT: Our operations team identified an additional case of the same issue for one TLS certificate (https://crt.sh/?id=6927218896) and verified that the affected TLS certificate profile was the only profile excluded in the 2022-01-14 at 08:21 PT change.
• 2022-06-14 at 15:10 PT: Our operations team applied a fix to update the OCSP publisher configuration that caused our OCSP responders to respond ‘good’ for the valid TLS certificate.

How and why the mistakes were made or bugs introduced:

• The OCSP configuration mentioned in our initial incident report was a shared configuration that impacted both S/MIME certificates and the affected TLS certificate.

How the mistakes avoided detection until now:

• Our strategy with our OCSP linter had been to lint a sample set of certificates from each CA, but that sample did not include certificates from all certificate profiles under a given CA. The sample set of certificates used by our OCSP linter did not include certificates from the affected TLS certificate profile and therefore did not detect the issue.

Remediations

• Our operations team reassigned the affected TLS certificate profile to our dedicated CABF OCSP publisher configuration on 2022-06-14 at 15:10 PT.
• Short term, by 2022-07-22, we are going to run the lint that checks for the correct OCSP status on every certificate we issue. Long-term, by 2022-09-23, we are going to examine the feasibility of running all lints on every certificate we issue.

We will provide further information on these remediations in next week’s update.

Comment 6

[certification_authority]

We are continuing to work on our short term remediation and are on track to complete this by 2022-07-22. We are monitoring this bug for comments or questions and will post another update within a week.

Comment 7

[certification_authority]

Work on our short term remediation is progressing well and remains on track for completion by 2022-07-22. We are monitoring this bug for comments or questions and will post another update within a week.

Comment 8

[certification_authority]

Apple Public CA continues to monitor this bug for comments or questions. We will post an update on our short term remediation on 2022-07-22.

Comment 9

[certification_authority]

We have completed work on our short-term remediation, and are now running the lint that checks for the correct OCSP status on every certificate we issue. We have begun work on our long-term remediation and will post another update next week. We will continue to monitor this bug for comments or questions.

Comment 10

[certification_authority]

We are continuing work on our long-term remediation to examine the feasibility of running all lints on every certificate we issue.

Apple Public CA continues to monitor this bug for comments or questions.

Comment 11

[certification_authority]

We are continuing work on our long-term remediation to examine the feasibility of running all lints on every certificate we issue. Apple Public CA continues to monitor this bug for comments or questions.

Comment 12

[certification_authority]

We have examined the feasibility of running all lints on every public certificate we issue, determined that it is feasible, and are now running all lints on every public certificate we issue.

We have no additional remediation items and consider this issue resolved unless there are further questions.

Comment 13

[Ben Wilson]

I'll close this next week sometime (9/26 - 9/30) unless there are any issues raised that still need to be resolved.