1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.
On May 23, 2022, an internal team brought to our attention that our OCSP validation service was responding ‘unknown’ instead of ‘good’ for valid S/MIME certificates.
2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.
- 2020-04-29: Effective date of the Apple Public CPS version 5.0, which first stated that OCSP responses for valid S/MIME certificates should be ‘good’ and not ‘unknown’.
- 2021-10-07 at 15:24 PT: We added our OCSP linter to our monitoring solution to check OCSP responses for TLS certificates, for conformance with RFC 6960. See Comment 21 of Apple: OCSP responders return responses with incorrect issuer for our OCSP lints and test cases during that time. We did not configure our OCSP linter for our S/MIME CAs at that time.
- 2022-01-14 at 08:21 PT: Our operations team changed the OCSP publisher configuration for S/MIME certificates to only publish when a certificate is revoked. The resulting behavior of that change is that the OCSP responder began responding as 'unknown' for issued non-revoked non-expired certificates instead of 'good'.
- 2022-05-23 at 10:56 PT: An internal team notified our support team that the OCSP response for their valid S/MIME certificate was ‘unknown’ instead of ‘good’.
- 2022-05-23 at 11:27 PT: Our operations team confirmed the behavior and identified the cause was the change made to the OCSP publisher for S/MIME certificates on 2022-01-14 at 08:21 PT.
- 2022-05-23 at 12:57 PT: Our compliance team confirmed that this was an incident because our OCSP validation service was responding ‘unknown’ instead of ‘good’ for valid S/MIME certificates, which did not meet our stated practice for OCSP responses in section 7.3.1 of the Apple Public CPS.
- 2022-05-23 at 15:22 PT: Our operations team applied a fix to update the OCSP publisher configuration that caused our OCSP responders to respond ‘good’ for valid S/MIME certificates.
- 2022-05-23 at 16:48 PT: The internal team was notified that the issue was fixed.
- 2022-05-26 at 13:05 PT: We held a Root Cause Analysis Review.
- 2022-05-26 at 15:38 PT: We filed the Bugzilla and posted our initial issue report.
- 2022-06-02 at 15:05 PT: We updated our OCSP linter to add a lint to indicate a response of ‘good’ should be returned for non-revoked certificates.
- 2022-06-08 at 21:21 PT: We added our OCSP linter to our S/MIME CAs so we can detect if our OCSP responses violate industry requirements/standards.
3. Whether your CA has stopped, or has not yet stopped, certificate issuance or the process giving rise to the problem or incident. A statement that you have stopped will be considered a pledge to the community; a statement that you have not stopped requires an explanation.
This did not affect certificate issuance. Our operations team applied a fix to update the OCSP publisher configuration that caused our OCSP responders to respond ‘good’ for valid certificates on 2022-05-23 at 15:22 PT.
4. In a case involving certificates, a summary of the problematic certificates. For each problem: the number of certificates, and the date the first and last certificates with that problem were issued. In other incidents that do not involve enumerating the affected certificates (e.g. OCSP failures, audit findings, delayed responses, etc.), please provide other similar statistics, aggregates, and a summary for each type of problem identified. This will help us measure the severity of each problem.
There were no problematic certificates issued in error, hence no certificates needed to be revoked and reissued. However, 230 S/MIME certificates were issued between the start of the incident and remediation of the incident that had incorrect OCSP responses:
- first affected cert issued at 2022-01-14 22:28:54 GMT
- last affected cert issued at 2022-05-20 18:12:00 GMT
5. In a case involving certificates, the complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem. In other cases not involving a review of affected certificates, please provide other similar, relevant specifics, if any.
There were no problematic certificates issued in error, hence no certificates needed to be revoked and reissued.
Because these are S/MIME certificates and are not logged to CT servers, they do not exist in crt.sh. In lieu of that, a file has been attached listing all certificates issued between the date ranges provided in response to question 4: affected_sha256.txt
6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
How and why the mistakes were made or bugs introduced:
- Our operations team made an intentional change to the OCSP publisher configuration for S/MIME certificates and did not realize it violated our stated practice for OCSP responses in the Apple Public CPS. The change was operationally driven to manage database size for our validation service.
How the mistakes avoided detection until now:
- Our OCSP linter, while configured for our TLS CAs, was not configured to monitor OCSP responses for our S/MIME CAs.
- Had our OCSP linter been monitoring OCSP responses for our S/MIME CAs, it would not have caught this issue because it was designed to check for statements like "MUST", "MUST NOT", "REQUIRED", "SHALL", and “SHALL NOT” in the standards and there is nothing in RFC 6960 that explicitly states an OCSP response MUST respond with the status ‘good’ for an issued non-revoked non-expired certificate.
7) List of steps your CA is taking to resolve the situation and ensure that such situation or incident will not be repeated in the future, accompanied with a binding timeline of when your CA expects to accomplish each of these remediation steps.
- Future changes to OCSP publishers will be reviewed by members of the team familiar with the practice for OCSP responses in the Apple Public CPS.
- Our OCSP linter has been added to our S/MIME CAs so we can proactively detect if our OCSP responses violate industry requirements/standards.
- Our OCSP linter was updated to add a lint to indicate a response of ‘good’ should be returned for issued non-revoked non-expired certificates.