1669618 - Apple: Empty SingleExtension in OCSP responses

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[certification_authority]

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0 Safari/605.1.15

Comment 1

[certification_authority]

We have identified that the Apple CA is running a version of EJBCA for OCSP responders that we believe has been impacted similarly to the GlobalSign: Empty SingleExtension in OCSP responses (https://bugzilla.mozilla.org/show_bug.cgi?id=1667944) bug. While we continue our investigation, we will work to provide a full incident report.

Comment 2

[certification_authority]

Full Incident Report

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.
   1. On October 5th at 13:55 PT based on the report issued by GlobalSign (GlobalSign: Empty SingleExtension in OCSP responses https://bugzilla.mozilla.org/show_bug.cgi?id=1667944) we identified that we might be impacted by the same bug. Upon further investigation, we confirmed that our current version of EJBCA exhibits the same non-compliant behavior of including an empty singleExtensions type in OCSP responses.
2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.
   1. 2020-Oct-05 13:55 PT: Based on the report issued by GlobalSign (GlobalSign: Empty SingleExtension in OCSP responses https://bugzilla.mozilla.org/show_bug.cgi?id=1667944), we identified that we might be impacted by the same bug. We began investigating the issue and the impact to our environment.
   2. 2020-Oct-06 08:37 PT: Contacted PrimeKey regarding the Empty SingleExtension issue.
   3. 2020-Oct-06 08:44 PT: Received information from PrimeKey about a software fix for EJBCA.
   4. 2020-Oct-06 09:00 PT: Confirmed the identified issue applied to our environment.
   5. 2020-Oct-06 13:21 PT: Updated OCSP lints and test cases to identify this issue going forward.
   6. 2020-Oct-07 10:01 PT: Received alpha version of EJBCA 7.4.3 from PrimeKey and deployed to UAT; verified with OCSP lints and test cases that the bug was fixed.
3. Whether your CA has stopped, or has not yet stopped, certificate issuance or the process giving rise to the problem or incident. A statement that you have stopped will be considered a pledge to the community; a statement that you have not stopped requires an explanation.
   1. Certificate issuance was not affected in this incident.
4. In a case involving certificates, a summary of the problematic certificates. For each problem: the number of certificates, and the date the first and last certificates with that problem were issued. In other incidents that do not involve enumerating the affected certificates (e.g. OCSP failures, audit findings, delayed responses, etc.), please provide other similar statistics, aggregates, and a summary for each type of problem identified. This will help us measure the severity of each problem.
   1. Certificate issuance was not affected in this incident.
5. In a case involving certificates, the complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem. In other cases not involving a review of affected certificates, please provide other similar, relevant specifics, if any.
   1. Certificate issuance was not affected in this incident.
6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
   1. We are running the latest version of EJBCA, version 7.4.2, which contains this bug.
   2. We have a set of OCSP lints and test cases for testing our OCSP software and they did not detect the issue. As part of our investigation, we determined that a library we use for ASN.1 decode operations is too lenient to ensure RFC compliance with respect to certain ASN.1 module syntax (e.g., SEQUENCE size limits).
7. List of steps your CA is taking to resolve the situation and ensure that such situation or incident will not be repeated in the future, accompanied with a binding timeline of when your CA expects to accomplish each of these remediation steps.
   1. We contacted PrimeKey to inquire about this bug and have been told that the fix is available in version 7.4.3 which can be expected to be released imminently.
   2. Due to the imminent availability of a software version which addresses this issue, as well as compliance fixes in the current software version which address recent BR changes, it was determined that rolling back to a prior software version would not be the best course of action.
   3. We have received the alpha version and have deployed it to our internal testing environment to prepare for the implementation to production OCSP servers as soon as possible once the final version is released.
   4. We immediately added a minimum SEQUENCE size check on the singleExtension field to our OCSP linting tool. We are also working on implementing our own strict ASN.1 module decoding operation for all ASN.1 structures contained in an OCSP response as defined in RFC 6960 (and those in RFC 5280 referenced by RFC 6960). This is expected to be implemented by the end of the calendar year.
   5. We have successfully run our full set of newly enhanced OCSP lints and test cases on the alpha version in UAT.

We expect to provide an update after we upgrade to version 7.4.3 in production, which is projected to occur by the end of the month.

Comment 3

[Ryan Sleevi]

(In reply to certification_authority from comment #2)

2. We have a set of OCSP lints and test cases for testing our OCSP software and they did not detect the issue. As part of our investigation, we determined that a library we use for ASN.1 decode operations is too lenient to ensure RFC compliance with respect to certain ASN.1 module syntax (e.g., SEQUENCE size limits).

Can you share more details about what library you're using?

I wasn't sure if this was something in-house using the SecAsn1 framework (or its spiritual replacements), an open-source library like asn1c, a commercial library like OSS's compiler, or a language integration like Golang.

Any one of these, or any details, could significantly help other CAs avoid similar issues.

Comment 4

[certification_authority]

Can you share more details about what library you're using?

The library referenced is Bouncy Castle.

Comment 5

[certification_authority]

The following tasks have been completed:

• We upgraded EJBCA to 7.4.3 in all environments.
• We updated our OCSP lints and test cases based on the findings of this incident. We additionally implemented our own strict ASN.1 module decoding operation for all ASN.1 structures contained in an OCSP response (except for Certificate and x400Address type of GeneralName based on risk and prioritization of resources).
• We verified compliance against the updated OCSP lints and test cases in production.

This completes the open tasks for this incident.

Comment 6

[Ben Wilson]

It appears that this bug can be closed. I'll schedule to close it on Monday, 21-December, unless there are any issues still to raise.