1667944 - GlobalSign: Empty SingleExtension in OCSP responses

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Paul Brown]

Incident report:

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

During gap analysis and impact assessment of the changes to the BR in the context of SC31 – Browser Alignment, we noted that our legacy platform, using EJBCA as issuance backend includes a SingleExtension with an empty SEQUENCE when no SingleExtensions are selected. This is not in compliance with RFC 6960 section 4.2.1 when read in conjunction with RFC 5280 section 4.1 which allows for SEQUENCE size 1..MAX

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

Date	Action
September 07 2020	Noticed issue with response, requested further testing
September 07 2020	Further testing showed that issue appeared to have been introduced in version 7.3.0 of EJBCA and that in particular it appeared to have come from EJBCAs OCSP Archive Cutoff addition.
September 08 2020	Informed Primekey of issue who created https://jira.primekey.se/browse/ECA-9426 for bug fix

3. Whether your CA has stopped, or has not yet stopped, certificate issuance or the process giving rise to the problem or incident. A statement that you have stopped will be considered a pledge to the community; a statement that you have not stopped requires an explanation.

N/A No certificates affected

If revocation is required, decision and rationale for delaying revocation. See "Timeline for revocation breached"

N/A

4. In a case involving certificates, a summary of the problematic certificates. For each problem: the number of certificates, and the date the first and last certificates with that problem were issued. In other incidents that do not involve enumerating the affected certificates (e.g. OCSP failures, audit findings, delayed responses, etc.), please provide other similar statistics, aggregates, and a summary for each type of problem identified. This will help us measure the severity of each problem.

N/A

5. In a case involving certificates, the complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem. In other cases not involving a review of affected certificates, please provide other similar, relevant specifics, if any.

N/A

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

In line with industry requirements have always kept our software up to date and do test all new releases from suppliers. However this particular issue was not spotted during this testing phase until now.

7. List of steps your CA is taking to resolve the situation and ensure that such situation or incident will not be repeated in the future, accompanied with a binding timeline of when your CA expects to accomplish each of these remediation steps.

We have informed Primekey of this issue and are awaiting release of new version which will solve the issue. We did consider rolling back however we required latest version in order to be compliant with changes implemented in Ballot SC31 for new Baseline Requirements.

For testing new releases we have added a full asn1 comparison check of OCSP responses from before/after to our testing procedures when OCSP changes are implemented.

Comment 1

[Ben Wilson]

Hi Paul,
Are there any updates before I close this? Did PrimeKey provide a fix for the issue?
Thanks,
Ben

Comment 2

[Paul Brown]

Hi Ben

Yes, Primekey released version 7.4.3 (https://doc.primekey.com/ejbca/ejbca-release-information/ejbca-release-notes/ejbca-7-4-3-release-notes) at the end of October which includes a fix for this issue. We are currently testing the release on our staging environment and will be updating our live systems upon successful completion of this testing.

Comment 3

[Ryan Sleevi]

There's been no update for two months. Are you still testing? If not, why no update?

Comment 4

[Paul Brown]

Hi Ryan

The release took longer than expected, we encountered some xml parsing errors during release testing on our staging environment. We are currently migrating a part of our infrastructure to new systems so instead of installing on EOL systems, this release has been bundled into these deployments and is due in 3rd and 4th week of January.

Comment 5

[Ryan Sleevi]

1. Can you be more precise about "XML parsing errors"
2. Why has GlobalSign failed to provide weekly updates, as described at https://wiki.mozilla.org/CA/Responding_To_An_Incident#Keeping_Us_Informed , including an update that things would be delayed, for example?

Comment 6

[Paul Brown]

The xml parsing errors were caused by one of our configuration attributes using a older data scheme and were unrelated to the OCSP solution so were never flagged as having to be reported. When the go-ahead to install on production was given, the new systems were updated since these were the systems the infra team are focusing on.

Given the migration, the feedback from the infrastructure team was delayed on the progress of this update. We shall provide a status update next week.

Comment 7

[Paul Brown]

To update, we are still on schedule to complete migration of all our affected OCSP Responders to new systems this week. As stated the new systems are running the updated EJBCA release. We will update upon completion of migration.

Comment 8

[Paul Brown]

To update, our migration is ongoing and our first batch of Responders from our new systems are live. We have verified that these responses do not include the empty single extension. We shall update upon completion.

Comment 9

[Paul Brown]

During our rollout we encountered an issue with one of our endpoints not serving an Expires header. We paused migration whilst we investigated this issue with Primekey. We now have a solution in place and have resumed migration.

Comment 10

[Paul Brown]

As update we confirm migration was completed for all OCSP responders.

Comment 11

[Ryan Sleevi]

(In reply to Paul Brown from comment #9)

During our rollout we encountered an issue with one of our endpoints not serving an Expires header. We paused migration whilst we investigated this issue with Primekey. We now have a solution in place and have resumed migration.

Can you share more details here about the investigation, cause, and mitigation?

Comment 12

[Paul Brown]

Historically GlobalSign did not respond with a nonce for OCSP requests which included a nonce. Towards the end of last year, however, we started receiving an increasing number of requests to include it for our QWAC/Qualified certificate offerings due to a significant proportion of European Financial Bodies not accepting a response without a nonce. After internal discussion we concluded that we could turn it on for some specific profiles.

Due to the upcoming migration, changes to profiles on our old systems had been frozen and the changes were designated for the profiles on our new systems. As we rolled over profiles to new systems, it was noted that the RFC5019 caching header was missing for some profiles. We stopped migration to investigate the reason and found this was due to the inclusion of the nonce.

Comment 13

[Ben Wilson]

Unless there are additional matters to discuss, I intend to close this matter on 5-March-2021.