1605372 - GlobalSign: OCSP responders found to respond signed by the default CA when passed an invalid issuer in request

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Paul Brown]

Incident Report

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

Wednesday, December 18 20:47 : Microsoft informed us of issue with their OCSP Responder had been reported by a security researcher

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

Wednesday, December 18 21:23 : Identified issue on OCSP cluster
Thursday , December 19 06:11 : Issue matched with EJBCA issue https://jira.primekey.se/browse/ECA-8620 and noted that fix was currently implemented on staging environmment for testing, but not ready for production
Thurdsay - ascertain whether it was a BR violation, or just a “best practice” violation
Thursday afternoon - identified workaround
Friday, December 20 10:53 : First OCSP Cluster updated with workaround
Friday, December 20 10:29 : Identified this was same issue as Apple OCSP Issue reported here
Monday, December 23 : Other clusters will be updated with workaround

3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

No non-compliant certificates were issued. The OCSP workaround has been pushed out to the first of our production OCSP clusters and scheduled for completion on other clusters by 23 December 2019.

4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

No non-compliant certificates were issued.

5. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

No non-compliant certificates were issued.

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

Issue was due to issue in EJBCA as identified on https://jira.primekey.se/browse/ECA-8620

7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

We have identified that the Apple OCSP Issue was originally not thoroughly investigated due to fact it was apparently caused by a very old version of EJBCA. In light of this we are investigatin best way to integrate all newly reported CA Certificate Compliance bugzilla tickets to our SOC reporting procedure in order that they are fully investigated as possibly affecting our own systems.

Comment 1

[Wayne Thayer]

I believe the incident report is referencing bug #1588001, an Apple incident report.

Paul: Thank you for the incident report. I have a few questions:

• Can you explain more about "Microsoft informed us of issue with their OCSP Responder" ? Is this a reference to a Microsoft product or an OCSP responder operated by or for Microsoft? How does it relate to this issue?

• Can you explain what the "workaround" is so that other CAs can benefit from this information?

• Did Primekey inform customers of this issue? In general, how does Globalsign ensure that you are aware of bugs in your COTS CA software that may require action?

Comment 2

[Ryan Sleevi]

Can you also indicate which OCSP responders were affected by this, so that they can be appropriately mapped to scope and impact. The incident response leaves it ambiguous (as Wayne mentioned) about which parties were affected and the scope.

Comment 3

[Paul Brown]

• Can you explain more about "Microsoft informed us of issue with their OCSP Responder" ? Is this a reference to a Microsoft product or an OCSP responder operated by or for Microsoft? How does it relate to this issue?

Oscar Karlsson (security researcher) informed Microsoft of issue. Microsoft informed GlobalSign (as service provider) of the issue

• Can you explain what the "workaround" is so that other CAs can benefit from this information?

We disabled the default OCSP signer, thus the system responds with Unauthorized (unsigned) vs. unknown signed from a default signer.

• Did Primekey inform customers of this issue? In general, how does Globalsign ensure that you are aware of bugs in your COTS CA software that may require action?

We do not find any Primekey notification regarding this release, however we have asked for confirmation whether one was sent
We employ various methods for bug awareness, information can come via either of the following mechanisms:
Security Assessment activities (e.g. Vulnerability Scanning, Penetration testing, Source code review etc.)
Vulnerability Alert Service
Security Advisories from vendors
Vulnerability disclosures by security researchers including zero day vulnerabilities.
Reported by Internal Employees
Internal audits and other compliance assessments

• Can you also indicate which OCSP responders were affected by this, so that they can be appropriately mapped to scope and impact. The incident response leaves it ambiguous (as Wayne mentioned) about which parties were affected and the scope.

For TLS only Microsoft and GlobalSign OCSP responders were affected

As part of this investigation however we have found that one of our infra team informed Primekey of this issue 4 days before the Apple incident, without realising this was a BR violation and due to the previously mentioned issue with lack of thorough investigation of the Apple incident, the seriousness of the issue was lost between two teams, so we will solve this by giving direct ownership to all sources of "alerts" to one team who will lead to conclusion.

Comment 4

[Wayne Thayer]

Monday, December 23 : Other clusters will be updated with workaround

Paul: please confirm that all remediation steps have been completed.

Comment 5

[Paul Brown]

Hi Wayne

All remediation steps - default responders were removed - were implemented for the OCSP issue.
Also all new CA Certificate Compliance bugzilla tickets from any CA now automatically create a ticket in our SOC Teams incident management system and these tickets have to be investigated and reported on by the relevant team before being signed off.

Comment 6

[Wayne Thayer]

It appears that all questions have been answered and remediation is complete.