1579509 - SSL.com: Precertificates without corresponding certificates return OCSP value of "Unknown"

Assignee

Description

•

5 years ago

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36

Steps to reproduce:

In light of recent CA issues involving precertificates, SSL.com's investigation revealed precertificates which 1) do not have corresponding issued certificates and 2) return a status of "Unknown".

Our research indicates this is an issue with EJBCA. We have opened a bug with PrimeKey and will apply remediation as soon as it is made available.

Examples of this issue:
https://crt.sh/?id=1720920023&opt=ocsp
https://crt.sh/?id=1677051376&opt=ocsp"

BugBot [:suhaib / :marco/ :calixte]

Updated

•

5 years ago

Status: UNCONFIRMED → ASSIGNED

Ever confirmed: true

Ryan Sleevi

Comment 1

•

5 years ago

Chris: Thanks for filing this issue early, as you became aware, rather than waiting until you believe it's been fully resolved.

It's unclear if this was meant to be the extent of the incident report or whether SSL.com is preparing further updates. If planning to provide additional details, can you provide a timeline for that?

Assignee: wthayer → support

Flags: needinfo?(support)

Whiteboard: [ca-compliance]

Thomas Zermeno

Assignee

Comment 2

•

5 years ago

Ryan: We wanted to inform the community of this issue as early as possible. We plan on fully following Mozilla best practices in addressing this issue, and filing this initial bug report was not meant to be the extent of our incident report.

At this point, we are waiting for feedback from Primekey. Regardless of their progress, we intend to submit a preliminary version of our report here as soon as possible, and definitely within the week.

Flags: needinfo?(support)

Ryan Sleevi

Updated

•

5 years ago

Flags: needinfo?(support)

Whiteboard: [ca-compliance] → [ca-compliance] Next Update - 13-September 2019

Thomas Zermeno

Assignee

Comment 3

•

5 years ago

This update is to note that our research into this issue is ongoing. A full report will be published to this bug when our investigation is complete.

Flags: needinfo?(support)

Wayne Thayer

Updated

•

5 years ago

Type: defect → task

Whiteboard: [ca-compliance] Next Update - 13-September 2019 → [ca-compliance]

Thomas Zermeno

Assignee

Comment 4

•

5 years ago

How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

As part of its participation in the greater Web PKI ecosystem, SSL.com monitors and contributes to the m.s.d.p. board. Upon review of the OCSP issues encountered by Let's Encrypt and GlobalSign, our investigation revealed that our OCSP responder is affected by the same issue.

A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

Immediately upon discovery, steps were taken in accordance with our Incident Management Policy (IMP) to investigate, mitigate and report this issue.

Timeline (format: yyyymmdd):

20190906: Reported as a potential issue upon reviewing GlobalSign’s m.s.d.p report.
20190906: Issue confirmed, preliminary crt.sh responses obtained showing problematic results:

https://crt.sh/?id=1746600571&opt=ocsp
https://crt.sh/?id=1720920023&opt=ocsp

20190906: Request for treatment as an incident under the IMP requested.
20190906: Per IMP, ticket opened, Incident declared, and Incident Manager assigned.
20190906: Bug opened in Bugzilla to explore this issue.
20190906: Ticket opened with PrimeKey to address the EJBCA issue.
20190909: Feedback from PrimeKey regarding the issue.
20190909: All affected certificates imported into the database, OCSP server no longer responds w/ “Unknown” status.
20190911: Notification made via m.d.s.p noting bug.
20190913: Per m.d.s.p. discussion, we note that all certificates required to return “Revoked” status. Testing revealed this was not the result for all items remediated using Primekey’s supplied solution.
20190913 - present: Consultation w/Primekey re: their remediation.
20190919: In-house remediation applied to resolve remaining issues.

Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

This issue is related to status reporting using OCSP, and there was no certificate issuance component to address. Remediation measures are designed to prevent recurrence of this issue by the OCSP server.

A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

52 SSL precertificates exhibited this issue. No certificates were actually issued or delivered to clients.

First (CST/Zulu): 20180521 1530 (20180521203053Z)
Last (CST/Zulu): 20190806 1552 (20190806205238Z)

The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.\

Links to crt.sh results for this population:

Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

The bug underlying this issue relates to how EJBCA handles generation of a certificate when the precertificate submission to CT logs fails. In particular, the precertificate generation introduces the promise to issue the corresponding certificate. However, if CT submission fails, EJBCA will not fulfill this promise and will discard the pre-certificate while stopping issuance. This results in providing the status “Unknown” for the aforementioned certificates.

SSL.com continuously monitors OCSP server operations to detect and correct issues, especially with replies returning "Unknown" as a result. These certificates were never issued and never delivered to a client, and no user ever requested validation information for these certificates, and our services thus never reported anomalous behavior.

List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

The issue has been resolved for the population covered by this report by inserting the certificate information into the EJBCA database manually and marking the certificate as revoked. EJBCA was immediately made aware of the new certificate, and the certificate status information was updated to “Revoked” for most certificates. An in-house solution was applied to correct status information for certificates not marked as “Revoked” by the Primekey-supplied solution.

In order to prevent the problem from happening again in the future, we have created a system to monitor failed CT submissions. In such cases, the certificate information is automatically inserted into the EJBCA database, and the responses are updated. This is an in-house developed solution that will be in effect until Primekey resolves the EJBCA problem.

Wayne Thayer

Comment 5

•

5 years ago

Thank you for the incident report. Given the outcome of the discussion on the mozilla.dev.security.policy list [1], I'm resolving this incident as INVALID.

[1] https://groups.google.com/d/msg/mozilla.dev.security.policy/LC_y8yPDI9Q/tPrL7rNkBAAJ

Status: ASSIGNED → RESOLVED

Closed: 5 years ago

Resolution: --- → INVALID

Nobody; OK to take it and work on it

Updated

•

1 year ago

Product: NSS → CA Program

Bugzilla

Quick Search

SSL.com: Precertificates without corresponding certificates return OCSP value of "Unknown"

Categories

(CA Program :: CA Certificate Compliance, task)

Tracking

(Not tracked)

People

(Reporter: support, Assigned: support)

References

Details

(Whiteboard: [ca-compliance])

Crash Data

Security

(public)

User Story

Description

Updated

Comment 1

Comment 2

Updated

Comment 3

Updated

Comment 4

How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.\

Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

Comment 5

Updated