1724276 - QuoVadis / PKIoverheid: incorrect OCSP response for precertificate

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[David Weissenberg]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Firefox/78.0

Comment 1

[David Weissenberg]

Logius is taking the next steps in the implementation of additional features of its post issuance linting tooling (VTT).

As a result, on July 22, as part of its supervision of CAs operating under the PKIoverheid hierarchy, Logius detected unusual OCSP responses for some pre-certificates issued by QuoVadis. This led to an extended investigation conducted by Logius, QuoVadis, and PrimeKey resulting in the filing of this bug. We have asked QuoVadis to provide an incident report with additional details.

Comment 2

[Stephen Davidson]

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

QuoVadis was contacted by Logius, the Policy Authority of PKIoverheid with an enquiry regarding OCSP responses for some of our precertificates. While investigating the request, as described in section 4, we discovered that revoked/certificateHold was returned as an OCSP response in certain circumstances for precertificates; certificateHold is prohibited in the Baseline Requirements for issued certificates (including precertificates) as of September 30, 2020. This in turn led us to identify an issue with the EJBCA CA software described below.

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

July 22 - Logius contacted QuoVadis with an enquiry regarding OCSP responses for precertificates, providing example precertificates that delivered an unexpected OCSP response.
July 23 - QuoVadis began investigating the issue; initial reviews of the CA database showed the system and OCSP performing as expected.
July 26 - QuoVadis found discrepancies in expected OCSP behavior for precerts.
July 29 - QuoVadis confirms existence of precertificates that are logged in CT and noted in audit logs but not present in CA database. QuoVadis starts working with PrimeKey to identify the root cause of the issue; those discussions are ongoing.
July 30 - QuoVadis starts Bugzilla documentation.
August 2 - QuoVadis informed Logius of the potential issue with EJBCA software. QuoVadis escalates the issue with PrimeKey.
August 3 - Configuration change made to change OCSP response to ‘Unauthorized’ for precertificates caught in this condition.
August 4 – Incident report updated with input from PrimeKey.
August 5 - Incident report updated with input from Logius.

3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

This is a very rare circumstance in EJBCA. No actual certificates are impacted, but in some cases precertificates for failed orders are not recorded in the CA database as described in section 4. Work with PrimeKey continues to identify affected certificates and in future to monitor potential reoccurance of this class of error.

4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

The issue is not with problematic certificates per se, but rather with an issue affecting EJBCA that can impact OCSP services for some precertificates.

To summarise the issue, when the QV Certificate Management System (CMS) requests a precertificate (Precert1) to be created by EJBCA, in very rare circumstances still under investigation, the CA unexpectedly rolls back the transaction, effectively removing the certificate from the CA database. The precertificate is recorded in QV audit logs and the CA reports “certificate issuance failed” to the QV CMS.

During this process Precert1 is submitted to Certificate Transparency logs, however, as result of the CA database roll back, it is not published to QV OCSP. Typically the failed order is then resubmitted by the QV CMS and a replacement Precert2 and Actualcert2 are successfully created and delivered. These new v2 certificates match the v1 certificates in all aspects other than serial number and OCSP response.

Because of the unexpected CA database rollback, the affected ‘reserved’ Precert1 serial numbers were determined to be ‘unused’ resulting in a revoked/certificateHold OCSP response.

BR v1.7.1 section 7.2.2 introduced a number of requirements pertaining to CRL and OCSP behavior, including forbidding the use of certificateHold as reason code for certificates issued on-or-after September 30, 2020.

At the time of BR 1.7.1 EJBCA offered revoked/certificateHold (showing the UNIX epoch date as the time of revocation) as the response for ‘unused’ serial numbers. This uses the ‘extended revoked’ setting described in section 4.4.8 of RFC 6960, and was described in the QV CPS to denote non-issuance and does not denote suspension which is not allowed in the QV PKI. For more discussion on this setting see https://groups.google.com/g/mozilla.dev.security.policy/c/LC_y8yPDI9Q/m/rLoTLWwdAQAJ

As a result the 4 known affected precertificates provided a disallowed OCSP response. For the majority of failed orders, QV OCSP provides the appropriate ‘good’ OCSP response as described in https://jira.primekey.se/browse/ECA-6979.

QV implemented a change on August 3 so that QV OCSP now provides an ‘Unauthorized’ response as an unsigned message for all ‘unused’ certificate serial numbers.

5. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

Our investigation of audit logs has identified the following affected precertificates issued by QV since September 1, 2020; however we are working with PrimeKey on additional methods to ensure completeness. This report will be updated if more affected precertificates found.

Precertificate SHA-1 fingerprints:
c7:4b:67:a9:80:b7:bd:e9:c9:b5:68:6c:b2:51:f9:d9:ec:ae:de:ff
8c:cf:7b:e6:a2:63:f4:bf:f0:c0:8b:e0:5e:ca:18:61:bd:1e:da:9b
4b:e8:46:fd:bd:52:55:52:f6:f3:eb:7f:4d:1e:ab:7d:54:27:e0:a2
d7:8a:a0:9d:4c:72:9d:73:0f:dd:b1:bc:f3:1e:9d:0f:d8:be:e4:ab

Two of these precertificates are under the QV hierarchy, two are under the PKIoverheid hierarchy.

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

QV reviews are based on the CA database which showed failed orders are handled as expected, with the appropriate ‘good’ OCSP responses for the failed Precert1. The issue was detected by testing of OCSP responses for certificates found in CT. This revealed the existence of the failed Precert1 cases which do not appear in the CA database and which provided the unexpected (and noncompliant at that time) OCSP response.

7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

We have updated our OCSP service and CPS to provide ‘Unauthorized’ as the response to all serial numbers that are ‘unused’ in the CA database.

The underlying issue is an unexpected EJBCA rollback wherein the original Precert1 does not appear in the CA database, but its replacement Precert2 does, although records for the processing of those precerts remain in the audit logs. We are working closely with PrimeKey to:

• Automate the correlation of audit logs to the CA database to highlight other classes of errors that might occur. This will be used:
  o to automate the identification of affected precerts from the audit logs and to restore them to the CA database.
  o to quickly detect potential re-occurrences until the EJBCA software is upgraded.
• Further identify the root causes and remediation in EJBCA in collaboration with PrimeKey. We anticipate a public EJBCA JIRA ticket will later be provided here.

Comment 3

[Andrew Ayer]

Thanks for the clear description of what went wrong.

I see that QV's OCSP responder is currently responding with "unauthorized" for the serial numbers in the 4 affected precertificates. However, since precertificates have been issued, the serial numbers should be considered "reserved" and the OCSP responses should be "good" as they are for other failed orders. Will this be fixed?

What's the timeline for implementing the remediation steps in question 7?

Comment 4

[Stephen Davidson]

Hi Andrew, for now affected 'reserved' serials are treated as 'unauthorized'. We continue to (a) investigate past issuance looking for similar affected precerts, and when complete will then (b) restore affected precerts to 'good' OCSP status. We will provide an update next week on the status of this effort.

In addition we continue to collaborate with PrimeKey to identify the root cause of the rollback. We will note here when PrimeKey publishes their own public EJBCA JIRA ticket. At that time, we will be able to finalize our CA update. In the meantime, we are monitoring for potential re-occurrences of the bug on our CAs until EJBCA is updated.

Comment 5

[Stephen Davidson]

PrimeKey have made public a related JIRA ticket related to this EJBCA issue at https://jira.primekey.se/browse/ECA-10215.
Additional information will be provided at the completion of our joint investigations.

Comment 6

[Stephen Davidson]

Attachment: QV_affectedprecerts_bug1724276.txt

QuoVadis has completed a detailed review using both our internal audit logs and CT to identify affected precertificates. The complete count is 54 affected precertificates. They were issued between September 07 2019 and May 27 2021.

The affected precertificates have been restored in the CA database and respond in a compliant manner in OCSP.

We are able to monitor for future occurrences of the unexpected CA database rollback until PrimeKey produces an update.

PrimeKey expects to deal with this issue in the next release of EJBCA (see their JIRA above).

We will post an update at the earlier of two weeks or when the PrimeKey update is released.

Comment 7

[Ryan Sleevi]

Ben: For a NextUpdate of 2021-08-30
Stephen: Absent a next update, please make sure to provide an update no later than 2021-08-20

Comment 8

[Stephen Davidson]

PrimeKey have provided more detail at https://jira.primekey.se/browse/ECA-10215 that may be useful for other CA operators that wish to investigate if they have encountered the same issue. We will update again on 8/30.

Comment 9

[Stephen Davidson]

We continue to monitor for reoccurrences of the EJBCA issue; there have been none.
PrimeKey is testing an EJBCA update to address the issue; we will update when we have more information on the availability of that update.

Comment 10

[Stephen Davidson]

We continue to monitor for reoccurrences of the EJBCA issue; there have been none.
PrimeKey is testing EJBCA 7.7.1 to address the issue; we will update when we have an implementation schedule (or by 9/30).

Comment 11

[Stephen Davidson]

The fix will now be named EJBCA 7.8.0 which is imminent for official release. We are already testing the release in our development environment, and plan to move it into production during a maintenance window during the weekend of October 16.

Comment 12

[Stephen Davidson]

The required updates have been made; there are no further steps to report for this disclosure.

Comment 13

[Ben Wilson]

I will close this on Friday 22-Oct-2021.