- How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.
QuoVadis was contacted by Logius, the Policy Authority of PKIoverheid with an enquiry regarding OCSP responses for some of our precertificates. While investigating the request, as described in section 4, we discovered that revoked/certificateHold was returned as an OCSP response in certain circumstances for precertificates; certificateHold is prohibited in the Baseline Requirements for issued certificates (including precertificates) as of September 30, 2020. This in turn led us to identify an issue with the EJBCA CA software described below.
- A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.
July 22 - Logius contacted QuoVadis with an enquiry regarding OCSP responses for precertificates, providing example precertificates that delivered an unexpected OCSP response.
July 23 - QuoVadis began investigating the issue; initial reviews of the CA database showed the system and OCSP performing as expected.
July 26 - QuoVadis found discrepancies in expected OCSP behavior for precerts.
July 29 - QuoVadis confirms existence of precertificates that are logged in CT and noted in audit logs but not present in CA database. QuoVadis starts working with PrimeKey to identify the root cause of the issue; those discussions are ongoing.
July 30 - QuoVadis starts Bugzilla documentation.
August 2 - QuoVadis informed Logius of the potential issue with EJBCA software. QuoVadis escalates the issue with PrimeKey.
August 3 - Configuration change made to change OCSP response to ‘Unauthorized’ for precertificates caught in this condition.
August 4 – Incident report updated with input from PrimeKey.
August 5 - Incident report updated with input from Logius.
- Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.
This is a very rare circumstance in EJBCA. No actual certificates are impacted, but in some cases precertificates for failed orders are not recorded in the CA database as described in section 4. Work with PrimeKey continues to identify affected certificates and in future to monitor potential reoccurance of this class of error.
- A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.
The issue is not with problematic certificates per se, but rather with an issue affecting EJBCA that can impact OCSP services for some precertificates.
To summarise the issue, when the QV Certificate Management System (CMS) requests a precertificate (Precert1) to be created by EJBCA, in very rare circumstances still under investigation, the CA unexpectedly rolls back the transaction, effectively removing the certificate from the CA database. The precertificate is recorded in QV audit logs and the CA reports “certificate issuance failed” to the QV CMS.
During this process Precert1 is submitted to Certificate Transparency logs, however, as result of the CA database roll back, it is not published to QV OCSP. Typically the failed order is then resubmitted by the QV CMS and a replacement Precert2 and Actualcert2 are successfully created and delivered. These new v2 certificates match the v1 certificates in all aspects other than serial number and OCSP response.
Because of the unexpected CA database rollback, the affected ‘reserved’ Precert1 serial numbers were determined to be ‘unused’ resulting in a revoked/certificateHold OCSP response.
BR v1.7.1 section 7.2.2 introduced a number of requirements pertaining to CRL and OCSP behavior, including forbidding the use of certificateHold as reason code for certificates issued on-or-after September 30, 2020.
At the time of BR 1.7.1 EJBCA offered revoked/certificateHold (showing the UNIX epoch date as the time of revocation) as the response for ‘unused’ serial numbers. This uses the ‘extended revoked’ setting described in section 4.4.8 of RFC 6960, and was described in the QV CPS to denote non-issuance and does not denote suspension which is not allowed in the QV PKI. For more discussion on this setting see https://groups.google.com/g/mozilla.dev.security.policy/c/LC_y8yPDI9Q/m/rLoTLWwdAQAJ
As a result the 4 known affected precertificates provided a disallowed OCSP response. For the majority of failed orders, QV OCSP provides the appropriate ‘good’ OCSP response as described in https://jira.primekey.se/browse/ECA-6979.
QV implemented a change on August 3 so that QV OCSP now provides an ‘Unauthorized’ response as an unsigned message for all ‘unused’ certificate serial numbers.
- The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.
Our investigation of audit logs has identified the following affected precertificates issued by QV since September 1, 2020; however we are working with PrimeKey on additional methods to ensure completeness. This report will be updated if more affected precertificates found.
Precertificate SHA-1 fingerprints:
Two of these precertificates are under the QV hierarchy, two are under the PKIoverheid hierarchy.
- Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
QV reviews are based on the CA database which showed failed orders are handled as expected, with the appropriate ‘good’ OCSP responses for the failed Precert1. The issue was detected by testing of OCSP responses for certificates found in CT. This revealed the existence of the failed Precert1 cases which do not appear in the CA database and which provided the unexpected (and noncompliant at that time) OCSP response.
- List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.
We have updated our OCSP service and CPS to provide ‘Unauthorized’ as the response to all serial numbers that are ‘unused’ in the CA database.
The underlying issue is an unexpected EJBCA rollback wherein the original Precert1 does not appear in the CA database, but its replacement Precert2 does, although records for the processing of those precerts remain in the audit logs. We are working closely with PrimeKey to:
- Automate the correlation of audit logs to the CA database to highlight other classes of errors that might occur. This will be used:
o to automate the identification of affected precerts from the audit logs and to restore them to the CA database.
o to quickly detect potential re-occurrences until the EJBCA software is upgraded.
- Further identify the root causes and remediation in EJBCA in collaboration with PrimeKey. We anticipate a public EJBCA JIRA ticket will later be provided here.