Open Bug 1468477 Opened 2 years ago Updated 3 months ago

QuoVadis (Freistaat Bayern): Non-BR-compliant Key Usage


(NSS :: CA Certificate Compliance, task)

Not set


(Not tracked)



(Reporter: rob, Assigned: s.davidson)


(Whiteboard: [ca-compliance] Next Update - 01-January 2020)

"Bayerische SSL-CA-2018-01", a technically-constrained Sub-CA under a QuoVadis root, has (mis)issued the following precertificate and corresponding certificate:

Whilst the intent appears to have been to issue an end-entity server authentication (pre)certificate, the Key Usage extension has the keyCertSign and cRLSign bits set.
Thanks Rob. 

We became aware of this certificate on 2018-06-12 at 20:30 UTC via an alert from our post-issuance linting system which scans TLS/SSL logged in CT that are part of our hierarchy - including the output of our technically-constrained external subCAs.

We immediately contacted the issuer, Freistaat Bayern.  The certificate was revoked 2018-06-13 at 05:32 UTC.

We are not aware of other certificates with this issue.

We are coordinating with Freistaat Bayern to understand the cause of the misissuance, and will file an incident report when complete.

Regards, Stephen
Whiteboard: [ca-compliance]
1.	How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in, a Bugzilla bug, or internal self-audit), and the time and date.

We became aware of this certificate on 2018-06-12 at 20:30 UTC via an alert from our post-issuance linting system which scans TLS/SSL logged in CT that are part of our hierarchy - including the output of our technically-constrained external subCAs.

2.	A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

We became aware of this certificate on 2018-06-12 at 20:30 UTC 
We immediately contacted the issuer, Freistaat Bayern via our contact email for such events.  The certificate was revoked by Freistaat Bayern on 2018-06-13 at 05:32 UTC.
We conducted calls with Freistaat Bayern on 2018-06-13/14/15 to identify the source of the issue and to gain confidence in the resolution.

3.	Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

The issue that caused this missuance was corrected as described below through tighter controls implemented at the CA.  Additional improvements to the CMS will be implemented c. end of June.

4.	A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

A review of certificates in the period identified only two certificates with issues, issued on June 6 and 10 2018 respectively.  Both were revoked within 12 hours of discovery.

5.	The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.,ocsp

6.	Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now. 

Controls for the BR and other standards are imposed where appropriate in a certificate management system (CMS/RA) and in the underlying CA.  Both use “off the shelf” PKI software provided and implemented by a well known vendor.  The CMS verifies different items in the CSR (Subject DN, key size and algorithm, hash algorithm etc) before the request is sent to the CA.  
In a specific process, a misconfiguration in the CMS allowed the Key Usage and Extended Key Usage in the submitted CSR to be passed to the CA. 
Working with the CA software vendor, the CA itself has been configured to replace submitted attributes in the Key Usage and Extended Key Usage fields with values set by the CA according to the certificate profile.

7.	List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

The issue that caused this missuance was corrected as described above through tighter controls implemented at the CA.  Additional improvements to the CMS identified in our discussions with Freistaat Bayern will be implemented via software patch by the external vendor, expected before end of June and following testing on a nonproduction environment.
Assignee: wthayer → sdavidson
Whiteboard: [ca-compliance] → [ca-compliance] - Next Update - 01-July 2018
Stephen: Is there a plan, as discussed in bug 1391063 for Siemens, to implement pre-issuance linting for this and other subordinate CAs chaining to Quovadis roots?
Freistaat Bayern is consulting with their vendor on the feasibility of pre-issuance linting.  

Some subCAs in our hierarchy are able to move more readily to pre-issuance linting due to vendor support.

Will revert.  In the meantime, we continue to conduct post-issuance linting of all SSL issuance under our hierarchy.
Wayne: It looks like this issue needs to get reassigned to a new CA contact, as Stephen's account is disabled.
Flags: needinfo?(wthayer)
Reassigning to Stephen's WISeKey address.

Stephen: please provide an update on the implementation of pre-issuance linting mentioned in comment 4.
Assignee: sdavidson → s.davidson
Flags: needinfo?(wthayer) → needinfo?(s.davidson)

Stephen: Do you have an update?

As described previously, for their name-constrained CA Freistaat Bayern uses an "off the shelf" PKI which does not, at this stage, support full pre-issuance linting.

However, their certificate management system (CMS/RA) does include a variety of filters and templates, which have been improved to better address this error and other BR requirements. It does not, however, deliver a full zLint coverage. Freistaat Bayern has focussed on changes to their CMS that would allow them to interface with a managed PKI for trusted TLS certificates, and we have agreed that they will transition to a managed PKI during 2019.

We continue post-issuance linting of Freistaat Bayern, as well as monitoring using an internally-developed tool.

Flags: needinfo?(s.davidson)

Trying to summarize this discussion.

  • The issue manifest as an invalid keyUsage for an end-entity certificate (Comment #0)
  • The root cause was diagnosed as the certificate management software copying over fields from the CSR into the final certificate without the appropriate review or configuration. (Comment #2)
  • The steps taken are:
    1. Implemented The CA software is now configured to configure keyUsage and extendedKeyUsage based on certificate profile (Comment #2)
    2. Implemented The RA/CMS software is also configured to validate parameters before sending to the CA (Comment #2)
    3. Implemented QuoVadis performs post-issuance linting and internal monitoring
    4. Pending - 2019 EOY The CA software will transition to a managed CA at some point during 2019 (Comment #8)

I think the only question from this is for more details on the post-issuance linting to be shared. This helps discover and develop industry best practice, by understanding how CAs are performing post-issuance linting and how they are designing such controls to operate reliably.

Flags: needinfo?(s.davidson)
Whiteboard: [ca-compliance] - Next Update - 01-July 2018 → [ca-compliance]

Hi Ryan: We monitor external CAs using several tools:

  • An internal tool aka CertNod which conducts post-issuance linting by checking CT logs for certs issued by a given subCA, and sends email alerts when certs with ERROR or FATAL are found. CertNod uses zLint as the source for its lints.
  • Daily sanity checks using
  • An internally developed monitoring tool which provides daily updates from the external subCA of domains in issued certs which are screened against the domains validated by QuoVadis and included in nameConstraints for the subCA. Although certs that defy the nameConstraints should not work in browsers, this alert gives insight into the subCA's compliance with agreements.
Flags: needinfo?(s.davidson)

Thanks Stephen! I think that is an incredibly helpful level of detail, and also highlights an expected practice being captured - namely, overseeing technically constrained sub-CAs for compliance.

Wayne, I think Comment #9 summarizes things, if you have any other questions

Flags: needinfo?(wthayer)

I have no further questions, but I do think this bug should remain open, and Quovadis should provide periodic updates, until the transition to managed infrastructure is completed.

Flags: needinfo?(wthayer)
Whiteboard: [ca-compliance] → [ca-compliance] Next Update - 01-January 2020

A contract was executed wherein Freistaat Bayern ceased issuance from this subCA as of July 10, 2019 and the subCA will be revoked by QuoVadis as of Dec 30, 2019. Technical changes were confirmed that prevent routine issuance of certificates from the subCA, and we continue to monitor to confirm that no issuance occurs through the time of revocation.

You need to log in before you can comment on or make changes to this bug.