1468477 - QuoVadis (Freistaat Bayern): Non-BR-compliant Key Usage

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Rob Stradling]

"Bayerische SSL-CA-2018-01", a technically-constrained Sub-CA under a QuoVadis root, has (mis)issued the following precertificate and corresponding certificate:
Precert: https://crt.sh/?id=517622425&opt=cablint,zlint
Cert: https://crt.sh/?id=523705088&opt=cablint,zlint

Whilst the intent appears to have been to issue an end-entity server authentication (pre)certificate, the Key Usage extension has the keyCertSign and cRLSign bits set.

Comment 1

[Stephen Davidson]

Thanks Rob. 

We became aware of this certificate on 2018-06-12 at 20:30 UTC via an alert from our post-issuance linting system which scans TLS/SSL logged in CT that are part of our hierarchy - including the output of our technically-constrained external subCAs.

We immediately contacted the issuer, Freistaat Bayern.  The certificate was revoked 2018-06-13 at 05:32 UTC.

We are not aware of other certificates with this issue.

We are coordinating with Freistaat Bayern to understand the cause of the misissuance, and will file an incident report when complete.

Regards, Stephen

Comment 2

[Stephen Davidson]

1.	How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

We became aware of this certificate on 2018-06-12 at 20:30 UTC via an alert from our post-issuance linting system which scans TLS/SSL logged in CT that are part of our hierarchy - including the output of our technically-constrained external subCAs.

2.	A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

We became aware of this certificate on 2018-06-12 at 20:30 UTC 
We immediately contacted the issuer, Freistaat Bayern via our contact email for such events.  The certificate was revoked by Freistaat Bayern on 2018-06-13 at 05:32 UTC.
We conducted calls with Freistaat Bayern on 2018-06-13/14/15 to identify the source of the issue and to gain confidence in the resolution.

3.	Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

The issue that caused this missuance was corrected as described below through tighter controls implemented at the CA.  Additional improvements to the CMS will be implemented c. end of June.

4.	A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

A review of certificates in the period identified only two certificates with issues, issued on June 6 and 10 2018 respectively.  Both were revoked within 12 hours of discovery.

5.	The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

https://crt.sh/?id=517622425&opt=cablint,ocsp
https://crt.sh/?id=509850542&opt=cablint

6.	Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now. 

Controls for the BR and other standards are imposed where appropriate in a certificate management system (CMS/RA) and in the underlying CA.  Both use “off the shelf” PKI software provided and implemented by a well known vendor.  The CMS verifies different items in the CSR (Subject DN, key size and algorithm, hash algorithm etc) before the request is sent to the CA.  
In a specific process, a misconfiguration in the CMS allowed the Key Usage and Extended Key Usage in the submitted CSR to be passed to the CA. 
Working with the CA software vendor, the CA itself has been configured to replace submitted attributes in the Key Usage and Extended Key Usage fields with values set by the CA according to the certificate profile.

7.	List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

The issue that caused this missuance was corrected as described above through tighter controls implemented at the CA.  Additional improvements to the CMS identified in our discussions with Freistaat Bayern will be implemented via software patch by the external vendor, expected before end of June and following testing on a nonproduction environment.

Comment 3

[Wayne Thayer]

Stephen: Is there a plan, as discussed in bug 1391063 for Siemens, to implement pre-issuance linting for this and other subordinate CAs chaining to Quovadis roots?

Comment 4

[Stephen Davidson]

Freistaat Bayern is consulting with their vendor on the feasibility of pre-issuance linting.  

Some subCAs in our hierarchy are able to move more readily to pre-issuance linting due to vendor support.

Will revert.  In the meantime, we continue to conduct post-issuance linting of all SSL issuance under our hierarchy.

Comment 5

[Ryan Sleevi]

Wayne: It looks like this issue needs to get reassigned to a new CA contact, as Stephen's account is disabled.

Comment 6

[Wayne Thayer]

Reassigning to Stephen's WISeKey address.

Stephen: please provide an update on the implementation of pre-issuance linting mentioned in comment 4.

Comment 7

[Ryan Sleevi]

Stephen: Do you have an update?

Comment 8

[Stephen Davidson]

As described previously, for their name-constrained CA Freistaat Bayern uses an "off the shelf" PKI which does not, at this stage, support full pre-issuance linting.

However, their certificate management system (CMS/RA) does include a variety of filters and templates, which have been improved to better address this error and other BR requirements. It does not, however, deliver a full zLint coverage. Freistaat Bayern has focussed on changes to their CMS that would allow them to interface with a managed PKI for trusted TLS certificates, and we have agreed that they will transition to a managed PKI during 2019.

We continue post-issuance linting of Freistaat Bayern, as well as monitoring using an internally-developed tool.

Comment 9

[Ryan Sleevi]

Trying to summarize this discussion.

• The issue manifest as an invalid keyUsage for an end-entity certificate (Comment #0)
• The root cause was diagnosed as the certificate management software copying over fields from the CSR into the final certificate without the appropriate review or configuration. (Comment #2)
• The steps taken are:
  1. Implemented The CA software is now configured to configure keyUsage and extendedKeyUsage based on certificate profile (Comment #2)
  2. Implemented The RA/CMS software is also configured to validate parameters before sending to the CA (Comment #2)
  3. Implemented QuoVadis performs post-issuance linting and internal monitoring
  4. Pending - 2019 EOY The CA software will transition to a managed CA at some point during 2019 (Comment #8)

I think the only question from this is for more details on the post-issuance linting to be shared. This helps discover and develop industry best practice, by understanding how CAs are performing post-issuance linting and how they are designing such controls to operate reliably.

Comment 10

[Stephen Davidson]

Hi Ryan: We monitor external CAs using several tools:

• An internal tool aka CertNod which conducts post-issuance linting by checking CT logs for certs issued by a given subCA, and sends email alerts when certs with ERROR or FATAL are found. CertNod uses zLint as the source for its lints.
• Daily sanity checks using crt.sh.
• An internally developed monitoring tool which provides daily updates from the external subCA of domains in issued certs which are screened against the domains validated by QuoVadis and included in nameConstraints for the subCA. Although certs that defy the nameConstraints should not work in browsers, this alert gives insight into the subCA's compliance with agreements.

Comment 11

[Ryan Sleevi]

Thanks Stephen! I think that is an incredibly helpful level of detail, and also highlights an expected practice being captured - namely, overseeing technically constrained sub-CAs for compliance.

Wayne, I think Comment #9 summarizes things, if you have any other questions

Comment 12

[Wayne Thayer]

I have no further questions, but I do think this bug should remain open, and Quovadis should provide periodic updates, until the transition to managed infrastructure is completed.

Comment 13

[Stephen Davidson]

A contract was executed wherein Freistaat Bayern ceased issuance from this subCA as of July 10, 2019 and the subCA will be revoked by QuoVadis as of Dec 30, 2019. Technical changes were confirmed that prevent routine issuance of certificates from the subCA, and we continue to monitor to confirm that no issuance occurs through the time of revocation.

Comment 14

[Stephen Davidson]

The following versions of the CA were revoked on 30 December.

ICA	Serial	Thumb
Bayerische SSL-CA-2018-01	44915f9e749ae3af4f9b67f6ff1c82b45f444bbf	52B63D1BD08E83BDC723D7B1FE962CEC1806E7F53F76F2C70858CA35293A1DC1
Bayerische SSL-CA-2018-01	22559aace2195a18cf8e404896b94132a8dc4ccf	C84AE01ECD202DAFBEEE1F0E679646DE8CCD653D7846718A3B5F4E129324298A
Bayerische SSL-CA-2018-01	5dced5064c9e3513c0524ad49972fbc5d37e7713	0F751035C18E1D392E9CC557C57E94A55D12FBB086F26A4529E2613625BFD13C

The subCA had been reissued several times within its validity amending nameConstraints. No prior versions of this CA remain valid.

We request that this bug be marked resolved.

Comment 15

[Ryan Sleevi]

Thanks. Confirming this shows as revoked (cessationOfOperation) for all three sub-CAs with this subject disclosed via CT via both OCSP and CRL.