1519265 - QuoVadis: Recap of BR Compliance in 2018 issuance by external subCAs

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Stephen Davidson]

During 2018, QuoVadis commenced post-issuance linting of SSL issuance using a combination of crt.sh and internally developed alerting tools. At year end 2018, we undertook an additional re-lint of 2018 issuance to ensure that all problem certificates were revoked and issues were remediated. In some cases, changes to available lints and presentation in reports such as crt.sh provide new focus on existing errors.

For the sake of clarity, QuoVadis implemented changes to our certificate management system (CMS) to resolve issues as they were picked up through linting. However, Bugzilla disclosures were not always made on issues. As the emphasis on disclosures has increased, for the sake of transparency, we are filing now filing a disclosure of 2018 problem certificates we identified (ie, triggering an ERROR in zLint).

A. Siemens

1. How your CA first became aware of the problem, and the time and date.

ERROR: The common name field in subscriber certificates must include only names from the SAN extension

The certificate was identified on May 3 (the day of issuance) using a internal linting tool we have in place to monitor Siemens SSL issuance.

2. A timeline of the actions your CA took in response.

Discovery of the certificate using lnting at 11:36 AST on May 3
QuoVadis contacts Siemens call list at 11:43 AST on May 3
Siemens revokes certificate at 2: 36 AST on May 3

3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

Measure have been put into place to reduce the risk of this reoccurring. See below.

4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

https://crt.sh/?id=437210034&opt=zlint issued on May 3, 2018.

5. The complete certificate data for the problematic certificates.

Above.

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

The Siemens CMS is typically thorough in enforcing BR requirements. This certificate was manually processed and thus missed many of the BR “quality checks”. Siemens:

• Reviewed all manually issued certificates in the prior 6 months (no issues found)
• Updated their checklists for manual SSL issuance.
• Integrated live issuance linting using zLint at their CA.

7. List of steps CA is taking to resolve the situation and ensure it will not be repeated.

Since pre-issuance linting was implemented Siemens has had a clear record of SSL issuance.

B. Freistaat Bayern

1. How your CA first became aware of the problem, and the time and date.

See also https://bugzilla.mozilla.org/show_bug.cgi?id=1468477

On June 10 a certificate was issued that was discovered using post-issuance linting that had multiple issues.

ERROR: if the keyCertSign bit is asserted, then the cA bit in the basic constraints extension MUST also be asserted
ERROR: Root and Sub CA Certificate: The CA field MUST be set to true.
ERROR: Subscriber Certificate: keyUsage if present, bit positions for keyCertSign and cRLSign MUST NOT be set.
ERROR: Subscriber Certificate: keyUsage if present, bit positions for keyCertSign and cRLSign MUST NOT be set.

2. A timeline of the actions your CA took in response.

Certificate issued on June 10 at 16:10 GMT
QuoVadis discovers certificate via linting on June 12 at 22:00 GMT and contacts Freistaat Bayern
Freistaat Bayern revokes certificate on June 13 at 05:30 GMT

3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

Measure have been put into place to reduce the risk of this reoccurring. See below.

4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

https://crt.sh/?id=523705088&opt=zlint,ocsp issued on June 10, 2018.

5. The complete certificate data for the problematic certificates.

Above.

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

Freistaat Bayern has a certificate management system that assists in enforcing BR requirements. In this case the end user provided a CSR with erroneous settings which was processed manually by an administrator who did not notice the issues.

Note that the Freistaat Bayern CA has the following for Basic Constraints
(Critical):
Subject Type=CA
Path Length Constraint=0

so no subordinates permitted under the ICA.

7. List of steps CA is taking to resolve the situation and ensure it will not be repeated.

QuoVadis is undertaking discussions with Freistaat Bayern regarding the possibility of bringing the SSL issuance to a managed PKI. The PKI software used by Freistaat Bayern does not currently support pre-issuance linting.

C. VR IDENT

See also https://bugzilla.mozilla.org/show_bug.cgi?id=1493760

1. How your CA first became aware of the problem, and the time and date.

QuoVadis became aware of the issue via post issuance linting on Sept 21 at 14:44.

ERROR: The country name field MUST contain the two-letter ISO code for the country or XX

2. A timeline of the actions your CA took in response.

QuoVadis contacted VR Ident, who removed the certificates from production and then revoked them as below. As a precaution, the CA was immediately taken offline until the issue could be identified.

3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

Yes, the issue has been remediated.

4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

https://crt.sh/?id=817516755&opt=zlint issued Sept 21 at 10:10 GMT – revoked Sept 25 at 7:54 GMT
https://crt.sh/?id=815260389&opt=zlint issued Sept 20 at 9:11 GMT – revoked Sept 25 at 7:51 GMT

5. The complete certificate data for the problematic certificates.

Above.

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

In the past, QuoVadis root signed external subCAs operated by VR IDENT. In order to improve controls, these were replaced by fully managed PKI using dedicated issuing CAs managed by QuoVadis in the days leading up this missuance.
As part of the transition, the previously validated Organisation details were bulk imported into our certificate management system. A previously unidentified caps sensitivity issue meant that a single record associated with this single organisation did not translate the text “Germany” into “de” in the template. The bulk import used in this case is a rare manual operation, and the issue was not spotted in testing.

7. List of steps CA is taking to resolve the situation and ensure it will not be repeated.

The issue was quickly identified and corrected for the single record, and then a review was made of caps sensitivity issues elsewhere in the certificate management system.

D. Swiss Government

1. How your CA first became aware of the problem, and the time and date.

QuoVadis became aware of the certificate using post issuance linting.

ERROR: Characters in labels of DNSNames MUST be alphanumeric, - , _ or *
ERROR: DNSNames must have a valid TLD.
ERROR: The common name field in subscriber certificates must include only names from the SAN extension

2. A timeline of the actions your CA took in response.

QuoVadis alerted the customer and the certificate was revoked. Plans were made to accelerate moving to a managed PKI in line with previously reported issues at https://bugzilla.mozilla.org/show_bug.cgi?id=1430909

3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

Yes, the Swiss Government transitioned to a managed PKI.

4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

https://crt.sh/?id=325970974&opt=zlint on Feb 8 at 11:57 GMT – revoked on Feb 12 at 14:15 GMT

5. The complete certificate data for the problematic certificates.

Above.

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

7. List of steps CA is taking to resolve the situation and ensure it will not be repeated.

The Swiss Government transitioned to a managed PKI, and have had clean linting since then.

E. DarkMatter

1. How your CA first became aware of the problem, and the time and date.

Discussed at https://bugzilla.mozilla.org/show_bug.cgi?id=1427262#c30

ERROR: The common name field in subscriber certificates must include only names from the SAN extension

2. A timeline of the actions your CA took in response.

Above.

3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

The CA has stopped issuing certificates with the problem.

4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

https://crt.sh/?id=492682805&opt=zlint issued on May 29 at 7:57 GMT – revoked on May 30 at 2:17 GMT

5. The complete certificate data for the problematic certificates.

Above.

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

DarkMatter has a considerable certificate management system to orchestrate validation and conformance of certificate requests with the BR. The certificate in question was processed manually by an administrator, which bypassed the check on repetition of the CN in SAN.

7. List of steps CA is taking to resolve the situation and ensure it will not be repeated.

DarkMatter undertook a review of all manually processed certificates and updated their administrator manuals. Most importantly DarkMatter has increased use of pre-issuance linting tools to catch issues before they are submitted to the CA.

Comment 1

[Ryan Sleevi]

• Regarding Comment #0, Incident A:

  "Updated their checklists for manual SSL issuance."

  Can you please provide more details regarding what the existing checklists contained and what they were updated to include?

• Regarding Comment #0, Incident B:

  QuoVadis is undertaking discussions with Freistaat Bayern regarding the possibility of bringing the SSL issuance to a managed PKI. The PKI software used by Freistaat Bayern does not currently support pre-issuance linting.

  Do you have an updated timetable?

• Regarding Comment #0, Incident C:

  As part of the transition, the previously validated Organisation details were bulk imported into our certificate management system.

  Can you clarify - who performed validation of this information? It sounds as if VR IDENT performed that validation, which QuoVadis subsequently imported as-is. Is this correct?

  It also sounds if the remediation only explored for case sensitivity. Could you explain if there has been any examination given to the freshness of the data and the method used to validate this data? Given the changes in Section 3.2 of the BRs, it seems important to ensure a full-fidelity import.

• Regarding Comment #0, Incident D:

  The Swiss Government transitioned to a managed PKI, and have had clean linting since then.

  When did this transition occur?

Broadly speaking, three of these incidents (A, B, E) were related to explicitly manual operations, and at least one tangentially (Incident C). We don't have a root cause for Incident D (Question 6 is empty).

Given this, what steps is QuoVadis (DigiCert?) taking to minimize the risk of manual issuance, both by first and by third parties?

Comment 2

[Stephen Davidson]

Regarding Comment #0, Incident A

The checklist was updated to highlight that in manual issuance it must be verified that the CN is repeated in the SAN. More importantly, pre-issuance linting is now in place.

Regarding Comment #0, Incident B: Do you have an updated timetable?

Not yet. Will update https://bugzilla.mozilla.org/show_bug.cgi?id=1468477 as needed/

Regarding Comment #0, Incident C: Can you clarify - who performed validation of this information?

VR Ident operated under a WebTrust seal. Nevertheless, the org and domain information was revalidated by QuoVadis before it was imported into our certificate management system. The caps issue in the affected record of that import was overlooked by our validation team.

Regarding Comment #0, Incident D: When did this transition occur?

See https://bugzilla.mozilla.org/show_bug.cgi?id=1430909#c5

What steps is QuoVadis (DigiCert?) taking to minimize the risk of manual issuance, both by first and by third parties?

1. In the case of QuoVadis own CAs, all requests are processed through our certificate management system. There is no manual SSL issuance.
2. In the case of external subCAs, we ceased offering root signing to new clients. We began using a tool to live track SSL issuance by external subCAs in 2012, and began using a tool to conduct post-issuance linting in 2018. Where possible, we have worked to move the customers to managed PKI for their SSL needs (and those discussions continue with our remaining external subCA customers).

Comment 3

[Wayne Thayer]

Stephen: thank you for providing this comprehensive report, it is very helpful. Since the remaining remediation is documented in another big, I'm resolving this one.