1393555 - GlobalSign: Non-BR-Compliant Certificate Issuance -- double-dots in dnsName

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Kathleen Wilson]

This bug is in regards to the problem: invalid dnsNames containing "double dots" 
https://groups.google.com/d/topic/mozilla.dev.security.policy/5bpr9yBgaYo/discussion
https://misissued.com/batch/13/
~~
To continue inclusion of your CA’s root certificates in Mozilla’s Root Store, you must respond in this bug to provide the following information:
1) How your CA first became aware of the problems listed below (e.g. via a Problem Report, via the discussion in mozilla.dev.security.policy, or via this Bugzilla Bug), and the date.
2) Prompt confirmation that your CA has stopped issuing TLS/SSL certificates with the problems listed below.
3) Complete list of certificates that your CA finds with each of the listed issues during the remediation process. The recommended way to handle this is to ensure each certificate is logged to CT and then attach a CSV file/spreadsheet of the fingerprints or crt.sh IDs, with one list per distinct problem.
4) Summary of the problematic certificates. For each problem listed below: number of certs, date first and last certs with that problem were issued.
5) Explanation about how and why the mistakes were made, and not caught and fixed earlier.
6) List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.
7) Regular updates to confirm when those steps have been completed.

Note Section 4.9.1.1 of the CA/Browser Forum’s Baseline Requirements, which states:
“The CA SHALL revoke a Certificate within 24 hours if one or more of the following occurs: …
9. The CA is made aware that the Certificate was not issued in accordance with these Requirements or the CA’s Certificate Policy or Certification Practice Statement; 
10. The CA determines that any of the information appearing in the Certificate is inaccurate or misleading; …
14. Revocation is required by the CA’s Certificate Policy and/or Certification Practice Statement; or 
15. The technical content or format of the Certificate presents an unacceptable risk to Application Software Suppliers or Relying Parties (e.g. the CA/Browser Forum might determine that a deprecated cryptographic/signature algorithm or key size presents an unacceptable risk and that such Certificates should be revoked and replaced by CAs within a given period of time).

However, it is not our intent to introduce additional problems by forcing the immediate revocation of certificates that are not BR compliant when they do not pose an urgent security concern. Therefore, we request that your CA perform careful analysis of the situation. If there is justification to not revoke the problematic certificates, then explain those reasons and provide a timeline for when the bulks of the certificates will expire or be revoked/replaced. 

We expect that your forthcoming audit statements will indicate the findings of these problems. If your CA will not be revoking the certificates within 24 hours in accordance with the BRs, then that will also need to be listed as a finding in your CA’s BR audit statement.

We expect that your CA will work with your auditor (and supervisory body, as appropriate) and the Root Store(s) that your CA participates in to ensure your analysis of the risk and plan of remediation is acceptable. If your CA will not be revoking the problematic certificates as required by the BRs, then we recommend that you also contact the other root programs that your CA participates in to acknowledge this non-compliance and discuss what expectations their Root Programs have with respect to these certificates.
~~

Comment 1

[douglas.beattie]

Problem report received and we're working on answers to the 7 questions above.

Comment 2

[douglas.beattie]

Hello Ryan,
Sorry, please let me expand and break it down point by point based on updated reporting and analysis.

1q) How your CA first became aware of the problems listed below (e.g. via a Problem Report, via the discussion in mozilla.dev.security.policy, or via this Bugzilla Bug), and the date.

1a) There were 2 discovery events:
9 March 2016 - We initially identified that some certificate OU fields that contained just meta data characters consisting of ".","-" and " " (i.e space) during one of our internal audits.

16 August 2017 - We identified that the character "_" had been used as the sole content for the OU field via our post issuance lint checker.

2q) Prompt confirmation that your CA has stopped issuing TLS/SSL certificates with the problems listed below.

2a) We stopped issuing certificates with OU fields containing only ".","-" and " " (i.e space) as of 25 October 2016. At the time that meta data characters in the OU field were found, it was not deamed a compliance issue due to the way the BRs defined the application of the meta data rule being applicable to "Other Subject Attributes". 

As far as the issue with "_" being included in the OU field: The fix will be installed on 28 August 2017. Operational work practices were immediately updated to prohibit the use of the identified character set and we will verify that no new certificates with metadata only in subject fields are issued.

3q) Complete list of certificates that your CA finds with each of the listed issues during the remediation process. The recommended way to handle this is to ensure each certificate is logged to CT and then attach a CSV file/spreadsheet of the fingerprints or crt.sh IDs, with one list per distinct problem.

3a) Please find a CSV with these fingerprints attached to this issue. The logging to CT should be finalized by end of the day 25 August 2017.

4q) Summary of the problematic certificates. For each problem listed below: number of certs, date first and last certs with that problem were issued.

4a) From 1 July 2012 through 24 August 2017, there were 1897 certificates issued with only meta data in a field of a certificate.

Breakdown:
- 1601 are either expired or revoked
- 296 remain active. Of the active certificates, 271 are OV with the following mix of fields with meta data only: 268 "OU" and 3 "S" Fields. The last will expire on 5 March 2020.
The remaining 25 active certificates are EV with the following mix of fields with meta data only: 1 "L", 1 "S", 17 "OU" and 6 "Address". The last will expire on 12 July 2019.

5q) Explanation about how and why the mistakes were made, and not caught and fixed earlier.

5a) The certificates were issued due to a following factors.

Up until 25 October 2016, meta data could be entered as the sole content in the OU field which included characters such as "-", " " (i.e. space) or "."

The fix applied on the 25 October 2016 covered the scope originally identified: ".","-" and " " (i.e space) and no additional certificates were issued with these characters since that date.

Meta data in the non OU field was entered incorrectly by the validation specialists, primarily due to the system requiring data to be entered into fields that were actually optional (S, L, and Address). The meta data was used to fill optional fields.

6q) List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

6a) We have recently implemented a post issuance certificate lint checker that provides notifications if a certificate was issued that isn't compliant with our checks which include cablint plus additional checks we have added. While cablint by itself would not have detected this issue, our certlint would, and does (as demonstrated above with the notification that we issued a certificate with "_" in an OU field).

We remain focused on adding a preissuance check and have been working with our CA vendor to provide the applicable hooks to validate precertificates as soon as feasible.

7q) Regular updates to confirm when those steps have been completed.

7a) We'll confirm the date of the current fix when it's put into production for more robust set of meta data characters. The  plan is for 28th of August 2017.

Revocation:
We're processing revocations of the 11 valid (not expired or revoked) certificates with meta data in fields other than the OU field and will notify you when this is complete. We're not planning to revoke certificates with meta data in the OU field.

Comment 3

[douglas.beattie]

Attachment: Fingerprint file

This file contains the list of fingerprints for the certificates referenced in this bug.  The certificates will be posted to CT logs by COB on 25 August.

Comment 4

[Ryan Sleevi]

Hi Doug,

Regarding Comment #2 and Comment #3, your responses are related to the "OU metadata" field, but this bug is about DNS double-dot. Could you please provide an update for this issue?

Comment 5

[douglas.beattie]

Yes, I realized I posted this in the wrong place earlier today, sorry about that.

There are 5 certificates issued under GlobalSign roots on this list referenced above: https://misissued.com/batch/13/

GlobalSign issued 3 certificates and Virginia tech, one of our Trusted Root customers, issued 2.

GlobalSign certificate status: We issued 3 certificates (the report lists 4, but 2 of the crt.sh links point to the same certificate.  The certificates were issued prior to February 2016 when we released a patch that addressed a number of CN and SAN validation deficiencies. This was documented in this prior incident discussion and report (which included these 3 certificates): https://groups.google.com/d/msg/mozilla.dev.security.policy/luxlU5TL2ew/bSmY3I1QCwAJ

We did another search for certificates issued with ".." and found none have been issued since the system patch in February 2016, so we consider this matter resolved (duplicate).

Virginia Tech certificate status: They are a Trusted Root customer who has a name constrained CA and we just received their full set of certificates this afternoon. We'll be processing those and reporting on the findings as soon as we can.

Comment 6

[douglas.beattie]

We received and processed the last 24 months of VT certificates and did not notice any issues with any of them.  We're asking them for all certificates issued under their SHA-256 CA which goes back about 30 months, so we're missing about 6 months (which is where the ".." issue was seen).  We hope to have them tommorrow.

Comment 7

[douglas.beattie]

Still waiting for the last batch of VT certificates, we have escalated and expect the certificates tomorrow.

Comment 8

[Ryan Sleevi]

Doug: It's been one week since the "tomorrow" comment on Comment #7 (itself a delay from Comment #6). Do you have any updates?

Comment 9

[douglas.beattie]

We've been chasing our POC at VT with limited success as he's been on vacation.  We've located a backup contact and have a call scheduled with them tomorrow.  I'm sorry for the delay.

Comment 10

[douglas.beattie]

Today we received a delivery of certs from the wrong CA (a private one) and then received one from hopefully the right CA, but the processing will have to wait until tomorrow due to timezone differences.

Comment 11

[douglas.beattie]

We've completed the review process of the certificates delivered to us and found the following:
- 2 certificates with ".." in the SAN (same as those that were reported), 
  - https://crt.sh/?id=175163894
  - https://crt.sh/?id=8299123 
- 1 certificate with the wildcard character that was not followed by a period, *testmx.vt.edu (expired) 
  - https://crt.sh/?id=206220245
- 1 certificate with "_" in the SAN, opendsa_server.cs.vt.edu.  We've asked them to revoke this. 
  - https://crt.sh/?id=206220246

The report covers all SHA-256 certificates issued under the Virginia Tech Global Qualified Server CA dating back to mid-2014.  This CA is: https://crt.sh/?caid=1366

Comment 12

[Ryan Sleevi]

Doug: Regarding Comment #5, the incident report in https://groups.google.com/forum/#!msg/mozilla.dev.security.policy/luxlU5TL2ew/bSmY3I1QCwAJ suggested that the full set was identified and enumerated, yet Comment #11 highlighted additional certificates (from both GlobalSign and VT)

With respect to understanding the incident report and timing, could you reply using the template? In particular, it's useful to understand the timelines, such as those mentioned in Comment #8, Comment #9, and Comment #10, to better understand holistically how GlobalSign is addressing this.

Thanks!

Comment 13

[douglas.beattie]

Ryan: Let's break this down to 2 different topics, those that GlobalSign issued and those that VT issued.

++ GlobalSign ++
For GlobalSign certificates, Comment #5 is accurate:

GlobalSign certificate status: We issued 3 certificates (the report lists 4, but 2 of the crt.sh links point to the same certificate.  The certificates were issued prior to February 2016 when we released a patch that addressed a number of CN and SAN validation deficiencies. This was documented in this prior incident discussion and report (which included these 3 certificates): https://groups.google.com/d/msg/mozilla.dev.security.policy/luxlU5TL2ew/bSmY3I1QCwAJ

We did another search for certificates issued with ".." and found none have been issued since the system patch in February 2016, so we consider this matter resolved (duplicate).  No incident report is necessary for this as one was already done.

++ Virginia Tech ++
For VT issued certificates, yes, it was hard to get accurate reports because of staff availability and the fact they archived off older certificates which made them hard to retrieve, thus the delay in getting full and complete reports from them.  We were notified of these 2 expired certificates on August 24th and it took 13 days to get the final 100% report of all certificates issued.

In the end, we confirmed that the 2 reported certificates with double-dots were the only 2 with this issue.  Both were expired.

During the course of the full audit, we found 2 more certificates with issues in the CN/SAN, see Comment #11.  We received the final report on September 6th.
- One certificate was expired: https://crt.sh/?id=206220245
- One certificate needed to be revoked, which was done on September 8th https://crt.sh/?id=206220246


+++ Remediation Plan +++
Based on this and other Name Constrained customers' misissuance problems, we've ended the practice of issuing SSL CAs to customers and will be moving all customers to in-house solutions by the end 2018.  In the meantime, we have moved from quarterly 3% audits to monthly 100% audits so we can review and validate compliance more timely and thoroughly. 


+++ VT report using the Mozilla 6 step template ++++

1) How your CA first became aware of the problems listed below (e.g. via a Problem Report, via the discussion in mozilla.dev.security.policy, or via this Bugzilla Bug), and the date.
 * Via this bug on August 24th

2) Prompt confirmation that your CA has stopped issuing TLS/SSL certificates with the problems listed below.
 * Confirmed that VT had not issued any with double-dots other than these 2 that were reported.  As discussed above, it took 13 days to get the complete report of historical certificates.

3) Complete list of certificates that your CA finds with each of the listed issues during the remediation process. The recommended way to handle this is to ensure each certificate is logged to CT and then attach a CSV file/spreadsheet of the fingerprints or crt.sh IDs, with one list per distinct problem.
 * The list of certificates is in Comment #11.

4) Summary of the problematic certificates. For each problem listed below: number of certs, date first and last certs with that problem were issued.
 * Comment #11 describes the 4 problematic certificates.

5) Explanation about how and why the mistakes were made, and not caught and fixed earlier.
 * These were local data validation mistakes and VT has since improved their process.  
 * 3 of the 4 were issued prior to mid 2015 and those issues have not reappeared which demonstrates improved data validation.  
 * The one issued in mid 2016 had an issue with _ in the host name which is a rather common issue (almost 200 in the past week, https://crt.sh/?cablint=62).  We've discussed this with VT and they will not issue any more with this character in the CN or SAN until it's permitted by the BRs in a replacement to failed Ballot 202.
 * GlobalSign did not catch these in our 3% quarterly audit.  We have moved to 100% monthly audit.

6) List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.
 * GlobalSign is doing 100% audits starting for the month of September.

7) Regular updates to confirm when those steps have been completed.
 * Steps have been completed and we believe this incident can be closed.

Comment 14

[Rob Stradling]

(In reply to douglas.beattie from comment #13)
<snip>
> For GlobalSign certificates, Comment #5 is accurate:
> 
> GlobalSign certificate status: We issued 3 certificates (the report lists 4,
> but 2 of the crt.sh links point to the same certificate.

Doug, I believe you're referring to these 2 crt.sh links:
https://crt.sh/?id=6760620
https://crt.sh/?id=6780133

The first is a precertificate; the second is the corresponding certificate.  Strictly speaking they're not the "same certificate", but neither are they entirely unrelated.  Maybe we could call them "conjoined twin certificates".  ;-)

Comment 15

[Gervase Markham [:gerv]]

Given GlobalSign's intent to "move all customers to in-house solutions by the end 2018", I will not be taking up the issue of the slowness of VT in providing necessary information. My last question is on this point:

> Meta data in the non OU field was entered incorrectly by the validation specialists, primarily due to the system requiring
> data to be entered into fields that were actually optional (S, L, and Address). The meta data was used to fill optional 
> fields.

Has the system now been updated to make optional fields actually optional?

Gerv

Comment 16

[douglas.beattie]

Yes, OU and related optional fields are no longer "required".