1391429 - GoDaddy: Non-BR-Compliant Certificate Issuance

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Kathleen Wilson]

The following problems have been found in certificates issued by your CA, and reported in the mozilla.dev.security.policy forum. Direct links to those discussions are provided for your convenience.

To continue inclusion of your CA’s root certificates in Mozilla’s Root Store, you must respond in this bug to provide the following information:
1) How your CA first became aware of the problems listed below (e.g. via a Problem Report, via the discussion in mozilla.dev.security.policy, or via this Bugzilla Bug), and the date.
2) Prompt confirmation that your CA has stopped issuing TLS/SSL certificates with the problems listed below.
3) Complete list of certificates that your CA finds with each of the listed issues during the remediation process. The recommended way to handle this is to ensure each certificate is logged to CT and then attach a CSV file/spreadsheet of the fingerprints or crt.sh IDs, with one list per distinct problem.
4) Summary of the problematic certificates. For each problem listed below: number of certs, date first and last certs with that problem were issued.
5) Explanation about how and why the mistakes were made, and not caught and fixed earlier.
6) List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.
7) Regular updates to confirm when those steps have been completed.

Note Section 4.9.1.1 of the CA/Browser Forum’s Baseline Requirements, which states:
“The CA SHALL revoke a Certificate within 24 hours if one or more of the following occurs: …
9. The CA is made aware that the Certificate was not issued in accordance with these Requirements or the CA’s Certificate Policy or Certification Practice Statement; 
10. The CA determines that any of the information appearing in the Certificate is inaccurate or misleading; …
14. Revocation is required by the CA’s Certificate Policy and/or Certification Practice Statement; or 
15. The technical content or format of the Certificate presents an unacceptable risk to Application Software Suppliers or Relying Parties (e.g. the CA/Browser Forum might determine that a deprecated cryptographic/signature algorithm or key size presents an unacceptable risk and that such Certificates should be revoked and replaced by CAs within a given period of time).

However, it is not our intent to introduce additional problems by forcing the immediate revocation of certificates that are not BR compliant when they do not pose an urgent security concern. Therefore, we request that your CA perform careful analysis of the situation. If there is justification to not revoke the problematic certificates, then explain those reasons and provide a timeline for when the bulks of the certificates will expire or be revoked/replaced. 

We expect that your forthcoming audit statements will indicate the findings of these problems. If your CA will not be revoking the certificates within 24 hours in accordance with the BRs, then that will also need to be listed as a finding in your CA’s BR audit statement.

We expect that your CA will work with your auditor (and supervisory body, as appropriate) and the Root Store(s) that your CA participates in to ensure your analysis of the risk and plan of remediation is acceptable. If your CA will not be revoking the problematic certificates as required by the BRs, then we recommend that you also contact the other root programs that your CA participates in to acknowledge this non-compliance and discuss what expectations their Root Programs have with respect to these certificates.


The problems reported for your CA in the mozilla.dev.security.policy forum are as follows:

** Failure to respond within 24 hours after Problem Report submitted
https://groups.google.com/d/msg/mozilla.dev.security.policy/PrsDfS8AMEk/w2AMK81jAQAJ
The problems were reported via your CA’s Problem Reporting Mechanism as listed here:
https://ccadb-public.secure.force.com/mozilla/CAInformationReport
Therefore, if this is the first time you have received notice of the problem(s) listed below, please review and fix your CA’s Problem Reporting Mechanism to ensure that it will work the next time someone reports a problem like this.

** Invalid dnsNames, containing a space at the beginning
https://groups.google.com/d/msg/mozilla.dev.security.policy/CfyeeybBz9c/lmmUT4x2CAAJ
https://groups.google.com/d/msg/mozilla.dev.security.policy/D0poUHqiYMw/Pf5p0kB7CAAJ
https://crt.sh/?id=93578583&opt=cablint
https://crt.sh/?id=110219638&opt=cablint
https://crt.sh/?id=20950698&opt=cablint
https://crt.sh/?id=25047430&opt=cablint
https://crt.sh/?id=108417123&opt=cablint

** double-dots in dnsName
https://groups.google.com/d/msg/mozilla.dev.security.policy/5bpr9yBgaYo/U-vZ3NNoBgAJ

Comment 1

[Wayne Thayer (old account)]

** Failure to respond within 24 hours after Problem Report submitted
https://groups.google.com/d/msg/mozilla.dev.security.policy/PrsDfS8AMEk/w2AMK81jAQAJ
The problems were reported via your CA’s Problem Reporting Mechanism as listed here:
https://ccadb-public.secure.force.com/mozilla/CAInformationReport
Therefore, if this is the first time you have received notice of the problem(s) listed below, please review and fix your CA’s Problem Reporting Mechanism to ensure that it will work the next time someone reports a problem like this.

We’ve included timelines below specific to each problem report.


** Invalid dnsNames, containing a space at the beginning
https://groups.google.com/d/msg/mozilla.dev.security.policy/CfyeeybBz9c/lmmUT4x2CAAJ
https://groups.google.com/d/msg/mozilla.dev.security.policy/D0poUHqiYMw/Pf5p0kB7CAAJ
https://crt.sh/?id=93578583&opt=cablint
https://crt.sh/?id=110219638&opt=cablint
https://crt.sh/?id=20950698&opt=cablint
https://crt.sh/?id=25047430&opt=cablint
https://crt.sh/?id=108417123&opt=cablint


1) How your CA first became aware of the problems listed below (e.g. via a Problem Report, via the discussion in mozilla.dev.security.policy, or via this Bugzilla Bug), and the date.

- We received a call reporting the issue to our support team on Sunday 30 July 2017. At that time, the problem was incorrectly escalated to our software engineering team for review.
- On Tuesday 1 Aug 2017, the error in handling the problem report was identified while reviewing the MDSP list.
- On Wednesday 2 Aug 2017, the 5 reported certificates were revoked. At this time, we also identified 4 more valid certificates containing spaces.
- On Thursday 3 Aug 2017 the remaining 4 certificates were revoked.

2) Prompt confirmation that your CA has stopped issuing TLS/SSL certificates with the problems listed below.

Our system has been updated to prevent issuance of certificates that contain a space in the CN or any SAN field.

3) Complete list of certificates that your CA finds with each of the listed issues during the remediation process. The recommended way to handle this is to ensure each certificate is logged to CT and then attach a CSV file/spreadsheet of the fingerprints or crt.sh IDs, with one list per distinct problem.

The additional certificates that we found to be unexpired and not revoked:
https://crt.sh/?id=192826740&opt=cablint
https://crt.sh/?id=192867474&opt=cablint
https://crt.sh/?id=192867566&opt=cablint
https://crt.sh/?id=192867982&opt=cablint

The entire list will be attached shortly. All of these certificates are, or will shortly, be logged.

4) Summary of the problematic certificates. For each problem listed below: number of certs, date first and last certs with that problem were issued.

Our investigation found a total of 58 certificates containing 34 unique instances of domain names that contain spaces. These certificates were issued between 19 Nov, 2014 and 2 April, 2017:

5) Explanation about how and why the mistakes were made, and not caught and fixed earlier.
Our system was trimming spaces from the beginning and end of the domain name, but was not detecting and removing whitespace characters from within the domain name, resulting in the issuance of certificates where the eTLD+1 label begins with whitespace when a CSR was submitted containing a typo. We have now added an explicit validation check for whitespace characters within domain names.

6) List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.
All of these certificates have been revoked. We are in the process of implementing a certlint check prior to issuance. We intend to complete this work by 30 Nov 2017. We have also reviewed the proper handling of problem reports received via telephone with our support team.

7) Regular updates to confirm when those steps have been completed.
We will update this ticket when cablint/x509lint validation has been added to our issuance process.


** double-dots in dnsName
https://groups.google.com/d/msg/mozilla.dev.security.policy/5bpr9yBgaYo/U-vZ3NNoBgAJ


1) How your CA first became aware of the problems listed below (e.g. via a Problem Report, via the discussion in mozilla.dev.security.policy, or via this Bugzilla Bug), and the date.

- I received the email posted to this list by Hanno Bock on 18 July 2017 containing the following certificates issued by GoDaddy:
https://crt.sh/?id=22835635
https://crt.sh/?id=8216255
https://crt.sh/?id=637932 (previously revoked)
- On 18 July 2017 we identified 2 more active certificates containing this problem.
- The certificates were revoked on 19 July 2017.

2) Prompt confirmation that your CA has stopped issuing TLS/SSL certificates with the problems listed below.

Our system has been updated to prevent issuance of certificates that contain an empty domain name label aka double-dot.

3) Complete list of certificates that your CA finds with each of the listed issues during the remediation process. The recommended way to handle this is to ensure each certificate is logged to CT and then attach a CSV file/spreadsheet of the fingerprints or crt.sh IDs, with one list per distinct problem.

The entire list will be attached shortly. All of these certificates are, or will shortly, be logged.

4) Summary of the problematic certificates. For each problem listed below: number of certs, date first and last certs with that problem were issued.

In our research, we found a total of 223 certificates. These certificates were issued between 24 March 2005 and 18 Oct 2016:

5) Explanation about how and why the mistakes were made, and not caught and fixed earlier.
Our system was not properly stripping empty labels from within domain names

6) List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.
All of these certs have been revoked. We are in the process of implementing a certlint check prior to issuance. We intend to complete this work by 30 Nov 2017.

7) Regular updates to confirm when those steps have been completed.
We will update this ticket when cablint/x509lint validation has been added to our issuance process.

Comment 2

[Wayne Thayer (old account)]

Attachment: List of issued certificates containing spaces in the domain name

Comment 3

[Wayne Thayer (old account)]

Attachment: List of issued certificates containing double-dots in the domain name

In my original post I stated that GoDaddy had issued 223 containing consecutive periods, aka double-dots in a domain name. I incorrectly included requests for certificates in that count. The correct count of issued certificates is 87 as detailed in the attached file.

Comment 4

[Ryan Sleevi]

(In reply to Wayne Thayer from comment #1)
> 5) Explanation about how and why the mistakes were made, and not caught and
> fixed earlier.
> Our system was trimming spaces from the beginning and end of the domain
> name, but was not detecting and removing whitespace characters from within
> the domain name, resulting in the issuance of certificates where the eTLD+1
> label begins with whitespace when a CSR was submitted containing a typo. We
> have now added an explicit validation check for whitespace characters within
> domain names.

While this is a positive change to preventing the immediate issue, as a mitigation for future issues, it seems insufficient.

That is, have you considered implementing within your system a validation control that the domain names appearing with the CN or SAN properly conform to the RFC 1034 preferred name syntax (as modified by RFC 1123), as specified in RFC 5280 Section 4.2.1.6?

Properly implemented, this would also have detected the other issue (double dots within domain names)

However, based on your response 6, it may be that you are implementing that full check - could you clarify?

> 
> 6) List of steps your CA is taking to resolve the situation and ensure such
> issuance will not be repeated in the future, accompanied with a timeline of
> when your CA expects to accomplish these things.
> All of these certificates have been revoked. We are in the process of
> implementing a certlint check prior to issuance. We intend to complete this
> work by 30 Nov 2017. We have also reviewed the proper handling of problem
> reports received via telephone with our support team.
> 

> ** double-dots in dnsName
> https://groups.google.com/d/msg/mozilla.dev.security.policy/5bpr9yBgaYo/U-
> vZ3NNoBgAJ
> 
> 5) Explanation about how and why the mistakes were made, and not caught and
> fixed earlier.
> Our system was not properly stripping empty labels from within domain names

Could you provide a more detailed explanation as to the root cause. For example, how was your system able to validate domains with empty-labels? Answers from other CAs have included examining only the authorization domain name, and having no validation on sub-ordinate labels provided by the customer.

That is, put differently, the goal is to understand "Why did this even work in the first place", examining each of the systems from ingestion (e.g. CSR) to issuance, and understanding whether or not controls could have detected or prevented this, or existing controls failed.

Comment 5

[Wayne Thayer (old account)]

(In reply to Ryan Sleevi from comment #4)
> (In reply to Wayne Thayer from comment #1)
> > 5) Explanation about how and why the mistakes were made, and not caught and
> > fixed earlier.
> > Our system was trimming spaces from the beginning and end of the domain
> > name, but was not detecting and removing whitespace characters from within
> > the domain name, resulting in the issuance of certificates where the eTLD+1
> > label begins with whitespace when a CSR was submitted containing a typo. We
> > have now added an explicit validation check for whitespace characters within
> > domain names.
> 
> While this is a positive change to preventing the immediate issue, as a
> mitigation for future issues, it seems insufficient.
> 
> That is, have you considered implementing within your system a validation
> control that the domain names appearing with the CN or SAN properly conform
> to the RFC 1034 preferred name syntax (as modified by RFC 1123), as
> specified in RFC 5280 Section 4.2.1.6?
> 
> Properly implemented, this would also have detected the other issue (double
> dots within domain names)
> 
> However, based on your response 6, it may be that you are implementing that
> full check - could you clarify?
> 

The regex we were using to validate RFC 1034 compliance was incomplete – specifically it was not checking for spaces in the domain name. We were stripping spaces from the beginning and end, but obviously that wouldn’t catch these certs. We now have an additional check in place for this specific condition, but per your suggestion, we are going to revisit our RFC 1034 validation logic again to ensure that we’re not missing anything else.

> 
> > ** double-dots in dnsName
> > https://groups.google.com/d/msg/mozilla.dev.security.policy/5bpr9yBgaYo/U-
> > vZ3NNoBgAJ
> > 
> > 5) Explanation about how and why the mistakes were made, and not caught and
> > fixed earlier.
> > Our system was not properly stripping empty labels from within domain names
> 
> Could you provide a more detailed explanation as to the root cause. For
> example, how was your system able to validate domains with empty-labels?
> Answers from other CAs have included examining only the authorization domain
> name, and having no validation on sub-ordinate labels provided by the
> customer.
> 
> That is, put differently, the goal is to understand "Why did this even work
> in the first place", examining each of the systems from ingestion (e.g. CSR)
> to issuance, and understanding whether or not controls could have detected
> or prevented this, or existing controls failed.

The root cause of this issue was the same inadequate verification of domain name input from CSRs that resulted in spaces in CNs and SANs. For the older certificates that we identified, your “why did this ever work” question is difficult to answer. For certificates issued in the past few years, this worked when validating domain control because our system was first removing the left most label if it was ‘*’ or ‘www’. Then periods were stripped from the beginning and end of the resulting string to obtain the authorization domain name. This logic was capable of validating domains that contained the empty label following a label of ‘*’ or ‘www’.

Comment 6

[Ryan Sleevi]

Below is my summary of the issues and remediation plan:

1) Leading spaces in DNS Names
- See Comment #0, Comment #1, Comment #5
- Existing Mitigations
  - Validation controls existed (regex-based) to verify consistency of domains. These controls failed because systems trimmed leading/trailing whitespace (See Comment #1), but did not consider whitespace within labels when trimmed to eTLD+1 (See Comment #1, Comment #5)
- Remediation
  - 2017-08-18 (or earlier) - System corrected to operate on the contents of the CN/SAN, and not just the authorization domain name
  - 2017-11-30 - certlint-based preissuance checks to be deployed

2) Double dots within DNS names
- See Comment #0, Comment #1, Comment #5
- Remediation
  - 2017-08-18 - System corrected to check for empty-labels within DNS names
  - 2017-11-30 - certlint-based preissuance checks to be deployed

Is this correct? If so, I believe we can consider this matter resolved, and the proposed prevention mechanism to be deployed on 2017-11-30 will help prevent future occurrences. I'm setting the Needs-Info flag so that Bugzilla will periodically remind you to provide updates on the progress and timing of this deployment.

Comment 7

[Wayne Thayer (old account)]

This is correct. We will update this bug when the certlint-based preissuance check is in place.

Comment 8

[Gervase Markham [:gerv]]

Reassigning to new GoDaddy rep.

Gerv

Comment 9

[Daymion Reynolds]

We are ready to close this open issue. As of today 11/30/2017 we have implemented CabLint as the certlint solution. All Certs being provisioned by our system are now linted, and will not be issued unless they pass the lint test.