1521950 - QuoVadis: BR Error - san dns name starts with period

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Stephen Davidson]

At 22:00 GMT Our post-issuance linting tool alerted 8 certificates issued today for the same Subscriber with the following uncommon but related zLint errors:

ERROR: Characters in labels of DNSNames MUST be alphanumeric, - , _ or *
ERROR: DNSName MUST NOT start with a period
ERROR: DNSNames should not have an empty label.

The certificates are:

https://crt.sh/?id=1133532965
https://crt.sh/?id=1133528454
https://crt.sh/?id=1133520969
https://crt.sh/?id=1133516374
https://crt.sh/?id=1133511797
https://crt.sh/?id=1133509223
https://crt.sh/?id=1133504710
https://crt.sh/?id=1133495997

In the interest of transparency, we are reporting them immediately. The cause has been identified and will be remediated, and the customer is being contacted regarding revocation. We will update later.

Comment 1

[Stephen Davidson]

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

The issue was discovered via post issuance linting on 1/22 at 22:00 GMT.

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

• The issue was discovered via post issuance linting on 1/22 at 22:00 GMT. The customer was informed.
• An investigation was immediately made of the certificate management system and a fix was deployed in Development systems on the evening of 1/22. That was moved to Production systems on 1/23.
• On 1/28 at 13:30 GMT the certificates were revoked.

3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

Yes, a fix has been deployed in our certificate management system.

4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

The certificates triggered the following zLints as they included a SAN with a leading “.” (equivalent to an empty label).

ERROR: Characters in labels of DNSNames MUST be alphanumeric, - , _ or *
ERROR: DNSName MUST NOT start with a period
ERROR: DNSNames should not have an empty label.

These requests were made sequentially on 1/22. A review was made of other QV issuance to verify that no other certificates of this type exist.

5. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

https://crt.sh/?id=1133532965
https://crt.sh/?id=1133528454
https://crt.sh/?id=1133520969
https://crt.sh/?id=1133516374
https://crt.sh/?id=1133511797
https://crt.sh/?id=1133509223
https://crt.sh/?id=1133504710
https://crt.sh/?id=1133495997

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

Our certificate management system did not have a filter in place for this, and had never had a previous request that included the leading dot, before these 8. The circumstance is so rare that Censys is only aware of 11 currently valid trusted certs with this error (including these 8).

https://censys.io/certificates/report?q=zlint.lints.e_san_dns_name_starts_with_period%3A+all&field=parsed.issuer.organization.raw&max_buckets=

7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

The fix was deployed in production systems on the evening of 1/23.

Comment 2

[Ryan Sleevi]

Stephen: What filters did you have in place, and how have they changed?

While this specific incident is low, it is part of a generic trend that has had substantial attention in 2018 - namely, the failure to validate that the name conforms to the DNS syntax. This has shown in a variety of misissuance reports; whether underscores, spaces, non-ASCII characters, double dots, etc.

For this reason, it's important to understand what steps QuoVadis had taken, if those steps had been reviewed in light of any of the dozen+ past incidents from other CAs, and what steps are now being taken.

Comment 3

[Stephen Davidson]

Hi Ryan – we have a broad array of filters in place in our certificate management system (and where appropriate, templates on the CAs) to enforce the BR and other requirements.

We pay close attention to public discussion (primarily m.d.s.p and CA Browser Forum), our linting of our issuance using zLint, and zLint development in order to keep an eye on the currency of these settings, including for the DNS syntax examples you mention.

However, there was not an effective filter for the leading dot, partly because we’d never received a request with one before getting 8 in the span of a few minutes. The problem was identified and remediated within 24 hours.

Overall our approach is proactive for issues in mainstream discussion, but recognizing resource constraints, perhaps reactive for rarities such as this.

Comment 4

[Ryan Sleevi]

Could you clarify what changed in those filters?

For example, a filter that was updated to merely prevent a leading period may still allow for other violations of the RFC1034/1123 rules. That was what was being alluded to in Comment #2.

I think an example highlighting how a CA can provide detail to questions like this can be found at https://bugzilla.mozilla.org/show_bug.cgi?id=1428877#c4 , which hopefully QuoVadis can provide to a similar level of detail about its processes and procedures.

Comment 5

[Stephen Davidson]

Hi Ryan: While we believe we have good coverage, we are reviewing our filters against the complete spreadsheet of lints provided for zLint. I anticipate providing a more clear information within the week.

Comment 6

[Stephen Davidson]

Thank you for your patience. Following is a summary of the checks relating to TLS SAN in our CMS, in no particular order.

1. Disallow leading dot or hyphen
2. Disallow double dots
3. If a wildcard (*) exists, check that there is only 1 and it is the first label; wildcard must be followed by a dot
4. Check value against RegEx - ^[a-zA-Z0-9-.]{0,100}?.[a-zA-Z0-9-]{2,26}?$
5. Check for at least 1 alphanumeric character (performed on every SAN field)
6. Disallow underscores
7. Verify TLD against allowed list (also prevents trailing dots, etc.)
8. Verify that IP address in SAN is iPAddress. Apart from dNSName, no other field types
9. If the 3-4th characters of the value are hyphens, check that the first 2 characters are “xn” (i.e., punycode) and label separators are U+002E

Please let me know if you wish clarification on any point.

Comment 7

[Stephen Davidson]

Pre-issuance linting has been implemented for QuoVadis trusted TLS policies effective April 16, 2019.
I think this resolves this bug.

Comment 8

[Wayne Thayer]

It appears that remediation is complete.