1716123 - e-commerce monitoring GmbH: CN domain not in SAN

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Michel Le Bihan]

Hello,
I found a leaf certificate: https://crt.sh/?id=4684929325&opt=zlint with the corresponding precertificate https://crt.sh/?id=4684924227&opt=zlint that doesn't have the domain in CN in SAN.

Comment 1

[Ryan Sleevi]

It's very disconcerting to see a compliance incident like this within months of this CA being trusted (Bug #1697071). I think there's going to be a lot said about the quality of the first incident report.

In particular, I find the following statement very concerning, from Bug 1627552, Comment #15:

DZ: e-commerce monitoring gmbh uses a pre issuance linting process (currently self developed with zlint and others under consideration), this is adressed by GCP 4.3. („reviewed and approved“ is intended to mean automatic checks as well as human four eyes principle at the same time). GCP 4.3 contains now a clarification, that "reviewed" means also technical implemented checks.

Comment 2

[Daniel Zens]

Thank you for pointing this out. GLOBALTRUST immediately stopped the issuance of SSL certificates until this issue is resolved. The certificate has been revoked without replacement. We are willing to share a transparent and comprehensible investigation of this case with you, and for this purpose we will publish a complete report until Friday 18th at the latest.

Comment 3

[Daniel Zens]

Attachment: Full incident report, 2021-06-16

Dear All,

we finished the examination and attached you will find a final incident report.

Comment 4

[Mathew Hodson]

(In reply to Daniel Zens from comment #3)

Q: List of steps your CA is taking to resolve the situation and ensure that such situation or
incident will not be repeated in the future, accompanied with a binding timeline of when
your CA expects to accomplish each of these remediation steps.

A: We extended our pre-checks and pre-issuance lint testings:

Can you provide more details about the pre-issuance and post-issuance linting you had on 2021-06-11 when the certificate was issued? As the crt.sh link shows, ZLint would have caught this. Why didn't GLOBALTRUST detect this issue?

a) In general, additional check routines were created to identify customer information that is
correct in individual fields but incorrect in combination with other information
b) Reports on individual errors were formulated in more detail in order to better support the
staff in making corrections
c) Incorrect customer entries were not replaced by default values, but must be corrected
manually in coordination with customer. Typical wrong entries are Space " " in Server Name
or SAN, ";" instead of "," in SAN, other Tags in SAN, than "DNS:" and "IP:", forget ":" in SAN
and so on
d) The check of the CN entries against the SAN entries is optimized in such way that all
incorrect user information must be cleaned up before the pre-issuance lint testing starts

The items described here sound like changes to the code that generates the certificate. Will your linting check that the commonName is contained in the list of SAN entries in the certificate? Do you use well-known linters such as ZLint or certlint?

Comment 5

[Daniel Zens]

(In reply to Mathew Hodson from comment #4)

Thank you Mathew for your questions:

Can you provide more details about the pre-issuance and post-issuance linting you had on 2021-06-11 when the certificate was issued? As the crt.sh link shows, ZLint would have caught this. Why didn't GLOBALTRUST detect this issue?

pre-issuance checks include but are not limited to:

• validation steps for the respective certificate profile done
• CCA records query
• Google Safe Browsing List check
• preferred name syntax as specified in applicable RFC
• no underscore characters
• no IDN
• asterisk in the left-most if present, no wildcards for EV
• CN may contain only 1 entry
• CN also in SAN (but please see below and the original incident report)
• IANAs Root Zone DB check to avoid Internal Names
• SAN entries reduced to DNS: and IP: (in case of EV only DNS:)
• validity period equal or less than 397 days
• allowed KU + EKU combination for example no keyAgreement for RSA keys, no mixture of serverAuthentication and eMailprotection (although this is also ensured on another level than pre issuance check)
• formal CSR check, allowed key parameters, no known weaknesses
• further checks include: show a warning if a certificate of the same customer had been revoked in the past
• post-issuance checks the certificate result against the expected values

I agree with you that zlint would have caught this.

The items described here sound like changes to the code that generates the certificate.

Strictly speaking, these checks takes place earlier, as the certificate generation can not even be initiated if there are errors.

Will your linting check that the commonName is contained in the list of SAN entries in the certificate?

Yes. It is checked if the CN contains an entry that is included 1:1 also in the SAN. Basically we have had this check before, but as we reported, this did not work in a specific constellation.

Do you use well-known linters such as ZLint or certlint?

Not yet, as Ryan had already cited me from https://bugzilla.mozilla.org/show_bug.cgi?id=1627552#c15
At this point in time we are using our own tool, that in basic has existed for almost 20 years, covering different certificate types and that is constantly being adopted in case of changing requirements, with some examples as mentioned above.

Comment 6

[Andrew Ayer]

The technical issuing process, including all tests, was tested internally with its own - internal - SubCA, but a complete run including ct entries is only possible with a real case. Since this product (wildcard) was issued for the first time, the error had not been discovered before.

Could GLOBALTRUST elaborate on why a "complete run" requires using a publicly-trusted CA? Why could wildcard issuance not have been tested using an untrusted test CA? Has GLOBALTRUST considered using test CT logs, like Google Solera or Let's Encrypt Testflume?

This combination affects customers who specify unsuitable server names for the CN and make different - but correct - SAN entries.

Why is it the customer's responsibility to specify a CN value? One way to avoid this error entirely would be for the system to automatically use the first SAN entry as the CN (provided it is not too long to fit in the CN). Or better yet, omit the CN entirely, considering it long since deprecated. Has GLOBALTRUST evaluated these options?

In this case, the wrong CN was corrected through the value www.[wildcard domain name], but this entry was not followed up in the SAN.

Could GLOBALTRUST elaborate on precisely what happened here? What values did the customer initially provide for the CN and SANs? If the customer had initially specified www.e-rating.at as the CN, would the request have been properly rejected?

If the customer initially provides invalid values and later corrects them, are there any other validation checks that can be, or could have been, bypassed? Are/were domain validation and CAA checking performed on corrected values?

This constellation was only possible for issuing wildcard-certificates.

Why is this?

In general, additional check routines were created to identify customer information that is correct in individual fields but incorrect in combination with other information

What new check routines have you added?

Incorrect customer entries were not replaced by default values, but must be corrected manually in coordination with customer. Typical wrong entries are Space " " in Server Name or SAN, ";" instead of "," in SAN, other Tags in SAN, than "DNS:" and "IP:", forget ":" in SAN and so on

In what format do customers specify the CN and SANs? At what stage in the issuance pipeline is the customer's specification parsed into the machine representations that are ultimately placed in the certificate? Is validation (such as checking that the CN is in the SANs, domain validation, CAA) done before or after this parsing occurs?

GLOBALTRUST previously stated that zlint was "under consideration". What is the current status of that consideration? Does GLOBALTRUST have a timeline for adopting zlint?

Comment 7

[Daniel Zens]

(In reply to Andrew Ayer from comment #6)

Could GLOBALTRUST elaborate on why a "complete run" requires using a publicly-trusted CA? Why could wildcard issuance not have been tested using an untrusted test CA? Has GLOBALTRUST considered using test CT logs, like Google Solera or Let's Encrypt Testflume?

I agree this is an extremely unlucky wording.
The emphasis should not have been on CT, nor a real CA would be necessary. However, we already know and have used test CT logs. Also,this was NOT a test certificate.
Rather it is always possible that issues occur in a real run-through, that were not yet present in test scenarios.

Also, there is no problem with the wildcard itself, but with the combination of a wildcard with a non-wildcard in case the customer specifies a wrong CN . Before this incident, we have checked at the time of order but not immediately before issuance .All test cases had a valid CN. Yes, we know that this is a serious gap - which we immediately closed as reported.

Why is it the customer's responsibility to specify a CN value? One way to avoid this error entirely would be for the system to automatically use the first SAN entry as the CN (provided it is not too long to fit in the CN).

It is rather his opportunity not his responsibility. As part of our customer journey, a customer can specify a range of SANs to be used by the machine but can choose which one he wants to be the prominent "main" name.

As described in the initial incident report, we do it vice versa and move the CN to the SAN, in case the applicant did not already specify it.
This approach has grown historically from a time when there were no SANs at all and is based, among other things, on the fact that we also have other certificate types where a CN is ordered, but (also) has to be moved somewhere else.

Or better yet, omit the CN entirely, considering it long since deprecated. Has GLOBALTRUST evaluated these options?

This is not quite as simple as abolishing the OU (no one really needs it, it doesn't hurt us, the doubts are justified.)
But of course I fully agree that abolishing the CN would immediately get to grips with all these cases! However so far we have not had any certainty how applications (beyond the major browsers) would react to a certificate without a CN in it. Do you have experience you can share with me?

Could GLOBALTRUST elaborate on precisely what happened here? What values did the customer initially provide for the CN and SANs?

The original order was:
CN: e-rating.at
SANs: e-rating.at, *.e-rating.at, mail.e-rating.at
Our system denied this CN because because it expected a CN to be a fully qualified domain name in the sense of having at least 2 portions other than the top level domain.
(We are aware that e-rating would have been an CN allowed by the BR)
The CN was than corrected to www.e-rating.at but the SAN was not followed up. Today this would lead to an error, in the sense that issuance is blocked, until you have a 1:1 matching of the CN and one of the SANs

If the customer had initially specified www.e-rating.at as the CN, would the request have been properly rejected?

No, because the correct CN would have been moved to the SAN automatically.

If the customer initially provides invalid values and later corrects them, are there any other validation checks that can be, or could have been, bypassed?

No.

Are/were domain validation and CAA checking performed on corrected values?

Yes.

This constellation was only possible for issuing wildcard-certificates.
Why is this?

Historically grown, we manage wildcard domain names and defined server names differently

What new check routines have you added?

As described above and in the original report

In what format do customers specify the CN and SANs?

We require the customer to specify defined fully qualified domain names or wildcard domain names or ipv4 addresses and separate them with commas without spaces. This is not yet optimal, in the past we even required to type DNS: or IP:.

At what stage in the issuance pipeline is the customer's specification parsed into the machine representations that are ultimately placed in the certificate? Is validation (such as checking that the CN is in the SANs, domain validation, CAA) done before or after this parsing occurs?

In general every application undergoes a process that is broken down into different defined status:

1. Order available
2. Order collected in our system
3. Order in work, this includes validation depending on certificate type
4. Validation completed, approved by first operator
5. Verification through second operator.
6. Issuance
7. Check of the issued certificate
8. Depends on type and situation, under normal circumstances delivery to customer.
9. Etc.

The machine represantation to be included in the certificate is created at stage 6. Issuance.

GLOBALTRUST previously stated that zlint was "under consideration". What is the current status of that consideration? Does GLOBALTRUST have a timeline for adopting zlint?

This is a matter of integration efforts and whether the technical conditions are practicable.
The specifications of the common linting tools including disucssions around them have also been part of our ongoing requirements monitoring.
In some issues, we also have to go much further then the linters would. For example, none of them checks BR 7.1.4.3.1 as an exhaustive list, or: for us the existence of both CRL and OCSP is mandatory.
We have also been learning a lot from other CA’s current and previous incidents.

Comment 8

[Andrew Ayer]

(In reply to Daniel Zens from comment #7)

In general every application undergoes a process that is broken down into different defined status:

1. Order available
2. Order collected in our system
3. Order in work, this includes validation depending on certificate type
4. Validation completed, approved by first operator
5. Verification through second operator.
6. Issuance
7. Check of the issued certificate
8. Depends on type and situation, under normal circumstances delivery to customer.
9. Etc.

At what stage is domain validation performed? Is it performed manually by operators?

The machine represantation to be included in the certificate is created at stage 6. Issuance.

It seems quite problematic that the machine representations are created after verification, since there's a risk that verification gets performed on different values than are included in the certificate.

GLOBALTRUST previously stated that zlint was "under consideration". What is the current status of that consideration? Does GLOBALTRUST have a timeline for adopting zlint?

This is a matter of integration efforts and whether the technical conditions are practicable.

This doesn't answer my question. If the current technical conditions are not practical, why not and what steps is GLOBALTRUST doing to make it practical, and with what timeline?

Comment 9

[Daniel Zens]

(In reply to Andrew Ayer from comment #8)

3. Order in work, this includes validation depending on certificate type
4. Validation completed, approved by first operator
5. Verification through second operator.

At what stage is domain validation performed? Is it performed manually by operators?

Step 3. Issuance is programatically blocked until Step 5 is done.
Any change in step 4 or 5 moves the application back, beginning at stage 3.

We use "constructed eMail" (BR 3.2.2.4.4), where the confirming response is verified by the operators. (again, this refers to steps 3-5)

It seems quite problematic that the machine representations are created after verification, since there's a risk that verification gets performed on different values than are included in the certificate.

We are convinced that in our process there is no risk for different (and thus unverified) machine representation to be created and used in the issuance. Also, no human input is possible at this stage.

However, an issued certificate must also be released for delivery (step 7). The release would not work if the the result differs from the expected certification data. In such a case, the process arranges non-delivery and immediate revocation. We are aware that this alone would not be enough to avoid misissuance. But this is an additional "just in case"-check.

This doesn't answer my question. If the current technical conditions are not practical, why not and what steps is GLOBALTRUST doing to make it practical, and with what timeline?

We are analyzing each single check provided by the well-known tools, examine how far it is already realized and how we can optimize our system. The timeline for implementing new or aligning existing checks always has to be adjusted to the effective date of the underlying requirement as the deadline, be it the BRs, any root store policies, ETSI requirements, or any self-imposed ones. There are no pending action items for us in order to achieve a compliant operation, as we are currently operating in a compliant way.

Comment 10

[Andrew Ayer]

(In reply to Daniel Zens from comment #9)

It seems quite problematic that the machine representations are created after verification, since there's a risk that verification gets performed on different values than are included in the certificate.

We are convinced that in our process there is no risk for different (and thus unverified) machine representation to be created and used in the issuance.

I encourage you to read up on parser differential bugs, which have caused a considerable number of security vulnerabilities over the years.

A concrete example of what could go wrong is that at some point you decide to accept space-separated lists in addition to comma-separated lists. You update the parser in step 6 to accept space separators, but forget to update the parser used during the validation step. An attacker requests a certificate for "www.example.com www.example.net". The validation step, which only accepts comma separators, thinks this is a one-SAN certificate for a subdomain of example.net, and requires validation to be completed on example.net only. But when the certificate is issued, it's issued as a two-SAN certificate containing www.example.com, which was unvalidated.

Even if you think this particular scenario is unlikely, it's just one possibility of what could go wrong if different parsers are used for validation and issuance.

We are analyzing each single check provided by the well-known tools, examine how far it is already realized and how we can optimize our system. The timeline for implementing new or aligning existing checks always has to be adjusted to the effective date of the underlying requirement as the deadline, be it the BRs, any root store policies, ETSI requirements, or any self-imposed ones. There are no pending action items for us in order to achieve a compliant operation, as we are currently operating in a compliant way.

Despite your "compliant operation", the very first non-test-website certificate that you issued was misissued. Complying with the BRs and root policy is merely the baseline and this incident and past incidents have made clear that it is not enough. Mozilla also considers what CAs do to minimize risk to Mozilla users. Your refusal to commit to using industry-standard linters even though they would have prevented this incident, as well as your use of manual domain validation despite the well-established risks, do not paint a picture of a CA that is making an effort to minimize risk to Mozilla users.

Comment 11

[Daniel Zens]

(In reply to Andrew Ayer from comment #10)

I encourage you to read up on parser differential bugs, which have caused a considerable number of security vulnerabilities over the years.
[...]

Thank you for your feedback including this concrete example. We have thoroughly reviewed and concluded that your concerns do not match our risk analysis.

[...] Your refusal to commit to using industry-standard linters even though they would have prevented this incident

I cannot see any refusal, as it was previously stated that we have already been imitating the linter's behavior. Despite the defective nature in this case.

[...] do not paint a picture of a CA that is making an effort to minimize risk to Mozilla users.

I deeply regret that you have this opinion. As mentioned, this does not correspond to our perspective, especially since it has already been mentioned that in some points we are going beyond the mere baseline of BRs and root policies.

Comment 12

[Ryan Sleevi]

Ben: Sending to you. Despite Comment #11, I have to agree with Andrew's analysis in Comment #10. This does not seem to be a mature CA operation that understands the risks it poses to users, or that follows industry good practices. I realize that both NIH and SEP are both two extremes of the same spectrum, but I can't help but feel the responses seem very squarely in the former, which is not ideal.

Comment 13

[Ben Wilson]

Can eCommerce Monitoring/GlobalTrust report other efforts it is taking to improve its CA operations?

Comment 14

[Daniel Zens]

I am off for education for tge rest of this week and get back with a detailed answer upcoming tuesday

Comment 15

[Daniel Zens]

Thank you for your patience.

In the meantime, efforts have been made to directly integrate the functions of certlint, zlint and x509lint. We expect a pre-issuance function by the third week of September, 2021.

Comment 16

[Daniel Zens]

The pre-issuance linting has been realized.

In case there are no other questions, could this bug be closed?

Comment 17

[Ben Wilson]

I'll put this on my schedule to close this on Wed. 15-Sept-2021.