Closed Bug 1705791 Opened 3 years ago Closed 3 years ago

Telekom Security: Multiple commonName in certificates

Categories

(CA Program :: CA Certificate Compliance, task)

Product:
CA Program
Component:
CA Certificate Compliance
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: michel, Assigned: Arnold.Essing)

Details

(Whiteboard: [ca-compliance] [ov-misissuance])

Attachments

(2 files)

hex_serials
8.67 KB, application/octet-stream
Details
crtsh_ids
10.39 KB, application/octet-stream
Details

Arnold: Could you please provide an incident response? This CA is disclosed by being operated by Deutsche Telekom Security, despite linking to DFN-Verein's audit / chaining through https://crt.sh/?id=23908438&opt=zlint

Assignee: bwilson → Arnold.Essing
Status: NEW → ASSIGNED
Summary: DFN-Verein: Multiple commonName in certificates → Telekom Security: Multiple commonName in certificates
Whiteboard: [ca-compliance]

I also found those certificates with the same issue that have a different intermediate, but chain to the same root:
https://crt.sh/?id=4162006962&opt=zlint
https://crt.sh/?id=4033884699&opt=zlint

Dear Ryan, thank you for the notification. We assume that you refer to the "Owner -> This CA"-entry in https://crt.sh/?id=25484751 which, according to our understanding, is wrong. The actual operator of that CA is the DFN-Verein and Jürgen Brauckmann will take over the further handling of this bug.
We are looking into why "Deutsche Telekom Security GmbH" is disclosed as the operator in crt.sh. In our understanding, the corresponding information in CCADB should be correct, if that is the source of the information.

Michel, could you please elaborate why you feel that this is a violation of Mozillas current requirements for CAs? I would assume that you refer to
BR section 7.1.4.2.2 Subject Distinguished Name Fields which states for subject:commonName

"If present, this field MUST contain a single IP address or Fully‐Qualified Domain Name that is one of the values contained in the
Certificate’s subjectAltNameextension"

All commonName fields in the certificates you mentioned contain a single IP address or Fully‐Qualified Domain Name.

The wording of 7.1.4.2.2 is part of the BRs since its earliest days; and our interpretation always was that it prevents monstrosities like:

CN=www.example.org,example.org

but does not forbid:

CN=www.example.org
CN=example.org

CN is deprecated, and thus multiple CNs are, well, deprecated all the more. We have been aware that we should change this behaviour of our
software environment for some time now. We've decided earlier this year that we will remove multiple CNs in server certificates later on, but we don't think that multiple CNs constitute a BR or Mozilla policy violation.

Jurgen: Can you clarify if you’ve examined past discussion on m.d.s.p. and the CA/Browser Forum before posting Comment #4?

It has never been allowed. The “single IP address or FQDN” has always been specifying one, and exactly one, instance of the CN field, since the earliest days of the BRs, precisely because of the ambiguities and security issues presented by multiple instances.

In the validation subcommittee of the CA/Browser Forum Server Certificate WG, this has been widely and uncontroversially understood. While the certificate profile work seeks to make all existing requirements unambiguous, this is not and has not been a reasonable interpretation. It equally defies a basic understanding of X.500 naming rules, or those of RFC 2818, so the argument “the BRs are ambiguous” also don’t hold, given the nearly three decade understanding here.

Flags: needinfo?(brauckmann)

Disclaimer: While we were issuing TLS certificates, we never did this and would have never done this. But since it is a remarkably interesting discussion and since I always try to learn more about the PKI eco system I would like to add the following:

RFC 2818 states

Otherwise, the (most specific) Common Name field in the Subject field of the certificate MUST be used.

I would understand the words in the parentheses "most specific" of this sentence as the proof that the authors of RFC 2818 were assuming that there might be more than one Common Name in the Subject Distinguished Name field.

(In reply to Ryan Sleevi from comment #5)

Jurgen: Can you clarify if you’ve examined past discussion on m.d.s.p. and the CA/Browser Forum before posting Comment #4?

We have been following the discussion on m.d.s.p. and the CA/Browser forum for years, and I don't remember and cannot find any discussion
about multi CNs. Is this something that was discussed in the EV guideline process?

It has never been allowed. The “single IP address or FQDN” has always been specifying one, and exactly one, instance of the CN field, since the earliest days of the BRs, precisely because of the ambiguities and security issues presented by multiple instances.

In the validation subcommittee of the CA/Browser Forum Server Certificate WG, this has been widely and uncontroversially understood. While the certificate profile work seeks to make all existing requirements unambiguous, this is not and has not been a reasonable interpretation. It equally defies a basic understanding of X.500 naming rules, or those of RFC 2818, so the argument “the BRs are ambiguous” also don’t hold, given the nearly three decade understanding here.

I don't dispute that our view might be plain wrong, but I currently don't see how it is that obvious.

I don't see where RFC 2818 expresses a single CN requirement (see also Rufus comment). RFC 6125 explicitely acknowledges the occurence of multi CNs, and limits its use with "SHOULD NOT" (not MUST NOT).

Also note that X.520 and RFC 4519 lists common name as multi valued. Multi valued attributes are of course different from multi occurences of
the attribute, but nevertheless, all over X.520 and LDAP a DN may have more than one CN value. Also RFC5280 does not limit this in any way.

What are we missing?

We are still trying to confirm if this is an incident and if we need to revoke the existing certs.

Flags: needinfo?(brauckmann)

Hmmm. I found https://groups.google.com/g/mozilla.dev.security.policy/c/HcIbeFRiyOU/m/APwTQbEgBAAJ that claims that they are allowed.

Ryan, could you please provide appropriate reference? It seems that the have stated this understanding previously and nobody claimed it to be wrong.

Flags: needinfo?(ryan.sleevi)

(In reply to Jürgen Brauckmann from comment #7)

We have been following the discussion on m.d.s.p. and the CA/Browser forum for years, and I don't remember and cannot find any discussion
about multi CNs. Is this something that was discussed in the EV guideline process?

Since you brought up multi-valued RDNs, here's the language from the BRs, v1.0, in which one field is allowed to be multi-valued, but another is not.

9.2.2 Issuer Common Name Field
Certificate Field: subject:commonName (OID 2.5.4.3)
Required/Optional: Optional
Contents: If present in a Certificate, the Common Name field MUST include a name that accurately identifies the Issuing CA.

9.2.3 Issuer Domain Component Field
Certificate Field: subject:domainComponent (OID 0.9.2342.19200300.100.1.25)
Required/Optional: Optional.
Contents: If present in a Certificate, the Domain Component field MUST include all components of the Issuing CA’s Registered Domain Name in ordered sequence, with the most significant component, closest to the root of the namespace, written last

Using the the logic of the answer given here, it would seem DFN is arguing that this would be "DC=foobarexample" for "foo.bar.example"

This language was discussed and introduced as part of Draft 48 of the Baseline Requirements, circulated by Ben Wilson (then at DigiCert) on 2011-10-27, based on further language that had been circulated by Wayne Thayer (then at GoDaddy). This understanding further held during the discussion of Ballot 92, on 2012-10-25. This related language was revised during the Ballot 102 discussion, before ultimately being removed in Ballot 148, along with the contemporaneous discussion.

At no point during any of these discussions, from the very first version, was the possible interpretation here held as valid. This was further discussed in the context of BR issues 15 and 29, circa 2012-09, around the handling of IDNs and gTLDs, in which the discussion has been that in a single certificate with both an IP address and a domain name, you can have either the IP or the domain name, but not both.

This was similarly discussed, in 2011-11.

Also note that X.520 and RFC 4519 lists common name as multi valued. Multi valued attributes are of course different from multi occurences of
the attribute, but nevertheless, all over X.520 and LDAP a DN may have more than one CN value. Also RFC5280 does not limit this in any way.

Within X.520 and RFC 4519, names are hierarchal in naming scope, and the context of any RDN is determined by its parent nodes. As you note, this is distinct from multi-valued attributes, in which all names are seen as interchangable. However, that's exactly the point here: a naming hierarchy of "foo.example in the context of bar.example" is directly at odds with DNS and ICP-3, while the context of "This name of this entity is interchangably foo.example or bar.example" (multi-valued), is both clear and explicitly forbidden.

Put differently, a name "CN=foo.example, CN=bar.example" (multiple occurrences, hierarchal) clearly doesn't make sense with (L)DAP naming, while a name "CN=foo.example+CN=bar.example" (multiple values, same level) is we seem to agree is forbidden (or, perhaps we don't agree?)

We are still trying to confirm if this is an incident and if we need to revoke the existing certs.

Yes.

I get that the argument here is "But we didn't know", and "we were given bad advice when this was previously raised", and I can even appreciate the argument "We didn't realize because we don't follow not-yet-adopted proposals to clarify language" and "We didn't realize our understanding was wrong, and so never thought to ask the question". But it's never been allowed, and had you gone through the historic evaluation - even just the BR versions to bisect - it would have been clear what was meant, as well as how/when it's changed.

Will certificate profiles help with this? Hopefully, yes. But there was never any accident in specifying a single instance as an intentional decisions, from the very beginning.

Flags: needinfo?(ryan.sleevi)

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in the MDSP mailing list, a Bugzilla bug, or internal self-audit), and the time and date.

This bug report was created on 2021-04-16 17:48 UTC. We became aware of it on the next workday, 2021-04-19 (approx 07:00 UTC).

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

2014-01-13 First version of internal specification document regarding DN attributes (see under 6. below). Multiple CNs were internally explicitely viewed as conforming to relevant requirements. All revisions to this internal spec (at least yearly) did not discover non-compliance of multiple CNs.
2017-10-29/30 Short discussion on m.s.d.p regarding multiple CNs https://groups.google.com/g/mozilla.dev.security.policy/c/HcIbeFRiyOU/m/APwTQbEgBAAJ Internal discussion concluded that our specification was still correct.

2021-04-16 17:48 UTC This bug report was created (no notification to our Problem Reporting Mechanism was made)
2021-04-19 approx 07:00 UTC got aware of this bug report
2021-04-19 10:00 UTC call with Deutsche Telekom Security
2021-04-19 11:45 UTC to 19:45 UTC discussion in this bug

2021-04-20 05:30 UTC internal call; handle as incident, commission system change to prevent multiple CNs.
2021-04-20 11:00 UTC system change finished.

3. Whether your CA has stopped, or has not yet stopped, certificate issuance or the process giving rise to the problem or incident. A statement that you have stopped will be considered a pledge to the community; a statement that you have not stopped requires an explanation.

We updated our issuance system at 2021-04-20, 11:00 UTC to prevent issuance of double CNs.

4. In a case involving certificates, a summary of the problematic certificates. For each problem: the number of certificates, and the date the first and last certificates with that problem were issued. In other incidents that do not involve enumerating the affected certificates (e.g. OCSP failures, audit findings, delayed responses, etc.), please provide other similar statistics, aggregates, and a summary for each type of problem identified. This will help us measure the severity of each problem.

As of 2021-04-20, there were 355 valid server certificates with more than one CN attribute in its DN and 1 valid user certificate with the same problem.

5. In a case involving certificates, the complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem. In other cases not involving a review of affected certificates, please provide other similar, relevant specifics, if any.

355 server certificates:
For now, I'll attach a list of serial numbers. I will post a list of crt.sh ids hopefully tomorrow.

The single user certificate is not disclosed/published. It's data is:
SHA256 Fingerprint=38:29:78:B1:C6:18:9C:FF:D0:DC:3C:4B:CC:A5:AA:B3:76:4F:5D:C4:A7:50:BD:BD:01:71:48:40:00:99:BA:63
notBefore=Jan 17 14:27:23 2019 GMT
notAfter=Jan 16 14:27:23 2022 GMT
serial=20600E57A2E1D840599BD72B
issuer=C = DE, O = Verein zur Foerderung eines Deutschen Forschungsnetzes e. V., OU = DFN-PKI, CN = DFN-Verein Global Issuing CA

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

Since ca. 2007, we documented the rules how our system shall handle different DN attributes (mandatory, optional, forbidden, syntax, cardinality) as part of our general software requirements.

At the end of 2013 we created a separate DN attribute specification to facilitate thorough compliance checking. This specification was created by compiling relevant regulations, cross-checking with the existing rules in our system, and adjusting our rules where necessary. Multiple CNs where considered compliant with all relevant regulations when this separate specification was created. All reviews of this specification since then (at least yearly) did not view/detect multiple CNs as non-compliant.

When this question came up in 2017 in a mail thread in m.s.d.p., our interpretation of rules governing multiple CNs seemed to be confirmed. https://groups.google.com/g/mozilla.dev.security.policy/c/HcIbeFRiyOU/m/APwTQbEgBAAJ

In 2019 we became aware that linters are issuing warnings (not errors) regarding multiple CN. We took this as further reinforcement (not as proof), that this is not a compliance issue, but a practice that should be avoided if possible.

We planned to eliminate issuing certificates with multiple CNs in the course of 2021, already before this bug report was filed.

7. List of steps your CA is taking to resolve the situation and ensure that such situation or incident will not be repeated in the future, accompanied with a binding timeline of when your CA expects to accomplish each of these remediation steps.

First steps:

  • Our issuance system was changed today at 2021-04-20 11:00 UTC so that issuance with multiple CNs is no longer possible.
  • Affected certificates will be revoked by 2021-04-24 07:00 UTC at the latest (5 days after we became aware of this incident. Please note that our Problem Reporting Mechanism was not used).

Further necessary steps to prevent repetition will be described until 2021-04-23.

Attached file hex_serials —

Hello,
Thanks for this detailed incident report.
Could you please tell me if you have discussed this issue with your auditors? If yes, could you please share their answer if possible?

Michel, we did not discuss with our auditors during this incident handling. This issue will of course be disclosed to them.

Attached file crtsh_ids — 
 7. List of steps your CA is taking to resolve the situation and ensure that such situation or incident will not be repeated in the future, accompanied with a binding timeline of when your CA expects to accomplish each of these remediation steps.

First steps:

- Our issuance system was changed today at 2021-04-20 11:00 UTC so that issuance with multiple CNs is no longer possible.
- Affected certificates will be revoked by 2021-04-24 07:00 UTC at the latest (5 days after we became aware of this incident. Please note that our 
Problem Reporting Mechanism was not used).  

Further necessary steps to prevent repetition will be described until 2021-04-23.

We identified two issues as root cause of this incident:

First root cause is a misunderstanding on our side of not explicitely stated definitions. Those definitions seem to be obvious for people working on the EVG and BR 10 years ago. Main point where we failed: The word "field" in the BRs and EVGs describes the union of all values of all attributes with that OID instead of a single attribute. This was not clear to us.

We agree that it can be easily concluded what is meant when revisiting the changes between Version 1.3 and 1.4 of the EVG (introduced in 2012):

EVG 1.3:
This field MUST contain one or more host Domain Name(s)...

EVG 1.4:
If present, this field MUST contain a single Domain Name(s)...

We don't issue EV certificates, so we are not that familiar with the EVGs and discussions surrounding changes.

We think that without knowledge of this change, just by interpreting the current language alone, it's still possible to err like we did. We did
revisit our interpretation on several occasions, and always came to the conclusion that multiple CNs are unusual, but are in line with the BRs and other relevant standards.

We are following current CA/B-Forum discussions, but we don't see that it's reasonable to expect crystal clear knowledge of +5 year old
discussions or old changes to guidelines. In addition, there already was a limited public discussion in 2017 about this issue, where a core member of CA/B-Forum contributed in a way that supported our view.

To conclude, we don't think that there is something systematically that we could change or improve to prevent this kind of misjudgement, as we
already executed a lot of care to this issue.

Second root cause:
Three facts:

  1. This issue was discussed with us back in 2017
  2. Current linters show warnings for multiple CNs
  3. We decided some time ago to stop this practice, but did not implement
    this decision in a tight schedule.

So, we were aware of this issue as some kind of "irregularity" or "anomaly" for some time. We should have taken the hints that multiple
CNs are an unwanted practice much earlier (also considering SHOULD NOT from RFC 6125), and pursuit a plan to stop issuing certs with this
property in a more aggressive way. We should have done this even in light of our (errornous) assessment that multiple CNs are conformant.

We will update our rules for assessing anomalies. The goal of the update is: If we detect an anomaly and find it conformant to standards, we should give much more weight to nevertheless eliminate the anomaly than to accept it.

We will update our rules for assessing anomalies by 2021-04-30.

Following this discussion, I was thinking if it would be possible to define an ABNF grammar of a TLS certificate that satisfies all formal rules to be a valid certificate according to Mozilla's root store policy (practically the CA/B guidelines). In the discussion above I saw a reference to upcoming "certificate profiles" but I am not sure how they are planned to look like.

(In reply to Rufus Buschart from comment #17)

Following this discussion, I was thinking if it would be possible to define an ABNF grammar of a TLS certificate that satisfies all formal rules to be a valid certificate according to Mozilla's root store policy (practically the CA/B guidelines). In the discussion above I saw a reference to upcoming "certificate profiles" but I am not sure how they are planned to look like.

ABNF would not be suitable for this.

However, https://github.com/sleevi/cabforum-docs/pull/36 is where this work is happening (to be replaced with a CA/B Forum PR this week, now that it's matured), as discussed on the validation@cabforum.org list at https://archive.cabforum.org/pipermail/validation/2021-March/001645.html

A step to take would be subscribing to the validation@cabforum.org list, even in a read-only capacity, as that's where significant discussion is happening these days.

(In reply to Ryan Sleevi from comment #18)

(In reply to Rufus Buschart from comment #17)

Following this discussion, I was thinking if it would be possible to define an ABNF grammar of a TLS certificate that satisfies all formal rules to be a valid certificate according to Mozilla's root store policy (practically the CA/B guidelines). In the discussion above I saw a reference to upcoming "certificate profiles" but I am not sure how they are planned to look like.

ABNF would not be suitable for this.

However, https://github.com/sleevi/cabforum-docs/pull/36 is where this work is happening (to be replaced with a CA/B Forum PR this week, now that it's matured), as discussed on the validation@cabforum.org list at https://archive.cabforum.org/pipermail/validation/2021-March/001645.html

A step to take would be subscribing to the validation@cabforum.org list, even in a read-only capacity, as that's where significant discussion is happening these days.

Thank you for the link to the GitHub PR! I didn't follow the Validation SC mailing list, but I'll subscribe it now.

All affected certificates are revoked by now.

(In reply to Ryan Sleevi from comment #1)

Arnold: Could you please provide an incident response? This CA is disclosed by being operated by Deutsche Telekom Security, despite linking to DFN-Verein's audit / chaining through https://crt.sh/?id=23908438&opt=zlint

(In reply to Arnold Essing from comment #3)

Dear Ryan, thank you for the notification. We assume that you refer to the "Owner -> This CA"-entry in https://crt.sh/?id=25484751 which, according to our understanding, is wrong. The actual operator of that CA is the DFN-Verein and Jürgen Brauckmann will take over the further handling of this bug.
We are looking into why "Deutsche Telekom Security GmbH" is disclosed as the operator in crt.sh. In our understanding, the corresponding information in CCADB should be correct, if that is the source of the information.

Hi Ryan. Arnold asked me to take a look at this.

TIL that, in the CCADB web interface, the "Subordinate CA Owner" tooltip says:
"Leave blank if BOTH control of the private key AND domain/IP/email validation activities are performed by the organization listed in the audit statement of the parent certificate."

crt.sh was (incorrectly) interpreting a blank "Subordinate CA Owner" field to mean that the CA Owner of the root certificate was the effective CA Owner. This is fixed by https://github.com/crtsh/certwatch_db/commit/4845f7dc4eb2c88fb0c6a6c99ac9fee911fee682.

Thanks Rob! And thanks Arnold for the follow-up and routing here :)

We have updated our rules for assessing anomalies to give more weight to the elimination of anomalies than to their acceptance.

We have completed the handling of this incident.

Are there any further questions/comments/recommendations?

Jürgen: Apologies for missing Comment #23. Note that, per https://wiki.mozilla.org/CA/Responding_To_An_Incident#Keeping_Us_Informed

You should also provide updates at least every week giving your progress, and confirm when the remediation steps have been completed - unless Mozilla representatives agree to a different schedule by setting a “Next Update” date in the “Whiteboard” field of the bug. Such updates should be posted to the MDSP mailing list, if there is one, and the Bugzilla bug. The bug will be closed when remediation is completed.

I realize that you've had your question lingering since Comment #23, but providing the weekly progress report - on all bugs - is the expectation, until they're closed out. They also help bump them up in the priority queue, since naturally, we want to pay close attention to recent bugs to make sure appropriate mitigations are in place.

One thing that stands out from Comment #11:

2021-04-16 17:48 UTC This bug report was created (no notification to our Problem Reporting Mechanism was made)
2021-04-19 approx 07:00 UTC got aware of this bug report

Ultimately, I realize that the present (BR) expectation is for CAs to be monitoring their problem reporting alias, and that the three-day delay was due, in part, to being misrouted (per Comment #3). But it seems there's an opportunity here to explore whether there are improvements that could be made to the Bugzilla problem monitoring (for both Telekom Security and DFN) that could improve response times.

For example, other CAs have adopted a process of monitoring all bugs (which can be done by subscribing to the CA Certificate Compliance component of NSS), and ensuring their NOC reviews all new bugs within 24 hours to examine if there is potential impact to the CA, as well as to monitor for improvements that are worth considering from other CAs' bugs, even if the immediate issue does not manifest for the CA. Could you describe a bit of the process in place for DFN? And similarly, Arnold, can you describe the process for Telekom Security?

Flags: needinfo?(brauckmann)
Flags: needinfo?(Arnold.Essing)

Hello Ryan, the process of Telekom Security is quite similar to what you described and also includes said subscription as well as documentation of the results. Every workday the compliance team checks for new bugs which also includes a first assessment whether there is potential impact to our CA or whether it is possible to derive other improvements. Depending on the results of the first assessment, a more detailed evaluation then follows in a timely manner and to the best of our knowledge. Bugs addressed to Telekom Security are obviously evaluated as soon as possible. However, we would like to emphasize that this process is limited to workdays while the bug was opened at 17:48 UTC / 19:48 CET on a Friday (2021-04-16). In our opinion, a reaction the very next workday is in fair time, but as always, we are open for discussion and good ideas.

Flags: needinfo?(Arnold.Essing)

DFN is subscribed to the CA Certificate Compliance component of this bugzilla. On workdays one member of the compliance team is reviewing new bugs until EOB. Bugs affecting us are handled immediately; bugs mentioning Telekom Security are checked with extra care if they affect our Sub-CA/our operations.

Topics affecting other CAs are reviewed for possible improvements for our own operations. Those with more "interesting" problems are followed closely to keep up with the state of the art.

Flags: needinfo?(brauckmann)

What, if any, are the remaining action items / remediation steps for this incident?

Flags: needinfo?(brauckmann)

I tried to express (and probably failed due to clumsy wording, sorry) in Comment #23 that we don't have any remaining action items or remediation steps. Which is also the reason why we have not given any further updates.

From our point of view, there is nothing left to do to handle this incident.

Flags: needinfo?(brauckmann)

I will schedule this bug for closure on Friday 4-June-2021.

Flags: needinfo?(bwilson)
Status: ASSIGNED → RESOLVED
Closed: 3 years ago
Flags: needinfo?(bwilson)
Resolution: --- → FIXED
Product: NSS → CA Program
Whiteboard: [ca-compliance] → [ca-compliance] [ov-misissuance]
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: