Closed Bug 1390981 Opened 7 years ago Closed 6 years ago

Comodo: Non-BR-Compliant Certificate Issuance

Categories

(CA Program :: CA Certificate Compliance, task)

task
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: kwilson, Assigned: rob)

References

Details

(Whiteboard: [ca-compliance] [ev-misissuance] [ov-misissuance] [dv-misissuance] [disclosure-failure])

The following problems have been found in certificates issued by your CA, and reported in the mozilla.dev.security.policy forum. Direct links to those discussions are provided for your convenience.

To continue inclusion of your CA’s root certificates in Mozilla’s Root Store, you must respond in this bug to provide the following information:
1) How your CA first became aware of the problems listed below (e.g. via a Problem Report, via the discussion in mozilla.dev.security.policy, or via this Bugzilla Bug), and the date.
2) Prompt confirmation that your CA has stopped issuing TLS/SSL certificates with the problems listed below.
3) Complete list of certificates that your CA finds with each of the listed issues during the remediation process. The recommended way to handle this is to ensure each certificate is logged to CT and then attach a CSV file/spreadsheet of the fingerprints or crt.sh IDs, with one list per distinct problem.
4) Summary of the problematic certificates. For each problem listed below: number of certs, date first and last certs with that problem were issued.
5) Explanation about how and why the mistakes were made, and not caught and fixed earlier.
6) List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.
7) Regular updates to confirm when those steps have been completed.

Note Section 4.9.1.1 of the CA/Browser Forum’s Baseline Requirements, which states:
“The CA SHALL revoke a Certificate within 24 hours if one or more of the following occurs: …
9. The CA is made aware that the Certificate was not issued in accordance with these Requirements or the CA’s Certificate Policy or Certification Practice Statement; 
10. The CA determines that any of the information appearing in the Certificate is inaccurate or misleading; …
14. Revocation is required by the CA’s Certificate Policy and/or Certification Practice Statement; or 
15. The technical content or format of the Certificate presents an unacceptable risk to Application Software Suppliers or Relying Parties (e.g. the CA/Browser Forum might determine that a deprecated cryptographic/signature algorithm or key size presents an unacceptable risk and that such Certificates should be revoked and replaced by CAs within a given period of time).

However, it is not our intent to introduce additional problems by forcing the immediate revocation of certificates that are not BR compliant when they do not pose an urgent security concern. Therefore, we request that your CA perform careful analysis of the situation. If there is justification to not revoke the problematic certificates, then explain those reasons and provide a timeline for when the bulks of the certificates will expire or be revoked/replaced. 

We expect that your forthcoming audit statements will indicate the findings of these problems. If your CA will not be revoking the certificates within 24 hours in accordance with the BRs, then that will also need to be listed as a finding in your CA’s BR audit statement.

We expect that your CA will work with your auditor (and supervisory body, as appropriate) and the Root Store(s) that your CA participates in to ensure your analysis of the risk and plan of remediation is acceptable. If your CA will not be revoking the problematic certificates as required by the BRs, then we recommend that you also contact the other root programs that your CA participates in to acknowledge this non-compliance and discuss what expectations their Root Programs have with respect to these certificates.


The problems reported for your CA in the mozilla.dev.security.policy forum are as follows:

** Failure to respond within 24 hours after Problem Report submitted
https://groups.google.com/d/msg/mozilla.dev.security.policy/PrsDfS8AMEk/w2AMK81jAQAJ
The problems were reported via your CA’s Problem Reporting Mechanism as listed here:
https://ccadb-public.secure.force.com/mozilla/CAInformationReport
Therefore, if this is the first time you have received notice of the problem(s) listed below, please review and fix your CA’s Problem Reporting Mechanism to ensure that it will work the next time someone reports a problem like this.

** Certificates with metadata-only subject fields (at least one subject field that only contains ASCII punctuation characters)
https://groups.google.com/d/msg/mozilla.dev.security.policy/Sae5lpT02Ng/-lsC11JnBwAJ
Prevent further issuance of certs with N/A and other metadata but revocation not necessary in this case. 

** Invalid dnsNames (e.g. invalid characters, internal names, and wildcards in the wrong position)
https://groups.google.com/d/msg/mozilla.dev.security.policy/CfyeeybBz9c/lmmUT4x2CAAJ
https://groups.google.com/d/msg/mozilla.dev.security.policy/D0poUHqiYMw/Pf5p0kB7CAAJ
Issue #1
Failure to respond within 24 hours from problem report.
https://groups.google.com/d/msg/mozilla.dev.security.policy/PrsDfS8AMEk/w2AMK81jAQAJ
I understand this to be referring to the problem reports that follow rather than being an issue in its own right, so I will address the timeline of each issue in the response for that issue.

The BRs give use that 24 hour figure, but I respectfully suggest that it is nearer to 
'Failure to revoke within 24 hours from receiving and verifying a problem report'

Times that I post here will be in BST (UK time, UTC+1, Eastern - 5) unless otherwise specified.

I will respond to each issue in a separate issue comment in case the separation proves useful.
Issue #2
https://groups.google.com/forum/#!searchin/mozilla.dev.security.policy/Certificates$20with$20common$20names$20not$20present$20in$20their$20SANs|sort:relevance/mozilla.dev.security.policy/K3sk5ZMv2DE/4oVzlN1xBgAJ

On Sat 05/08/2017 15:28
Alex Gaynor reported to Comodo's sslabuse@comodo.com email address that the following certificates
- https://crt.sh/?id=14729289&opt=cablint
- https://crt.sh/?id=13354292&opt=cablint
- https://crt.sh/?id=12859256&opt=cablint
- https://crt.sh/?id=13941102&opt=cablint
- https://crt.sh/?id=15882340&opt=cablint
"violate the CABF BR (7.1.4.2.2) that the common name must be included in the Subject Alternative Names.
The CABForum guidelines require misissued certs to be revoked within 24 hours."

These certificates are issued from a name-constrained CA operated by Intel Corporation.
Comodo acknowledges its responsibility for this CA’s compliance with Mozilla’s CA policy.

It took us (Comodo) a few days to forward this report to Intel.
Although the report was circulated within Comodo we were not able to revoke these end entity certificates since we do not operate that CA.  
Those within Comodo who would otherwise have revoked the end entity certificates did not initially feed back that the certificates needed external action.

We apologize for the delay in forwarding this report to the CA for action.

Intel revoked the reported certificates by Thu 10/08/2017 17:36

They confirmed that the issue which allowed these certificates to be created had already been corrected so that no further such certificates would be issued.
They are reviewing the body of previously issued certificates to find and revoke and replace any other end entity certificates that exhibit this problem.
We will report back on this bug within 7 days of posting this message to provide a complete list of certificates exhibiting this problem and their revocation status.

We responded to Alex at Thu 10/08/2017 22:27
Issue #3
Certificates with metadata-only subject fields (at least one subject field that only contains ASCII punctuation characters)
https://groups.google.com/d/msg/mozilla.dev.security.policy/Sae5lpT02Ng/-lsC11JnBwAJ
Prevent further issuance of certs with N/A and other metadata but revocation not necessary in this case.

The email to mdsp is dated 9th August.
We became aware of it on 10th August.

The initial report, as linked above, reasonably quotes from the BRs and makes its statement that the certificates listed have at least one subject field that only contains ASCII punctuation characters.

We followed the discussion on the list with interest.

We have subsequently identified shortcomings in our subject attribute checking code which permitted certificate requests to proceed through validation and on to issuance with an OU field containing only meta characters.
Although the prior checking code did identify and stop or correct a number of cases where inappropriate meta characters were present, it did not catch the case where only meta characters were present. 
Patches are being applied to our system today to prevent any further such issuance.

We will append to this bug a full list of the problematic certificates with this issue, and a summary thereof, within 24 hours of this post.
Issue #4
Invalid dnsNames (e.g. invalid characters, internal names, and wildcards in the wrong position)
https://groups.google.com/d/msg/mozilla.dev.security.policy/CfyeeybBz9c/lmmUT4x2CAAJ

(I can’t relate this 2nd linked email to Comodo’s certificates
https://groups.google.com/d/msg/mozilla.dev.security.policy/D0poUHqiYMw/Pf5p0kB7CAAJ
)

Jonathan posted to mdsp on 13th August.
We became aware of the post on 14th August.

The post refers to
Comodo 
    Intel External Basic Issuing CA 3B (1) 
and https://misissued.com/batch/8/ linked to https://crt.sh/?id=8884169&opt=cablint

This certificate contains a wildcard dns name where it is not the case that the wildcard appears as the first character of the dns name and is immediately followed by a period.

Intel revoked this certificate on Tue 15/08/2017
I will add times of day when I have them.

We are verifying that no further malformed wildcard certificates could be issued from this name-constrained CA, and will provide a complete list of nonconforming certificates within 24 hours.  At the time of writing only the originally reported certificate is known to exhibit this problem.

We note that that the failure of the CABF's Ballot 202 https://cabforum.org/pipermail/public/2017-July/011749.html has made this issue less well defined than it might otherwise be.
(In reply to Robin Alden from comment #2)
> Issue #2
> https://groups.google.com/forum/#!searchin/mozilla.dev.security.policy/
> Certificates$20with$20common$20names$20not$20present$20in$20their$20SANs|sort
> :relevance/mozilla.dev.security.policy/K3sk5ZMv2DE/4oVzlN1xBgAJ
> 
> On Sat 05/08/2017 15:28
> Alex Gaynor reported to Comodo's sslabuse@comodo.com email address that the
> following certificates
> - https://crt.sh/?id=14729289&opt=cablint
> - https://crt.sh/?id=13354292&opt=cablint
> - https://crt.sh/?id=12859256&opt=cablint
> - https://crt.sh/?id=13941102&opt=cablint
> - https://crt.sh/?id=15882340&opt=cablint
> "violate the CABF BR (7.1.4.2.2) that the common name must be included in
> the Subject Alternative Names.
> The CABForum guidelines require misissued certs to be revoked within 24
> hours."
> 
> These certificates are issued from a name-constrained CA operated by Intel
> Corporation.
> Comodo acknowledges its responsibility for this CA’s compliance with
> Mozilla’s CA policy.
> 
> It took us (Comodo) a few days to forward this report to Intel.
> Although the report was circulated within Comodo we were not able to revoke
> these end entity certificates since we do not operate that CA.  
> Those within Comodo who would otherwise have revoked the end entity
> certificates did not initially feed back that the certificates needed
> external action.
> 
> We apologize for the delay in forwarding this report to the CA for action.
> 
> Intel revoked the reported certificates by Thu 10/08/2017 17:36
> 
> They confirmed that the issue which allowed these certificates to be created
> had already been corrected so that no further such certificates would be
> issued.
> They are reviewing the body of previously issued certificates to find and
> revoke and replace any other end entity certificates that exhibit this
> problem.
> We will report back on this bug within 7 days of posting this message to
> provide a complete list of certificates exhibiting this problem and their
> revocation status.
> 
> We responded to Alex at Thu 10/08/2017 22:27

Thanks for responding. I think it's still necessary to provide additional detail.

That is, we can look at the problem from two dimensions: The problem itself, and the systemic issues that allowed the problem to manifest. Your description focuses on the resolution of the problem, but doesn't indicate any systemic changes have been made. As a consequence, it does not help the community feel that your CA has taken steps to reduce the risk of future violations (of any requirement) in the future. That is, one dimension is "Did you fix the bug", but another dimension is "How was the bug introduced, why was it not detected, and what steps are you taking to prevent future bugs"

To understand how you might approach this problem, consider https://groups.google.com/d/msg/mozilla.dev.security.policy/vl5eq0PoJxY/W1D4oZ__BwAJ and how it provided a timeline of events, the steps that were already in place (and with substantial detail), where there were controls missing or mistakes made, details about the steps being taken (e.g. "We fixed the bug" is not sufficient detail to understand), and a holistic, systemic awareness of how the CA is managed and the opportunities for errors.


For this specific case:
---
- Why did it take a few days to forward this response to Intel?
- What steps have been taken to ensure future communications can be done in a timely fashion?
- What was the underlying issue that permitted these certificates to be issued?
- What steps have been taken to ensure that these certificates cannot be issued in the future?
- What systemic steps have been taking to ensure future certificates issued by this CA comply with your CA's CP, CPS, and the Baseline Requirements?

While it is encouraging that this is a Technically-Constrained Sub-CA, and that the risk of such mismatch is itself mitigated by RFC2818 and 6125's requirement to ignore the commonName when a dNSName SAN is present, it is important to understand the systemic failures and how they're being addressed and mitigated in the future.
(In reply to Robin Alden from comment #4)
> The post refers to
> Comodo 
>     Intel External Basic Issuing CA 3B (1) 
> and https://misissued.com/batch/8/ linked to
> https://crt.sh/?id=8884169&opt=cablint
> 
> This certificate contains a wildcard dns name where it is not the case that
> the wildcard appears as the first character of the dns name and is
> immediately followed by a period.

Note that the non-compliance reported here (from the link you provided) indicates that BR certificates must include the certificatePolicies extension, as per Section 7.1.2.3 (a) of the Baseline Requirements.

In this case, the position of the wildcard is not what's being reported as non-compliance, and we apologize for the miscommunication - although, it's worth noting, this was also indicated at the link provided.

That said, this certificate was issued on October 17, 2014. When was Comodo's stated compliance and adherence to the Baseline Requirements? When were technically constrained sub-CAs appropriately audited by Comodo for compliance and adherence, in line with Section 8.1 and 8.7 of the Baseline Requirements? Providing an explicit timeline for these events can hopefully help the community understand how Comodo has historically (and presently) oversees its relationship with Technically Constrained Sub-CAs, and what steps, if any, Comodo is taking to reduce the risk of future non-compliance from their Technically Constrained Sub-CAs.
(In reply to Ryan Sleevi from comment #6)
> (In reply to Robin Alden from comment #4)
> > The post refers to
> > Comodo 
> >     Intel External Basic Issuing CA 3B (1) 
> > and https://misissued.com/batch/8/ linked to
> > https://crt.sh/?id=8884169&opt=cablint
> > 
> > This certificate contains a wildcard dns name where it is not the case that
> > the wildcard appears as the first character of the dns name and is
> > immediately followed by a period.
> 
> Note that the non-compliance reported here (from the link you provided)
> indicates that BR certificates must include the certificatePolicies
> extension, as per Section 7.1.2.3 (a) of the Baseline Requirements.

After sampling some of the certificates issued by this intermediate, I believe most if not all are not BR-compliant due to the missing certificatePolicies extension.

Additionally, many are missing an OCSP URL.

Here's a list of the certificates issued by this intermediate that are known to CT: https://crt.sh/?Identity=%25&iCAID=1192
A certificate with a notBefore of 2017-08-16 was issued by Comodo with a stateOrProvinceName of '/'.

https://crt.sh/?id=191688041&opt=cablint
A certificate with a notBefore of 2017-08-17 was issued by "Intel External Issuing CA 6B" with a malformed wildcard.

https://crt.sh/?id=193365931&opt=cablint
I apologize for the delay in providing a further substantive reply.  We are preparing full responses to address each of these issues.  I aim to have these out today.

As an immediate step we have instructed Intel to suspend issuance of SSL certificates from their CAs that chain to our roots.
(In reply to Robin Alden from comment #2)
> Issue #2
> "the common name must be included in the Subject Alternative Names.”
> 

> 1) How your CA first became aware of the problems listed 
> below (e.g. via a Problem Report, via the discussion in 
> mozilla.dev.security.policy, or via this Bugzilla Bug), and 
> the date.
On Sat 05/Aug/17 15:28, Alex Gaynor submitted a Problem Report to Comodo's sslabuse@comodo.com address.


> 2) Prompt confirmation that your CA has stopped issuing 
> TLS/SSL certificates with the problems listed below.
The certificates with this issue were issued from a name-constrained sub-CA issued to a 3rd party.
The CA was asked to cease issuing Server Authentication certificates until additional controls were in place to prevent this and other issues.

> 3) Complete list of certificates that your CA finds with 
> each of the listed issues during the remediation process. 
> The recommended way to handle this is to ensure each 
> certificate is logged to CT and then attach a CSV 
> file/spreadsheet of the fingerprints or crt.sh IDs, with one 
> list per distinct problem.
There are two distinct sub-issues affecting these certificates.
(a) certificates with no SAN 
https://docs.google.com/spreadsheets/d/14z0N3e0jH5nB9pfZ9p1KqZsMbA8eqaYqlpmtJqDk-YE/edit?usp=sharing
(b) commonName is not present in a SAN:dnsName
https://docs.google.com/spreadsheets/d/1jiboOr2nohVEvUG1ikAG0lrK-bHSEwPtxAnNK8uSupY/edit?usp=sharing

> 4) Summary of the problematic certificates. For each problem 
> listed below: number of certs, date first and last certs with 
> that problem were issued.
There are two issues affecting these certificates.
(a) There are 233 certificates with no SAN, issued between 02/Nov/15 and 16/Aug/16 
(b) There are 457 certificates where the commonName is not present in a SAN:dnsName, issued between 02/Nov/15 and 18/Jul/17

> 5) Explanation about how and why the mistakes were made, and 
> not caught and fixed earlier.
There was an inadequate understanding of the requirements by the 3rd party CA.
Comodo's audit of the CA had not identified the issue (commonName not present in SAN:dnsNames).  
Comodo’s audit had previously identified the issue of the completely missing SAN extension and communicated it to the CA, but the remedy was not adequately prioritized. 
Due to the name constraints, Comodo did not push as hard as we could have done to force the issue to be fixed sooner.


> 6) List of steps your CA is taking to resolve the situation 
> and ensure such issuance will not be repeated in the future, 
> accompanied with a timeline of when your CA expects to 
> accomplish these things.
The 3rd party CA identified and corrected the issue with missing SAN extensions 12 months ago.
The 3rd party CA has now ceased issuing server authentication certificates and will not resume.

> 7) Regular updates to confirm when 
> those steps have been completed.
We will report progress with revocation of the affected certificates.
(In reply to Robin Alden from comment #3)
> Issue #3
> Certificates with metadata-only subject fields (at least one subject field
> that only contains ASCII punctuation characters)

> 1) How your CA first became aware of the problems listed 
> below (e.g. via a Problem Report, via the discussion in 
> mozilla.dev.security.policy, or via this Bugzilla Bug), and 
> the date.
We had been aware of some aspects of this issue for some time and had taken previous steps to addresss it.
We became aware of the remaining issue as reported here through Jonathan Rudenberg’s post to m.d.s.p on 9/Aug/17

> 2) Prompt confirmation that your CA has stopped issuing 
> TLS/SSL certificates with the problems listed below.
Confirmed.  
Our systems were patched during 17/Aug/17 to prevent the issuance of certificates with subject fields consisting entirely of ASCII punctuation characters.
We are undertaking further work to address the challenges posed by UTF-8 meta-characters.

> 3) Complete list of certificates that your CA finds with 
> each of the listed issues during the remediation process. 
> The recommended way to handle this is to ensure each 
> certificate is logged to CT and then attach a CSV 
> file/spreadsheet of the fingerprints or crt.sh IDs, with one 
> list per distinct problem.
There are two distinct classes of issue.
(a) Subject field only contains metadata
https://docs.google.com/spreadsheets/d/1FpEZTYn_nmpj01zcvXM0Dr03FasMxPO5WCrIyy_BlRE/edit?usp=sharing
(b) Trailing whitespace in OU name
https://docs.google.com/spreadsheets/d/1uJZNjIMjPKK7KKF7ZH0TRnL8sVKqI0vc6pnUrzN5eR8/edit?usp=sharing

> 4) Summary of the problematic certificates. For each problem 
> listed below: number of certs, date first and last certs with 
> that problem were issued.
There are two distinct classes of issue.
(a) There are 50 certificates where a subject field only contains metadata, issued between 26/Aug/14 and 16/Aug/17
(b) There are 2 certificates with trailing whitespace in an OU name, issued between 25/May/16 and 2/Nov/16

> 5) Explanation about how and why the mistakes were made, and 

> not caught and fixed earlier.
We had checks in place to address exactly this problem of metadata in subject fields and yet a deficiency of software engineering allowed some classes of this problem to get through to issued certificates.

> 6) List of steps your CA is taking to resolve the situation 
> and ensure such issuance will not be repeated in the future, 
> accompanied with a timeline of when your CA expects to 
> accomplish these things.
As well as the patches which went into our system during 17/Aug/17, we will take measures to incorporate a separate set of certificate policy checks into our certificate issuance flow.
Specifically we will incorporate the use of the excellent cablint and x509lint tools into our issuance pipeline such that a warning or error indication from those tools will prevent the issuance of any SSL certificate.  Any such certificate generating a warning or error will be treated as high risk and escalated for manual review.
We reserve the right to disregard specific warnings or errors from these 3rd party tools if we encounter them frequently and, after careful consideration, consider them to be extraneous.  We will of course continue to offer patches to improve the tools where we feel that the tools themselves are deficient.

We anticipate that it will take 7 days to incorporate these tools into our issuance flow.

We are working to revoke the affected certificates.

> 7) Regular updates to confirm when 
> those steps have been completed.
We will further update this issue when the tools have been incorporated into our issuance flow.
We will report progress with the revocation of the affected certificates.
(In reply to Robin Alden from comment #4)
> Issue #4
> Invalid dnsNames (e.g. invalid characters, internal names, and wildcards in
> the wrong position)
The report covered several CAs, but in Comodo’s case a better description would have been
“Invalid dnsNames (e.g. double dots, trailing hyphen in labels, and wildcards in the wrong position)”
Subsequently clarified to include:
Missing certificatePolicies extension.

> 1) How your CA first became aware of the problems listed 
> below (e.g. via a Problem Report, via the discussion in 
> mozilla.dev.security.policy, or via this Bugzilla Bug), and 
> the date.
Jonathan Rudenburg posted to mdsp on 13/Aug/17
We became aware of the post on 14/Aug/17.

> 2) Prompt confirmation that your CA has stopped issuing 
> TLS/SSL certificates with the problems listed below.
We confirm we have taken measures to prevent further issuance of certificates with these issues.
See also answer #6, below.
A majority of affected certificates with this issue were issued from a name-constrained sub-CA issued to a 3rd party.
The CA was asked to cease issuing Server Authentication certificates until additional controls were in place to prevent this and other issues.
The issue of trailing hyphens in FQDN labels, when subordinate to the registered domain, e.g. ab-.example.com, remains unaddressed.  It does not comply with the RFCs, but such names resolve in DNS and can and do provide service on the internet.

> 3) Complete list of certificates that your CA finds with 
> each of the listed issues during the remediation process. 
> The recommended way to handle this is to ensure each 
> certificate is logged to CT and then attach a CSV 
> file/spreadsheet of the fingerprints or crt.sh IDs, with one 
> list per distinct problem.
There are 2 distinct issues being reported here.
(a) Invalid dnsNames (e.g. double dots, trailing hyphen in labels, and wildcards in the wrong position).
Some ‘double dots’ in dnsNames had been previously reported and those are included in this list for completeness.
https://docs.google.com/spreadsheets/d/164bZTqfnisgGxF_f7_dPXbKFwyJq56HYHFlJtENX4NM/edit?usp=sharing
(b) Omitted certificatePolicies extension
https://docs.google.com/spreadsheets/d/17uLJOcLByHUcWrcMGQbTqb6JV37GQ7rjf6t-qnSWqfE/edit?usp=sharing

> 4) Summary of the problematic certificates. For each problem 
> listed below: number of certs, date first and last certs with 
> that problem were issued.
(a) 41 certificates with Invalid dnsNames (e.g. double dots, trailing hyphen in labels, and wildcards in the wrong position) were issued between 19/Sep/14 and 17/Aug/17.
(b) 2482 certificates with omitted certificatePolicies extension were issued between 03/Sep/14 and 10/Apr/17.

> 5) Explanation about how and why the mistakes were made, and 
> not caught and fixed earlier.
(a) certificates with Invalid dnsNames:
The double dots in sub-domains were an issue with inadequate checking of FQDN syntax when removing sub-domains to identify the Authorization Domain.
The misplaced wildcard characters were due to a name-constrained external sub-CA relying on the letter of the Baseline Requirements rather than the accepted best practice which is yet to be clarified in the BRs through a ballot.
(b) certificates with omitted certificatePolicies extension: 
Comodo identified that the name-constrained external sub-CA which issued these certificates was omitting the certificatePolicies extension.  We failed to adequately prioritize their remedy of the issue.

> 6) List of steps your CA is taking to resolve the situation 
> and ensure such issuance will not be repeated in the future, 
> accompanied with a timeline of when your CA expects to 
> accomplish these things.
(a) The 3rd party CA corrected the issue with missing certificatePolicies extensions 4 months ago.
The 3rd party CA has now ceased issuing server authentication certificates and will not resume. 
(b) For Comodo’s CA’s issues with invalid dnsNames, we will take measures to incorporate a separate set of certificate policy checks into our certificate issuance flow.
Specifically we will incorporate the use of the excellent cablint and x509lint tools into our issuance pipeline such that a warning or error indication from those tools will prevent the issuance of any SSL certificate.  Any such certificate generating a warning or error will be treated as high risk and escalated for manual review.
We reserve the right to disregard specific warnings or errors from these 3rd party tools if we encounter them frequently and, after careful consideration, consider them to be extraneous.  We will of course continue to offer patches to improve the tools where we feel that the tools themselves are deficient.

We anticipate that it will take 7 days to incorporate these tools into our issuance flow.

We will report the progress with revocation of the affected certificates.


> 7) Regular updates to confirm when 
> those steps have been completed.
Will do.
(In reply to Ryan Sleevi from comment #5)
> (In reply to Robin Alden from comment #2)
> > Issue #2
> Thanks for responding. I think it's still necessary to provide additional
> detail.
> 
...
> For this specific case:
> ---
> - Why did it take a few days to forward this response to Intel?

The report was received on a Saturday.
There are fewer senior staff at Comodo monitoring the sslabuse@comodo.net mailbox at weekends.
We did not have a pre-identified out-of-hours contact for Intel's CA abuse reporting.
There was no documented procedure in place to handle a report of CA non-compliance.  Informal processes have worked well in the past, but not this time.
On further investigation it appears that we got a report to the CA on Sunday 6th, and received an initial response late that same day.

> - What steps have been taken to ensure future communications can be done in 
> a timely fashion?

We have identified the need for us to have a documented procedure in place to handle a report of CA non-compliance whenever it is received.  
That document is being worked on now.
Although reports don't have to be handled within minutes, time is short so hours must not be lost.

A definitive contact list for external subordinate CA abuse and compliance operators will be available for those operating that new procedure.

> - What was the underlying issue that permitted these certificates to be
> issued?

I think your other questions map to Kathleen’s questions #5 and #6.  I have now provided answers to Kathleen’s questions for this issue in comment #11
(In reply to Ryan Sleevi from comment #6)
> (In reply to Robin Alden from comment #4)
> > The post refers to
> > Comodo 
> >     Intel External Basic Issuing CA 3B (1) 
> > and https://misissued.com/batch/8/ linked to
> > https://crt.sh/?id=8884169&opt=cablint
> > 
> > This certificate contains a wildcard dns name where it is not the case that
> > the wildcard appears as the first character of the dns name and is
> > immediately followed by a period.
> 
> Note that the non-compliance reported here (from the link you provided)
> indicates that BR certificates must include the certificatePolicies
> extension, as per Section 7.1.2.3 (a) of the Baseline Requirements.
> 
> In this case, the position of the wildcard is not what's being reported as
> non-compliance, and we apologize for the miscommunication - although, it's
> worth noting, this was also indicated at the link provided.
> 
> That said, this certificate was issued on October 17, 2014. 
> When was Comodo's stated compliance and adherence to the Baseline Requirements? 

Comodo's stated compliance to the BRs was prior to October 17, 2014.

> When were technically constrained sub-CAs appropriately audited by Comodo for
> compliance and adherence, in line with Section 8.1 and 8.7 of the Baseline
> Requirements? 

Comodo audited constrained sub-CAs on a quarterly basis, with a randomly selected sample of three percent of the Certificates issued by the Subordinate CA.

> Providing an explicit timeline for these events can hopefully
> help the community understand how Comodo has historically (and presently)
> oversees its relationship with Technically Constrained Sub-CAs, and what
> steps, if any, Comodo is taking to reduce the risk of future non-compliance
> from their Technically Constrained Sub-CAs.

I have now provided answers to Kathleen’s questions for this issue in comment #13.
I hope those answers more fully address your concern.
(In reply to Jonathan Rudenberg from comment #7)
> (In reply to Ryan Sleevi from comment #6)
> > (In reply to Robin Alden from comment #4)
> > > The post refers to
> > > Comodo 
> > >     Intel External Basic Issuing CA 3B (1) 
> > > and https://misissued.com/batch/8/ linked to
> > > https://crt.sh/?id=8884169&opt=cablint
> > > 
> 
> Additionally, many are missing an OCSP URL.
> 
> Here's a list of the certificates issued by this intermediate that are known
> to CT: https://crt.sh/?Identity=%25&iCAID=1192

It took us until well into 2014 to get OCSP set up for this CA.
I am not aware of any unexpired server certificates from this CA which don’t have an OCSP URI.
(In reply to Robin Alden from comment #13)
> (a) The 3rd party CA corrected the issue with missing certificatePolicies
> extensions 4 months ago.
> The 3rd party CA has now ceased issuing server authentication certificates
> and will not resume. 

Given the cessation of (TLS server) operation, would it make sense to include this subordinate CA within OneCRL, to ensure technical controls within Mozilla products to prevent the issuance of server authentication certificates?

I can imagine that issues with that are:
1) Existing certificates that are not revoked/remain valid

Does Comodo have a measure of the unexpired, unrevoked TLS server certificates issued by this subordinate to allow Mozilla to assess the impact of such a proposal?

(In reply to Robin Alden from comment #14)
> > - What steps have been taken to ensure future communications can be done in 
> > a timely fashion?
> 
> We have identified the need for us to have a documented procedure in place
> to handle a report of CA non-compliance whenever it is received.  
> That document is being worked on now.
> Although reports don't have to be handled within minutes, time is short so
> hours must not be lost.
> 
> A definitive contact list for external subordinate CA abuse and compliance
> operators will be available for those operating that new procedure.

I think this sounds like a reasonable mitigation/response to the issues presented. From evaluating issues at other CAs, one suggestion is to ensure a multi-party playbook/checklist, as is done for mission critical systems, to ensure the procedure is followed. As human error is still an unfortunate possibility with such execution, a multi-party control to ensure the procedures are followed and confirmed seems an appropriate mitigation. For example, two party confirmation that an e-mail was sent, to ensure there is no confusion about who is 'following' the playbook.

With respect to detection and prevention of issues arising from Technically-Constrained Sub-CAs, has Comodo made any steps to ensure more automated compliance - for example, by requiring that TCSCs disclose issued certificates, either publicly or to Comodo, to allow for automated controls and review? This seems like a reasonable mitigation to address some of the uncertainty around both volume and comprehensiveness of the dataset. If disclosure is done directly to Comodo, it would also be worthwhile to ensure independent confirmation of public disclosure being consistent with internal disclosure (e.g. that a certificate does not appear in a public log that has not been disclosed to Comodo)

These represent suggestions for how to more comprehensively detect and/or mitigate future issues.

(In reply to Jonathan Rudenberg from comment #8)
> A certificate with a notBefore of 2017-08-16 was issued by Comodo with a
> stateOrProvinceName of '/'.
> 
> https://crt.sh/?id=191688041&opt=cablint

Did I miss a response to this issue? It still appears to be outstanding.
(In reply to Ryan Sleevi from comment #17)
> (In reply to Robin Alden from comment #13)
> > (a) The 3rd party CA corrected the issue with missing certificatePolicies
> > extensions 4 months ago.
> > The 3rd party CA has now ceased issuing server authentication certificates
> > and will not resume. 
> 
> Given the cessation of (TLS server) operation, would it make sense to
> include this subordinate CA within OneCRL, to ensure technical controls
> within Mozilla products to prevent the issuance of server authentication
> certificates?

At some point, yes, it would.

> I can imagine that issues with that are:
> 1) Existing certificates that are not revoked/remain valid
> 
Yes, agreed.  

> Does Comodo have a measure of the unexpired, unrevoked TLS server
> certificates issued by this subordinate to allow Mozilla to assess the
> impact of such a proposal?

We do have the metrics.  
We see a couple of things going on here.
There are two  bodies of certificates which are currently unrevoked and unexpired.
There are the end entity certificates which, having been scanned by the lint tools, do not exhibit the problems identified in this bug - or any other problems that have been identified, i.e. they are OK.
The issuing organization has told us that they do not wish to replace and re-issue the entirety of these certificates if it can possibly be avoided.

There are the end entity certificates which, having been scanned by the lint tools, do exhibit problems and have been published to CT and included in the lists and summaries in this bug.
The issuing organization has told us that they are working hard to replace and re-issue end entity certificates where they can but that, particularly for the 2482 certificates with the missing certificatePolicies extension, to revoke all of those end entity certificates will represent an incredible amount of churn and will terribly impact operations across the world for them.  
We do not have a timescale by which the last of these end entity certificates can be revoked.

However, since the SSL CAs have now stopped issuing we can look ahead to the time when these end entity certificates have all expired.  Those dates are in
https://docs.google.com/spreadsheets/d/1SnQ4z1pIc5uFpf2EaxRCa0w03lAp_ykRSv92LWawd_Y/edit?usp=sharing

> 
> (In reply to Robin Alden from comment #14)
> > > - What steps have been taken to ensure future communications can be done in 
> > > a timely fashion?
> > 
> > We have identified the need for us to have a documented procedure in place
> > to handle a report of CA non-compliance whenever it is received.  
> > That document is being worked on now.
> > Although reports don't have to be handled within minutes, time is short so
> > hours must not be lost.
> > 
> > A definitive contact list for external subordinate CA abuse and compliance
> > operators will be available for those operating that new procedure.
> 
> I think this sounds like a reasonable mitigation/response to the issues
> presented. From evaluating issues at other CAs, one suggestion is to ensure
> a multi-party playbook/checklist, as is done for mission critical systems,
> to ensure the procedure is followed. As human error is still an unfortunate
> possibility with such execution, a multi-party control to ensure the
> procedures are followed and confirmed seems an appropriate mitigation. For
> example, two party confirmation that an e-mail was sent, to ensure there is
> no confusion about who is 'following' the playbook.

That is a useful thought, thanks.  

> 
> With respect to detection and prevention of issues arising from
> Technically-Constrained Sub-CAs, has Comodo made any steps to ensure more
> automated compliance - for example, by requiring that TCSCs disclose issued
> certificates, either publicly or to Comodo, to allow for automated controls
> and review? This seems like a reasonable mitigation to address some of the
> uncertainty around both volume and comprehensiveness of the dataset. If
> disclosure is done directly to Comodo, it would also be worthwhile to ensure
> independent confirmation of public disclosure being consistent with internal
> disclosure (e.g. that a certificate does not appear in a public log that has
> not been disclosed to Comodo)
> 
> These represent suggestions for how to more comprehensively detect and/or
> mitigate future issues.

Our initial response to these problem reports with this TCSC was to ask them to stop issuing until further controls were in place.
We subsequently stood up a web service which allows a TBSCertificate to be passed through x509lint and cablint.  There is some detail on that service in this post from Rob Stradling: https://groups.google.com/d/msg/mozilla.dev.security.policy/oTQ9OYgS8D4/KRFUWERDAgAJ

As a gate to resuming signing from these CAs we moved to require the TCSC to submit all of their TBSCertificates to this service before resuming signing, and also require them not to sign anything where there were issues marked WARNING, ERROR or FATAL.  We had in mind a non-automated appeal process for those occasional false positives from the automated lint checks.
We plan to require this same automated compliance test of all external TCSCs.
As mentioned above, in this case they have elected not to resume signing from these CA certificates.

> 
> (In reply to Jonathan Rudenberg from comment #8)
> > A certificate with a notBefore of 2017-08-16 was issued by Comodo with a
> > stateOrProvinceName of '/'.
> > 
> > https://crt.sh/?id=191688041&opt=cablint
> 
> Did I miss a response to this issue? It still appears to be outstanding.

That certificate was the last offending one to be issued before we patched our systems to prevent certificates with meta-data-only subject fields.
I did respond, in that it is the first certificate appearing in the first list I provided in comment #12, i.e. https://docs.google.com/spreadsheets/d/1FpEZTYn_nmpj01zcvXM0Dr03FasMxPO5WCrIyy_BlRE/edit?usp=sharing
but I should have specifically called that out in response to comment #8.
Below is my summary of all issues and remediation plans:

1) Certificates with metadata-only subject fields
- See Comment #0, Comment #3, Comment #8, Comment #12, Comment #18
- Existing Mitigations
  - Had some software controls related to metadata, but they were incomplete (See Comment #12)
- Remediation
  - 2017-08-17 - Hotfix deployed to existing metadata controls to address the identified issues
  - 2017-08-28 - Integrating pre-issuance checks of certlint and x509lint (with manual overrides) to prevent issuance

2) Missing certificate policies
- See Comment #0, Comment #6, Comment #7, Comment #10, Comment #13, Comment #15
- Remediation
  - 2017-08-21 - Technically Constrained Subordinate CA has been contractually requested to cease issuance (See Comment #10)

3) Invalid dNSNames (wildcards and hyphens)
- See Comment #0, Comment #4, Comment #9, Comment #10, Comment #13, Comment #15
- Remediation
  - 2017-08-21 - Technically Constrained Subordinate CA has been contractually requested to cease issuance (See Comment #10)

4) Common Name missing from SAN / SAN missing
- See Comment #2, Comment #5, Comment #11, Comment #14
- Remediation
  - 2016-08-XX - Missing SAN was identified and remediated (see Comment #11)
  - 2017-08-21 - Technically Constrained Subordinate CA has been contractually requested to cease issuance (See Comment #10)

In addition to these specific issues, Comodo is developing an incident response playbook for both issues with its CA and for its subordinate CAs, with no ETA provided for its review and completion (See Comment #14)

Is this a correct summary?

I think there's still some questions related to the proposed remediation and whether or not they sufficiently address the underlying or systemic causes. That's not to suggest the remediations don't address the surface cause, but whether or not there are additional systemic issues is unclear.

- Comment #8 and Comment #12 identified an example of an EV certificate that was issued with a '/' in a stateOrProvinceName. Unlike other types of certificates, EV is required to undergo a "four-eyes" review system to ensure compliance. The identified certificate did have a jurisdictionLocalityName. As the address is Ljubljana, in the City Municipality of Ljubljana, it's unclear why this wasn't identified during either review, or what steps have been taken here to ensure proper review and consistency (e.g. setting aside the technical control section)
- Comment #15 indicated quarterly audits had been performed for its TCSCs, but apparently had missed a number of the non-conformities (see Comment #6 and Comment #7). While the technical detection tools in Comment #18 provide useful robustness checks, I feel it does raise the question as to what exactly was being audited/examined and why these issues weren't detected sooner (particularly given that some issues - such as missing SANs - were identified, as evidenced by Comment #11). It feels like there's still analysis necessary as to the root causes of these failures, as otherwise, the mitigation may entirely rest on technical controls and thus fail to detect issues of a non-technical nature or which the tools do not yet address (for example, improper or invalid OCSP responses, as a hypothetical)
Flags: needinfo?(robin)
Robin: do you have comments on Ryan's most recent set of questions, posted 20 days ago now?

Gerv
Ryan, Gerv,

I know Robin was working on this last week.  He's away all this week but I expect he will post his reply early next week.  Sorry for the delay.
Robin and Rob: it has been a further 16 days. Please provide an update ASAP.

Gerv
Flags: needinfo?(rob)
(In reply to Gerv from comment #22)
Sorry it took me so long to respond.

(In reply to Ryan Sleevi from comment #19)
> Below is my summary of all issues and remediation plans:
> 
> 1) Certificates with metadata-only subject fields
> - See Comment #0, Comment #3, Comment #8, Comment #12, Comment #18
> - Existing Mitigations
>   - Had some software controls related to metadata, but they were incomplete

These checks are heuristic in nature and hence are always incomplete, but we accept the point that there was more we could have done had we thought of it.

> (See Comment #12)
> - Remediation
>   - 2017-08-17 - Hotfix deployed to existing metadata controls to address
> the identified issues
>   - 2017-08-28 - Integrating pre-issuance checks of certlint and x509lint
> (with manual overrides) to prevent issuance

We missed our target date of 2017-08-28 and it was actually 2017-09-06 before pre-issuance checks of certlint and x509lint were integrated into our issuance pipeline.
Rob got side-tracked into also integrating zlint, hence the overrun :-)

We have ended up without a manual per-certificate override for the lint checks, but we do have a short list of lint check 'reason' codes which we consider to be false positives and which we therefore ignore.
Currently the only reason that we consider to be a false positive is:
'[x509lint] ERROR: Name entry contains an invalid type'
and the reason we need to FP that is that for certain customers we add in an unstructuredName attribute to the certificate subject.  The content of that field is identical to the commonName.

<snip>
>
> Is this a correct summary?

With the above slight amendment, yes.

> I think there's still some questions related to the proposed remediation and
> whether or not they sufficiently address the underlying or systemic causes.
> That's not to suggest the remediations don't address the surface cause, but
> whether or not there are additional systemic issues is unclear.
> 
> - Comment #8 and Comment #12 identified an example of an EV certificate that
> was issued with a '/' in a stateOrProvinceName. Unlike other types of
> certificates, EV is required to undergo a "four-eyes" review system to
> ensure compliance. The identified certificate did have a
> jurisdictionLocalityName. As the address is Ljubljana, in the City
> Municipality of Ljubljana, it's unclear why this wasn't identified during
> either review, or what steps have been taken here to ensure proper review
> and consistency (e.g. setting aside the technical control section)

We believe that the technical control, i.e. linting checks, are a vital step to address these.

I think the particular challenge here is that the trained eyes are looking to compare information in one place (the certificate request) with information in another place; in this case the registration details of the subject organization.
That's what they spend their days doing.
They failed to do what we hoped here, in my opinion, because the field in the certificate request did not contain information of a form they expected.
Should they have rejected it rather than decided it was a match? - Yes, definitely, and we have asked them to positively identify and reject cases where there is nonsense in a field.
I think this issue was caused by the presence of metadata in a place where human validators were expecting data and I think we can best help them to do their job to the highest standard possible by removing the metadata.

I'm not trying to detract further from the significance of the resultant issuance, but there are a number of variations of meta data which could have been included, and we have seen included in other certificate requests, which our human validators could not have spotted such as leading or trailing whitespace or the inclusion of non-printable characters in fields.

> - Comment #15 indicated quarterly audits had been performed for its TCSCs,
> but apparently had missed a number of the non-conformities (see Comment #6
> and Comment #7). 

a) Wildcard DNS name where the wildcard character is in the 'wrong' position.
This issue was not present in the audit sample.
The particular malformation of having a single '*' in the first label but not as the first and only character is arguably a valid interpretation of the words in the BRs, although had we spotted it we would have asked Intel to change behaviour.

b) missing certificatePolicies extension
This was spotted at audit.
We raised the issue with the CA.
We failed to adequately prioritize their remedy of the issue.
The CA corrected the issue in April 2017.
The CA did not immediately begin to revoke the affected end entity certificates.  

c) missing an OCSP URL
This was spotted at audit.
We implemented interfaces between Comodo's CA systems and this CA's systems so that Comodo was able to generate and disseminate OCSP responses for the leaf certificates issued by this CA.
This system was activated and OCSP URLs were included in their leaf certificates from 24th July 2014.
This missed the target date for supporting OCSP for leaf certificates issued through external name-constrained CAs according to Mozilla's policy and Wiki, which was 15th May 2014.

> While the technical detection tools in Comment #18 provide
> useful robustness checks, I feel it does raise the question as to what
> exactly was being audited/examined and why these issues weren't detected
> sooner (particularly given that some issues - such as missing SANs - were
> identified, as evidenced by Comment #11). 

d) Missing SAN extension
This was spotted at audit.
The CA introduced it as a new bug that we spotted in Jan 2016.
We failed to report this to the CA for remediation due to an internal communication breakdown.

> It feels like there's still
> analysis necessary as to the root causes of these failures, as otherwise,
> the mitigation may entirely rest on technical controls and thus fail to
> detect issues of a non-technical nature or which the tools do not yet
> address (for example, improper or invalid OCSP responses, as a hypothetical)

OCSP - see (c) above.

Our detection rate was actually pretty good, but our enforcement record unfortunately was not.

Had this been an unconstrained external sub-CA we would have been stricter over the prioritization of actions required of them.  With the benefit of hindsight, our more relaxed attitude to this NC sub-CA was ill-advised.

Actions for Comodo:
For future audits of NC sub-CAs We will formalize and improve the recording and reporting of audit findings to the operator of the NC sub-CA.
We will record and more precisely communicate to the operator of the NC sub-CA our expectation of them in regard to any audit findings.
We will insist on prompt remediation of any issues that have led to audit findings.
We will insist on prompt revocation of any non-compliant certificates as per 4.9.1.1 of the BRs.
We will terminate and/or revoke any NC sub-CA that does not respond with the required actions in a timely manner.
Flags: needinfo?(robin)
Flags: needinfo?(rob)
(In reply to Robin Alden from comment #11)
> (In reply to Robin Alden from comment #2)
> > Issue #2
> > "the common name must be included in the Subject Alternative Names.”
> > 
<snip>
> > 7) Regular updates to confirm when 
> > those steps have been completed.
> We will report progress with revocation of the affected certificates.

The revocation status of some of these certificates may be seen in real time at:
https://misissued.com/batch/21/

I have updated these spreadsheets with the revocation status at the time I post this.
> There are two distinct sub-issues affecting these certificates.
> (a) certificates with no SAN 
> https://docs.google.com/spreadsheets/d/14z0N3e0jH5nB9pfZ9p1KqZsMbA8eqaYqlpmtJqDk-YE/edit?usp=sharing
> (b) commonName is not present in a SAN:dnsName
> https://docs.google.com/spreadsheets/d/1jiboOr2nohVEvUG1ikAG0lrK-bHSEwPtxAnNK8uSupY/edit?usp=sharing
> 
As you will see, almost all of those end entity certificates are now revoked.
The CA is targeting the 1st week of November to complete the revocations.
(In reply to Robin Alden from comment #24)
> The CA is targeting the 1st week of November to complete the revocations.

Robin: time for a status update?

Gerv
Flags: needinfo?(robin)
Hi Gerv, 
I will post a substantial reply to this by the end of tomorrow, December 8.
Regards
Robin
(In reply to Robin Alden from comment #24)
> (In reply to Robin Alden from comment #11)
> > (In reply to Robin Alden from comment #2)
> > > Issue #2
> > > "the common name must be included in the Subject Alternative Names.”
> > > 
> <snip>
> > > 7) Regular updates to confirm when 
> > > those steps have been completed.
> > We will report progress with revocation of the affected certificates.
> 
> The revocation status of some of these certificates may be seen in real time
> at:
> https://misissued.com/batch/21/
> 
> I have updated these spreadsheets with the revocation status at the time I
> post this.
> > There are two distinct sub-issues affecting these certificates.
> > (a) certificates with no SAN 
> > https://docs.google.com/spreadsheets/d/14z0N3e0jH5nB9pfZ9p1KqZsMbA8eqaYqlpmtJqDk-YE/edit?usp=sharing
> > (b) commonName is not present in a SAN:dnsName
> > https://docs.google.com/spreadsheets/d/1jiboOr2nohVEvUG1ikAG0lrK-bHSEwPtxAnNK8uSupY/edit?usp=sharing
> > 
> As you will see, almost all of those end entity certificates are now revoked.
> The CA is targeting the 1st week of November to complete the revocations.

All of these end entity certificates are now revoked.
The revocation was completed on November 8th.
Flags: needinfo?(robin)
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
Product: NSS → CA Program
Whiteboard: [ca-compliance] → [ca-compliance] [ev-misissuance] [ov-misissuance] [dv-misissuance] [disclosure-failure]
You need to log in before you can comment on or make changes to this bug.