Camerfirma: Non-BR-Compliant Certificate Issuance

Status

NSS
CA Certificate Mis-Issuance
4 months ago
2 days ago

People

(Reporter: Kathleen Wilson, Assigned: Ramiro Muñoz Muñoz, NeedInfo)

Tracking

(Blocks: 1 bug)

trunk

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [ca-compliance])

Attachments

(3 attachments)

(Reporter)

Description

4 months ago
The following problems have been found in certificates issued by your CA, and reported in the mozilla.dev.security.policy forum. Direct links to those discussions are provided for your convenience.

To continue inclusion of your CA’s root certificates in Mozilla’s Root Store, you must respond in this bug to provide the following information:
1) How your CA first became aware of the problems listed below (e.g. via a Problem Report, via the discussion in mozilla.dev.security.policy, or via this Bugzilla Bug), and the date.
2) Prompt confirmation that your CA has stopped issuing TLS/SSL certificates with the problems listed below.
3) Complete list of certificates that your CA finds with each of the listed issues during the remediation process. The recommended way to handle this is to ensure each certificate is logged to CT and then attach a CSV file/spreadsheet of the fingerprints or crt.sh IDs, with one list per distinct problem.
4) Summary of the problematic certificates. For each problem listed below: number of certs, date first and last certs with that problem were issued.
5) Explanation about how and why the mistakes were made, and not caught and fixed earlier.
6) List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.
7) Regular updates to confirm when those steps have been completed.

Note Section 4.9.1.1 of the CA/Browser Forum’s Baseline Requirements, which states:
“The CA SHALL revoke a Certificate within 24 hours if one or more of the following occurs: …
9. The CA is made aware that the Certificate was not issued in accordance with these Requirements or the CA’s Certificate Policy or Certification Practice Statement; 
10. The CA determines that any of the information appearing in the Certificate is inaccurate or misleading; …
14. Revocation is required by the CA’s Certificate Policy and/or Certification Practice Statement; or 
15. The technical content or format of the Certificate presents an unacceptable risk to Application Software Suppliers or Relying Parties (e.g. the CA/Browser Forum might determine that a deprecated cryptographic/signature algorithm or key size presents an unacceptable risk and that such Certificates should be revoked and replaced by CAs within a given period of time).

However, it is not our intent to introduce additional problems by forcing the immediate revocation of certificates that are not BR compliant when they do not pose an urgent security concern. Therefore, we request that your CA perform careful analysis of the situation. If there is justification to not revoke the problematic certificates, then explain those reasons and provide a timeline for when the bulks of the certificates will expire or be revoked/replaced. 

We expect that your forthcoming audit statements will indicate the findings of these problems. If your CA will not be revoking the certificates within 24 hours in accordance with the BRs, then that will also need to be listed as a finding in your CA’s BR audit statement.

We expect that your CA will work with your auditor (and supervisory body, as appropriate) and the Root Store(s) that your CA participates in to ensure your analysis of the risk and plan of remediation is acceptable. If your CA will not be revoking the problematic certificates as required by the BRs, then we recommend that you also contact the other root programs that your CA participates in to acknowledge this non-compliance and discuss what expectations their Root Programs have with respect to these certificates.


The problems reported for your CA in the mozilla.dev.security.policy forum are as follows:

** Failure to respond within 24 hours after Problem Report submitted
https://groups.google.com/d/msg/mozilla.dev.security.policy/PrsDfS8AMEk/w2AMK81jAQAJ
The problems were reported via your CA’s Problem Reporting Mechanism as listed here:
https://ccadb-public.secure.force.com/mozilla/CAInformationReport
Therefore, if this is the first time you have received notice of the problem(s) listed below, please review and fix your CA’s Problem Reporting Mechanism to ensure that it will work the next time someone reports a problem like this.

** Invalid dnsNames (e.g. invalid characters, internal names, and wildcards in the wrong position)
https://groups.google.com/d/msg/mozilla.dev.security.policy/CfyeeybBz9c/lmmUT4x2CAAJ
https://groups.google.com/d/msg/mozilla.dev.security.policy/D0poUHqiYMw/Pf5p0kB7CAAJ

** URI in dNSName SAN
https://groups.google.com/d/msg/mozilla.dev.security.policy/etp2Yz2fmM4/ayBTsfJnBgAJ
(Assignee)

Comment 1

4 months ago
Created attachment 8898277 [details]
Requerimiento Agosto 2017.docx
(Assignee)

Comment 2

4 months ago
Hi 
Here you are AC Camerfirma answers to this request (there is a doc attached to this bug):

AC Camerfirma was aware via the problem reporting email address places in https://ccadb-public.secure.force.com/mozilla/CAInformationReport.

Ac Camerfirma stop issuing more certificates with this problem and replace the problematic certificates issued coordinating with our customers. 

Problematic certificates issued are to do with the DNS name that include HTTP://, HTTPS:// or characters not allowed. We consider that those certificate do not produce a security critical problem to our users, so we avoid to revoke the certificates immediately without notifying our customers, and replacing the wrong certificate.
ID	DNS Name	Problema	CT?	REVOKED
19953712	sede electrónica del excmo ayuntamiento de palos de la frontera	DNSNAME ERROR	NO	YES
42531587	http://sede.palmasport.es/sedeelectronica/	DNSNAME ERROR	NO	YES
12555916	carpeta.colmenarviejo.es/gdcarpetaciudadano	DNSNAME ERROR	NO	YES
11796736	https://sede.ayto-meco.es	DNSNAME ERROR	NO	YES
116010910	one vision imaging ltd	DNSNAME ERROR	NO	YES
118466681	one vision imaging ltd	DNSNAME ERROR	NO	YES
112942986	ccc holding gmbh	DNSNAME ERROR	NO	YES
114181316	ccc holding gmbh	DNSNAME ERROR	NO	YES
112929021	ccc holding gmbh	DNSNAME ERROR	NO	YES
 	 	 	 	 
7388249	sede.apalmeria.gob.es/	DNSNAME ERROR	NO	PENDING
 	 	 	 	 
8322063	autodiscover.asisa.es , autodiscover.clinicacitralosnaranjos.es , autodiscover.grupoasisa.local , autodiscover.hospitalmoncloa.es , correo.clinicacitralosnaranjos.es , correo.grupoasisa.local , correo.hospitalmoncloa.es , correo365.asisa.es , asisa.es , asisa.pt , autodiscover.asisa.pt , asisadental.pt , autodiscover.asisadental.pt , correo.asisa.pt , correo.asisadental.pt , correo.asisa.es 	Warning. 
Cannot find the special name in SAN	NO	NO
 	 	 	 	 
8680123	asisa.es , autodiscover.asisa.es , autodiscover.clinicacitralosnaranjos.es , autodiscover.grupoasisa.local , autodiscover.hospitalmoncloa.es , correo.clinicacitralosnaranjos.es , correo.grupoasisa.local , correo.hospitalmoncloa.es , correo365.asisa.es , asisa.pt , autodiscover.asisa.pt , asisadental.pt , autodiscover.asisadental.pt , correo.asisa.pt , correo.asisadental.pt , correo.asisercard.es , autodiscover.asisercard.es , correo.clinicavistahermosa.es , autodiscover.clinicavistahermosa.es , correo.asisa.es	Warning. 
Cannot find the special name in SAN	NO	NO
 	 	 	 	 
5129200	https://sede.arandadeduero.es  - https://crt.sh/?id=5129200&opt=cablint	DNSNAME ERROR	NO	YES
42531587	http://sede.palmasport.es/sedeelectronica/ - https://crt.sh/?id=42531587&opt=cablint	DNSNAME ERROR	NO	YES
112929021	CCC HOLDING GMBH - https://crt.sh/?id=112929021&opt=cablint	DNSNAME ERROR	SI	Precertificate

AC Camerfirma are going to enforce a strong control over the request form field to avoid this kind of errors in a future, and inform to the AR operators to pay special attention to this issue.

Comment 3

4 months ago
(In reply to Ramiro Muñoz Muñoz from comment #2)
> Hi 
> Here you are AC Camerfirma answers to this request (there is a doc attached
> to this bug):
> 
> AC Camerfirma was aware via the problem reporting email address places in
> https://ccadb-public.secure.force.com/mozilla/CAInformationReport.
> 
> Ac Camerfirma stop issuing more certificates with this problem and replace
> the problematic certificates issued coordinating with our customers. 
> 
> Problematic certificates issued are to do with the DNS name that include
> HTTP://, HTTPS:// or characters not allowed. We consider that those
> certificate do not produce a security critical problem to our users, so we
> avoid to revoke the certificates immediately without notifying our
> customers, and replacing the wrong certificate.
> ID	DNS Name	Problema	CT?	REVOKED
> 19953712	sede electrónica del excmo ayuntamiento de palos de la frontera
> DNSNAME ERROR	NO	YES
> 42531587	http://sede.palmasport.es/sedeelectronica/	DNSNAME ERROR	NO	YES
> 12555916	carpeta.colmenarviejo.es/gdcarpetaciudadano	DNSNAME ERROR	NO	YES
> 11796736	https://sede.ayto-meco.es	DNSNAME ERROR	NO	YES
> 116010910	one vision imaging ltd	DNSNAME ERROR	NO	YES
> 118466681	one vision imaging ltd	DNSNAME ERROR	NO	YES
> 112942986	ccc holding gmbh	DNSNAME ERROR	NO	YES
> 114181316	ccc holding gmbh	DNSNAME ERROR	NO	YES
> 112929021	ccc holding gmbh	DNSNAME ERROR	NO	YES
>  	 	 	 	 
> 7388249	sede.apalmeria.gob.es/	DNSNAME ERROR	NO	PENDING
>  	 	 	 	 
> 8322063	autodiscover.asisa.es , autodiscover.clinicacitralosnaranjos.es ,
> autodiscover.grupoasisa.local , autodiscover.hospitalmoncloa.es ,
> correo.clinicacitralosnaranjos.es , correo.grupoasisa.local ,
> correo.hospitalmoncloa.es , correo365.asisa.es , asisa.es , asisa.pt ,
> autodiscover.asisa.pt , asisadental.pt , autodiscover.asisadental.pt ,
> correo.asisa.pt , correo.asisadental.pt , correo.asisa.es 	Warning. 
> Cannot find the special name in SAN	NO	NO
>  	 	 	 	 
> 8680123	asisa.es , autodiscover.asisa.es ,
> autodiscover.clinicacitralosnaranjos.es , autodiscover.grupoasisa.local ,
> autodiscover.hospitalmoncloa.es , correo.clinicacitralosnaranjos.es ,
> correo.grupoasisa.local , correo.hospitalmoncloa.es , correo365.asisa.es ,
> asisa.pt , autodiscover.asisa.pt , asisadental.pt ,
> autodiscover.asisadental.pt , correo.asisa.pt , correo.asisadental.pt ,
> correo.asisercard.es , autodiscover.asisercard.es ,
> correo.clinicavistahermosa.es , autodiscover.clinicavistahermosa.es ,
> correo.asisa.es	Warning. 
> Cannot find the special name in SAN	NO	NO
>  	 	 	 	 
> 5129200	https://sede.arandadeduero.es  -
> https://crt.sh/?id=5129200&opt=cablint	DNSNAME ERROR	NO	YES
> 42531587	http://sede.palmasport.es/sedeelectronica/ -
> https://crt.sh/?id=42531587&opt=cablint	DNSNAME ERROR	NO	YES
> 112929021	CCC HOLDING GMBH - https://crt.sh/?id=112929021&opt=cablint
> DNSNAME ERROR	SI	Precertificate
> 
> AC Camerfirma are going to enforce a strong control over the request form
> field to avoid this kind of errors in a future, and inform to the AR
> operators to pay special attention to this issue.

Thanks for responding. I think it's still necessary to provide additional detail.

That is, we can look at the problem from two dimensions: The problem itself, and the systemic issues that allowed the problem to manifest. Your description focuses on the resolution of the problem, but doesn't indicate any systemic changes have been made. As a consequence, it does not help the community feel that your CA has taken steps to reduce the risk of future violations (of any requirement) in the future. That is, one dimension is "Did you fix the bug", but another dimension is "How was the bug introduced, why was it not detected, and what steps are you taking to prevent future bugs"

To understand how you might approach this problem, consider https://groups.google.com/d/msg/mozilla.dev.security.policy/vl5eq0PoJxY/W1D4oZ__BwAJ and how it provided a timeline of events, the steps that were already in place (and with substantial detail), where there were controls missing or mistakes made, details about the steps being taken (e.g. "We fixed the bug" is not sufficient detail to understand), and a holistic, systemic awareness of how the CA is managed and the opportunities for errors.

In this particular case:
- It sounds like AR operators are allowed to directly enter the names in the certificates, is that correct?
- What technical controls exist to prevent mistakes like this?
- If no technical controls exist, what technical controls are being introduced to prevent issues like this/
- Systemically, why did AC Camerfirma not detect these issues, for example, as part of its internal review procedures?
- What changes are being made to its review procedures to provide assurance to the community that it will be more proactive in monitoring and responding to misissuance?
- What steps are being taken, systemically, to prevent misissued certificates? For example, the introduction of tools such as certlint to review the tbsCertificate prior to issuance?
- What steps are being taken, systemically, to provide transparency to the community about AC Camerfirma's operations and commitment to security? For example, logging all present and future valid certificates to Certificate Transparency logs.
- Please review steps 3-6 on the original report to ensure there are sufficiently detailed answers to these requests.
Alex: please can you give the details of your submitted Problem Report and the associated timelines, so any problems there can also be investigated in the context of this bug?

Gerv
(Assignee)

Comment 5

4 months ago
Hi
Our AR operators can modify request from our PKI platform before issuing a certificate when they detect any mistake in the request.

We have many AR working (STARTCOM included) and they can validate SSL OV, EV certificates. AC Camerfirma also have its own RA, and do a second verification process in all EV certificates.

We are going to work with our auditor to provide you with a fully answer as you request. At the moment our auditors are not available due to the summer holiday. Hope to have our report in a couple of week if this is ok for you, let me know it if not.

All the EV certificates issued by camerfirma has the CT controls, also most of the OV certificates have the CT verification. AC Camerfirma do not issue DV certificates. We are going to extend this year the CT verification to all SSL certificates issued

AC Camerfirma are to include a certlint automatic verification for any SSL certificate issued (this will take us some time), meanwhile we are going to have a daily extra verification process by our internal AR Operators over all SSL certificated issued to detect any possible mistake made by an external RA operator.

We already have revoked  the wrong certificates (Only one left to be revoked). 

Regards
Ramiro

Comment 6

4 months ago
For reference, I submitted a problem report via email with these headers:

  From: Jonathan Rudenberg <jonathan@titanous.com>
  Subject: Misissuance - invalid dnsNames
  Message-Id: <93BAE42F-65E5-45E7-A7C6-80D79DAEB0C1@titanous.com>
  Date: Sun, 13 Aug 2017 00:32:39 -0400
  To: operaciones@camerfirma.com

Comment 7

4 months ago
(In reply to Ramiro Muñoz Muñoz from comment #5)
> We are going to work with our auditor to provide you with a fully answer as
> you request. At the moment our auditors are not available due to the summer
> holiday. Hope to have our report in a couple of week if this is ok for you,
> let me know it if not.

Thanks for your reply. However, I'm uncertain why your auditor needs to be involved. That is, many of these questions relate to how Camerfirma is operating and maintaining its systems, and while your auditors may be involved to confirm the changes are as you describe, certainly, describing your system and the mitigations you have, why they failed, and their future plan should be something possible. This is certainly true that in the event of more pressing misissuance, these are the same questions that would be asked, and for which it would be time-critical to provide answers to (even if incomplete or subsequently updated with additional details)

I'll note that many CAs have provided replies within the past 24 hours to the questions asked. It would be good if you could reply in-line (using the "Reply" button available on Comment #0 and Comment #3) to ensure that each question has been answered.
Gerv,

On July 30th, I sent an email to Camerfirma (from my personal email) listing three certificates that were non-compliant due to invalid hostnames, and requesting they be revoked. Camerfirma did not respond to that email at any time. If more metadata for this email is required, I can dig it up.

The only other direct contact I had with them was the email thread you started, "Problematic Camerfirma certs", from August 15th.
(Assignee)

Comment 9

4 months ago
Alex,

I received you email On July 30th, ok, but you should have an answer from me.
I wrote an email to Gerv and you on August 16th confirming the revocation of those certificates. I was delayed in my answer since I was on holidays those days and also we had to get in contact with our customers to replace the wrong certificates.
I got an email from you on August 16th notifying that one of this certificates was not revoked and also there is an email from me to you on August 18th answering you that it was a precertificate and no revocation was issued. Nevertheless this precertificates is already revoked. At the moment all certificates you report are revoked.
Sorry for the delay due to the vacation period and also because we cannot revoke a certificate without notifying our customers and replacing it for a new one much less when the wrong certificate no means a security risk neither for our customer nor the end user.
Now we are writing the report as is requested by Kathleen and Ryan. Hope finishing along this Spanish-time morning.
Best regards
Ramiro
(Assignee)

Comment 10

4 months ago
Created attachment 8900677 [details]
answer3.csv - Complete list of certificates that your CA finds with each of the listed issues during the remediation process
(Assignee)

Comment 11

4 months ago
Created attachment 8900678 [details]
answer4.xlsx - Summary of the problematic certificates
(Assignee)

Comment 12

4 months ago
1)	How your CA first became aware of the problems listed below (e.g. via a Problem Report, via the discussion in mozilla.dev.security.policy, or via this Bugzilla Bug), and the date.
** URI in dNSName SAN
https://groups.google.com/d/msg/mozilla.dev.security.policy/etp2Yz2fmM4/ayBTsfJnBgAJ
We received information by email form Gervase Markham gerv@mozilla.org and Buzilla 20/7. 21/07 I answered Gerv and Katleen. 
Certificates was revoked once we could replace for new ones.
** Invalid dnsNames (e.g. invalid characters, internal names, and wildcards in the wrong position)
https://groups.google.com/d/msg/mozilla.dev.security.policy/CfyeeybBz9c/lmmUT4x2CAAJ
We received an email fom Jonathan Rudenberg <jonathan@titanous.com> 16th/08.
** https://groups.google.com/d/msg/mozilla.dev.security.policy/D0poUHqiYMw/Pf5p0kB7CAAJ
email from Kethleen Bugzilla 16th/8


2) Prompt confirmation that your CA has stopped issuing TLS/SSL certificates with the problems listed below.
Ac Camerfirma stopped issuing more certificates with this problem and replace the problematic certificates issued coordinating with our customers.
Problematic certificates issued are to do with the DNS name that include HTTP://, HTTPS:// or characters not allowed. We consider that those certificate do not produce a security risk to our customers, so we avoid to revoke the certificates immediately without notifying our customers, and replacing the wrong certificates.

3) Complete list of certificates that your CA finds with each of the listed issues during the remediation process. The recommended way to handle this is to ensure each certificate is logged to CT and then attach a CSV file/spreadsheet of the fingerprints or crt.sh IDs, with one list per distinct problem.
Attached file answer3.csv. Not all certificate has logged to CT.

4) Summary of the problematic certificates. For each problem listed below: number of certs, date first and last certs with that problem were issued.
Attached file answer4.xlsx

5) Explanation about how and why the mistakes were made, and not caught and fixed earlier.
There is no technical control about the DNSName input field in the request form. AR Operators have to review that the DNSName fulfil the policy requirement. Those certificates have a random review from our approval AR operators but none of the wrong certificates was detected. We neider have received incident reports from our customers. 
We can say that the origin of this problem is an external AR operator mistake. 

6) List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

•	Our approval AR Operators will carry out a full review of all DNSName certificates field issued. (Immediately).
•	Develop a technical control in the DNSName request field. (3 months).
•	Develop a control for each SSL/TSL certificates issued running a CA/B Forum lint and x509 lint in order to detect DNSName and other errors immediately after issuing a certificate. (3 months) 
•	All SSL/TSL Certificates issued at the moment have CT controls.
•	We will review our Problem Reporting Mechanism email address. We have been noticed to my personal email address, not by the established channel. We will create a new problem reporting email to notify to all the key people.

7) Regular updates to confirm when those steps have been completed.

We already started to verify in a daily basis all DNSNames issued. We do not issue a big number of SSL/TSL certificate so we can afford this.
Technical control will take us 2 or 3 month to be working.
Once we can detect any wrong certificate AC Camerfirma will proceed to notify before 24 hours and revoke them.

Best Regards
Ramiro Muñoz

Comment 13

4 months ago
> 4) Summary of the problematic certificates. For each problem listed below:
> number of certs, date first and last certs with that problem were issued.
> Attached file answer4.xlsx

https://crt.sh/?id=8322063&opt=cablint is not revoked, but contains two internal server names (correo.grupoasisa.local and autodiscover.grupoasisa.local). Per the Baseline Requirements 7.1.4.2.1 (using BR version 1.3.2 as an example), this was required to be revoked as of 1 October 2016.

https://crt.sh/?id=8680123&opt=cablint is not revoked, and also contains the same two internal server names.

The Baseline Requirements stated that, as of the effective date, CAs SHALL NOT issue such certificates with an Expiry Date later than 1 November 2015, yet these certificates expire in 2018. Why is that?

> 6) List of steps your CA is taking to resolve the situation and ensure such
> issuance will not be repeated in the future, accompanied with a timeline of
> when your CA expects to accomplish these things.
> 
> •	Our approval AR Operators will carry out a full review of all DNSName
> certificates field issued. (Immediately).

- Is this a technical review, or a non-technical review?
- Are there any assurances that the domain itself was appropriately validated? Based on the nature of these incidents, it appears that Camerfirma does not have the necessary or technical controls to ensure appropriate compliance.

> •	Develop a technical control in the DNSName request field. (3 months).

- Why will this take 3 months? Please provide more detail, given other CAs have been able to deploy changes within hours, days, or weeks.

> •	Develop a control for each SSL/TSL certificates issued running a CA/B
> Forum lint and x509 lint in order to detect DNSName and other errors
> immediately after issuing a certificate. (3 months) 

- Why is this not done prior to issuance? Checking after issuance is still misissuance, so while this is an important secondary detection mechanism, it is not a prevention or mitigation mechanism.

> 7) Regular updates to confirm when those steps have been completed.
> 
> We already started to verify in a daily basis all DNSNames issued.

- Please indicate how you are verifying these domains, and what you're verifying?
  - Compliance with RFC5280 and the 'preferred name syntax'
  - An appropriate demonstration of domain control
  - That the domain does not contain internal server names (which appeared to have been undetected by Camerfirma's compliance team in reviewing the flagged certificates).

> We do not
> issue a big number of SSL/TSL certificate so we can afford this.
> Technical control will take us 2 or 3 month to be working.
> Once we can detect any wrong certificate AC Camerfirma will proceed to
> notify before 24 hours and revoke them.

- As a mitigation, this would not be suitable. Not issuing the certificate in the first place is the expected result.
- Echoing the earlier remark, can you explain why it will take 2-3 months to be implemented?

For example, a suitable mitigation may be to require automatic demonstration of domain control, multi-party review in the case where that's not acceptable, automated checking of the well-formedness of the name (wildcards, preferred name syntax, valid TLD, etc) - in addition to the certlint/x509lint mitigations.
Flags: needinfo?(ramirom)
(Assignee)

Comment 14

4 months ago
(In reply to Ryan Sleevi from comment #13)
> > 4) Summary of the problematic certificates. For each problem listed below:
> > number of certs, date first and last certs with that problem were issued.
> > Attached file answer4.xlsx
> 
> https://crt.sh/?id=8322063&opt=cablint is not revoked, but contains two
> internal server names (correo.grupoasisa.local and
> autodiscover.grupoasisa.local). Per the Baseline Requirements 7.1.4.2.1
> (using BR version 1.3.2 as an example), this was required to be revoked as
> of 1 October 2016.
> 
> https://crt.sh/?id=8680123&opt=cablint is not revoked, and also contains the
> same two internal server names.
> 
> The Baseline Requirements stated that, as of the effective date, CAs SHALL
> NOT issue such certificates with an Expiry Date later than 1 November 2015,
> yet these certificates expire in 2018. Why is that?

Obviously is an error in the validation process. We will revoke those certificates.

> > 6) List of steps your CA is taking to resolve the situation and ensure such
> > issuance will not be repeated in the future, accompanied with a timeline of
> > when your CA expects to accomplish these things.
> > 
> > •	Our approval AR Operators will carry out a full review of all DNSName
> > certificates field issued. (Immediately).
> 
> - Is this a technical review, or a non-technical review?
> - Are there any assurances that the domain itself was appropriately
> validated? Based on the nature of these incidents, it appears that
> Camerfirma does not have the necessary or technical controls to ensure
> appropriate compliance.

It is a non-technical control, but this second validation proposed could mitigate the problem. Internal approval AR operators are more experienced staff. 
In this specific case (DNS Name) there are no technical controls since our AR Operators validate one by one each DNSName. 
We propose this action (second validation)to avoid errors since it can be done immediately. Technical control deployment take us some more time.

 > > •	Develop a technical control in the DNSName request field. (3 months).
> 
> - Why will this take 3 months? Please provide more detail, given other CAs
> have been able to deploy changes within hours, days, or weeks.
We are to include all BR requirement validation over all DNSNames found in the request form and this take some codification time. It was the answer from our PKI developers, anyway I will try to give priority to deploy this technical controls in weeks.
> > •	Develop a control for each SSL/TSL certificates issued running a CA/B
> > Forum lint and x509 lint in order to detect DNSName and other errors
> > immediately after issuing a certificate. (3 months) 
> 
> - Why is this not done prior to issuance? Checking after issuance is still
> misissuance, so while this is an important secondary detection mechanism, it
> is not a prevention or mitigation mechanism.

After certificate issuing we can use the cablint tool, this can speed up the process. We would revoke immediately problematic certificates before distribute it, so in some way it could be a mitigation mechanism. Otherwise we should code all those controls before issuing the certificate and this take us an unpredictable development time.

Nevertheless a technical control over the DNSname as we proposed before could solve the DNSName problem.

> > 7) Regular updates to confirm when those steps have been completed.
> > 
> > We already started to verify in a daily basis all DNSNames issued.
> 
> - Please indicate how you are verifying these domains, and what you're
> verifying?
>   - Compliance with RFC5280 and the 'preferred name syntax'
>   - An appropriate demonstration of domain control
>   - That the domain does not contain internal server names (which appeared
> to have been undetected by Camerfirma's compliance team in reviewing the
> flagged certificates).

Our Internal approval AR operators will validate the correct syntax and other validations in the request. Domain control is also verified (notifying domain contacts). We propose enforce the second validation process over OV SSL/TSL certificates to reduce those problems.

> > We do not
> > issue a big number of SSL/TSL certificate so we can afford this.
> > Technical control will take us 2 or 3 month to be working.
> > Once we can detect any wrong certificate AC Camerfirma will proceed to
> > notify before 24 hours and revoke them.
> 
> - As a mitigation, this would not be suitable. Not issuing the certificate
> in the first place is the expected result.
> - Echoing the earlier remark, can you explain why it will take 2-3 months to
> be implemented?
> 
> For example, a suitable mitigation may be to require automatic demonstration
> of domain control, multi-party review in the case where that's not
> acceptable, automated checking of the well-formedness of the name
> (wildcards, preferred name syntax, valid TLD, etc) - in addition to the
> certlint / x509lint mitigations.

Automatic demonstration of the domain control is already done by the PKI platform. We send automatically a communication to the domain contacts and wait for approval using a random code, without this we cannot validate a request.
Multi-party is what we are proposing while our developers deploy the automatic BR Domain Name validation. 
We also have an automatic double validation for our EV certificates that we could use this process in the OV validation process.
Flags: needinfo?(ramirom)

Comment 15

4 months ago
(In reply to Ramiro Muñoz Muñoz from comment #14)
> (In reply to Ryan Sleevi from comment #13)
> > The Baseline Requirements stated that, as of the effective date, CAs SHALL
> > NOT issue such certificates with an Expiry Date later than 1 November 2015,
> > yet these certificates expire in 2018. Why is that?
> 
> Obviously is an error in the validation process. We will revoke those
> certificates.

Well, yes, this is an error.

The question, however, is:
1) Why weren't these revoked when originally reported?
2) Why weren't these revoked when previously required?
3) Why weren't these limited in their validity period?

That is, an answer of "mistakes were made" does not demonstrate the level of introspection, analysis, and transparency necessary for public trust. It's clear from the evidence that mistakes were made, what is not clear is why these mistakes they were made, how they were made, and what steps are being taken to prevent these future mistakes?

That is, in the report, the certlint information was available. What analysis was done that lead to the conclusion not to revoke these certificates? What level of secondary review was conducted? Were your auditors (or conformance assessment body) involved in that decision?

Understanding how the CA has made the decision not to revoke, and how it will make future decisions regarding compliance, is an essential part of public trust.

> > > •	Develop a control for each SSL/TSL certificates issued running a CA/B
> > > Forum lint and x509 lint in order to detect DNSName and other errors
> > > immediately after issuing a certificate. (3 months) 
> > 
> > - Why is this not done prior to issuance? Checking after issuance is still
> > misissuance, so while this is an important secondary detection mechanism, it
> > is not a prevention or mitigation mechanism.
> 
> After certificate issuing we can use the cablint tool, this can speed up the
> process. We would revoke immediately problematic certificates before
> distribute it, so in some way it could be a mitigation mechanism. 

Note that solely replying on mitigation mechanisms, such as revocation, are insufficient for continued trust. CAs are expected to avoid these issues in the first place, particularly when technical controls to prevent issuance exist.

Are you stating that your systems are that unknown that you cannot provide an estimate as to when you can implement technical controls to adhere to the Baseline Requirements, and that your issuance practices will rely on the expertise of your RAs to understand and internalize all of the direct requirements of the Baseline Requirements and indirect requirements from the relevant specifications?

> Otherwise
> we should code all those controls before issuing the certificate and this
> take us an unpredictable development time.

Can you not integrate a single control over the tbsCertificate prior to signing?

If not, can you please share further details about your CA operation software. For example, is it using a 'common off the shelf' system (such as EJBCA or Microsoft CA server), or is it using an in-house developed CA?

The trustworthiness of a CA organization is, in part, derived from the trustworthiness of its infrastructure. If a CA finds it difficult to implement or integrate controls, this poses both direct risk to the ecosystem (from situations such as this) and indirect risk (such as due to delaying requirements to allow such CAs sufficient time), and may be better suited by migrating trust away from such CAs.

I say that to help emphasize that public trust is a rather serious matter, and an important part of that trust is taking all reasonable steps to ensure a globally-secure system and infrastructure is in place. If the costs of such a system are not practical, it may be that providing global trust is not an appropriate solution.

> > > 7) Regular updates to confirm when those steps have been completed.
> > > 
> > > We already started to verify in a daily basis all DNSNames issued.
> > 
> > - Please indicate how you are verifying these domains, and what you're
> > verifying?
> >   - Compliance with RFC5280 and the 'preferred name syntax'
> >   - An appropriate demonstration of domain control
> >   - That the domain does not contain internal server names (which appeared
> > to have been undetected by Camerfirma's compliance team in reviewing the
> > flagged certificates).
> 
> Our Internal approval AR operators will validate the correct syntax and
> other validations in the request. Domain control is also verified (notifying
> domain contacts). We propose enforce the second validation process over OV
> SSL/TSL certificates to reduce those problems.

To confirm: Your answer indicates that RA staff are validating everything by hand, and that compliance and consistency are solely determined by human factors, is that correct? As such, any failure of training, or any human error, can lead to misissued certificates.

> > For example, a suitable mitigation may be to require automatic demonstration
> > of domain control, multi-party review in the case where that's not
> > acceptable, automated checking of the well-formedness of the name
> > (wildcards, preferred name syntax, valid TLD, etc) - in addition to the
> > certlint / x509lint mitigations.
> 
> Automatic demonstration of the domain control is already done by the PKI
> platform. We send automatically a communication to the domain contacts and
> wait for approval using a random code, without this we cannot validate a
> request.

This is a useful detail. In the spirit of analyzing the root issue, if this process was followed, how was it possible for a certificate such as https://crt.sh/?q=8680123 (which is _still_ not revoked, despite Comment #14) to be issued? There is no valid way for this domain to have been validated, which suggests that you have one or more critical security vulnerabilities in how you validate domain names, or that the answer may be incomplete?

> Multi-party is what we are proposing while our developers deploy the
> automatic BR Domain Name validation. 
> We also have an automatic double validation for our EV certificates that we
> could use this process in the OV validation process.
Flags: needinfo?(ramirom)
(Assignee)

Comment 16

4 months ago
(In reply to Ryan Sleevi from comment #15)
> (In reply to Ramiro Muñoz Muñoz from comment #14)
> > (In reply to Ryan Sleevi from comment #13)
> > > The Baseline Requirements stated that, as of the effective date, CAs SHALL
> > > NOT issue such certificates with an Expiry Date later than 1 November 2015,
> > > yet these certificates expire in 2018. Why is that?
> > 
> > Obviously is an error in the validation process. We will revoke those
> > certificates.
> 
> Well, yes, this is an error.
> 
> The question, however, is:
> 1) Why weren't these revoked when originally reported?
> 2) Why weren't these revoked when previously required?
> 3) Why weren't these limited in their validity period?
> 
> That is, an answer of "mistakes were made" does not demonstrate the level of
> introspection, analysis, and transparency necessary for public trust. It's
> clear from the evidence that mistakes were made, what is not clear is why
> these mistakes they were made, how they were made, and what steps are being
> taken to prevent these future mistakes?
> 
> That is, in the report, the certlint information was available. What
> analysis was done that lead to the conclusion not to revoke these
> certificates? What level of secondary review was conducted? Were your
> auditors (or conformance assessment body) involved in that decision?
> 
> Understanding how the CA has made the decision not to revoke, and how it
> will make future decisions regarding compliance, is an essential part of
> public trust.
> 
As we already explain on previous answers, we need to inform to our customer before certificate revocation, in order to avoid any damage. Our customer has imposed us not revoke the certificate until the replacement was complete. We as a CA are in the middle between our customer and the browsers requirement. We have opened an issue with our legal department to include in the contract with our customers a speedy certificate revocation procedure. 
RA Operator mistakes are possible so that why we review the 3% of the “all” SSL/TSL certificates issued, in this case this certificate was not detected in this 3% review.
AC Camerfirma is right now reviewing “all” SSL/TSL certificates issued (much more that the 3%).
AC Camerfirma is going to deploy this week a technical control to satisfy the BR requirement about the DNSName.

With these controls, we think that the DNSName issue is fixed and enough to guarantee the public trust.
Keep in mind that Ac Camerfirma also include in all SSL/TSL certificates the STC from the certificate transparency.

> > > > •	Develop a control for each SSL/TSL certificates issued running a CA/B
> > > > Forum lint and x509 lint in order to detect DNSName and other errors
> > > > immediately after issuing a certificate. (3 months) 
> > > 
> > > - Why is this not done prior to issuance? Checking after issuance is still
> > > misissuance, so while this is an important secondary detection mechanism, it
> > > is not a prevention or mitigation mechanism.
> > 
> > After certificate issuing we can use the cablint tool, this can speed up the
> > process. We would revoke immediately problematic certificates before
> > distribute it, so in some way it could be a mitigation mechanism. 
> 
> Note that solely replying on mitigation mechanisms, such as revocation, are
> insufficient for continued trust. CAs are expected to avoid these issues in
> the first place, particularly when technical controls to prevent issuance
> exist.
> 
> Are you stating that your systems are that unknown that you cannot provide
> an estimate as to when you can implement technical controls to adhere to the
> Baseline Requirements, and that your issuance practices will rely on the
> expertise of your RAs to understand and internalize all of the direct
> requirements of the Baseline Requirements and indirect requirements from the
> relevant specifications?

Well now we rely on the RA expertise and a second operator validation to adhere to BR, I do know if technical control are mandatory. Our number of SSL certificates issued is not so high so we can afford this. Nevertheless I think that technical control can help us to speed up the issuing process and we are going to integrate cablint control in our PKI platform. This can cover every certificate issued and detect any codification problem. 
> 
> > Otherwise
> > we should code all those controls before issuing the certificate and this
> > take us an unpredictable development time.
> 
> Can you not integrate a single control over the tbsCertificate prior to
> signing?

Yes we can do it, but we need to evaluate the effort. That’s what our programmers are doing right now. I have not all information but it appeared that is not a simple integration. 
> 
> If not, can you please share further details about your CA operation
> software. For example, is it using a 'common off the shelf' system (such as
> EJBCA or Microsoft CA server), or is it using an in-house developed CA?

Is an in-house developed CA
> 
> The trustworthiness of a CA organization is, in part, derived from the
> trustworthiness of its infrastructure. If a CA finds it difficult to
> implement or integrate controls, this poses both direct risk to the
> ecosystem (from situations such as this) and indirect risk (such as due to
> delaying requirements to allow such CAs sufficient time), and may be better
> suited by migrating trust away from such CAs.
> 
> I say that to help emphasize that public trust is a rather serious matter,
> and an important part of that trust is taking all reasonable steps to ensure
> a globally-secure system and infrastructure is in place. If the costs of
> such a system are not practical, it may be that providing global trust is
> not an appropriate solution.

AC Camerfirma has been working with this technology for 17 years without problems and we have dedicated a big effort improving it along those years. 

It’s not a matter of difficulty to integrate new controls, we can do it, but we first of all need to evaluate changes, that why we cannot provide a specific date.

> 
> > > > 7) Regular updates to confirm when those steps have been completed.
> > > > 
> > > > We already started to verify in a daily basis all DNSNames issued.
> > > 
> > > - Please indicate how you are verifying these domains, and what you're
> > > verifying?
> > >   - Compliance with RFC5280 and the 'preferred name syntax'
> > >   - An appropriate demonstration of domain control
> > >   - That the domain does not contain internal server names (which appeared
> > > to have been undetected by Camerfirma's compliance team in reviewing the
> > > flagged certificates).
> > 
> > Our Internal approval AR operators will validate the correct syntax and
> > other validations in the request. Domain control is also verified (notifying
> > domain contacts). We propose enforce the second validation process over OV
> > SSL/TSL certificates to reduce those problems.
> 
> To confirm: Your answer indicates that RA staff are validating everything by
> hand, and that compliance and consistency are solely determined by human
> factors, is that correct? As such, any failure of training, or any human
> error, can lead to misissued certificates.


Yes that’s why we are doing now a second validation process for all SSL/TSL certificated issued, meanwhile we have already place into operation the (DNSName) technical controls.

> 
> > > For example, a suitable mitigation may be to require automatic demonstration
> > > of domain control, multi-party review in the case where that's not
> > > acceptable, automated checking of the well-formedness of the name
> > > (wildcards, preferred name syntax, valid TLD, etc) - in addition to the
> > > certlint / x509lint mitigations.
> > 
> > Automatic demonstration of the domain control is already done by the PKI
> > platform. We send automatically a communication to the domain contacts and
> > wait for approval using a random code, without this we cannot validate a
> > request.
> 
> This is a useful detail. In the spirit of analyzing the root issue, if this
> process was followed, how was it possible for a certificate such as
> https://crt.sh/?q=8680123 (which is _still_ not revoked, despite Comment
> #14) to be issued? 

This certificate was issued on Jul 23 2015 I am afraid that these controls was not working by those dates.

> There is no valid way for this domain to have been
> validated, which suggests that you have one or more critical security
> vulnerabilities in how you validate domain names, or that the answer may be
> incomplete?
> 
It was a mistake from our RA Operator that was not controlled by the 3% review process. We already have explained what to do, and what we have done to avoid this problem in the future.

> > Multi-party is what we are proposing while our developers deploy the
> > automatic BR Domain Name validation. 
> > We also have an automatic double validation for our EV certificates that we
> > could use this process in the OV validation process.
Flags: needinfo?(ramirom)

Comment 17

3 months ago
(In reply to Ramiro Muñoz Muñoz from comment #16)
> As we already explain on previous answers, we need to inform to our customer
> before certificate revocation, in order to avoid any damage. Our customer
> has imposed us not revoke the certificate until the replacement was
> complete. We as a CA are in the middle between our customer and the browsers
> requirement. We have opened an issue with our legal department to include in
> the contract with our customers a speedy certificate revocation procedure. 

Could you explain what provisions you have that prevent you from revoking?

That is, the Baseline Requirements require revoking within 24 hours. They also require that, in the existence of a conflict between the BRs and your CP/CPS, the BRs override what's in your CP/CPS. Your subscriber agreement is required to include provisions for revocation based on your CP/CPS.

As such, there should be no reason that you _can't_ revoke, only when you choose not to revoke.

In this case, I was asking for the analysis about why this wasn't revoked as originally scheduled, and what policies and practices are in place to ensure future issues won't happen. These are all retrospective analyses that need to be done and that haven't been done, and look at the controls in place (which weren't sufficient) - and about understanding holistically how those controls failed.

Put differently, it's good to understand mistakes were made, and it's good to have fixes for those mistakes - but it's better to understand why the mistakes were made, and it's good to prevent future mistakes. That's missing here.

> RA Operator mistakes are possible so that why we review the 3% of the “all”
> SSL/TSL certificates issued, in this case this certificate was not detected
> in this 3% review.
> AC Camerfirma is right now reviewing “all” SSL/TSL certificates issued (much
> more that the 3%).
> AC Camerfirma is going to deploy this week a technical control to satisfy
> the BR requirement about the DNSName.

Has this been deployed yet?

> Well now we rely on the RA expertise and a second operator validation to
> adhere to BR, I do know if technical control are mandatory. Our number of
> SSL certificates issued is not so high so we can afford this. Nevertheless I
> think that technical control can help us to speed up the issuing process and
> we are going to integrate cablint control in our PKI platform. This can
> cover every certificate issued and detect any codification problem. 

This can address technical compliance, but doesn't address policy compliance. And given that the technical controls failed with the current system, it's difficult to imagine how the policy compliance will not have the same issues.


> > > Automatic demonstration of the domain control is already done by the PKI
> > > platform. We send automatically a communication to the domain contacts and
> > > wait for approval using a random code, without this we cannot validate a
> > > request.
> > 
> > This is a useful detail. In the spirit of analyzing the root issue, if this
> > process was followed, how was it possible for a certificate such as
> > https://crt.sh/?q=8680123 (which is _still_ not revoked, despite Comment
> > #14) to be issued? 
> 
> This certificate was issued on Jul 23 2015 I am afraid that these controls
> was not working by those dates.

So you anticipate that there are still-valid certificates issued with insufficient controls and thus may not meet either the policy or technical requirements of the Baseline Requirements?

I think more detail then "we had an issue" is warranted here, because the whole goal of this issue is to demonstrate Camerfirma's complete understanding of the issue (and their approach to past issues), to ensure that assurances given about future issues can be relied upon.
Flags: needinfo?(ramirom)

Comment 18

3 months ago
Attempting to summarize the information to date:

1) Invalid DNS Names (Invalid Characters - URLs)
- See Comment #0, Comment #1, Comment #12
- Remediation:
  - 2017-08-24 - Retrain staff on conformance requirements (see Comment #12)
  - 2017-08-24 - Require additional review after issuance (see Comment #12)
  - 2017-11-XX - Automatic compliance checking

2) Invalid DNS names (Invalid Names - e.g. company names)
- See Comment #0, Comment #1, Comment #12
- Remediation:
  - 2017-08-24 - Retrain staff on conformance requirements (see Comment #12)
  - 2017-08-24 - Require additional review after issuance (see Comment #12)
  - 2017-11-XX - Automatic compliance checking

Is this a correct summary so far?

Regarding the certificates you have not yet revoked, but contain internal server names - as these present critical security risk to the ecosystem, please revoke these immediately. This is the same that has been requested and expected of other CAs for this same issue. It is particularly concerning that, despite being presented with these certificates, Camerfirma did not detect the non-compliance, even under manual review. This raises significant concerns about the remediation process taken, as well as concerns for the remediations proposed.

Kathleen, I'm not sure how you want to handle this. I'm also concerned that certificates that were said were going to be revoked (for being internal server names - see Comment #14 "Obviously is an error in the validation process. We will revoke those certificates." - have not been revoked - see Comment #16), even as we're now a week later. I also don't believe these remediation plans demonstrate the necessary awareness of the root causes or provide sufficient assurance to prevent future issues.
Flags: needinfo?(kwilson)
(Assignee)

Comment 19

3 months ago
(In reply to Ryan Sleevi from comment #17)
> (In reply to Ramiro Muñoz Muñoz from comment #16)
> > As we already explain on previous answers, we need to inform to our customer
> > before certificate revocation, in order to avoid any damage. Our customer
> > has imposed us not revoke the certificate until the replacement was
> > complete. We as a CA are in the middle between our customer and the browsers
> > requirement. We have opened an issue with our legal department to include in
> > the contract with our customers a speedy certificate revocation procedure. 
> 
> Could you explain what provisions you have that prevent you from revoking?
> 
> That is, the Baseline Requirements require revoking within 24 hours. They
> also require that, in the existence of a conflict between the BRs and your
> CP/CPS, the BRs override what's in your CP/CPS. Your subscriber agreement is
> required to include provisions for revocation based on your CP/CPS.
> 
> As such, there should be no reason that you _can't_ revoke, only when you
> choose not to revoke.
> 
> In this case, I was asking for the analysis about why this wasn't revoked as
> originally scheduled, and what policies and practices are in place to ensure
> future issues won't happen. These are all retrospective analyses that need
> to be done and that haven't been done, and look at the controls in place
> (which weren't sufficient) - and about understanding holistically how those
> controls failed.
> 
> Put differently, it's good to understand mistakes were made, and it's good
> to have fixes for those mistakes - but it's better to understand why the
> mistakes were made, and it's good to prevent future mistakes. That's missing
> here.
> 

We are working with our customer to revoke the certificates. One of them is already revoked and there is one left to revoke that will be revoked on September the 11th (already agreed with our customer). AC Camerfirma has replaced this certificate and must be installed before revoking the old one, as our customer imposed us.  There are many domains inside the certificate a many services affected.
AC Camerfirma want to avoid that this situation again so we are going to clarify our subscriber agreement to make clear this situation to our customers and protect AC Camerfirma legal responsibility when we unilaterally revoke the certificate.
Regarding the mistake, we have been trying to understand the problem, as you said. By the dates we issue the certificate our RA Operator, understand from our CPS, that the internal names was allowed until October 2016, a misunderstanding that we already clarifyed in our late CPS version. 

> > RA Operator mistakes are possible so that why we review the 3% of the “all”
> > SSL/TSL certificates issued, in this case this certificate was not detected
> > in this 3% review.
> > AC Camerfirma is right now reviewing “all” SSL/TSL certificates issued (much
> > more that the 3%).
> > AC Camerfirma is going to deploy this week a technical control to satisfy
> > the BR requirement about the DNSName.
> 
> Has this been deployed yet?
> 
Yes.
So, at the moment we have the DNSName technical control working and we review 100% the TSL/SSL certificates issued. I think this can offer enough guaranties to detect any future problem about this issue.
>
> > Well now we rely on the RA expertise and a second operator validation to
> > adhere to BR, I do know if technical control are mandatory. Our number of
> > SSL certificates issued is not so high so we can afford this. Nevertheless I
> > think that technical control can help us to speed up the issuing process and
> > we are going to integrate cablint control in our PKI platform. This can
> > cover every certificate issued and detect any codification problem. 
> 
> This can address technical compliance, but doesn't address policy
> compliance. And given that the technical controls failed with the current
> system, it's difficult to imagine how the policy compliance will not have
> the same issues.
> 
> 
> > > > Automatic demonstration of the domain control is already done by the PKI
> > > > platform. We send automatically a communication to the domain contacts and
> > > > wait for approval using a random code, without this we cannot validate a
> > > > request.
> > > 
> > > This is a useful detail. In the spirit of analyzing the root issue, if this
> > > process was followed, how was it possible for a certificate such as
> > > https://crt.sh/?q=8680123 (which is _still_ not revoked, despite Comment
> > > #14) to be issued? 
> > 
> > This certificate was issued on Jul 23 2015 I am afraid that these controls
> > was not working by those dates.
> 
> So you anticipate that there are still-valid certificates issued with
> insufficient controls and thus may not meet either the policy or technical
> requirements of the Baseline Requirements?
>
Obviously not. 
We have been deploying some technical control along the time, but control were carry out by other means, as our auditors said in the audit reports we have been providing over time.
>
> I think more detail then "we had an issue" is warranted here, because the
> whole goal of this issue is to demonstrate Camerfirma's complete
> understanding of the issue (and their approach to past issues), to ensure
> that assurances given about future issues can be relied upon.
I can assure you that AC Camerfirma understand completely the issue and we are working hard to offer a quality and secure services to our customers and to the community as our certification in ISO27001, ISO20000, WEBTRUST CA BR EV, ETSI 319401, ETSI 319411-1, ETSI 319411-2, ETSI 319 41121,  demonstrate.

> 
> I think more detail then "we had an issue" is warranted here, because the
> whole goal of this issue is to demonstrate Camerfirma's complete
> understanding of the issue (and their approach to past issues), to ensure
> that assurances given about future issues can be relied upon.
>
I can assure you that AC Camerfirma understand completely the issue and we are working hard to offer a quality and secure services to our customers and to the community as our audit process in ISO27001, ISO20000, WEBTRUST CA-BR-EV, ETSI-319401, ETSI-319411-1, ETSI-319411-2, ETSI-319421, state.
Flags: needinfo?(ramirom)
(Reporter)

Comment 20

3 months ago
(In reply to Ryan Sleevi from comment #18)
> Regarding the certificates you have not yet revoked, but contain internal
> server names - as these present critical security risk to the ecosystem,
> please revoke these immediately. This is the same that has been requested
> and expected of other CAs for this same issue. It is particularly concerning
> that, despite being presented with these certificates, Camerfirma did not
> detect the non-compliance, even under manual review. This raises significant
> concerns about the remediation process taken, as well as concerns for the
> remediations proposed.
> 
> Kathleen, I'm not sure how you want to handle this. I'm also concerned that
> certificates that were said were going to be revoked (for being internal
> server names - see Comment #14 "Obviously is an error in the validation
> process. We will revoke those certificates." - have not been revoked - see
> Comment #16), even as we're now a week later. 


In the description of this bug I said: it is not our intent to introduce additional problems by forcing the immediate revocation of certificates that are not BR compliant when they do not pose an urgent security concern. 

Clearly, certificates that present security risk to the ecosystem must be revoked and replaced very promptly. But I understand the need for CAs to work with their customers to resolve such problems in an orderly manner, so they do not introduce potentially worse problems.


> I also don't believe these
> remediation plans demonstrate the necessary awareness of the root causes or
> provide sufficient assurance to prevent future issues.

It is up to the CA to demonstrate this via their response in this bug.

If a CA does not demonstrate their ability to respond to such incidents and does not provide sufficient assurance to prevent future issues, then we will add the CA to our list of CAs to investigate and discuss more thoroughly, as we have begun doing in the mozilla.dev.security.policy forum.
Flags: needinfo?(kwilson)
Ramiro: these questions still remain outstanding, 3 months after the last comment in this bug:

(In reply to Ryan Sleevi from comment #15)
> The question, however, is:
> 1) Why weren't these revoked when originally reported?
> 2) Why weren't these revoked when previously required?
> 3) Why weren't these limited in their validity period?
> 
> That is, an answer of "mistakes were made" does not demonstrate the level of
> introspection, analysis, and transparency necessary for public trust. It's
> clear from the evidence that mistakes were made, what is not clear is why
> these mistakes they were made, how they were made, and what steps are being
> taken to prevent these future mistakes?

I suggest you read up on "root cause analysis" before attempting to give another answer. And I would pay careful attention to what Kathleen said at the end of comment #20.

Gerv
Flags: needinfo?(ramirom)

Comment 22

16 days ago
Hello all,

We were not aware that the bug was pending on our side. 

We apologize for this. 

We'll answer as soon as possible taking into account your indications.

Kind regards
Juan Angel
(Assignee)

Comment 23

14 days ago
Hi Gerv
Sorry for the delay. We have held a meeting with the people involved in this issue trying, as you suggest, analyzing and understanding the root cause.

We focus the three questions about the use of internal names.
1)	Why weren't these revoked when originally reported?
2)	Why weren't these revoked when previously required?
3)	Why weren't these limited in their validity period?

Affected certificates (2) were issued on July 2015 in a renewal process when local domains were permitted. So no problem was detected in this operation. Nevertheless, control over the limit date (November 2015) allowed to internal domains was not developed into the platform, so platform was not aware of this time limit, and neither when we should have finaly revoked them (October 2016).

No more internal names were issued after November 2015, so our platform managed well the requirement to avoid internal DNS names but not the transition period.

We were aware of this misisuance on August 2017 by means of some remarks in this Bugzilla bug.

We revoked those certificates on 1st and 12th of September 2017 after notifying and arranged a date for revocation with the affected user.

On September 2017 we activated a control in our PKI platform to avoid the DNSName bad spellings, check their availability and therefore avoid internal names. This control cover the problems Ryan sumarized in Comment#18 
 
Getting back to your questions:

4)	Why weren't these revoked when originally reported?
We were aware of this problem, as I said before, on August 24th 2017, we revoke them on September the 1st and 12th after arrange with the customer a date to revoke.

5)	Why weren't these revoked when previously required?
If you means the time where BR state, we were not aware of this misissuance neither November 2015 nor October 2016. Please confirm it were the question.

6)	Why weren't these limited in their validity period?
Our PKI platform works with predefined profiles and the time limit is a fixed value, so is complex to change it.

Hope we have answered your questions and keep working and ready to improve our tools and procedures in order to add value to the community.

Ramiro
Flags: needinfo?(ramirom)
Ramiro,

The point is that the deadline for ceasing issuance of internal names, and the further deadline by which all such certificates need to be revoked, was known for several years before they actually arrived. Other CAs did one or more of the following:

1) Made sure no internal name certificates had lifetimes which extended beyond the revocation deadline
2) Did an internal scan just before or on the revocation date, and revoked all outstanding certs they found

If 1) was not possible for you due to your predefined profiles, I would expect you to have done 2) as a matter of BR compliance.

Gerv
Flags: needinfo?(ramirom)
(Assignee)

Comment 25

11 days ago
Hi Gerv,

Yes, we should have been done that (2). We did not. AC Camerfirma pass Webtrust CA, EV and BR yearly to detect any non-compliance procedures and certificates. Auditors detected any problem in 2016 audit because only a percentage of certificates were checked. Some bad certificates were detected in 2017 audit because bad DNSName spelling. In this case, crt.sh was a helpful tool. A couple of those certificates contain internal names (8322063, 8680123). Even in our attachment 8898277 [details] in this bug , we made a comment saying that we did not find any problem with DNSNames spelling in those certificates. That was when we were aware of the “internal name” problem (August 2017).

Yearly audit wasn't enough for a detective-control, as we firstly thought, so we carry out a more strict observance today.
As we remarked in this bug, now “every single SSL OV & EV” certificates issued is checked by crt.sh .A reactive control meanwhile a preventive and automatic control is placed in our PKI platform (being developed and planned to deploy this month). We will notify in this bug when this control was working and our auditors will check that this control is already deployed in the next audit report.

Ramiro
Flags: needinfo?(ramirom)
(In reply to Ramiro Muñoz Muñoz from comment #25) 
> Yes, we should have been done that (2). We did not. AC Camerfirma pass
> Webtrust CA, EV and BR yearly to detect any non-compliance procedures and
> certificates. 

All the checks you talk about in this message seem to be checks at issuance time. But the point here is that the BRs were updated to require a set of certs to be revoked at a certain time, and AC Camerfirma did not incorporate that requirement into their processes.

So what changes are you making to make sure that, in future, all ramifications of all changes to the BRs are incorporated into your operations?

Gerv
(Assignee)

Comment 27

9 days ago
Hi Gerv

We have increase our resources and involvement in SSL/TSL certificate management, human and technical.  We hope, in this way, to eliminate any anomalous certificate issuances in the future. To be precise we already have:

•	A quality operator in charge of reviewing every single SSL issued.
•	A PKI senior expert in charge of CABFORUM distribution list, MDSP, CCADB, self-assessment and communication management.
•	Increased our requirements for external AR audits. 
•	A PKI Developer in charge to deploy new technical control in our PKI platform (DNSName (done), cablint (in process), CAA check (in process)...etc).

We will ask to our auditors to confirm that all of these actions are working in the next yearly report.

Ramiro.

Comment 28

2 days ago
Hi Ramiro,

I'm not sure your reply in Comment #27 really instills confidence or responds to the concerns in Comment #26. Further, the response in Comment #23 is what is being discussed in Comment #26 - that answer - and analysis, is insufficient.

Your proposed changes - in Comment #27 - thus seem grossly inadequate, because as a CA, Camerfirma has not demonstrated it fully understands what went wrong, why it went wrong, and how it will be improved. Comment #25 underscores that you admit 'something went wrong' - but it fails to respond to any of the substance expected of a CA incident report.

The expectation is that, as a globally trusted CA, you had staffed adequately, and competently, to ensure that deadlines would be met. What are the process and procedure failures that caused Camerfirma to overlook this very simple, very critical requirement - as a trusted CA? What reason, if any, should the community believe that in the future, Camerfirma will do better.

Comment #27 does not instill confidence in understanding the systemic failures that caused this issue ("Why didn't you do (2) in Comment #25"), and thus does not instill confidence in Camerfirma as a CA.

Updated

2 days ago
Flags: needinfo?(ramirom)
You need to log in before you can comment on or make changes to this bug.