1390977 - Camerfirma: Non-BR-Compliant Certificate Issuance

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Kathleen Wilson]

The following problems have been found in certificates issued by your CA, and reported in the mozilla.dev.security.policy forum. Direct links to those discussions are provided for your convenience.

To continue inclusion of your CA’s root certificates in Mozilla’s Root Store, you must respond in this bug to provide the following information:
1) How your CA first became aware of the problems listed below (e.g. via a Problem Report, via the discussion in mozilla.dev.security.policy, or via this Bugzilla Bug), and the date.
2) Prompt confirmation that your CA has stopped issuing TLS/SSL certificates with the problems listed below.
3) Complete list of certificates that your CA finds with each of the listed issues during the remediation process. The recommended way to handle this is to ensure each certificate is logged to CT and then attach a CSV file/spreadsheet of the fingerprints or crt.sh IDs, with one list per distinct problem.
4) Summary of the problematic certificates. For each problem listed below: number of certs, date first and last certs with that problem were issued.
5) Explanation about how and why the mistakes were made, and not caught and fixed earlier.
6) List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.
7) Regular updates to confirm when those steps have been completed.

Note Section 4.9.1.1 of the CA/Browser Forum’s Baseline Requirements, which states:
“The CA SHALL revoke a Certificate within 24 hours if one or more of the following occurs: …
9. The CA is made aware that the Certificate was not issued in accordance with these Requirements or the CA’s Certificate Policy or Certification Practice Statement; 
10. The CA determines that any of the information appearing in the Certificate is inaccurate or misleading; …
14. Revocation is required by the CA’s Certificate Policy and/or Certification Practice Statement; or 
15. The technical content or format of the Certificate presents an unacceptable risk to Application Software Suppliers or Relying Parties (e.g. the CA/Browser Forum might determine that a deprecated cryptographic/signature algorithm or key size presents an unacceptable risk and that such Certificates should be revoked and replaced by CAs within a given period of time).

However, it is not our intent to introduce additional problems by forcing the immediate revocation of certificates that are not BR compliant when they do not pose an urgent security concern. Therefore, we request that your CA perform careful analysis of the situation. If there is justification to not revoke the problematic certificates, then explain those reasons and provide a timeline for when the bulks of the certificates will expire or be revoked/replaced. 

We expect that your forthcoming audit statements will indicate the findings of these problems. If your CA will not be revoking the certificates within 24 hours in accordance with the BRs, then that will also need to be listed as a finding in your CA’s BR audit statement.

We expect that your CA will work with your auditor (and supervisory body, as appropriate) and the Root Store(s) that your CA participates in to ensure your analysis of the risk and plan of remediation is acceptable. If your CA will not be revoking the problematic certificates as required by the BRs, then we recommend that you also contact the other root programs that your CA participates in to acknowledge this non-compliance and discuss what expectations their Root Programs have with respect to these certificates.


The problems reported for your CA in the mozilla.dev.security.policy forum are as follows:

** Failure to respond within 24 hours after Problem Report submitted
https://groups.google.com/d/msg/mozilla.dev.security.policy/PrsDfS8AMEk/w2AMK81jAQAJ
The problems were reported via your CA’s Problem Reporting Mechanism as listed here:
https://ccadb-public.secure.force.com/mozilla/CAInformationReport
Therefore, if this is the first time you have received notice of the problem(s) listed below, please review and fix your CA’s Problem Reporting Mechanism to ensure that it will work the next time someone reports a problem like this.

** Invalid dnsNames (e.g. invalid characters, internal names, and wildcards in the wrong position)
https://groups.google.com/d/msg/mozilla.dev.security.policy/CfyeeybBz9c/lmmUT4x2CAAJ
https://groups.google.com/d/msg/mozilla.dev.security.policy/D0poUHqiYMw/Pf5p0kB7CAAJ

** URI in dNSName SAN
https://groups.google.com/d/msg/mozilla.dev.security.policy/etp2Yz2fmM4/ayBTsfJnBgAJ

Comment 1

[Ramiro Muñoz Muñoz]

Attachment: Requerimiento Agosto 2017.docx

Comment 2

[Ramiro Muñoz Muñoz]

Hi 
Here you are AC Camerfirma answers to this request (there is a doc attached to this bug):

AC Camerfirma was aware via the problem reporting email address places in https://ccadb-public.secure.force.com/mozilla/CAInformationReport.

Ac Camerfirma stop issuing more certificates with this problem and replace the problematic certificates issued coordinating with our customers. 

Problematic certificates issued are to do with the DNS name that include HTTP://, HTTPS:// or characters not allowed. We consider that those certificate do not produce a security critical problem to our users, so we avoid to revoke the certificates immediately without notifying our customers, and replacing the wrong certificate.
ID	DNS Name	Problema	CT?	REVOKED
19953712	sede electrónica del excmo ayuntamiento de palos de la frontera	DNSNAME ERROR	NO	YES
42531587	http://sede.palmasport.es/sedeelectronica/	DNSNAME ERROR	NO	YES
12555916	carpeta.colmenarviejo.es/gdcarpetaciudadano	DNSNAME ERROR	NO	YES
11796736	https://sede.ayto-meco.es	DNSNAME ERROR	NO	YES
116010910	one vision imaging ltd	DNSNAME ERROR	NO	YES
118466681	one vision imaging ltd	DNSNAME ERROR	NO	YES
112942986	ccc holding gmbh	DNSNAME ERROR	NO	YES
114181316	ccc holding gmbh	DNSNAME ERROR	NO	YES
112929021	ccc holding gmbh	DNSNAME ERROR	NO	YES
 	 	 	 	 
7388249	sede.apalmeria.gob.es/	DNSNAME ERROR	NO	PENDING
 	 	 	 	 
8322063	autodiscover.asisa.es , autodiscover.clinicacitralosnaranjos.es , autodiscover.grupoasisa.local , autodiscover.hospitalmoncloa.es , correo.clinicacitralosnaranjos.es , correo.grupoasisa.local , correo.hospitalmoncloa.es , correo365.asisa.es , asisa.es , asisa.pt , autodiscover.asisa.pt , asisadental.pt , autodiscover.asisadental.pt , correo.asisa.pt , correo.asisadental.pt , correo.asisa.es 	Warning. 
Cannot find the special name in SAN	NO	NO
 	 	 	 	 
8680123	asisa.es , autodiscover.asisa.es , autodiscover.clinicacitralosnaranjos.es , autodiscover.grupoasisa.local , autodiscover.hospitalmoncloa.es , correo.clinicacitralosnaranjos.es , correo.grupoasisa.local , correo.hospitalmoncloa.es , correo365.asisa.es , asisa.pt , autodiscover.asisa.pt , asisadental.pt , autodiscover.asisadental.pt , correo.asisa.pt , correo.asisadental.pt , correo.asisercard.es , autodiscover.asisercard.es , correo.clinicavistahermosa.es , autodiscover.clinicavistahermosa.es , correo.asisa.es	Warning. 
Cannot find the special name in SAN	NO	NO
 	 	 	 	 
5129200	https://sede.arandadeduero.es  - https://crt.sh/?id=5129200&opt=cablint	DNSNAME ERROR	NO	YES
42531587	http://sede.palmasport.es/sedeelectronica/ - https://crt.sh/?id=42531587&opt=cablint	DNSNAME ERROR	NO	YES
112929021	CCC HOLDING GMBH - https://crt.sh/?id=112929021&opt=cablint	DNSNAME ERROR	SI	Precertificate

AC Camerfirma are going to enforce a strong control over the request form field to avoid this kind of errors in a future, and inform to the AR operators to pay special attention to this issue.

Comment 3

[Ryan Sleevi]

(In reply to Ramiro Muñoz Muñoz from comment #2)
> Hi 
> Here you are AC Camerfirma answers to this request (there is a doc attached
> to this bug):
> 
> AC Camerfirma was aware via the problem reporting email address places in
> https://ccadb-public.secure.force.com/mozilla/CAInformationReport.
> 
> Ac Camerfirma stop issuing more certificates with this problem and replace
> the problematic certificates issued coordinating with our customers. 
> 
> Problematic certificates issued are to do with the DNS name that include
> HTTP://, HTTPS:// or characters not allowed. We consider that those
> certificate do not produce a security critical problem to our users, so we
> avoid to revoke the certificates immediately without notifying our
> customers, and replacing the wrong certificate.
> ID	DNS Name	Problema	CT?	REVOKED
> 19953712	sede electrónica del excmo ayuntamiento de palos de la frontera
> DNSNAME ERROR	NO	YES
> 42531587	http://sede.palmasport.es/sedeelectronica/	DNSNAME ERROR	NO	YES
> 12555916	carpeta.colmenarviejo.es/gdcarpetaciudadano	DNSNAME ERROR	NO	YES
> 11796736	https://sede.ayto-meco.es	DNSNAME ERROR	NO	YES
> 116010910	one vision imaging ltd	DNSNAME ERROR	NO	YES
> 118466681	one vision imaging ltd	DNSNAME ERROR	NO	YES
> 112942986	ccc holding gmbh	DNSNAME ERROR	NO	YES
> 114181316	ccc holding gmbh	DNSNAME ERROR	NO	YES
> 112929021	ccc holding gmbh	DNSNAME ERROR	NO	YES
>  	 	 	 	 
> 7388249	sede.apalmeria.gob.es/	DNSNAME ERROR	NO	PENDING
>  	 	 	 	 
> 8322063	autodiscover.asisa.es , autodiscover.clinicacitralosnaranjos.es ,
> autodiscover.grupoasisa.local , autodiscover.hospitalmoncloa.es ,
> correo.clinicacitralosnaranjos.es , correo.grupoasisa.local ,
> correo.hospitalmoncloa.es , correo365.asisa.es , asisa.es , asisa.pt ,
> autodiscover.asisa.pt , asisadental.pt , autodiscover.asisadental.pt ,
> correo.asisa.pt , correo.asisadental.pt , correo.asisa.es 	Warning. 
> Cannot find the special name in SAN	NO	NO
>  	 	 	 	 
> 8680123	asisa.es , autodiscover.asisa.es ,
> autodiscover.clinicacitralosnaranjos.es , autodiscover.grupoasisa.local ,
> autodiscover.hospitalmoncloa.es , correo.clinicacitralosnaranjos.es ,
> correo.grupoasisa.local , correo.hospitalmoncloa.es , correo365.asisa.es ,
> asisa.pt , autodiscover.asisa.pt , asisadental.pt ,
> autodiscover.asisadental.pt , correo.asisa.pt , correo.asisadental.pt ,
> correo.asisercard.es , autodiscover.asisercard.es ,
> correo.clinicavistahermosa.es , autodiscover.clinicavistahermosa.es ,
> correo.asisa.es	Warning. 
> Cannot find the special name in SAN	NO	NO
>  	 	 	 	 
> 5129200	https://sede.arandadeduero.es  -
> https://crt.sh/?id=5129200&opt=cablint	DNSNAME ERROR	NO	YES
> 42531587	http://sede.palmasport.es/sedeelectronica/ -
> https://crt.sh/?id=42531587&opt=cablint	DNSNAME ERROR	NO	YES
> 112929021	CCC HOLDING GMBH - https://crt.sh/?id=112929021&opt=cablint
> DNSNAME ERROR	SI	Precertificate
> 
> AC Camerfirma are going to enforce a strong control over the request form
> field to avoid this kind of errors in a future, and inform to the AR
> operators to pay special attention to this issue.

Thanks for responding. I think it's still necessary to provide additional detail.

That is, we can look at the problem from two dimensions: The problem itself, and the systemic issues that allowed the problem to manifest. Your description focuses on the resolution of the problem, but doesn't indicate any systemic changes have been made. As a consequence, it does not help the community feel that your CA has taken steps to reduce the risk of future violations (of any requirement) in the future. That is, one dimension is "Did you fix the bug", but another dimension is "How was the bug introduced, why was it not detected, and what steps are you taking to prevent future bugs"

To understand how you might approach this problem, consider https://groups.google.com/d/msg/mozilla.dev.security.policy/vl5eq0PoJxY/W1D4oZ__BwAJ and how it provided a timeline of events, the steps that were already in place (and with substantial detail), where there were controls missing or mistakes made, details about the steps being taken (e.g. "We fixed the bug" is not sufficient detail to understand), and a holistic, systemic awareness of how the CA is managed and the opportunities for errors.

In this particular case:
- It sounds like AR operators are allowed to directly enter the names in the certificates, is that correct?
- What technical controls exist to prevent mistakes like this?
- If no technical controls exist, what technical controls are being introduced to prevent issues like this/
- Systemically, why did AC Camerfirma not detect these issues, for example, as part of its internal review procedures?
- What changes are being made to its review procedures to provide assurance to the community that it will be more proactive in monitoring and responding to misissuance?
- What steps are being taken, systemically, to prevent misissued certificates? For example, the introduction of tools such as certlint to review the tbsCertificate prior to issuance?
- What steps are being taken, systemically, to provide transparency to the community about AC Camerfirma's operations and commitment to security? For example, logging all present and future valid certificates to Certificate Transparency logs.
- Please review steps 3-6 on the original report to ensure there are sufficiently detailed answers to these requests.

Comment 4

[Gervase Markham [:gerv]]

Alex: please can you give the details of your submitted Problem Report and the associated timelines, so any problems there can also be investigated in the context of this bug?

Gerv

Comment 5

[Ramiro Muñoz Muñoz]

Hi
Our AR operators can modify request from our PKI platform before issuing a certificate when they detect any mistake in the request.

We have many AR working (STARTCOM included) and they can validate SSL OV, EV certificates. AC Camerfirma also have its own RA, and do a second verification process in all EV certificates.

We are going to work with our auditor to provide you with a fully answer as you request. At the moment our auditors are not available due to the summer holiday. Hope to have our report in a couple of week if this is ok for you, let me know it if not.

All the EV certificates issued by camerfirma has the CT controls, also most of the OV certificates have the CT verification. AC Camerfirma do not issue DV certificates. We are going to extend this year the CT verification to all SSL certificates issued

AC Camerfirma are to include a certlint automatic verification for any SSL certificate issued (this will take us some time), meanwhile we are going to have a daily extra verification process by our internal AR Operators over all SSL certificated issued to detect any possible mistake made by an external RA operator.

We already have revoked  the wrong certificates (Only one left to be revoked). 

Regards
Ramiro

Comment 6

[Jonathan Rudenberg]

For reference, I submitted a problem report via email with these headers:

  From: Jonathan Rudenberg <[e-mail]>
  Subject: Misissuance - invalid dnsNames
  Message-Id: <[e-mail]>
  Date: Sun, 13 Aug 2017 00:32:39 -0400
  To: [e-mail]

Comment 7

[Ryan Sleevi]

(In reply to Ramiro Muñoz Muñoz from comment #5)
> We are going to work with our auditor to provide you with a fully answer as
> you request. At the moment our auditors are not available due to the summer
> holiday. Hope to have our report in a couple of week if this is ok for you,
> let me know it if not.

Thanks for your reply. However, I'm uncertain why your auditor needs to be involved. That is, many of these questions relate to how Camerfirma is operating and maintaining its systems, and while your auditors may be involved to confirm the changes are as you describe, certainly, describing your system and the mitigations you have, why they failed, and their future plan should be something possible. This is certainly true that in the event of more pressing misissuance, these are the same questions that would be asked, and for which it would be time-critical to provide answers to (even if incomplete or subsequently updated with additional details)

I'll note that many CAs have provided replies within the past 24 hours to the questions asked. It would be good if you could reply in-line (using the "Reply" button available on Comment #0 and Comment #3) to ensure that each question has been answered.

Comment 8

[Alex Gaynor [:Alex_Gaynor]]

Gerv,

On July 30th, I sent an email to Camerfirma (from my personal email) listing three certificates that were non-compliant due to invalid hostnames, and requesting they be revoked. Camerfirma did not respond to that email at any time. If more metadata for this email is required, I can dig it up.

The only other direct contact I had with them was the email thread you started, "Problematic Camerfirma certs", from August 15th.

Comment 9

[Ramiro Muñoz Muñoz]

Alex,

I received you email On July 30th, ok, but you should have an answer from me.
I wrote an email to Gerv and you on August 16th confirming the revocation of those certificates. I was delayed in my answer since I was on holidays those days and also we had to get in contact with our customers to replace the wrong certificates.
I got an email from you on August 16th notifying that one of this certificates was not revoked and also there is an email from me to you on August 18th answering you that it was a precertificate and no revocation was issued. Nevertheless this precertificates is already revoked. At the moment all certificates you report are revoked.
Sorry for the delay due to the vacation period and also because we cannot revoke a certificate without notifying our customers and replacing it for a new one much less when the wrong certificate no means a security risk neither for our customer nor the end user.
Now we are writing the report as is requested by Kathleen and Ryan. Hope finishing along this Spanish-time morning.
Best regards
Ramiro

Comment 10

[Ramiro Muñoz Muñoz]

Attachment: answer3.csv - Complete list of certificates that your CA finds with each of the listed issues during the remediation process

Comment 11

[Ramiro Muñoz Muñoz]

Attachment: answer4.xlsx - Summary of the problematic certificates

Comment 12

[Ramiro Muñoz Muñoz]

1)	How your CA first became aware of the problems listed below (e.g. via a Problem Report, via the discussion in mozilla.dev.security.policy, or via this Bugzilla Bug), and the date.
** URI in dNSName SAN
https://groups.google.com/d/msg/mozilla.dev.security.policy/etp2Yz2fmM4/ayBTsfJnBgAJ
We received information by email form Gervase Markham [e-mail] and Buzilla 20/7. 21/07 I answered Gerv and Katleen. 
Certificates was revoked once we could replace for new ones.
** Invalid dnsNames (e.g. invalid characters, internal names, and wildcards in the wrong position)
https://groups.google.com/d/msg/mozilla.dev.security.policy/CfyeeybBz9c/lmmUT4x2CAAJ
We received an email fom Jonathan Rudenberg <[e-mail]> 16th/08.
** https://groups.google.com/d/msg/mozilla.dev.security.policy/D0poUHqiYMw/Pf5p0kB7CAAJ
email from Kethleen Bugzilla 16th/8


2) Prompt confirmation that your CA has stopped issuing TLS/SSL certificates with the problems listed below.
Ac Camerfirma stopped issuing more certificates with this problem and replace the problematic certificates issued coordinating with our customers.
Problematic certificates issued are to do with the DNS name that include HTTP://, HTTPS:// or characters not allowed. We consider that those certificate do not produce a security risk to our customers, so we avoid to revoke the certificates immediately without notifying our customers, and replacing the wrong certificates.

3) Complete list of certificates that your CA finds with each of the listed issues during the remediation process. The recommended way to handle this is to ensure each certificate is logged to CT and then attach a CSV file/spreadsheet of the fingerprints or crt.sh IDs, with one list per distinct problem.
Attached file answer3.csv. Not all certificate has logged to CT.

4) Summary of the problematic certificates. For each problem listed below: number of certs, date first and last certs with that problem were issued.
Attached file answer4.xlsx

5) Explanation about how and why the mistakes were made, and not caught and fixed earlier.
There is no technical control about the DNSName input field in the request form. AR Operators have to review that the DNSName fulfil the policy requirement. Those certificates have a random review from our approval AR operators but none of the wrong certificates was detected. We neider have received incident reports from our customers. 
We can say that the origin of this problem is an external AR operator mistake. 

6) List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

•	Our approval AR Operators will carry out a full review of all DNSName certificates field issued. (Immediately).
•	Develop a technical control in the DNSName request field. (3 months).
•	Develop a control for each SSL/TSL certificates issued running a CA/B Forum lint and x509 lint in order to detect DNSName and other errors immediately after issuing a certificate. (3 months) 
•	All SSL/TSL Certificates issued at the moment have CT controls.
•	We will review our Problem Reporting Mechanism email address. We have been noticed to my personal email address, not by the established channel. We will create a new problem reporting email to notify to all the key people.

7) Regular updates to confirm when those steps have been completed.

We already started to verify in a daily basis all DNSNames issued. We do not issue a big number of SSL/TSL certificate so we can afford this.
Technical control will take us 2 or 3 month to be working.
Once we can detect any wrong certificate AC Camerfirma will proceed to notify before 24 hours and revoke them.

Best Regards
Ramiro Muñoz

Comment 13

[Ryan Sleevi]

> 4) Summary of the problematic certificates. For each problem listed below:
> number of certs, date first and last certs with that problem were issued.
> Attached file answer4.xlsx

https://crt.sh/?id=8322063&opt=cablint is not revoked, but contains two internal server names (correo.grupoasisa.local and autodiscover.grupoasisa.local). Per the Baseline Requirements 7.1.4.2.1 (using BR version 1.3.2 as an example), this was required to be revoked as of 1 October 2016.

https://crt.sh/?id=8680123&opt=cablint is not revoked, and also contains the same two internal server names.

The Baseline Requirements stated that, as of the effective date, CAs SHALL NOT issue such certificates with an Expiry Date later than 1 November 2015, yet these certificates expire in 2018. Why is that?

> 6) List of steps your CA is taking to resolve the situation and ensure such
> issuance will not be repeated in the future, accompanied with a timeline of
> when your CA expects to accomplish these things.
> 
> •	Our approval AR Operators will carry out a full review of all DNSName
> certificates field issued. (Immediately).

- Is this a technical review, or a non-technical review?
- Are there any assurances that the domain itself was appropriately validated? Based on the nature of these incidents, it appears that Camerfirma does not have the necessary or technical controls to ensure appropriate compliance.

> •	Develop a technical control in the DNSName request field. (3 months).

- Why will this take 3 months? Please provide more detail, given other CAs have been able to deploy changes within hours, days, or weeks.

> •	Develop a control for each SSL/TSL certificates issued running a CA/B
> Forum lint and x509 lint in order to detect DNSName and other errors
> immediately after issuing a certificate. (3 months) 

- Why is this not done prior to issuance? Checking after issuance is still misissuance, so while this is an important secondary detection mechanism, it is not a prevention or mitigation mechanism.

> 7) Regular updates to confirm when those steps have been completed.
> 
> We already started to verify in a daily basis all DNSNames issued.

- Please indicate how you are verifying these domains, and what you're verifying?
  - Compliance with RFC5280 and the 'preferred name syntax'
  - An appropriate demonstration of domain control
  - That the domain does not contain internal server names (which appeared to have been undetected by Camerfirma's compliance team in reviewing the flagged certificates).

> We do not
> issue a big number of SSL/TSL certificate so we can afford this.
> Technical control will take us 2 or 3 month to be working.
> Once we can detect any wrong certificate AC Camerfirma will proceed to
> notify before 24 hours and revoke them.

- As a mitigation, this would not be suitable. Not issuing the certificate in the first place is the expected result.
- Echoing the earlier remark, can you explain why it will take 2-3 months to be implemented?

For example, a suitable mitigation may be to require automatic demonstration of domain control, multi-party review in the case where that's not acceptable, automated checking of the well-formedness of the name (wildcards, preferred name syntax, valid TLD, etc) - in addition to the certlint/x509lint mitigations.

Comment 14

[Ramiro Muñoz Muñoz]

(In reply to Ryan Sleevi from comment #13)
> > 4) Summary of the problematic certificates. For each problem listed below:
> > number of certs, date first and last certs with that problem were issued.
> > Attached file answer4.xlsx
> 
> https://crt.sh/?id=8322063&opt=cablint is not revoked, but contains two
> internal server names (correo.grupoasisa.local and
> autodiscover.grupoasisa.local). Per the Baseline Requirements 7.1.4.2.1
> (using BR version 1.3.2 as an example), this was required to be revoked as
> of 1 October 2016.
> 
> https://crt.sh/?id=8680123&opt=cablint is not revoked, and also contains the
> same two internal server names.
> 
> The Baseline Requirements stated that, as of the effective date, CAs SHALL
> NOT issue such certificates with an Expiry Date later than 1 November 2015,
> yet these certificates expire in 2018. Why is that?

Obviously is an error in the validation process. We will revoke those certificates.

> > 6) List of steps your CA is taking to resolve the situation and ensure such
> > issuance will not be repeated in the future, accompanied with a timeline of
> > when your CA expects to accomplish these things.
> > 
> > •	Our approval AR Operators will carry out a full review of all DNSName
> > certificates field issued. (Immediately).
> 
> - Is this a technical review, or a non-technical review?
> - Are there any assurances that the domain itself was appropriately
> validated? Based on the nature of these incidents, it appears that
> Camerfirma does not have the necessary or technical controls to ensure
> appropriate compliance.

It is a non-technical control, but this second validation proposed could mitigate the problem. Internal approval AR operators are more experienced staff. 
In this specific case (DNS Name) there are no technical controls since our AR Operators validate one by one each DNSName. 
We propose this action (second validation)to avoid errors since it can be done immediately. Technical control deployment take us some more time.

 > > •	Develop a technical control in the DNSName request field. (3 months).
> 
> - Why will this take 3 months? Please provide more detail, given other CAs
> have been able to deploy changes within hours, days, or weeks.
We are to include all BR requirement validation over all DNSNames found in the request form and this take some codification time. It was the answer from our PKI developers, anyway I will try to give priority to deploy this technical controls in weeks.
> > •	Develop a control for each SSL/TSL certificates issued running a CA/B
> > Forum lint and x509 lint in order to detect DNSName and other errors
> > immediately after issuing a certificate. (3 months) 
> 
> - Why is this not done prior to issuance? Checking after issuance is still
> misissuance, so while this is an important secondary detection mechanism, it
> is not a prevention or mitigation mechanism.

After certificate issuing we can use the cablint tool, this can speed up the process. We would revoke immediately problematic certificates before distribute it, so in some way it could be a mitigation mechanism. Otherwise we should code all those controls before issuing the certificate and this take us an unpredictable development time.

Nevertheless a technical control over the DNSname as we proposed before could solve the DNSName problem.

> > 7) Regular updates to confirm when those steps have been completed.
> > 
> > We already started to verify in a daily basis all DNSNames issued.
> 
> - Please indicate how you are verifying these domains, and what you're
> verifying?
>   - Compliance with RFC5280 and the 'preferred name syntax'
>   - An appropriate demonstration of domain control
>   - That the domain does not contain internal server names (which appeared
> to have been undetected by Camerfirma's compliance team in reviewing the
> flagged certificates).

Our Internal approval AR operators will validate the correct syntax and other validations in the request. Domain control is also verified (notifying domain contacts). We propose enforce the second validation process over OV SSL/TSL certificates to reduce those problems.

> > We do not
> > issue a big number of SSL/TSL certificate so we can afford this.
> > Technical control will take us 2 or 3 month to be working.
> > Once we can detect any wrong certificate AC Camerfirma will proceed to
> > notify before 24 hours and revoke them.
> 
> - As a mitigation, this would not be suitable. Not issuing the certificate
> in the first place is the expected result.
> - Echoing the earlier remark, can you explain why it will take 2-3 months to
> be implemented?
> 
> For example, a suitable mitigation may be to require automatic demonstration
> of domain control, multi-party review in the case where that's not
> acceptable, automated checking of the well-formedness of the name
> (wildcards, preferred name syntax, valid TLD, etc) - in addition to the
> certlint / x509lint mitigations.

Automatic demonstration of the domain control is already done by the PKI platform. We send automatically a communication to the domain contacts and wait for approval using a random code, without this we cannot validate a request.
Multi-party is what we are proposing while our developers deploy the automatic BR Domain Name validation. 
We also have an automatic double validation for our EV certificates that we could use this process in the OV validation process.

Comment 15

[Ryan Sleevi]

(In reply to Ramiro Muñoz Muñoz from comment #14)
> (In reply to Ryan Sleevi from comment #13)
> > The Baseline Requirements stated that, as of the effective date, CAs SHALL
> > NOT issue such certificates with an Expiry Date later than 1 November 2015,
> > yet these certificates expire in 2018. Why is that?
> 
> Obviously is an error in the validation process. We will revoke those
> certificates.

Well, yes, this is an error.

The question, however, is:
1) Why weren't these revoked when originally reported?
2) Why weren't these revoked when previously required?
3) Why weren't these limited in their validity period?

That is, an answer of "mistakes were made" does not demonstrate the level of introspection, analysis, and transparency necessary for public trust. It's clear from the evidence that mistakes were made, what is not clear is why these mistakes they were made, how they were made, and what steps are being taken to prevent these future mistakes?

That is, in the report, the certlint information was available. What analysis was done that lead to the conclusion not to revoke these certificates? What level of secondary review was conducted? Were your auditors (or conformance assessment body) involved in that decision?

Understanding how the CA has made the decision not to revoke, and how it will make future decisions regarding compliance, is an essential part of public trust.

> > > •	Develop a control for each SSL/TSL certificates issued running a CA/B
> > > Forum lint and x509 lint in order to detect DNSName and other errors
> > > immediately after issuing a certificate. (3 months) 
> > 
> > - Why is this not done prior to issuance? Checking after issuance is still
> > misissuance, so while this is an important secondary detection mechanism, it
> > is not a prevention or mitigation mechanism.
> 
> After certificate issuing we can use the cablint tool, this can speed up the
> process. We would revoke immediately problematic certificates before
> distribute it, so in some way it could be a mitigation mechanism. 

Note that solely replying on mitigation mechanisms, such as revocation, are insufficient for continued trust. CAs are expected to avoid these issues in the first place, particularly when technical controls to prevent issuance exist.

Are you stating that your systems are that unknown that you cannot provide an estimate as to when you can implement technical controls to adhere to the Baseline Requirements, and that your issuance practices will rely on the expertise of your RAs to understand and internalize all of the direct requirements of the Baseline Requirements and indirect requirements from the relevant specifications?

> Otherwise
> we should code all those controls before issuing the certificate and this
> take us an unpredictable development time.

Can you not integrate a single control over the tbsCertificate prior to signing?

If not, can you please share further details about your CA operation software. For example, is it using a 'common off the shelf' system (such as EJBCA or Microsoft CA server), or is it using an in-house developed CA?

The trustworthiness of a CA organization is, in part, derived from the trustworthiness of its infrastructure. If a CA finds it difficult to implement or integrate controls, this poses both direct risk to the ecosystem (from situations such as this) and indirect risk (such as due to delaying requirements to allow such CAs sufficient time), and may be better suited by migrating trust away from such CAs.

I say that to help emphasize that public trust is a rather serious matter, and an important part of that trust is taking all reasonable steps to ensure a globally-secure system and infrastructure is in place. If the costs of such a system are not practical, it may be that providing global trust is not an appropriate solution.

> > > 7) Regular updates to confirm when those steps have been completed.
> > > 
> > > We already started to verify in a daily basis all DNSNames issued.
> > 
> > - Please indicate how you are verifying these domains, and what you're
> > verifying?
> >   - Compliance with RFC5280 and the 'preferred name syntax'
> >   - An appropriate demonstration of domain control
> >   - That the domain does not contain internal server names (which appeared
> > to have been undetected by Camerfirma's compliance team in reviewing the
> > flagged certificates).
> 
> Our Internal approval AR operators will validate the correct syntax and
> other validations in the request. Domain control is also verified (notifying
> domain contacts). We propose enforce the second validation process over OV
> SSL/TSL certificates to reduce those problems.

To confirm: Your answer indicates that RA staff are validating everything by hand, and that compliance and consistency are solely determined by human factors, is that correct? As such, any failure of training, or any human error, can lead to misissued certificates.

> > For example, a suitable mitigation may be to require automatic demonstration
> > of domain control, multi-party review in the case where that's not
> > acceptable, automated checking of the well-formedness of the name
> > (wildcards, preferred name syntax, valid TLD, etc) - in addition to the
> > certlint / x509lint mitigations.
> 
> Automatic demonstration of the domain control is already done by the PKI
> platform. We send automatically a communication to the domain contacts and
> wait for approval using a random code, without this we cannot validate a
> request.

This is a useful detail. In the spirit of analyzing the root issue, if this process was followed, how was it possible for a certificate such as https://crt.sh/?q=8680123 (which is _still_ not revoked, despite Comment #14) to be issued? There is no valid way for this domain to have been validated, which suggests that you have one or more critical security vulnerabilities in how you validate domain names, or that the answer may be incomplete?

> Multi-party is what we are proposing while our developers deploy the
> automatic BR Domain Name validation. 
> We also have an automatic double validation for our EV certificates that we
> could use this process in the OV validation process.

Comment 16

[Ramiro Muñoz Muñoz]

(In reply to Ryan Sleevi from comment #15)
> (In reply to Ramiro Muñoz Muñoz from comment #14)
> > (In reply to Ryan Sleevi from comment #13)
> > > The Baseline Requirements stated that, as of the effective date, CAs SHALL
> > > NOT issue such certificates with an Expiry Date later than 1 November 2015,
> > > yet these certificates expire in 2018. Why is that?
> > 
> > Obviously is an error in the validation process. We will revoke those
> > certificates.
> 
> Well, yes, this is an error.
> 
> The question, however, is:
> 1) Why weren't these revoked when originally reported?
> 2) Why weren't these revoked when previously required?
> 3) Why weren't these limited in their validity period?
> 
> That is, an answer of "mistakes were made" does not demonstrate the level of
> introspection, analysis, and transparency necessary for public trust. It's
> clear from the evidence that mistakes were made, what is not clear is why
> these mistakes they were made, how they were made, and what steps are being
> taken to prevent these future mistakes?
> 
> That is, in the report, the certlint information was available. What
> analysis was done that lead to the conclusion not to revoke these
> certificates? What level of secondary review was conducted? Were your
> auditors (or conformance assessment body) involved in that decision?
> 
> Understanding how the CA has made the decision not to revoke, and how it
> will make future decisions regarding compliance, is an essential part of
> public trust.
> 
As we already explain on previous answers, we need to inform to our customer before certificate revocation, in order to avoid any damage. Our customer has imposed us not revoke the certificate until the replacement was complete. We as a CA are in the middle between our customer and the browsers requirement. We have opened an issue with our legal department to include in the contract with our customers a speedy certificate revocation procedure. 
RA Operator mistakes are possible so that why we review the 3% of the “all” SSL/TSL certificates issued, in this case this certificate was not detected in this 3% review.
AC Camerfirma is right now reviewing “all” SSL/TSL certificates issued (much more that the 3%).
AC Camerfirma is going to deploy this week a technical control to satisfy the BR requirement about the DNSName.

With these controls, we think that the DNSName issue is fixed and enough to guarantee the public trust.
Keep in mind that Ac Camerfirma also include in all SSL/TSL certificates the STC from the certificate transparency.

> > > > •	Develop a control for each SSL/TSL certificates issued running a CA/B
> > > > Forum lint and x509 lint in order to detect DNSName and other errors
> > > > immediately after issuing a certificate. (3 months) 
> > > 
> > > - Why is this not done prior to issuance? Checking after issuance is still
> > > misissuance, so while this is an important secondary detection mechanism, it
> > > is not a prevention or mitigation mechanism.
> > 
> > After certificate issuing we can use the cablint tool, this can speed up the
> > process. We would revoke immediately problematic certificates before
> > distribute it, so in some way it could be a mitigation mechanism. 
> 
> Note that solely replying on mitigation mechanisms, such as revocation, are
> insufficient for continued trust. CAs are expected to avoid these issues in
> the first place, particularly when technical controls to prevent issuance
> exist.
> 
> Are you stating that your systems are that unknown that you cannot provide
> an estimate as to when you can implement technical controls to adhere to the
> Baseline Requirements, and that your issuance practices will rely on the
> expertise of your RAs to understand and internalize all of the direct
> requirements of the Baseline Requirements and indirect requirements from the
> relevant specifications?

Well now we rely on the RA expertise and a second operator validation to adhere to BR, I do know if technical control are mandatory. Our number of SSL certificates issued is not so high so we can afford this. Nevertheless I think that technical control can help us to speed up the issuing process and we are going to integrate cablint control in our PKI platform. This can cover every certificate issued and detect any codification problem. 
> 
> > Otherwise
> > we should code all those controls before issuing the certificate and this
> > take us an unpredictable development time.
> 
> Can you not integrate a single control over the tbsCertificate prior to
> signing?

Yes we can do it, but we need to evaluate the effort. That’s what our programmers are doing right now. I have not all information but it appeared that is not a simple integration. 
> 
> If not, can you please share further details about your CA operation
> software. For example, is it using a 'common off the shelf' system (such as
> EJBCA or Microsoft CA server), or is it using an in-house developed CA?

Is an in-house developed CA
> 
> The trustworthiness of a CA organization is, in part, derived from the
> trustworthiness of its infrastructure. If a CA finds it difficult to
> implement or integrate controls, this poses both direct risk to the
> ecosystem (from situations such as this) and indirect risk (such as due to
> delaying requirements to allow such CAs sufficient time), and may be better
> suited by migrating trust away from such CAs.
> 
> I say that to help emphasize that public trust is a rather serious matter,
> and an important part of that trust is taking all reasonable steps to ensure
> a globally-secure system and infrastructure is in place. If the costs of
> such a system are not practical, it may be that providing global trust is
> not an appropriate solution.

AC Camerfirma has been working with this technology for 17 years without problems and we have dedicated a big effort improving it along those years. 

It’s not a matter of difficulty to integrate new controls, we can do it, but we first of all need to evaluate changes, that why we cannot provide a specific date.

> 
> > > > 7) Regular updates to confirm when those steps have been completed.
> > > > 
> > > > We already started to verify in a daily basis all DNSNames issued.
> > > 
> > > - Please indicate how you are verifying these domains, and what you're
> > > verifying?
> > >   - Compliance with RFC5280 and the 'preferred name syntax'
> > >   - An appropriate demonstration of domain control
> > >   - That the domain does not contain internal server names (which appeared
> > > to have been undetected by Camerfirma's compliance team in reviewing the
> > > flagged certificates).
> > 
> > Our Internal approval AR operators will validate the correct syntax and
> > other validations in the request. Domain control is also verified (notifying
> > domain contacts). We propose enforce the second validation process over OV
> > SSL/TSL certificates to reduce those problems.
> 
> To confirm: Your answer indicates that RA staff are validating everything by
> hand, and that compliance and consistency are solely determined by human
> factors, is that correct? As such, any failure of training, or any human
> error, can lead to misissued certificates.


Yes that’s why we are doing now a second validation process for all SSL/TSL certificated issued, meanwhile we have already place into operation the (DNSName) technical controls.

> 
> > > For example, a suitable mitigation may be to require automatic demonstration
> > > of domain control, multi-party review in the case where that's not
> > > acceptable, automated checking of the well-formedness of the name
> > > (wildcards, preferred name syntax, valid TLD, etc) - in addition to the
> > > certlint / x509lint mitigations.
> > 
> > Automatic demonstration of the domain control is already done by the PKI
> > platform. We send automatically a communication to the domain contacts and
> > wait for approval using a random code, without this we cannot validate a
> > request.
> 
> This is a useful detail. In the spirit of analyzing the root issue, if this
> process was followed, how was it possible for a certificate such as
> https://crt.sh/?q=8680123 (which is _still_ not revoked, despite Comment
> #14) to be issued? 

This certificate was issued on Jul 23 2015 I am afraid that these controls was not working by those dates.

> There is no valid way for this domain to have been
> validated, which suggests that you have one or more critical security
> vulnerabilities in how you validate domain names, or that the answer may be
> incomplete?
> 
It was a mistake from our RA Operator that was not controlled by the 3% review process. We already have explained what to do, and what we have done to avoid this problem in the future.

> > Multi-party is what we are proposing while our developers deploy the
> > automatic BR Domain Name validation. 
> > We also have an automatic double validation for our EV certificates that we
> > could use this process in the OV validation process.

Comment 17

[Ryan Sleevi]

(In reply to Ramiro Muñoz Muñoz from comment #16)
> As we already explain on previous answers, we need to inform to our customer
> before certificate revocation, in order to avoid any damage. Our customer
> has imposed us not revoke the certificate until the replacement was
> complete. We as a CA are in the middle between our customer and the browsers
> requirement. We have opened an issue with our legal department to include in
> the contract with our customers a speedy certificate revocation procedure. 

Could you explain what provisions you have that prevent you from revoking?

That is, the Baseline Requirements require revoking within 24 hours. They also require that, in the existence of a conflict between the BRs and your CP/CPS, the BRs override what's in your CP/CPS. Your subscriber agreement is required to include provisions for revocation based on your CP/CPS.

As such, there should be no reason that you _can't_ revoke, only when you choose not to revoke.

In this case, I was asking for the analysis about why this wasn't revoked as originally scheduled, and what policies and practices are in place to ensure future issues won't happen. These are all retrospective analyses that need to be done and that haven't been done, and look at the controls in place (which weren't sufficient) - and about understanding holistically how those controls failed.

Put differently, it's good to understand mistakes were made, and it's good to have fixes for those mistakes - but it's better to understand why the mistakes were made, and it's good to prevent future mistakes. That's missing here.

> RA Operator mistakes are possible so that why we review the 3% of the “all”
> SSL/TSL certificates issued, in this case this certificate was not detected
> in this 3% review.
> AC Camerfirma is right now reviewing “all” SSL/TSL certificates issued (much
> more that the 3%).
> AC Camerfirma is going to deploy this week a technical control to satisfy
> the BR requirement about the DNSName.

Has this been deployed yet?

> Well now we rely on the RA expertise and a second operator validation to
> adhere to BR, I do know if technical control are mandatory. Our number of
> SSL certificates issued is not so high so we can afford this. Nevertheless I
> think that technical control can help us to speed up the issuing process and
> we are going to integrate cablint control in our PKI platform. This can
> cover every certificate issued and detect any codification problem. 

This can address technical compliance, but doesn't address policy compliance. And given that the technical controls failed with the current system, it's difficult to imagine how the policy compliance will not have the same issues.


> > > Automatic demonstration of the domain control is already done by the PKI
> > > platform. We send automatically a communication to the domain contacts and
> > > wait for approval using a random code, without this we cannot validate a
> > > request.
> > 
> > This is a useful detail. In the spirit of analyzing the root issue, if this
> > process was followed, how was it possible for a certificate such as
> > https://crt.sh/?q=8680123 (which is _still_ not revoked, despite Comment
> > #14) to be issued? 
> 
> This certificate was issued on Jul 23 2015 I am afraid that these controls
> was not working by those dates.

So you anticipate that there are still-valid certificates issued with insufficient controls and thus may not meet either the policy or technical requirements of the Baseline Requirements?

I think more detail then "we had an issue" is warranted here, because the whole goal of this issue is to demonstrate Camerfirma's complete understanding of the issue (and their approach to past issues), to ensure that assurances given about future issues can be relied upon.

Comment 18

[Ryan Sleevi]

Attempting to summarize the information to date:

1) Invalid DNS Names (Invalid Characters - URLs)
- See Comment #0, Comment #1, Comment #12
- Remediation:
  - 2017-08-24 - Retrain staff on conformance requirements (see Comment #12)
  - 2017-08-24 - Require additional review after issuance (see Comment #12)
  - 2017-11-XX - Automatic compliance checking

2) Invalid DNS names (Invalid Names - e.g. company names)
- See Comment #0, Comment #1, Comment #12
- Remediation:
  - 2017-08-24 - Retrain staff on conformance requirements (see Comment #12)
  - 2017-08-24 - Require additional review after issuance (see Comment #12)
  - 2017-11-XX - Automatic compliance checking

Is this a correct summary so far?

Regarding the certificates you have not yet revoked, but contain internal server names - as these present critical security risk to the ecosystem, please revoke these immediately. This is the same that has been requested and expected of other CAs for this same issue. It is particularly concerning that, despite being presented with these certificates, Camerfirma did not detect the non-compliance, even under manual review. This raises significant concerns about the remediation process taken, as well as concerns for the remediations proposed.

Kathleen, I'm not sure how you want to handle this. I'm also concerned that certificates that were said were going to be revoked (for being internal server names - see Comment #14 "Obviously is an error in the validation process. We will revoke those certificates." - have not been revoked - see Comment #16), even as we're now a week later. I also don't believe these remediation plans demonstrate the necessary awareness of the root causes or provide sufficient assurance to prevent future issues.

Comment 19

[Ramiro Muñoz Muñoz]

(In reply to Ryan Sleevi from comment #17)
> (In reply to Ramiro Muñoz Muñoz from comment #16)
> > As we already explain on previous answers, we need to inform to our customer
> > before certificate revocation, in order to avoid any damage. Our customer
> > has imposed us not revoke the certificate until the replacement was
> > complete. We as a CA are in the middle between our customer and the browsers
> > requirement. We have opened an issue with our legal department to include in
> > the contract with our customers a speedy certificate revocation procedure. 
> 
> Could you explain what provisions you have that prevent you from revoking?
> 
> That is, the Baseline Requirements require revoking within 24 hours. They
> also require that, in the existence of a conflict between the BRs and your
> CP/CPS, the BRs override what's in your CP/CPS. Your subscriber agreement is
> required to include provisions for revocation based on your CP/CPS.
> 
> As such, there should be no reason that you _can't_ revoke, only when you
> choose not to revoke.
> 
> In this case, I was asking for the analysis about why this wasn't revoked as
> originally scheduled, and what policies and practices are in place to ensure
> future issues won't happen. These are all retrospective analyses that need
> to be done and that haven't been done, and look at the controls in place
> (which weren't sufficient) - and about understanding holistically how those
> controls failed.
> 
> Put differently, it's good to understand mistakes were made, and it's good
> to have fixes for those mistakes - but it's better to understand why the
> mistakes were made, and it's good to prevent future mistakes. That's missing
> here.
> 

We are working with our customer to revoke the certificates. One of them is already revoked and there is one left to revoke that will be revoked on September the 11th (already agreed with our customer). AC Camerfirma has replaced this certificate and must be installed before revoking the old one, as our customer imposed us.  There are many domains inside the certificate a many services affected.
AC Camerfirma want to avoid that this situation again so we are going to clarify our subscriber agreement to make clear this situation to our customers and protect AC Camerfirma legal responsibility when we unilaterally revoke the certificate.
Regarding the mistake, we have been trying to understand the problem, as you said. By the dates we issue the certificate our RA Operator, understand from our CPS, that the internal names was allowed until October 2016, a misunderstanding that we already clarifyed in our late CPS version. 

> > RA Operator mistakes are possible so that why we review the 3% of the “all”
> > SSL/TSL certificates issued, in this case this certificate was not detected
> > in this 3% review.
> > AC Camerfirma is right now reviewing “all” SSL/TSL certificates issued (much
> > more that the 3%).
> > AC Camerfirma is going to deploy this week a technical control to satisfy
> > the BR requirement about the DNSName.
> 
> Has this been deployed yet?
> 
Yes.
So, at the moment we have the DNSName technical control working and we review 100% the TSL/SSL certificates issued. I think this can offer enough guaranties to detect any future problem about this issue.
>
> > Well now we rely on the RA expertise and a second operator validation to
> > adhere to BR, I do know if technical control are mandatory. Our number of
> > SSL certificates issued is not so high so we can afford this. Nevertheless I
> > think that technical control can help us to speed up the issuing process and
> > we are going to integrate cablint control in our PKI platform. This can
> > cover every certificate issued and detect any codification problem. 
> 
> This can address technical compliance, but doesn't address policy
> compliance. And given that the technical controls failed with the current
> system, it's difficult to imagine how the policy compliance will not have
> the same issues.
> 
> 
> > > > Automatic demonstration of the domain control is already done by the PKI
> > > > platform. We send automatically a communication to the domain contacts and
> > > > wait for approval using a random code, without this we cannot validate a
> > > > request.
> > > 
> > > This is a useful detail. In the spirit of analyzing the root issue, if this
> > > process was followed, how was it possible for a certificate such as
> > > https://crt.sh/?q=8680123 (which is _still_ not revoked, despite Comment
> > > #14) to be issued? 
> > 
> > This certificate was issued on Jul 23 2015 I am afraid that these controls
> > was not working by those dates.
> 
> So you anticipate that there are still-valid certificates issued with
> insufficient controls and thus may not meet either the policy or technical
> requirements of the Baseline Requirements?
>
Obviously not. 
We have been deploying some technical control along the time, but control were carry out by other means, as our auditors said in the audit reports we have been providing over time.
>
> I think more detail then "we had an issue" is warranted here, because the
> whole goal of this issue is to demonstrate Camerfirma's complete
> understanding of the issue (and their approach to past issues), to ensure
> that assurances given about future issues can be relied upon.
I can assure you that AC Camerfirma understand completely the issue and we are working hard to offer a quality and secure services to our customers and to the community as our certification in ISO27001, ISO20000, WEBTRUST CA BR EV, ETSI 319401, ETSI 319411-1, ETSI 319411-2, ETSI 319 41121,  demonstrate.

> 
> I think more detail then "we had an issue" is warranted here, because the
> whole goal of this issue is to demonstrate Camerfirma's complete
> understanding of the issue (and their approach to past issues), to ensure
> that assurances given about future issues can be relied upon.
>
I can assure you that AC Camerfirma understand completely the issue and we are working hard to offer a quality and secure services to our customers and to the community as our audit process in ISO27001, ISO20000, WEBTRUST CA-BR-EV, ETSI-319401, ETSI-319411-1, ETSI-319411-2, ETSI-319421, state.

Comment 20

[Kathleen Wilson]

(In reply to Ryan Sleevi from comment #18)
> Regarding the certificates you have not yet revoked, but contain internal
> server names - as these present critical security risk to the ecosystem,
> please revoke these immediately. This is the same that has been requested
> and expected of other CAs for this same issue. It is particularly concerning
> that, despite being presented with these certificates, Camerfirma did not
> detect the non-compliance, even under manual review. This raises significant
> concerns about the remediation process taken, as well as concerns for the
> remediations proposed.
> 
> Kathleen, I'm not sure how you want to handle this. I'm also concerned that
> certificates that were said were going to be revoked (for being internal
> server names - see Comment #14 "Obviously is an error in the validation
> process. We will revoke those certificates." - have not been revoked - see
> Comment #16), even as we're now a week later. 


In the description of this bug I said: it is not our intent to introduce additional problems by forcing the immediate revocation of certificates that are not BR compliant when they do not pose an urgent security concern. 

Clearly, certificates that present security risk to the ecosystem must be revoked and replaced very promptly. But I understand the need for CAs to work with their customers to resolve such problems in an orderly manner, so they do not introduce potentially worse problems.


> I also don't believe these
> remediation plans demonstrate the necessary awareness of the root causes or
> provide sufficient assurance to prevent future issues.

It is up to the CA to demonstrate this via their response in this bug.

If a CA does not demonstrate their ability to respond to such incidents and does not provide sufficient assurance to prevent future issues, then we will add the CA to our list of CAs to investigate and discuss more thoroughly, as we have begun doing in the mozilla.dev.security.policy forum.

Comment 21

[Gervase Markham [:gerv]]

Ramiro: these questions still remain outstanding, 3 months after the last comment in this bug:

(In reply to Ryan Sleevi from comment #15)
> The question, however, is:
> 1) Why weren't these revoked when originally reported?
> 2) Why weren't these revoked when previously required?
> 3) Why weren't these limited in their validity period?
> 
> That is, an answer of "mistakes were made" does not demonstrate the level of
> introspection, analysis, and transparency necessary for public trust. It's
> clear from the evidence that mistakes were made, what is not clear is why
> these mistakes they were made, how they were made, and what steps are being
> taken to prevent these future mistakes?

I suggest you read up on "root cause analysis" before attempting to give another answer. And I would pay careful attention to what Kathleen said at the end of comment #20.

Gerv

Comment 22

[Juan Angel Martin]

Hello all,

We were not aware that the bug was pending on our side. 

We apologize for this. 

We'll answer as soon as possible taking into account your indications.

Kind regards
Juan Angel

Comment 23

[Ramiro Muñoz Muñoz]

Hi Gerv
Sorry for the delay. We have held a meeting with the people involved in this issue trying, as you suggest, analyzing and understanding the root cause.

We focus the three questions about the use of internal names.
1)	Why weren't these revoked when originally reported?
2)	Why weren't these revoked when previously required?
3)	Why weren't these limited in their validity period?

Affected certificates (2) were issued on July 2015 in a renewal process when local domains were permitted. So no problem was detected in this operation. Nevertheless, control over the limit date (November 2015) allowed to internal domains was not developed into the platform, so platform was not aware of this time limit, and neither when we should have finaly revoked them (October 2016).

No more internal names were issued after November 2015, so our platform managed well the requirement to avoid internal DNS names but not the transition period.

We were aware of this misisuance on August 2017 by means of some remarks in this Bugzilla bug.

We revoked those certificates on 1st and 12th of September 2017 after notifying and arranged a date for revocation with the affected user.

On September 2017 we activated a control in our PKI platform to avoid the DNSName bad spellings, check their availability and therefore avoid internal names. This control cover the problems Ryan sumarized in Comment#18 
 
Getting back to your questions:

4)	Why weren't these revoked when originally reported?
We were aware of this problem, as I said before, on August 24th 2017, we revoke them on September the 1st and 12th after arrange with the customer a date to revoke.

5)	Why weren't these revoked when previously required?
If you means the time where BR state, we were not aware of this misissuance neither November 2015 nor October 2016. Please confirm it were the question.

6)	Why weren't these limited in their validity period?
Our PKI platform works with predefined profiles and the time limit is a fixed value, so is complex to change it.

Hope we have answered your questions and keep working and ready to improve our tools and procedures in order to add value to the community.

Ramiro

Comment 24

[Gervase Markham [:gerv]]

Ramiro,

The point is that the deadline for ceasing issuance of internal names, and the further deadline by which all such certificates need to be revoked, was known for several years before they actually arrived. Other CAs did one or more of the following:

1) Made sure no internal name certificates had lifetimes which extended beyond the revocation deadline
2) Did an internal scan just before or on the revocation date, and revoked all outstanding certs they found

If 1) was not possible for you due to your predefined profiles, I would expect you to have done 2) as a matter of BR compliance.

Gerv

Comment 25

[Ramiro Muñoz Muñoz]

Hi Gerv,

Yes, we should have been done that (2). We did not. AC Camerfirma pass Webtrust CA, EV and BR yearly to detect any non-compliance procedures and certificates. Auditors detected any problem in 2016 audit because only a percentage of certificates were checked. Some bad certificates were detected in 2017 audit because bad DNSName spelling. In this case, crt.sh was a helpful tool. A couple of those certificates contain internal names (8322063, 8680123). Even in our attachment 8898277 [details] in this bug , we made a comment saying that we did not find any problem with DNSNames spelling in those certificates. That was when we were aware of the “internal name” problem (August 2017).

Yearly audit wasn't enough for a detective-control, as we firstly thought, so we carry out a more strict observance today.
As we remarked in this bug, now “every single SSL OV & EV” certificates issued is checked by crt.sh .A reactive control meanwhile a preventive and automatic control is placed in our PKI platform (being developed and planned to deploy this month). We will notify in this bug when this control was working and our auditors will check that this control is already deployed in the next audit report.

Ramiro

Comment 26

[Gervase Markham [:gerv]]

(In reply to Ramiro Muñoz Muñoz from comment #25) 
> Yes, we should have been done that (2). We did not. AC Camerfirma pass
> Webtrust CA, EV and BR yearly to detect any non-compliance procedures and
> certificates. 

All the checks you talk about in this message seem to be checks at issuance time. But the point here is that the BRs were updated to require a set of certs to be revoked at a certain time, and AC Camerfirma did not incorporate that requirement into their processes.

So what changes are you making to make sure that, in future, all ramifications of all changes to the BRs are incorporated into your operations?

Gerv

Comment 27

[Ramiro Muñoz Muñoz]

Hi Gerv

We have increase our resources and involvement in SSL/TSL certificate management, human and technical.  We hope, in this way, to eliminate any anomalous certificate issuances in the future. To be precise we already have:

•	A quality operator in charge of reviewing every single SSL issued.
•	A PKI senior expert in charge of CABFORUM distribution list, MDSP, CCADB, self-assessment and communication management.
•	Increased our requirements for external AR audits. 
•	A PKI Developer in charge to deploy new technical control in our PKI platform (DNSName (done), cablint (in process), CAA check (in process)...etc).

We will ask to our auditors to confirm that all of these actions are working in the next yearly report.

Ramiro.

Comment 28

[Ryan Sleevi]

Hi Ramiro,

I'm not sure your reply in Comment #27 really instills confidence or responds to the concerns in Comment #26. Further, the response in Comment #23 is what is being discussed in Comment #26 - that answer - and analysis, is insufficient.

Your proposed changes - in Comment #27 - thus seem grossly inadequate, because as a CA, Camerfirma has not demonstrated it fully understands what went wrong, why it went wrong, and how it will be improved. Comment #25 underscores that you admit 'something went wrong' - but it fails to respond to any of the substance expected of a CA incident report.

The expectation is that, as a globally trusted CA, you had staffed adequately, and competently, to ensure that deadlines would be met. What are the process and procedure failures that caused Camerfirma to overlook this very simple, very critical requirement - as a trusted CA? What reason, if any, should the community believe that in the future, Camerfirma will do better.

Comment #27 does not instill confidence in understanding the systemic failures that caused this issue ("Why didn't you do (2) in Comment #25"), and thus does not instill confidence in Camerfirma as a CA.

Comment 29

[Ramiro Muñoz Muñoz]

Hi Ryan

I am out of the office some days, sorry for the delay in our response.
We are writing a new report trying to build better our explanation.

Regards

Comment 30

[Ramiro Muñoz Muñoz]

Dear Ryan and Mozilla community:

Our apologies if something was not clear in the previous comments and thus not really instills confidence or responds to the concerns in previous comments.

We conducted internal analysis of the problems reported but probably we have failed to provide the transparency necessary. So, with this report we are trying to provide all required answers to the community that perhaps were not clear in our previous comments.

AC Camerfirma is a small CA that uses an in-house developed CA software and is supported by our RA trusted operators and the continuous training they receive. AC Camerfirma has been working with this technology and staff for 17 years without problems and we have dedicated a big effort improving it along those years. Due to the number of TLS certificates issued on previous years, this working method was appropriate and suitable for us. The increase of certificates issued, increasing technical requirements and the analysis of these reported problems have helped us to realize the need to extend the technical controls in the CA application to cover all possible and new requirements and introduce a new person in charge of coordinating these implementations.

Regarding the problems reported in this Bugzilla bug itself and as stated before in comment#23, affected certificates with internal names were issued on July 2015 in a renewal process when local domains were permitted. The modification of lifetime of certificates in our CA application is complex due to it’s a fixed parameter for each profile and not customizable per each certificate. So, as this was a very few certificate requests we review the BR requirements and concluded wrongly we can issued this certificates with an expiry date later than 1 November 2015 if we revoke them before 1 October 2016, so we permitted this issuance. Due to this root cause identified, Camerfirma introduced in the application new controls regarding the issuance of this type of certificates so no more internal names were issued after November 2015. However, these certificates had to been revoked in October 2016 but as this problem has shown, they were not. According to our conclusions this situation should have been registered in our ticketing tool to generate an alert on the scheduled date, but it was not due to human error. As this was an exceptional case (internal Names in dnsNames), and the certificates were issued more than a year in the past we didn’t realize the need to conduct an internal scan as we didn’t expect/remember to have this type of certificates. In addition, this was not detected in the periodic sample certificate audits.We were aware of this mistake on August 2017 by means of some remarks in this Bugzilla bug. We revoked those certificates on 1st  and 12th  September 2017 after notifying and arranged a date for revocation with the affected user.

We have analyzed again our database (as per comment#13) in order to find other valid certificates with the problems reported in this Bugzilla bug and we can confirm that the certificates in answer3.csv (comment#10) and answer4.xlsx (comment#11) are the only ones affected by these problems and all of them are revoked at this time.

In order to improve our operations and avoid repetition of this issue, we have increased our resources and involvement in TSL certificate management, both human and technical. We have incorporated to our team a PKI senior expert in charge of CAB FORUM distribution list, MDSP, CCADB, self-assessment and communication management. This person is in charge of identifying changes to make sure that, in future, all ramifications of all changes to the BRs are incorporated into our operations both procedural and technical. This includes conducting internal scans previous to each BR relevant date regardless of the expected result. Tasks performed by this person will be registered in our task management tool and reviewed periodically by technical management. Besides, a quality operator is in charge of reviewing that all modified operations are conformant with all requirements (BR, ETSI, eIDAS, etc.) and reviewing every single TSL certificate issued. Therefore, an additional control is on place from now on.
We are aware that it is not the best solution, but it is a realistic workaround while we develop and implement all required controls (not only the ones to avoid the reported problems in this Bugzilla bug) and evaluate the integration of the crt.sh control in order to analyze the precertificate for every server certificate issued (OV, EV).

Considering the main topics “Invalid dnsNames” and “URI in dNSName SAN”, there was not a strict technical control about the DNSName input field in the request form. RA Operators had to review that the DNSName fulfills the policy requirement. This manual method was appropriate and suitable for us due the number of TLS certificates issued. However as explained before, we realize the need to extend the technical controls in the application to cover all possible requirements (BR, ETSI, etc.). 
All those certificates were issued for the same external RA and unfortunately were not been detected in our periodic sample audit. We proceeded to provided new training and evaluation in order to verify all validation requirements are fully understand by the external RA.

In addition, on September 2017, we activated a control in our PKI platform to avoid the DNSName bad spellings check, their availability and therefore avoid internal names, in order to prevent any issues. This control covers the problems Ryan summarized in Comment#18 

The additional list of steps that Camerfirma is taking to solve the situation are:

1.	Internal RA Operators will carry out a second validation of all DNSName certificates field issued (non-technical control). Our Internal approval RA operators will validate the correct syntax and other validations in the request. Domain control is also verified (notifying domain contacts). Timeline: Immediately from August 24th, 2017.

2.	Develop a technical control in the DNSName request field. We are to include all BR requirement validation over all DNSNames found in the request form and this take some codification time. We gave priority to deploy this technical controls so at this time they are implemented. Timeline: implemented on September 15th, 2017.

3.	Develop a control for each TSL certificates issued running a CA/B Forum lint and x509 lint in order to detect DNSName and other errors immediately after issuing a certificate. After certificate issuing we can use the cablint tool, this can speed up the process. We would revoke immediately problematic certificates before distributing it, so in some way it could be a mitigation mechanism. Timeline: Implemented on August 24th 17.

4.	All TSL Certificates issued at the moment have CT controls. Timeline: Immediately from August 24th 2017.

5.	Develop technical controls for all BR requirements in the requesting form (including gTLD, CAA, Leading whitespaces, etc.). Timeline:(January 19th, 2018)

6.	Integrate crt.sh control in order to analyze precertificates prior to the certificate issuance. Timeline: December 29th, 2017)

Considering the main topic “Failure to respond within 24 hours after Problem Report submitted” as stated in comments before, we initiated the investigation of the problems reported within 24 hours, but we had to get in contact with our customers to replace the wrong certificates and due to it was vacation period those days it took more than desirable. For this reason, the response about problem report submitted was delayed. (See timeline in Comment#9).We are sorry for the delay due to the vacation period and because we cannot revoke a certificate without notifying our customers and replacing it for a new one (The certificates were installed in production environments valid domains). We understand this was the root cause of this problem.

To ensure this will not be repeated in the future, Camerfirma is taking the following actions:

1.	We have reviewed this problem timeline and updated the CCADB in order to update the contact point. Timeline: September 1st, 2017.

2.	We have updated the subscriber agreement to make clear this situation to our customers and protect AC Camerfirma legal responsibility when we unilaterally revoke the certificate. Timeline: September 15th, 2017.

3.	We have reviewed our Problem Response procedure in order to clarify the expected timelines and aligned it to the new subscriber agreement. Timeline: September 15th, 2017.

Hope we have answered your questions and keep working and ready to improve our tools and procedures in order to add value to the community.

Ramiro

Comment 31

[Firefox Bug Husbandry Bot]

Changing QA contact per https://bugzilla.mozilla.org/show_bug.cgi?id=1438254

Comment 32

[Wayne Thayer]

Ramiro: have all the changes defined in comment #30 been implemented?

Comment 33

[Ramiro Muñoz Muñoz]

Hi Wayne 
Here I will describe the current situation about changes described in our comment #30.

1.Internal RA Operators will carry out a second validation of all DNSName certificates field issued (non-technical control). Our Internal approval RA operators will validate the correct syntax and other validations in the request. Domain control is also verified (notifying domain contacts). Timeline: Immediately from August 24th, 2017.

This non-technical control was effective from the date reported (August 2017).

2. Develop a technical control in the DNSName request field. We are to include all BR requirement validation over all DNSNames found in the request form and this take some codification time. We gave priority to deploy this technical controls so at this time they are implemented. Timeline: implemented on September 15th, 2017.

Technical control DNSNames was delayed till December 2017. This control has been working since then. An issue arrived on February 2018 https://crt.sh/?id=336248268&opt=cablint so at the moment we are reviewing the control to improve it. We expect to be in production next week.

3.	Develop a control for each TSL certificates issued running a CA/B Forum lint and x509 lint in order to detect DNSName and other errors immediately after issuing a certificate. After certificate issuing we can use the cablint tool, this can speed up the process. We would revoke immediately problematic certificates before distributing it, so in some way it could be a mitigation mechanism. Timeline: Implemented on August 24th 17.

We have been working in this way from the date reported. An operator have been checking daily the cablint tool looking for any issue detected.

4.	All TSL Certificates issued at the moment have CT controls. Timeline: Immediately from August 24th 2017.

This control is effective from April 2017.

5.	Develop technical controls for all BR requirements in the requesting form (including gTLD, CAA, Leading whitespaces, etc.). Timeline:(January 19th, 2018).

At the moment a non-technical check over the CAA value is done by the RA Operator before issue any certificate. We plan to have the CAA technical control in March 2018, the rest of controls are covered by the cablint technical control (next point).

6.	Integrate crt.sh control in order to analyze precertificates prior to the certificate issuance. Timeline: December 29th, 2017)

Cablint & x509lint technical control was delayed to February the 14th 2018. A problem appeared after install the control when detecting a wrong request, certificate issuing wasn't block properly. This is already fixed and working.   


To ensure this will not be repeated in the future, Camerfirma is taking the following actions:

1.	We have reviewed this problem timeline and updated the CCADB in order to update the contact point. Timeline: September 1st, 2017

Done

2.	We have updated the subscriber agreement to make clear this situation to our customers and protect AC Camerfirma legal responsibility when we unilaterally revoke the certificate. Timeline: September 15th, 2017.

Done

3.	We have reviewed our Problem Response procedure in order to clarify the expected timelines and aligned it to the new subscriber agreement. Timeline: September 15th, 2017.

Done

Best Regards
Ramiro

Comment 34

[Wayne Thayer]

With the exception to fixes for #2 described above, it appears that all of the action items have been completed. I'm marking this bug resolved.