Camerfirma: Unrevocation of MULTICERT SSL Certification Authority 001 certificate
Categories
(CA Program :: CA Certificate Compliance, task)
Tracking
(Not tracked)
People
(Reporter: eusebio.herrera, Assigned: eusebio.herrera)
Details
(Whiteboard: [ca-compliance] [crl-failure])
- How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.
2019-02-19 18:43 CAMERFIRMA received a call from MULTICERT informing that they were receiving revocation alerts about CA 'MULTICERT SSL Certification Authority 001'. This is one Camerfirma's intermediate CA (root='Global Chambersign Root - 2008', intermediate CA serial number 01A4B767C5A58D8E).
- A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.
2019-02-19 18:44 Start the procedure to detect the error causing the reported problem.
2019-02-19 18:55 The reason for the error is detected. Human failure due to confusion generated because MULTICERT has two CA certificates, both with the same CN (MULTICERT SSL Certification Authority 001).
https://ccadb.force.com/0011J00001HcKaN
https://ccadb.force.com/0011J00001HcKZo
In addition, the similarity of the serial numbers of both certificates (01A4B767C5A58D8E and 01A844E66A6C0D8E) caused this difference not to be detected.
2018-02-19 19:10 The OCSP service responses provided by CAMERFIRMA (wich is established as main validation method in Camerfirma CPS) weren't affected and were correct at all times.
2019-02-19 19:35 CAMERFIRMA communicated to MULTICERT the reason of the problem. It's proposed the effective revocation of this intermediate CA and the MULTICERT informed CAMERFIRMA that there is a high number of affected customers and that online payment systems in Portugal were paralyzed for this reason so it was urgent to establish an alternative solution.
2019-02-19 20:16 A new CRL was issued with the serial number of the correctly revoked CA (01A844E66A6C0D8E) included and without the incorrectly revoked CA (01A4B767C5A58D8E).
- Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.
N/A
- A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.
N/A
- The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.
N/A
- Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
6.1) One error in the serial number incorporated in the CRL that had not been detected by the validator.
6.2) One error in the publication procedure of the CRLs of some of our CAs.
- List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.
2019-02-20 The CRL publishing procedure has been improved: The team responsible for CRLs' publication mechanism will not proceed with this task if it doesn't receive confirmation of its correction by two different persons in charge of the validation task.
Updated•5 years ago
|
Comment 1•5 years ago
|
||
Please be aware: This is a MAJOR incident. Unrevoking a CA is not something that should even be technically possible for a CA - that such a method exists, and was used, represents a significant and grave concern. Revocations should be immutable - and while it's understandable they can be disruptive, that's a consequence for a CA revoking.
It is equally deeply problematic the similarity of the serial numbers, as these were issued following the BRs requirement for entropy.
The level of seriousness is significant enough that this may represent a root CA disqualifying event.
Please provide a thorough analysis for the following issues revealed in this incident report:
- Why the OCSP services did not reflect the CRL statuses
- Why the serial numbers for these certificates are similar
- Why it is even technically possible to "unrevoke" a certificate
Assignee | ||
Updated•5 years ago
|
Comment 2•5 years ago
|
||
Eusebio, it has now been one week since I highlighted this was a MAJOR issue.
Please provide a response ASAP.
Assignee | ||
Comment 3•5 years ago
|
||
Sorry for the delay, we are working on providing answers to the questions raised with all the staff involved.
Tomorrow morning (Spanish time) we will offer these answers.
Comment 4•5 years ago
|
||
Hello Ryan, sorry for the delay in aswering you:
We understand the the severity of the case but while technically is an 'unrevocation', it was actually a misissued CRL publication.
We issued a wrong CRL on Feb 14 17:18:56 2019 GMT and published it on Feb 19 13:12 2019 GMT to Feb 19 19:21 2019 GMT. Once we received the alerts from Multicert we analysed the options to overcome this issue.
We talked with our Authority Policy to evaluate this scenario including the definitive revocation of the subCA certificate. We spoke with Multicert and it was clear that revocation would be a major problem. Some of their customers are Banks and Electronic Payments companies. The consequence of definitively revoking this subCA would have been the collapse of Portugal electronic payments. We also review similar cases reported to the community. Definitly we decided to make a rollback moreover taking into account that there wasn’t any reason (key compromise, cessation of operation, ...) for revoking the subCA certificate.
Our CA Root management system (wich issues subCA certificates and CRLs) is an off-line system, which is always switched off in a safe room, except the time for issuing subCA certificates or issuing the CRLs.
On 2018-06-29 In an off-line ceremony we issued the first subCA certificate for "MULTICERT SSL Certification Authority 001" (Signature Hash Algorithm: SHA256WithRSA).
Few days after, our customer realized that one key usage was missing in the certificate, so a second ceremony for issuing a second certificate was needed.
On 2018-07-03, the second ceremony took place by a different team since some of the original members was out. The serial number used wasn't incorporated into the certificate profile template file correctly due to a reuse of the previous template.
On 2018-07-17, the first intermediate CA certificate was revoked successfully. So there was a new ceremony in which a new CRL was issued in our CA Root management system and later, our OCSP responder was updated with this new revocation.
On 2019-02-14 17:18 GMT, in a routine task, a new ceremony for issuing a new CRL was made (the wrong one). There wasn’t any new revocation, so it wasn’t needed an OCSP responder service update. That is the reason because our main validation method has been working right.
We’ve improved this process:
1.- We'll increase the controls to avoid template reuse.
2.- We’ll increase the controls to assure that the serial number in this offline process had the right entropy (much more than 64 bits of entropy) and this code doesn't produce any serial numbers similar to those already available in none of the certificates of CAs (root CA or intermediate CA).
3.- For intermediate CAs CRL a new operator checking process will assure that the CRL is OK before publishing.
Comment 5•5 years ago
|
||
I'm not sure I fully understand what occurred in the events on 2018-07-17 and 2019-02-14. Could you describe a bit more fully about what happened, especially on the 14?
Comment 6•5 years ago
|
||
On 2018-07-17, the first intermediate CA certificate was revoked successfully. So there was a new ceremony in which a new CRL was issued in our CA Root management system and later, our OCSP responder was updated with this new revocation.
2018-07-17 The certificate issued on 2018-06-29 was revoked.
For this offline ceremony was used one template file that included by mistake the serial number of the certificate that was issued on 2018-07-03 instead of the one issued on 2018-06-29.
Subsequently the ceremony was executed and in the review prior to the issuance of this CRL it was detected that the template incorporated the wrong serial number (intermediate CA issued on 2018-07-03).
This template file was corrected and the CRL was issued with the serial number of the CA we wanted to revoke (intermediate CA issued on 2018-06-29).
We also incorporated the change in the status of this certificate in our OCSP service.
On 2019-02-14 17:18 GMT, in a routine task, a new ceremony for issuing a new CRL was made (the wrong one). There wasn’t any new revocation, so it wasn’t needed an OCSP responder service update. That is the reason because our main validation method has been working right.
2019-02-14 Camerfirma issues CRLs for some of our CAs, roots and intermediate CAs non-issuing end entity certificates, on an annually basis (<365). In order to avoid a great number of CRL's issuing ceremonies it was performed one ceremony that issues all of these CRLs.
Please note that the team that performed the ceremony on 2018-07-17 wasn't the same as on 2019-02-14. In this ceremony the configuration file used had the erroneous serial number (intermediate CA issued on 2018-07-03) and this error wasn't detected.
It was assumed that the new CRL shouldn't include any difference from the previous one (revokedCertificates) and for that reason the status of any of the certificates issued wasn't changed into the OCSP service.
Comment 7•5 years ago
|
||
Juan: Thank you for this response. I now have a good understanding of what happened. Given the very serious nature of this incident, I do not think that Camerfirma's response is adequate. Specifically:
- It appears to me that one cause of this incident was the existence of a second certificate due to the missing key usage. What has been done to prevent issuance of intermediate certificates with incorrect attributes?
- From your response, I believe that the second certificate contains a serial number that does not meet BR entropy requirements. How will this be prevented in the future?
- What is being done to detect and prevent Camerfirma's CRL and OCSP services from providing inconsistent revocation information?
- What will be done to ensure that Camerfirma never "unrevokes" a certificate again, even if human error causes an inadvertent revocation?
Simply adding more human review seems unlikely to really "fix" the underlying causes of this incident.
Comment 8•5 years ago
|
||
- It appears to me that one cause of this incident was the existence of a second certificate due to the missing key usage. What has been done to prevent issuance of intermediate certificates with incorrect attributes?
It was not a mistake but a change of customer’s requirements. Prior to the issuance of the first CA certificate, a prototype certificate was generated for the confirmation of MULTICERT. Once MULTICERT confirmed that the certificate met his needs, the first CA certificate was issued by CAMERFIRMA. Subsequently MULTICERT detected the new need to include the EKU TLS Web Client Authentication. CAMERFIRMA issued the second certificate in response to this new need and proceeded to revoke the first one.
- From your response, I believe that the second certificate contains a serial number that does not meet BR entropy requirements. How will this be prevented in the future?
Currently CAMERFIRMA haS changed the CA’s certificate generation protocol so that the ceremony script has a control to make sure that serial number is greater than zero (0) containing at least 64 bits of output from a CSPRNG.
CAMERFIRMA's definitive solution will be to incorporate EJBCA for offline CAs’ automated management (within a secure room and without internet connection) and that only sign other CA certificates, CRLs and OCSP responders. In the initial configuration we will incorporate at least a serial number size of 9 octets.
- What is being done to detect and prevent Camerfirma's CRL and OCSP services from providing inconsistent revocation information?
CAMERFIRMA will reduce the period of time in which the automatic controls that detect inconsistencies between CRL and OCSP are done.
CAMERFIRMA will incorporate to the automatic process that publishes the new CRL for not issuing CAs certificates for final entities the update of OCSP service’s information so that inconsistencies do not exist.
- What will be done to ensure that Camerfirma never "unrevokes" a certificate again, even if human error causes an inadvertent revocation?
CAMERFIRMA is explicitly incorporating in the terms & conditions of new contracts for Subordinate CA certificates that in case of a revocation it is not possible to reverse the revocation. For existing contracts, we are discussing similar amendments.
Comment 9•5 years ago
|
||
What is the timeline for changes in Comment #8? Have they been implemented?
I am concerned that there may be contractual renegotiations required in order to ensure that CAMERFIRMA does not violate the BRs in the future. Have those changes been completed?
Comment 10•5 years ago
|
||
Emailed POCs on 2019-07-04 regarding this issue, highlighting https://wiki.mozilla.org/CA/Responding_To_An_Incident#Keeping_Us_Informed
Comment 11•5 years ago
|
||
CAMERFIRMA's definitive solution will be to incorporate EJBCA for offline CAs’ automated management (within a secure room and without internet connection) and that only sign other CA certificates, CRLs and OCSP responders. In the initial configuration we will incorporate at least a serial number size of 9 octets.
2019-07-31 EJBCA pre-production environment deployment completed.
2019-08-16 EJBCA production environment deployment completed.
I am concerned that there may be contractual renegotiations required in order to ensure that CAMERFIRMA does not violate the BRs in the future. Have those changes been completed?
I'll update about this question this week.
Comment 12•5 years ago
|
||
(In reply to Juan Angel Martin from comment #11)
I am concerned that there may be contractual renegotiations required in order to ensure that CAMERFIRMA does not violate the BRs in the future. Have those changes been completed?
I'll update about this question this week.
Do you have an update?
Assignee | ||
Comment 13•5 years ago
|
||
Our Legal Department is reviewing the contracts terms with our subCAs in order to add the amendments about the imposibility of unrevocating revoked subCAs.
In a few hours we will update this question with more info from our Legal Department.
Assignee | ||
Comment 14•5 years ago
|
||
Hi Ryan,
We completely agree with you that it wouldn't be neccesary to add amendments about the imposibility of unrevocating revoked subCAs.
The aim of our Legal Department for adding this clause is to raise awareness among our subCAs of the effects of possible non-compliance with the BRs.
Best regards,
Eusebio
Comment 15•5 years ago
|
||
Update:
CAMERFIRMA's definitive solution will be to incorporate EJBCA for offline CAs’ automated management (within a secure room and without internet > connection) and that only sign other CA certificates, CRLs and OCSP responders. In the initial configuration we will incorporate at least a serial number size of 9 octets.
2019-07-31 EJBCA pre-production environment deployment completed.
2019-08-16 EJBCA production environment deployment completed.
2019-08-31 EJBCA pre-production environment deployment completed.
2019-09-16 EJBCA production environment deployment completed.
Comment 16•5 years ago
|
||
Update:
CAMERFIRMA's definitive solution will be to incorporate EJBCA for offline CAs’ automated management (within a secure room and without internet > connection) and that only sign other CA certificates, CRLs and OCSP responders. In the initial configuration we will incorporate at least a serial number size of 9 octets.
2019-08-31 EJBCA pre-production environment deployment completed.
2019-09-16 EJBCA production environment deployment completed.
There have been no changes to the planned dates
Comment 17•4 years ago
|
||
Update: After a change in the company’s direction’s perspective, the solution implemented to manage the CAs has been developing an internal tool.
This tool improves the CAs’ management and avoids the problems that generated this bug. As important aspects, we can say that it ensures the entropy of the serial numbers, generates serial numbers with at least 64 bits of output from a CSPRNG and, to avoid the problems caused by the manual management of the CAs, it shows the active CAs and, in case of revocation, offers the information related to the CA and requires a double check to continue, which will avoid revocations and operations over wrong CAs.
Comment 18•4 years ago
|
||
Ana: I understand that the solution described in comment #16 was never deployed as planned? If not, why not, and why was this information not communicated sooner?
What is the current status of the new internal tool? Is it being used to manage the issuance of all new CA certificates?
Comment 19•4 years ago
|
||
Hi Wayne,
The solution described in comment #16 was not deployed using the tool that we had planned in the beginning, but the final solution developed internally was deployed in the production environment within the dates indicated in our planning and it incorporates the same features as EJBCA which can assure that the error that originated the bug will not be repeated in the future.
We decided to change to our solution because we considered more critical to meet the deadline and control all the new certificates to avoid errors and, unfortunately, we overlooked to update the information in the bug.
This fact has demonstrated us that our mechanism to follow the bugs and maintain them updated by using the alerts by e-mail is not a good option for some cases like this and, that is why, we have started to maintain daily internal meetings to manage all our open bugs from Jan 2nd 2020.
Regarding the status of this internal tool, it was moved to the production environment on Sept 16th 2019 and we have been using it to manage the issuance of all CA certificates from that date.
In fact, we already used the tool to issue the following new certificates in November:
-
InfoCert Organization Validation 2019 CA 3 (22/11/2019) https://ccadb.force.com/0011J00001SDJ1qQAH
-
Intesa Sanpaolo Organization Validation 2019 CA (22/11/2019) https://ccadb.force.com/0011J00001SDJ20QAH
-
CAMERFIRMA COLOMBIA SAS CERTIFICADOS - 001 (12/11/2019) https://ccadb.force.com/0011J00001SBTY6QAP
-
CAMERFIRMA COLOMBIA SAS CERTIFICADOS - 002 (12/11/2019) https://ccadb.force.com/0011J00001D6AeiQAF
Comment 20•4 years ago
|
||
It appears that all questions have been answeered and remediation is complete.
Updated•1 year ago
|
Updated•1 year ago
|
Description
•