Microsec: Failure to revoke noncompliant ICA within 7 days
Categories
(CA Program :: CA Certificate Compliance, task)
Tracking
(Not tracked)
People
(Reporter: szoke.sandor, Assigned: szoke.sandor)
Details
(Whiteboard: [ca-compliance] [ca-revocation-delay] )
Attachments
(2 files)
User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.87 Safari/537.36
Steps to reproduce:
Microsec got the notification about misissued OCSP responder certificates on 2020-07-01, details can be seen in the following bug:
https://bugzilla.mozilla.org/show_bug.cgi?id=1649947
Actual results:
Microsec investigated the problem, the details can be seen in the referenced bug.
2 of the affected ICA certificates have already been revoked, but two others - which belong to the same key - could not been revoked within 7 days due to the huge impact on the users.
Expected results:
Microsec is working on the issue and looking for the solution which will be acceptable for the community of the web PKI and also for the users of our time stamps.
Microsec will publish the detailed plans in a form of incident report within a week.
|
Assignee | |
Updated•4 years ago
|
|
||
Updated•4 years ago
|
|
|
Assignee | |
Comment 1•4 years ago
|
||
We are still working on the issue, we have developed several alternative solutions.
The most perfect solution would take 4 months, so the affected TSA CA key would be destroyed in late November.
The solution includes issuing public documents twice, creating new CAs, having a conformity assessment audit, publishing the new TSA CA certificates on the Hungarian Trust List, all of which takes time.
In parallel, we can develop software to support the revocation of the TSA CA certificates.
To speed up the process, we are working on an alternative temporary solution that can run in parallel to the complete solution and can give much earlier results.
If I understand the incident well, the main problem is the existence of the affected TSA CA private key, because an attacker can issue false OCSP responses even for the affected TSA CA certificate. If we do not have to revoke the affected TSA CA certificates immediately, we may be able to destroy the affected TSA CA key by the end of August or even earlier.
The detailed incident report will be released tomorrow.
|
|
Assignee | |
Comment 2•4 years ago
|
||
Let us provide you our detailed incident report.
-
How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.
Microsec first became aware of the problem by the email "[Bug 1649947] New: Microsec: Incorrect OCSP Delegated Responder Certificate" received at 03:43 (Hungarian time) on July 2, 2020 (2020-07-02 01:43 UTC).
-
A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.
2019-08-28
The auditor raised the issue of OCSP chaining according to the Microsoft requirements. Microsec had 3 technically constrained intermediate CA certificates in the ECC based hierarchy containing Time Stamping or Code Signing EKU without having OCSP Signing EKU.
2019-09-04
Our auditor opened a thread in this issue:
https://groups.google.com/forum/#!topic/mozilla.dev.security.policy/XQd3rNF4yOo
2019-09-10
Based on the received answers we agreed that there is no problem with the OCSP Signing EKU in the CA certificates and issued new CA certificates with OCSP Signing EKU. The earlier CA certificates have not been revoked so presently they are still valid.
The audit in September 2019 covered both versions of the affected CA certificates and the present attestation letter contains all of them.
2020-07-02
- Microsec received the email about the incompatibility.
- Microsec began the investigation on the issue, a local team of experts was set up.
- Microsec confirmed that the problematic certificate is a subordinate TSA CA certificate, which is intended to issue only certificates for time stamping. The TSA CA is technically constrained, because it contains the Time Stamping EKU as required by Mozilla. This TSA CA has a delegated OCSP Responder. The TSA CA issues end entity OCSP Responder certificates for the delegated OCSP Responder, which certificates contain only the OCSP Signing EKU. The problematic TSA CA certificate contains the OCSP Signing EKU only to fulfil the requirement of the EKU chaining. It is not intended to be used as a delegated OCSP responder of the Root CA. It does not have the Digital Signature KU bit set.
- Microsec checked its CA hierarchies and identified, that altogether 4 ICA certificates were issued with the same EKU problem on 2019-09-10. The 2 Code Signing CA is managed separately in the following bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1649947. This bug is focusing on the TSA CA incident.
- Microsec decided to make a risk assessment to identify the real level of risk in its system caused by this problem.
- Microsec informed the conformity assessment auditor about the incident.
- Microsec informed the supervisory authority about the incident.
- Microsec confirmed the receipt of the problem report in the [Bug 1649947].
- Microsec begun to set up a plan to solve this security issue.
- Microsec continuously follows the messages on the following threads to make the proper steps from the beginning:
https://groups.google.com/forum/#!topic/mozilla.dev.security.policy/EzjIkNGfVEE
https://groups.google.com/forum/#!topic/mozilla.dev.security.policy/XQd3rNF4yOo
2020-07-06
Microsec conducted a security risk assessment to identify the real risk of misuse of the affected Microsec ICA keys.
Microsec agrees that these ICA certificates with the OCSP Signing EKU may cause problems in the client application who do not complete the full validation of the received OCSP response, if an OCSP response is signed by them.
Microsec has found that the risk of key misuse on the server side at Microsec is very low due to the following facts:
- the affected keys belong to intermediate CA certificates intended to issue End Entity certificates for Code Signing and Time Stamping
- the affected keys and the CA certificates are managed by Microsec, and no third parties are involved in the operation in any way
- keys are generated and managed throughout their lifecycle by FIPS 140-2 Level 3 certified HSM devices
- our CA software and OCSP responder software are separate applications
- the CA software is unable to issue OCSP responses
- all applications only have access to those keys that are necessary for their operation
- OCSP responders do not have access to CA signing keys
- OCSP responders verify OCSP responder certificates before signing the OCSP response. OCSP Responders check the following criteria, among others:
- the certificate is an end-user certificate (CA bit is false)
- the certificate has the OCSP Signing EKU
- the certificate does not have any EKU other than OCSP Signing
- the certificate has the digitalSignature Key Usage bit set
The OCSP response is signed only if all the above requirements are met. In our case, only the second requirement is met, thus, a potential OSCP request with these CA certificates will be rejected.
Issuing an OCSP response signed with any of these CA keys and certificates requires a deep attack on the Microsec IT system that would compromise the entire CA service.
Of course, there is a risk to this as well, but it is so low that it can be managed as a residual risk till fully solving the issue on the server side.
Based on the result of the risk assessment Microsec is working to resolve this security problem as soon as possible.
2020-07-07
Microsec decided to set up a new TSA CA with 2 TSA CA certificates as soon as possible and migrate the time stamping services from the affected CAs.
Microsec will issue the new ICA certificates without the OCSP Signing EKU.
2020-07-08
Microsec generated new keys for the planned new CA, OCSP responder and TSUs.
Issuance of the new CA certificates will happen only after solving the urgent issues.
The immediate revocation of the Time Stamping CA certificates is not possible due to the huge harm it may cause for the customers.
About 150 million qualified time stamps were issued based on this TS CA key and TS CA certificate, and the revocation of this TS CA certificate would cause problems in the validation of all the digital signatures protected by these time stamps.
Our qualified time stamps are widely used among others in the Hungarian company registration system, by all the Hungarian state notaries and most of the Hungarian lawyers to protect the validity of their electronically signed documents for near 12 years.
A separate action plan need to be worked out how to protect these users and solving this security issue at the same time.
Microsec published the normal incident report in the [Bug 1649947].
2020-07-09
Microsec opened a separate incident report for the late revocation of the affected TSA CA certificates in the [Bug 1651632]
2020-07-10
Microsec accepted the plan how to solve the security incident. Microsec will issue new TSA CA certificates and will set up a new system for issuing time stamps.
The solution includes issuing public documents twice, having a conformity assessment audit, publishing the new TSA CA certificates on the Hungarian Trust List, all of which takes time. This most perfect solution would take 4 months, so the affected TSA CA key would be destroyed only in late November, 2020.
Seing the long process Microsec also decided to work out an alternative solution which makes possible the key destruction earlier.
2020-07-15
Microsec announced the new draft version of the public documents which will makes possible the issuance of ICA certificates without the OCSP Signing EKU.
The effect date according to the requirements will be 2020-08-14.
Another public document will take affect on 2020-07-22, which allows the issuance of special closing OCSP responder certificates with long lifetime in special cases.
2020-07-16
Microsec accepted an alternative solution, which makes possible the key destruction latest at the end of August 2020, depending on the publication of the new Trust List.
2020-07-16
Microsec published this detailed incident report in the present bug.
-
Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.
The problematic ICA certificates have been issued by our Root CAs. Microsec will not issue other ICA certificates with his OCSP Signing EKU problem.
The Time Stamping CA is used only to issue TSA certificates for the time stamping service, it happens typically once a year and only a few certificates yearly. There will be no further issuance of new TSA certificates by using the affected key.
The Time Stamping units are still in operation and they issue about a half million qualified time stamps daily. They can not be stopped and it is not possible to set up new time stamping units quickly due to regulatory issues. Microsec is working on the issue with the auditors and the supervisory authority. -
A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.
The number of the problematic TSA ICA certificates is 2, they belong to 1 keys.
They were issued on 2019-09-10. -
The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.
e-Szigno TSA CA 2017 https://crt.sh/?id=2411247733
e-Szigno TSA CA 2017 https://crt.sh/?id=2411246057 -
Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
As we described in point 2., the OCSP Signing EKU was not present in our ICA certificates before 2019-09-10, even in the technically constrained ICA certificates. During our audit in 2019 our auditor raised the issue of EKU chaining as required by Microsoft.
Our auditor opened a discussion on Mozilla forum, and we interpreted the answers, that Microsoft requires the presence of the OCSP Signing EKU in the technically constrained CA certificates and Mozilla is not against it.
Microsec issued the 2 affected TSA ICA certificates based on the public discussion on 2019-09-10 including the OCSP Signing EKU.
We had no information about the existence of this problem before receiving the email from Mozilla on 2020-07-02.
We haven't got any incident report from our users or from any other third parties. -
List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.
Microsec decided to start two processes in parallel.
One of them will completely solve the problem of time stamping and will make it possible to issue time stamps for 12 years.
The other workaround will focus on the incident itself and enables destruction of the affected TSA CA key much earlier. It will cause some problems in the time stamping service, but we hope to be able to manage them.The main steps of the second process, which is relevant for the present incident:
2020-07-17
Issuance of new TSA certificates under one of the currently existing ICA (not technically constrained).
Request the inclusion of the new TSA certificates in the Hungarian Trust List.
2020-08-17
Publication of the new Trust List with the new TSA certificates. If this happens earlier, all the following steps can be done earlier.
2020-08-18
Activate the new TSA keys and certificates in the time stamping service.
2020-08-19
Destroying the old TSU keys.
2020-08-25
Terminate the TSA CA 2017, including the issuance of closing CRL and a closing OCSP Responder Certificate.
2020-08-25
Destroy the affected TSA CA key.
2020-09-25 (or even later)
Revocation of the misissued TSA ICA certificates. The original TSA ICA certificate without OCSP Signing EKU will not be revoked, this way the validation of the issued time stamps will be possible. This date is not critical, because the affected key has already been destroyed. Our customers may need more time to develop their client applications to properly manage the case of revoked TSA CA certificates.
|
||
Comment 3•4 years ago
|
||
Thanks for the very thorough explanation. I'm going to set the Next Update for 20-Sept-2020.
|
|
Assignee | |
Comment 4•4 years ago
|
||
Thanks for the confirmation.
We will keep you informed about the main milestones.
|
|
Assignee | |
Comment 5•4 years ago
|
||
Progress report
I sumarize the main new events regarding the Microsec OCSP Signing EKU incident.
2020-07-21
Issuance of new TSU certificates under the "Qualified e-Szigno QCP CA".
2020-07-22
The publication of the new TSU certificates requested from the supervisory authority with high priority.
2020-07-22
The new version of the public documents were published which made possible the usage of special closing OCSP responder certificates with long lifetime in special cases.
2020-07-30
The new TSU certificates were published on the Hungarian Trust List.
2020-08-03
The Codesigning CA keys were deleted from the life system (all files were deleted). The keys are available only on the archive CD-s stored in safe in a secure environment.
2020-08-06
Time stamping services have been moved to the new TSU keys and certificates.
2020-08-14
The new version of public documents was released that allowed the issuance of new ICA certificates without the OCSP Signing EKU.
2020-08-25
Microsec has issued the new ICA certificates for the following CAs:
- e-Szigno Root CA 2017 >> e-Szigno Class3 CodeSigning CA 2020
- e-Szigno Root CA 2017 >> e-Szigno Class2 CodeSigning CA 2020
- e-Szigno Root CA 2017 >> e-Szigno TSA CA 2020
- Microsec e-Szigno Root CA 2009 >> e-Szigno TSA CA 2020
2020-08-27
The issuance of new TSU certificates has been disabled in the "e-Szigno TSA CA 2017".
The CA has since only issued OCSP responder certificates.
2020-08-28
The 3 new CA units have been set up with new keys and new ICA certificates.
2020-08-29
Microsec has begun the process of re-timestamping all archived documents in the preservation service.
2020-08-31
The issuance of the long term OCSP responder certificate has been cancelled due to security concerns (no revocation would be possible in case of any problem).
2020-09-01
The NIST certificate status of our HSM devices has been changed to "Historical".
Microsec immediately decided to suspend any provider's key generation on its HSM devices.
Microsec is studying the situation how to create new provider's keys for the Time Stamping OCSP service.
This HSM situation is currently blocking the transition process.
2020-09-12
During the on-site audit, Microsec agreed with the auditor on an acceptable solution for the special OCSP service.
Instead of issuing an OCSP responder certificate with extreme long validity, Microsec will pre-generate OCSP responder certificates that are valid only for 24 hours, but till the validity end date of the issuing CA. In the event of a problem, Microsec will be able to terminate the OCSP service for the "e-Szigno TSA CA 2017".
Events that block the process
Due to the HSM certification issue, Microsec is unable to generate new keys for the OCSP responder.
This blocks the most critical process, but some other independent processes can continue.
|
|
||
Updated•4 years ago
|
|
|
Assignee | |
Comment 6•4 years ago
|
||
Progress report
I sumarize the main new events regarding the Microsec OCSP Signing EKU incident.
2020-10-09
- Microsec revoked the misissued ICA certificates
-- e-Szigno TSA CA 2017 (issued from "e-Szigno Root CA 2017" on 2019-09-10)
-- e-Szigno TSA CA 2017 (issued from "Microsec e-Szigno Root CA 2009" on 2019-09-10) - The revocation was recorded in CCADB
Final task
2020-10-16
Microsec will destroy all ICA keys affected in this incident with the presence and supervision of an independent auditor.
|
|
Assignee | |
Comment 7•4 years ago
|
||
Final report
2020-10-16
Microsec destroyed all ICA keys affected in this incident with the presence and supervision of an independent auditor.
Present Status
All affected keys have been destroyed and all misissued ICA certificates have been revoked.
|
|
||
Comment 8•4 years ago
|
||
Is there some type of audit attestation or other document that you can share by uploading as an attachment here?
|
|
||
Updated•4 years ago
|
|
|
Assignee | |
Comment 9•4 years ago
|
||
Yes, of course, we have the Minutes of key destruction ceremony which is signed by the independent auditor too.
It is a paper based document, so I can upload the scanned version of it.
It is Hungarian, I prepare the draft English translation.
|
|
Assignee | |
Comment 10•4 years ago
|
||
|
|
Assignee | |
Comment 11•4 years ago
|
||
|
|
||
Updated•3 years ago
|
|
|
||
Comment 12•3 years ago
|
||
I intend to close this matter next week (Dec.7-11) unless there are other issues to be discussed.
|
|
||
Updated•3 years ago
|
|
||
Updated•2 years ago
|
|
||
Updated•1 year ago
|
Description
•