Closed Bug 1649947 Opened 4 years ago Closed 4 years ago

Microsec: Incorrect OCSP Delegated Responder Certificate

Categories

(CA Program :: CA Certificate Compliance, task)

Product:
CA Program
Component:
CA Certificate Compliance
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: ryan.sleevi, Assigned: szoke.sandor)

Details

(Whiteboard: [ca-compliance] [ocsp-failure])

The following was originally reported to m.d.s.p. at https://www.mail-archive.com/dev-security-policy@lists.mozilla.org/msg13493.html

Microsec has issued one or more OCSP Delegated Responders, as defined within RFC 6960, Section 2.6 and Section 4.2.2.2, without including the id-pkix-ocsp-nocheck response, as required by the Baseline Requirements, Version 1, Section 13.2.5 through Version 1.7.0, Section 4.9.9

Example certificate: https://crt.sh/?id=2411246057

Please provide an incident report, including the timeline for revocation.

Microsec confirms receipt of this report and is investigating the issue.

Let us provide you our incident report.

  1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

    Microsec first became aware of the problem by the email "[Bug 1649947] New: Microsec: Incorrect OCSP Delegated Responder Certificate" received at 03:43 (Hungarian time) on July 2, 2020 (2020-07-02 01:43 UTC).

  2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.
    2019-08-28
    The auditor raised the issue of OCSP chaining according to the Microsoft requirements. Microsec had 3 technically constrained intermediate CA certificates in the ECC based hierarchy containing Time Stamping or Code Signing EKU without having OCSP Signing EKU.
    2019-09-04
    Our auditor opened a thread in this issue:
    https://groups.google.com/forum/#!topic/mozilla.dev.security.policy/XQd3rNF4yOo

2019-09-10
Based on the received answers we agreed that there is no problem with the OCSP Signing EKU in the CA certificates and issued new CA certificates with OCSP Signing EKU. The earlier CA certificates have not been revoked so presently they are still valid.
The audit in September 2019 covered both versions of the affected CA certificates and the present attestation letter contains all of them.

2020-07-02
Microsec received the email about the incompatibility.

	Microsec began the investigation on the issue, a local team of experts was set up.

	Microsec confirmed that the problematic certificate is a subordinate TSA CA certificate, which is intended to issue only certificates for time stamping. The TSA CA is technically constrained, because it contains the Time Stamping EKU as required by Mozilla. This TSA CA has a delegated OCSP Responder. The TSA CA issues end entity OCSP Responder certificates for the delegated OCSP Responder, which certificates contain only the OCSP Signing EKU. The problematic TSA CA certificate contains the OCSP Signing EKU only to fulfil the requirement of the EKU chaining. It is not intended to be used as a delegated OCSP responder of the Root CA. It does not have the Digital Signature KU bit set.

	Microsec checked its CA hierarchies and identified, that altogether 4 ICA certificates were issued with the same EKU problem on 2019-09-10.

	Microsec decided to make a risk assessment to identify the real level of risk in its system caused by this problem.

	Microsec informed the conformity assessment auditor about the incident.

	Microsec informed the supervisory authority about the incident.

	Microsec confirmed the receipt of the problem report in the [Bug 1649947].

	Microsec begun to set up a plan to solve this security issue.

	Microsec continuously follows the messages on the following threads to make the proper steps from the beginning:
	https://groups.google.com/forum/#!topic/mozilla.dev.security.policy/EzjIkNGfVEE
	https://groups.google.com/forum/#!topic/mozilla.dev.security.policy/XQd3rNF4yOo

2020-07-06
Microsec conducted a security risk assessment to identify the real risk of misuse of the affected Microsec ICA keys.
Microsec agrees that these ICA certificates with the OCSP Signing EKU may cause problems in the client application who do not complete the full validation of the received OCSP response, if an OCSP response is signed by them.

	Microsec has found that the risk of key misuse on the server side at Microsec is very low due to the following facts:
	- the affected keys belong to intermediate CA certificates intended to issue End Entity certificates for Code Signing and Time Stamping
	- the affected keys and the CA certificates are managed by Microsec, and no third parties are involved in the operation in any way
	- keys are generated and managed throughout their lifecycle by FIPS 140-2 Level 3 certified HSM devices
	- our CA software and OCSP responder software are separate applications
	- the CA software is unable to issue OCSP responses
	- all applications only have access to those keys that are necessary for their operation
	- OCSP responders do not have access to CA signing keys
	- OCSP responders verify OCSP responder certificates before signing the OCSP response. OCSP Responders check the following criteria, among others:
		- the certificate is an end-user certificate (CA bit is false)
		- the certificate has the OCSP Signing EKU
		- the certificate does not have any EKU other than OCSP Signing
		- the certificate has the digitalSignature Key Usage bit set
		The OCSP response is signed only if all the above requirements are met. In our case, only  the second requirement is met, thus, a potential OSCP request with these CA certificates will be rejected.

	Issuing an OCSP response signed with any of these CA keys and certificates requires a deep attack on the Microsec IT system that would compromise the entire CA service.
	Of course, there is a risk to this as well, but it is so low that it can be managed as a residual risk till fully solving the issue on the server side.

	Based on the result of the risk assessment Microsec is working to resolve this security problem as soon as possible.

2020-07-07
Microsec decided to set up 3 new CA-s with 4 CA certificates as soon as possible and migrate the services from the affected CAs.
It is still not absolutely clear what are the exact EKU chaining requirements in case of OCSP Signing EKU. What type of CA certificates shall be issued to replace the present problematic CA certificates? Is the same CA certificate profile but without OCSP Signing EKU acceptable by each root programs?
If there will be no other information Microsec will issue the new ICA certificates without the OCSP Signing EKU.

2020-07-08
Microsec generated new keys for the planned 3 new CAs.

	Microsec decided to immediately terminate the 2 affected Code Signing CAs in the ECC hierarchy.
	Termination of the 2 Code Signing CAs:
	- there was no valid end entity certificate issued by them
	- issuance of closing CRLs
	- stop OCSP service for these CAs
	- revoke all 4 ECC based Code Signing CA certificates belonging to the affected keys (with and without OCSP Signing EKU)
	- refresh the root crl
	- upgrading CCADB info
	- the 2 affected CA keys will be destroyed as soon as possible but at a later time - It is not clear, that we need an independent witness or can we do it ourselves. Our auditor is in Germany and presently we have travel difficulties due to the COVID-19 situation.

	Issuance of the new CA certificates will happen later after solving the urgent issues and specifying the clear requirements.
	It is not clear that for the new CA certificates do we need an extra audit or is it enough to cover it with our next audit in September.

	The immediate revocation of the Time Stamping CA certificates is not possible due to the huge harm it may cause for the customers. 
	About 150 million qualified time stamps were issued based on this TS CA key and TS CA certificate, and the revocation of this TS CA certificate would cause problems in the validation of all the digital signatures protected by these time stamps. 
	Our qualified time stamps are widely used among others in the Hungarian company registration system, by all the Hungarian state notaries and most of the Hungarian lawyers to protect the validity of their electronically signed documents for near 12 years.
	A separate action plan need to be worked out how to protect these users and solving this security issue at the same time.

	Microsec publishes the present incident report in the [Bug 1649947].

  1. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

     The problematic ICA certificates have been issued by our Root CAs. Microsec will not issue other ICA certificates with his EKU problem.
 The Code Signing CAs are in our new ECC based hierarchy which is still not used to issue End Entity certificates. 
 The Time Stamping CA is used only to issue TSA certificates for the time stamping service, it happens typically once a year and only a few certificates yearly. Presently there is no issuance of new TSA certificates.
 The Time Stamping units are still in operation and they issue about a half million qualified time stamps daily. They can not be stopped and it is not possible to set up new time stamping units quickly due to regulatory issues. Microsec is working on the issue with the auditors and the supervisory authority.

  2. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

     The number of the problematic ICA certificates is 4, they belong to 3 keys.
 They were issued on 2019-09-10.

  3. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

     e-Szigno Class2 CodeSigning CA 2017		https://crt.sh/?id=3059070922
 e-Szigno Class3 CodeSigning CA 2017		https://crt.sh/?id=3059070921
 e-Szigno TSA CA 2017					https://crt.sh/?id=2411247733
 e-Szigno TSA CA 2017					https://crt.sh/?id=2411246057

  4. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

    As we described in point 2., the OCSP Signing EKU was not present in our ICA certificates before 2019-09-10, even in the technically constrained ICA certificates. During our audit in 2019 our auditor raised the issue of EKU chaining as required by Microsoft.
    Our auditor opened a discussion on Mozilla forum, and we interpreted the answers, that Microsoft requires the presence of the OCSP Signing EKU in the technically constrained CA certificates and Mozilla is not against it.
    Microsec issued 4 ICA certificates based on the public discussion on 2019-09-10 including the OCSP Signing EKU.
    We had no information about the existence of this problem before receiving the email from Mozilla on 2020-07-02.
    We haven't got any incident report from our users or from any other third parties.

  5. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

    Microsec will not issue any new CA certificates before having an agreement on the expected content of the technically constrained CA certificates with delegated OCSP responder.
    We hope that this will be fixed very soon, and makes it possible to proceed with the migration of our time stamping service to the new system.

Microsec opened a separate incident report for the late revocation of the TSA CA certificates:
https://bugzilla.mozilla.org/show_bug.cgi?id=1651632

(In reply to dr. Sándor SZŐKE from comment #2)

2019-09-10
Based on the received answers we agreed that there is no problem with the OCSP Signing EKU in the CA certificates and issued new CA certificates with OCSP Signing EKU. The earlier CA certificates have not been revoked so presently they are still valid.
The audit in September 2019 covered both versions of the affected CA certificates and the present attestation letter contains all of them.

How does that square with https://groups.google.com/d/msg/mozilla.dev.security.policy/XQd3rNF4yOo/bXYjt1mZAwAJ ? Which calls out the problem being referenced here?

Flags: needinfo?(szoke.sandor)

You are right, this comment includes the following sentence:
"An 'obvious' solution would have been to re-issue the Intermediate CA certificate and include id-kp-OCSPSigning in its EKU. Obviously wrong in this case since had we done so
<<the Intermediate CA (i.e. our customer) would then have been able to sign OCSP responses for any certificate issued by our Root CA.>>"

We could not recognize this problem and we made a bad decision in 2019. We issued 3 new ICA certificates with OCSP Signing EKU and in this way we caused problems to our root what we had not had before.

We now understand that there is no any formal CABF or Root Store Operator's requirement to add OCSP Signing EKU to an ICA due to EKU chaining, this only comes from the behavior of Microsoft CA software.

Flags: needinfo?(szoke.sandor)

I am willing to close this bug and consolidate further discussion under Microsec's bug for delayed revocation, Bug #1651632. The comments in this bug contain many valuable disclosures and observations which are preserved for cross-reference in that bug. However, before I close this bug, I would like to understand what steps Microsec takes to ensure it is following and participating in discussions that highlight these types of issues. For instance, I assume that Microsec follows and occasionally participates in discussions about the CA incidents of other CAs in Bugzilla; changes and discussions of CA/Browser Forum Guidelines, and in particular, certificate profiles; and m.d.s.p posts, which discuss developments and interpretations of the applicable standards for CAs. (Section 2.1 of the Mozilla Root Store Policy states, "CAs MUST follow and be aware of discussions in the mozilla.dev.security.policy forum, where Mozilla's root program is coordinated. They are encouraged, but not required, to contribute to those discussions.") I think that understanding steps you take to ensure this expectation is met, in order to detect and prevent future issues, is useful to ensure that no future interpretation issues arise.

I confirm that Microsec follows and occasionally participates in discussions about the CA incidents of other CAs in Bugzilla; changes and discussions of CA/Browser Forum Guidelines, and in particular, certificate profiles; and m.d.s.p posts, which discuss developments and interpretations of the applicable standards for CAs.
We have some pending tasks regarding our affected Code Signing CAs, we will put this information to Bug #1651632 in the future.

Closing this bug. Please refer to Bug #1651632 for further proceedings.

Status: ASSIGNED → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
Product: NSS → CA Program
Whiteboard: [ca-compliance] → [ca-compliance] [ocsp-failure]
You need to log in before you can comment on or make changes to this bug.