Closed Bug 1649945 Opened 4 years ago Closed 3 years ago

HARICA: Incorrect OCSP Delegated Responder Certificate

Categories

(CA Program :: CA Certificate Compliance, task)

Product:
CA Program
Component:
CA Certificate Compliance
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: ryan.sleevi, Assigned: jimmy)

Details

(Whiteboard: [ca-compliance] [ocsp-failure])

Attachments

(1 file)

Key Destruction Attestation
345.22 KB, application/pdf
Details

The following was originally reported to m.d.s.p. at https://www.mail-archive.com/dev-security-policy@lists.mozilla.org/msg13493.html

HARICA has issued one or more OCSP Delegated Responders, as defined within RFC 6960, Section 2.6 and Section 4.2.2.2, without including the id-pkix-ocsp-nocheck response, as required by the Baseline Requirements, Version 1, Section 13.2.5 through Version 1.7.0, Section 4.9.9

Example certificate: https://crt.sh/?id=2858289782

Please provide an incident report, including the timeline for revocation.

We have started our investigation on this topic and will provide our response.

Thanks for confirming. I noticed you went and CC'd yourself on all the other bugs related to this, and I'm encouraged to see that degree of proactiveness in learning from other CAs, although I do hope we'll see meaningfully distinct independent responses :)

Note that the recommended way for CAs to stay aware of incidents and incident responses is by subscribing to the "CA Certificates Component". If you click your profile (in the top right of Bugzilla), under "Preferences" there is a tab for "Component Watching". From there, you can monitor the Product of "NSS" with the Component of "CA Certificate Compliance", which will ensure you are notified on all compliance issues, subject to the "Email Preferences" tab. I would strongly encourage a CA to make sure they're subscribed to all comments, at a minimum.

I was already subscribed to that component but didn't get any of the messages and incidents you opened. At first, I thought there was something wrong with my email, I checked and it was ok.

It turned out that Bugzilla disabled my email delivery due to delivery issues back in June.

I have now re-enabled it so we should be good.

Regarding the incident, we are still preparing our response and discussing our mitigation plan, while evaluating the practical security impact of the issue and existing controls.

Our management is evaluating a proposed mitigation plan and will provide a summary tomorrow.

We are continuously evaluating more options in a short period of time to provide a thorough preliminary report. We plan on posting it Friday 2020-07-09.

Apologies for the date, it should be no later than Friday 2020-07-10. We also created ticket https://bugzilla.mozilla.org/show_bug.cgi?id=1651465 for the delayed revocation.

1. Incident Report Analysis

1.1 How HARICA first became aware of the problem

HARICA is monitoring the Mozilla dev-security-policy Forum. A post on 2020-07-02 reported a violation of the BRs for several CAs including HARICA. Later that day, bug 1649945 was created.

1.2 A timeline of the actions HARICA took in response

(A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.)

HARICA initiated an investigation to assess the issue and possible security implications on Relying Parties. Our top priority was to verify that these CA Certificates were not allowed to sign OCSP responses in our OCSP responder services. After reviewing configuration changes (all the way back to the issue date of the oldest affected CA Certificate) of our OCSP responder services, we determined that these CA Certificates were never enabled to sign OCSP responses.
As our investigation was progressing, we documented conflicting interpretations of the requirements by experts and a lot of contradictory messages in the discussion that followed in m.d.s.p. There has been a lot of information to process in a short period of time. We started working on a revocation/key destruction scenario and adjusted it based on the information that was analyzed.
Here is a detailed timeline:

  • 2020-07-02: Initial report of the incident
  • 2020-07-02: The team re-examined Baseline Requirements, RFC 5280 and RFC 6960 along with previous discussions in m.d.s.p. which acknowledged that some implementations would fail without “EKU chaining”, without calling out such specific security risks for a CA Certificate including this EKU. First analysis revealed two parts of the incident:
    • a compliance part (because of the lack of an extension which is mandatory for delegated OCSP responders and triggers a violation in the BRs despite the affected CA Certificates being Technically Constrained and out of BR scope) , and
    • a security part (because some OCSP clients accepted delegated OCSP responses regardless if the signing Certificate is an end-entity or a CA Certificate). We also started a separate investigation to discover how many popular Browsers are affected by this security issue. We quickly identified that Mozilla Firefox and implementations using NSS mozilla::pkix library are not affected.
  • 2020-07-02: We identified the full set of non-expired non-revoked CA Certificates affected by this incident. Since all HARICA’s SubCAs are operated internally, the risk of a third-party CA operating an affected SubCA that includes the id-kp-OCSPSigning EKU and sign OCSP responses on behalf of its issuing Root CA. As the CA keys were constantly protected inside HSMs and under strict management, the likelihood of this event taking place was very low. Regardless of this fact, we continued our investigation to collect evidence of our OCSP configurations since 2018-05-23 which is the notBefore date of the oldest affected CA Certificate.
  • 2020-07-03: Given the fact that Mozilla Firefox was not affected by this incident (and possibly other User Agents) and that HARICA had the affected CA keys under audited control (outside OCSP responder systems) at all times, the team started to draft a mitigation plan, assessing the damage of revoking all CA Certificates and associated end-entity Certificates within the 7 day period per section 4.9.1.2 of the BRs. The approach to create replacement CA Certificates without the id-kp-OCSPSigning EKU was considered (using same KeyPairs), in order to meet the 7-day revocation requirement of the BRs, but was dismissed by the team as it did address the compliance issue, but not the security one.
  • 2020-07-04: We searched the Certificate database for the exact number of affected non-expired, non-revoked non-TLS Subscriber Certificates from the affected subCAs, trying to assess the complexity and constraints of replacing them. Our investigation showed that a substantial percentage of these had private keys generated in FIPS crypto-devices which makes quick replacement a more difficult task compared to replacement of TLS certificates.
  • 2020-07-05: After considering all the compliance, security and business continuity implications, we agreed on main course of action and started drafting a complete mitigation plan. The first step of the plan was to stop issuing certificates from the affected SubCAs and plan for creating new Issuing CAs.
  • 2020-07-06: We stopped issuing Certificates from the affected subCAs. A preliminary revocation plan was created trying to balance the difficulty in replacing these non-TLS certificates with keys in FIPS devices and the risks associated with this incident. We contacted our Subscribers to evaluate the operational risks of revocation and determined that a large number of these Certificates are used in mission-critical Academic operations and their immediate revocation would interrupt important services of the Academic sector.
  • 2020-07-06: In parallel, we continued our investigation to identify affected clients as a means to assess the possible impact on relying parties. In particular, our analysis on Chromium code revealed that Chromium-based Browsers are vulnerable to this security issue. It also located a relevant comment in the source code: “Not all properties of the certificate are verified, only the signature and EKU. Can full RFC 5280 validation be used, or are there compatibility concerns?”
  • 2020-07-07: An audit log analysis was performed to collect evidence on our OCSP responder system configuration parameters, including all changes since 2018-05-23 (the notBefore date of the oldest affected SubCA). Our CA Software provides cryptographically verifiable audit logs, evaluated and certified by Common Criteria (Audit logging is specified in 6.1: Security Audit) (see more details from PrimeKey), which can be used both by us and by external qualified auditors to evaluate and confirm completeness (unbroken sequence of logs) and integrity. These audit logs confirmed that the Internal OCSP responder system was never configured to use any of the affected CA Certificates to sign OCSP responses. We also contacted our CAB to discuss possible special audit activities to provide public assurance on specific assertions regarding CA and OCSP Responder configuration history.
  • 2020-07-08: Since issuance of new Certificates has stopped for the affected subCAs, we continued exploring options to disable the affected CA keys in our HSMs and enable them before the end of the nextUpdate to sign new CRLs, until the day we revoke and destroy the keys. Fresh revocation information would primarily be delivered via our delegated OCSP responder service. We are still examining possible compliance issues with this plan.
  • 2020-07-09: We decided to speed up the design and implementation of tools, which were already part of our 2020 plan, to allow for faster replacement of non-TLS Certificates, especially those that require their associated private keys to be generated in crypto-devices. The updated plan is to deliver the first version of these tools by the end of July, test them thoroughly in August and roll out in September.
  • 2020-07-10: Discussions with affected Subscribers continued, especially those that had difficulties replacing Issuing CAs in production systems. HARICA analyzed some special use cases of these certificates in various applications, offered solutions and a replacement plan was agreed for every special use case. These special use cases dominated the final revocation deadline and key destruction of all affected subCAs.

1.3 Has HARICA stopped, or has not yet stopped, certificate issuance or the process giving rise to the problem or incident?

(A statement that you have stopped will be considered a pledge to the community; a statement that you have not stopped requires an explanation).

Yes, issuance of new Certificates for Subscribers by the affected Issuing CAs has stopped since 2020-07-06.

1.4 In a case involving certificates, a summary of the problematic certificates

(For each problem: the number of certificates, and the date the first and last certificates with that problem were issued. In other incidents that do not involve enumerating the affected certificates (e.g. OCSP failures, audit findings, delayed responses, etc.), please provide other similar statistics, aggregates, and a summary for each type of problem identified. This will help us measure the severity of each problem.)

We counted 39 non-expired, non-revoked CA Certificates affected by this incident. These are non-TLS, Technically Constrained subCA Certificates.

1.5 The complete certificate data for the problematic certificates

(In a case involving certificates, the complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem. In other cases not involving a review of affected certificates, please provide other similar, relevant specifics, if any)

The entire CA certificate database was examined. The serial numbers of affected unexpired, unrevoked CA Certificates that contain the id-kp-OCSPSigning KeyPurposeId in their EKU extension are:

  • 6a042ec821d7be27
  • 225f61e114a253ba
  • 7987f879914d8e5a
  • 34138f19ad11938b
  • 6f54453aaa35942e665e715c911c514c
  • cabeb1fdd8bf54fadb4fda2f1dbfbf5
  • 777ce06d1120233381548777a9a01992
  • 434e7a1c96eac47444917d264778553e
  • 7f55382ec12749e0483e807dcddcf5c8
  • 1f0fee987b70762683aac7f513cadab5
  • 6062ec64ac31f2ba4e1df9f9ad32aaba
  • 638679695ebaf6648eb4a82345d87387
  • 2a29da7fb6b3c731628bd14291b18813
  • 217d97e4f49d23d6e4bd42c5288c7a35
  • 36c676021860fee9bb22820629e38ffa
  • 5c75cb1eb88cc0a02aa354497356ff2b
  • 7c41c8a6f221887d512082ca63a6a4f3
  • 396cbb1871dd3628da0b98c80b16437a
  • 5ac593830e1ef6fcf468e7da996fcf0
  • 2a802df52d4e69b9471c59abc78b5ee1
  • 1920cf7f72fabfacfdc8adc6a4495a3b
  • 781bb21753dcb9200cd0f01b136828fb
  • 66e931e4788e35e93ce3a656ac2d00ae
  • 56ffb91d4a66b4cb44711cba6e64dc8f
  • 638b242226277dfc5b772c81fab4922c
  • 3a57ac850d47eab182b377e8d7bf734c
  • 58d22f7a371dd2b68d7a0fbebafedb69
  • 3604be85ae18a1db22ff2be3b1024c2e
  • 3495669547a5f4b77058aab0e1371b68
  • 98dd7772d4982fd293a954715a1ed34
  • 51be44c52732763902c5f9eb8a2971e6
  • 2bc9d246094e9524ac1900677c1a8ef8
  • 6c4330bfc874ac7ec0cbd55230f0f0d
  • 3f53f31cf0acb0dc9b3d1d5751a4f6a6
  • 1e0b491d8fa481ef2d52b24578abcf18
  • 5451504d35ad11c9a6b846e79532bffb
  • 7d6c907fb2b5282762e484a8153fdbaf
  • 34700b6cc7b7493f5ed0db5c92cf5c52
  • 35a11942e5a78a05107f3ed1a9963b11

Most of these CA Certificates are disclosed in CCADB despite the fact that they are Technically Constrained. HARICA decided to add non-TLS CAs to CCADB once this became a requirement from Mozilla for unconstrained non-TLS CA Certificates. All HARICA CA Certificates are disclosed in https://repo.harica.gr.

We will provide crt.sh links on a later update. In the meantime, most of these certificates can be discovered via https://crt.sh as https://crt.sh/?serial=XXXXX.

1.6 Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

The interpretation of the combination of applicable requirements (Root Store program requirements, BRs, the enforcement of both RFC 5280 and 6960) was that the inclusion of id-kp-OCSPsigning KeyPurposeId in technically constrained subCAs was allowed. In addition, according to BRs section 7.1.2.2, it would further protect Relying Parties. The protection of Relying Parties and the proper management/isolation of risks are the main reasons that HARICA extensively uses the nameConstraints and EKU extensions in Issuing CA Certificates, which is to limit the unintended use of associated CA Keys.

The issue was also discussed in November 2019 (during our monitoring of a relevant discussion on m.d.s.p. regarding interoperability issues of Microsoft’s CA software), but the result of that discussion, in our understanding, was inconclusive.

1.7 List of steps HARICA is taking to resolve the situation and ensure that such situation or incident will not be repeated in the future

(List of steps your CA is taking to resolve the situation and ensure that such situation or incident will not be repeated in the future, accompanied with a binding timeline of when your CA expects to accomplish each of these remediation steps)

HARICA makes extensive use of both EKU and nameConstraint extensions to technically constrain its ICAs in accordance with section 7.1.5 of Baseline Requirements and Mozilla Root Store Policy (for S/MIME Certificates), as a technically enforceable means of protecting Relying Parties and managing/isolating risks.

HARICA operates all its Root and Intermediate CA Certificates on its own; all affected Intermediate CA Certificates are operated internally under the same security environment and by the same trusted personnel as our Root CA Certificates. All our CAs, including the technically constrained ones, are regularly audited against the "baseline" ETSI standards (i.e. ETSI EN 319 401, ETSI EN 319 411-1 which apply to all certificate types and policies). All affected CAs are included in audit reports without gaps, as attested by the qualified external auditors.

After thorough investigation of our cryptographically verifiable audit logs (the audit mechanism has been evaluated/certified by Common Criteria (Audit logging is specified in 6.1: Security Audit), see more details from PrimeKey), we have confirmed that the affected CA Certificates have never been configured in our OCSP responder services that would enable them to issue a malicious OCSP response on behalf of their parent (RootCA). We have also contacted our CAB to explore the possibility of an external audit team to examine the above and to issue an independent letter of opinion (attestation). Our CAB confirmed that it is able to provide this independent assessment if requested, as an additional measure of assurance to Relying Parties.

We consider that the above facts significantly decrease the probability of a successful attack and result in very low security risk. Revocation within the BR required timeframe for our affected non-TLS certificates would cause significant operational risks for a large number of our subscribers and Relying Parties (who depend on those subscribers), causing more damage to the ecosystem than is warranted by the inherent risk. Thus, we designed a more balanced approach with regards to the revocation time-line of the affected CA certificates.

To further mitigate the threat, until all affected CA Certificates are revoked and their keys destroyed:

  1. The id-kp-OCSPSigning KeyPurposeId has been removed from all CA Certificate profiles.
  2. We have switched issuance of new Subscriber Certificates to compliant ICAs. Only revocation information will be signed using the keys of the affected CAs.
  3. We are already making progress contacting our Subscribers in order to replace the certificates issued by the affected Issuing CAs, so that we revoke each ICA as soon as possible. We are already building tools and updating/improving our procedures to accommodate the large volume of operations.
  4. 2020-10-31 has been set as the deadline for revoking all affected Issuing CA Certificates. We expect all affected subscribers to have replaced their non-TLS certificates by the end of October. The decision for this deadline was based on the difficulties of replacing some end-entity certificates. This will be further explained in the corresponding delayed revocation bug.
  5. Keys of the affected CAs will be destroyed during the first week of November. This will be witnessed by our CAB, along with examination of audit logs (OCSP responder system configuration) for the period of time up to the destruction of keys.

Learning from past experience with replacing TLS certificates (and also during the serialNumber entropy incident), HARICA has already drafted plans that would accelerate the replacement of non-TLS Certificates when necessary. These efforts were already underway before this incident, but they were not completed nor tested extensively. Some portions of this plan can and will be used during the course of this incident. We will need to “fast track” the remaining pieces that will enable this replacement to happen timely and we have already allocated the necessary resources. A significant number of affected certificates, with keys included in FIPS devices, will be replaced in a relatively short period of time, because during provisioning of the devices we pre-generated additional keys in case of compromise or disaster.

Considering the fact that different trust frameworks show increasingly different expectations, we have a plan to create separate hierarchies for different uses, especially hierarchies related to server TLS. This is scheduled for Q4 2020.

2. Incident Impact

This incident has unintended and unexpected consequences on a number of HARICA's non-TLS Subscribers, where certificates have to be replaced. No other security impact was detected for Subscribers or Relying Parties since existing controls already protected against the new attack vectors introduced by this vulnerability.

3. Conclusions and Recommendations

We consider the past and existing controls reasonable to assure Relying Parties that the affected CA Certificates were not used to sign any OCSP Responses on behalf of their Issuer (i.e. delegated OCSP responses). This new vulnerability makes these affected issuing CAs more "attractive" to attackers, because getting access to them would not only allow them to issue end-entity certificates, but to also have the "unrevoke" capability if the Root CA were to revoke such an ICA. However, all of HARICA's affected ICAs are Technically Constrained to specific email domains and organizations via the nameConstraints extension, making them less "attractive" to attackers.

The proposed mitigations which result in the destruction of the Keys associated with the affected CA Certificates, will provide further assurances going forward.

Separation of the TLS hierarchy from other usages will isolate the TLS risks to the TLS hierarchy only.

This the first time that HARICA decided to violate the revocation timelines of the Baseline Requirements, in an effort to act responsibly to this security incident and balance the ecosystem damage from mass revocation of non-TLS Certificates. Mozilla and Microsoft Root store Program managers acknowledged the difficulty in mass revocation of certificates within 7 days. HARICA had an additional difficulty that the only Certificates affected by end users were non-TLS, some of which with keys generated in crypto-devices. Even in the serialNumber entropy incident that affected a great number of Certificates, HARICA managed to complete the revocations in the required timeline and already has the necessary controls to contact affected Subscribers and help them replace TLS Certificates timely.

This incident is an opportunity for us to extend our existing tools for certificate replacement in non-TLS environments, and communicate these tools to our Subscribers depending on the use cases. We also plan to notify our non-TLS users and explain the risks of building "custom" use cases that "pin" or "depend" on specific ICAs and end-entity certificates, as these may need replacement after a short notice in case of a security-related event going forward. This will provide improved agility to our non-TLS services for our Subscribers and Relying Parties.

(In reply to Dimitris Zacharopoulos from comment #7)

After thorough investigation of our cryptographically verifiable audit logs (the audit mechanism has been evaluated/certified by Common Criteria (Audit logging is specified in 6.1: Security Audit), see more details from PrimeKey), we have confirmed that the affected CA Certificates have never been configured in our OCSP responder services that would enable them to issue a malicious OCSP response on behalf of their parent (RootCA). We have also contacted our CAB to explore the possibility of an external audit team to examine the above and to issue an independent letter of opinion (attestation). Our CAB confirmed that it is able to provide this independent assessment if requested, as an additional measure of assurance to Relying Parties.

I'm hoping you can provide more specific details here. I appreciate the detail you've provided, but I'm having trouble squaring this.

Does the HSM provide these audit logs? Or is it simply your CA software that provides these audit logs? How is your system designed in a way that interactions are made to an HSM, how are those interactions authenticated, etc.

As mentioned on m.d.s.p., it is not the auditor's opinion that provides assurance. It's an understanding of the facts that lead to that auditor's opinion that are, demonstrated time again, more relevant, and ETSI ESI-based audits are already significantly limited in that regard, because of the inherent limitations of design of ETSI audits and their design.

As has been suggested to other CAs, the notion of assurance needed is one of a Detailed Control Report equivalent, for which no such thing exists for ETSI, and I think there would be significant concerns were it to be invented bespoke in response to this.

The goal of this report should be to holistically help not just browsers, but relying parties, understand your architecture, why you believe there are a sufficient number of controls, why those controls themselves are sufficient, how this can be objectively verified regardless of the auditor (e.g. by thorough documentation about the factors the auditor considered and the system design reviewed), etc.

You have the foundations of something useful, and I want to acknowledge that; however, there's a large element of "trust us" (and/or "trust our auditor"), and given the security factors involved, it's not unreasonable to ask "show us why".

Flags: needinfo?(jimmy)

(In reply to Dimitris Zacharopoulos from comment #7)

  • 2020-07-02: The team re-examined Baseline Requirements, RFC 5280 and RFC 6960 along with previous discussions in m.d.s.p. which acknowledged that some implementations would fail without “EKU chaining”, without calling out such specific security risks for a CA Certificate including this EKU.

I read through the discussion at the MDSP link, and I see two messages that call out the risks of a CA certificate having an OCSP signing EKU. I pasted below:

On Wednesday, September 4, 2019 at 4:18:28 p.m. UTC-4, Jakob Bohm wrote:

This Microsoft requirement is highly unfortunate, because unless there
is an explicit RFC permission that allows OCSP clients to reject OCSP
certificates from delegated OCSP responder certificates with CA:TRUE,
yet accept them from issuing CA certificates with CA:TRUE; clients will
be required by RFCs to accept bogus OCSP responses signed by SubCA
certificates that contain the EKU for Microsoft compatibility.

This is especially bad if the SubCA is controlled by an entity other
than its direct parent CA.

On Tuesday, September 10, 2019 at 9:53:47 a.m. UTC-4, ro...@comodoca.com wrote:

An 'obvious' solution would have been to re-issue the Intermediate CA certificate and include id-kp-OCSPSigning in its EKU. Obviously wrong in this case since had we done so the Intermediate CA (i.e. our customer) would then have been able to sign OCSP responses for any certificate issued by our Root CA.

(In reply to Ryan Sleevi from comment #8)

(In reply to Dimitris Zacharopoulos from comment #7)

After thorough investigation of our cryptographically verifiable audit logs (the audit mechanism has been evaluated/certified by Common Criteria (Audit logging is specified in 6.1: Security Audit), see more details from PrimeKey), we have confirmed that the affected CA Certificates have never been configured in our OCSP responder services that would enable them to issue a malicious OCSP response on behalf of their parent (RootCA). We have also contacted our CAB to explore the possibility of an external audit team to examine the above and to issue an independent letter of opinion (attestation). Our CAB confirmed that it is able to provide this independent assessment if requested, as an additional measure of assurance to Relying Parties.

I'm hoping you can provide more specific details here. I appreciate the detail you've provided, but I'm having trouble squaring this.

Does the HSM provide these audit logs? Or is it simply your CA software that provides these audit logs? How is your system designed in a way that interactions are made to an HSM, how are those interactions authenticated, etc.

As mentioned on m.d.s.p., it is not the auditor's opinion that provides assurance. It's an understanding of the facts that lead to that auditor's opinion that are, demonstrated time again, more relevant, and ETSI ESI-based audits are already significantly limited in that regard, because of the inherent limitations of design of ETSI audits and their design.

As has been suggested to other CAs, the notion of assurance needed is one of a Detailed Control Report equivalent, for which no such thing exists for ETSI, and I think there would be significant concerns were it to be invented bespoke in response to this.

The goal of this report should be to holistically help not just browsers, but relying parties, understand your architecture, why you believe there are a sufficient number of controls, why those controls themselves are sufficient, how this can be objectively verified regardless of the auditor (e.g. by thorough documentation about the factors the auditor considered and the system design reviewed), etc.

You have the foundations of something useful, and I want to acknowledge that; however, there's a large element of "trust us" (and/or "trust our auditor"), and given the security factors involved, it's not unreasonable to ask "show us why".

Our CA software performs the signing operations and each signing operation is authenticated by the HSM and logged in a tamper-proof auditlog as described. Our HSM requires authentication before allowing key signing operations and, of course, it resides in a high security zone. Therefore, the integrity of the auditlog is guaranteed and provides the entire history of any interaction between the CA software and the HSM.

As explained in our preliminary report, we did not rely solely to the auditor to verify and confirm the expected settings of the OCSP Responder configurations, we scanned the logs ourselves and confirmed the expected behavior. The meaning of our proposal for an external witnessing of the destruction of the keys is to provide a separate, independent opinion which provides additional assurance to relying parties.

Regarding the ETSI critique about attestations, we believe that ISO 17065 and ETSI EN 319 403 set the proper foundations for independent audit attestations for special issues covered in the ETSI EN 319 401 and 411-1, including CA Private Key management and all technical and procedural aspects of setting up and operating OCSP responders. Therefore, it should be expected for ETSI auditors to be in a position to perform an independent assessment of the issues at hand, produce attestation or opinion letters that the CA can provide to Relying Parties as an independently “witnessed” report. It is similar to the key generation reports and the key destruction reports that have been in existence. Special audits are tools in the ETSI scheme to check and verify corrective actions but also for evaluation of significant system changes including new services (see 7.10 in ISO 17065 and ETSI EN 319 403/403-1).

This new vulnerability introduced by CA Certificates that include the id-kp-OCSPSigning EKU, poses a very low (almost zero) additional threat to HARICA due to the fact that all affected subCAs are operated internally, the RootCA OCSP responders (with the “proper” delegated OCSP responder end-entity Certificates) are also operated internally, all controls protecting these operations are continuously monitored by HARICA. In addition to doing our own homework, these have been audited and independently evaluated by our CAB for years.

(In reply to Mathew Hodson from comment #9)

(In reply to Dimitris Zacharopoulos from comment #7)

  • 2020-07-02: The team re-examined Baseline Requirements, RFC 5280 and RFC 6960 along with previous discussions in m.d.s.p. which acknowledged that some implementations would fail without “EKU chaining”, without calling out such specific security risks for a CA Certificate including this EKU.

I read through the discussion at the MDSP link, and I see two messages that call out the risks of a CA certificate having an OCSP signing EKU. I pasted below:

On Wednesday, September 4, 2019 at 4:18:28 p.m. UTC-4, Jakob Bohm wrote:

This Microsoft requirement is highly unfortunate, because unless there
is an explicit RFC permission that allows OCSP clients to reject OCSP
certificates from delegated OCSP responder certificates with CA:TRUE,
yet accept them from issuing CA certificates with CA:TRUE; clients will
be required by RFCs to accept bogus OCSP responses signed by SubCA
certificates that contain the EKU for Microsoft compatibility.

This is especially bad if the SubCA is controlled by an entity other
than its direct parent CA.

On Tuesday, September 10, 2019 at 9:53:47 a.m. UTC-4, ro...@comodoca.com wrote:

An 'obvious' solution would have been to re-issue the Intermediate CA certificate and include id-kp-OCSPSigning in its EKU. Obviously wrong in this case since had we done so the Intermediate CA (i.e. our customer) would then have been able to sign OCSP responses for any certificate issued by our Root CA.

As you correctly mentioned, these specific CA risks were called out as “especially bad if the SubCA is controlled by an entity other than its direct parent CA”. HARICA has not been operating the affected subCAs externally, therefore it is did not seem to be affected by this threat at a level that required immediate revocations.

As already mentioned, we shall manage residual risks by revoking and destroying the keys of the affected subCAs. Separate hierarchies shall also be created in order to erase any doubt going forward.

This week we focused on improving and testing our Certificate replacement tools (mainly the ones stored on FIPS devices) and some changes to our Registration Authority. We arranged for a special audit by our CAB to review these changes and confirm they are aligned with the applicable standards.

We continue to engage with subscribers that have special uses of their non-TLS certificates associated with private keys in FIPS devices. We have proposed solutions that need to be thorougly tested to ensure that no critical operations are disrupted.

We also sent out notifications to affected Subscribers with Certificates not associated with private keys in FIPS devices, with instructions for acquiring new certificates to replace existing ones.

Finally, we continue to evaluate the possibility of bringing the private keys of the affected CAs “offline” and sign CRLs on a weekly basis. We will try to get some clarity on the existing requirements regarding “Revocation Information” and whether providing this information via OCSP continuously but CRLs on a weekly basis, is considered compliant or not. This compliance question is more closely coupled with Qualified Certificates as it relates to status information for non-TLS Certificates issued from Technically Constrained non-TLS Issuing CAs.

We continue working on the issue and will provide another update next week.

(In reply to Dimitris Zacharopoulos from comment #10)

(In reply to Ryan Sleevi from comment #8)

Does the HSM provide these audit logs? Or is it simply your CA software that provides these audit logs? How is your system designed in a way that interactions are made to an HSM, how are those interactions authenticated, etc.

Our CA software performs the signing operations and each signing operation is authenticated by the HSM and logged in a tamper-proof auditlog as described. Our HSM requires authentication before allowing key signing operations and, of course, it resides in a high security zone. Therefore, the integrity of the auditlog is guaranteed and provides the entire history of any interaction between the CA software and the HSM.

This doesn't really answer the "how" that I was previously asking. You acknowledge that it is, but the goal here is not just a statement "it is", but a holistic description of the system design and architecture. Consider bugs like Bug 1556948 or Bug 1595921 that help build a holistic picture about how a CA is operating and what they're changing to fix.

Regarding the ETSI critique about attestations, we believe that ISO 17065 and ETSI EN 319 403 set the proper foundations for independent audit attestations for special issues covered in the ETSI EN 319 401 and 411-1, including CA Private Key management and all technical and procedural aspects of setting up and operating OCSP responders. Therefore, it should be expected for ETSI auditors to be in a position to perform an independent assessment of the issues at hand, produce attestation or opinion letters that the CA can provide to Relying Parties as an independently “witnessed” report. It is similar to the key generation reports and the key destruction reports that have been in existence. Special audits are tools in the ETSI scheme to check and verify corrective actions but also for evaluation of significant system changes including new services (see 7.10 in ISO 17065 and ETSI EN 319 403/403-1).

This is conflating several things, although I do appreciate your specific references. This conflates a key destruction report, which I agree in theory an ETSI-accredited auditor could provide, with a more general report about what controls the auditor is testing, the results of those tests, etc. You provide reference to how an auditor can offer their opinion, but as I mentioned, without understanding the underlying facts and basis of that opinion, it's not useful. This is especially true given the rather sizable differences of opinion with respect to ETSI auditors and browser expectations.

As I tried to highlight in Comment #8, this is the opportunity for HARICA to provide more detail about its architecture, and then be able to provide more detail about how the controls can test that nothing is amiss in that architecture. Obviously, this has to be holistic and multi-layered. For example, if the HSM will sign whatever an authenticated channel will send it, how do you ensure that whatever the authenticated channel is correct? If, for example, you've enabled arbitrary software to be automatically downloaded and installed (e.g. enabling OS updates), how are you managing and monitoring that software?

I realize you've set your plan for October/November of this year, and that's quite encouraging, and better than several other CAs. However, what I'm trying to make sure we capture now, and think about going forward, is how we can document the "Here's how our system is designed, here's how we have multiple layers to detect anything goes wrong, here's how we'll be monitoring and getting independent confirmation", and going forward. In particular, I want to make sure that if, in the unfortunate likelihood, there's ever an incident "like" this (e.g. latent risk relying upon CA's correct operation, with a near-impossible ability to detect malfeasance by the public), we have better tools next time.

I want to commend HARICA for the detail of it's reporting, and updates: this is much better than a number of CAs, and encouraging to see. For the most part, I want to believe that HARICA has this under control, but I want to make sure we get an better degree of transparency before shifting the next-update.

We have prepared most of our next progress report and expect to have it ready no later than Monday July 27.

(In reply to Ryan Sleevi from comment #13)

(In reply to Dimitris Zacharopoulos from comment #10)

(In reply to Ryan Sleevi from comment #8)

Does the HSM provide these audit logs? Or is it simply your CA software that provides these audit logs? How is your system designed in a way that interactions are made to an HSM, how are those interactions authenticated, etc.

Our CA software performs the signing operations and each signing operation is authenticated by the HSM and logged in a tamper-proof auditlog as described. Our HSM requires authentication before allowing key signing operations and, of course, it resides in a high security zone. Therefore, the integrity of the auditlog is guaranteed and provides the entire history of any interaction between the CA software and the HSM.

This doesn't really answer the "how" that I was previously asking. You acknowledge that it is, but the goal here is not just a statement "it is", but a holistic description of the system design and architecture. Consider bugs like Bug 1556948 or Bug 1595921 that help build a holistic picture about how a CA is operating and what they're changing to fix.

The design follows one of the recommended architectures using external OCSP Responders. All the details about how a “VA” (i.e. delegated OCSP responder) interacts with the CA part is described in the CA software’s documentation which is publicly available. The external VA has no access to the CA signing components; only one-way synchronization takes place (direction from CA → to VA) regarding the status of Certificates issued by the CA for which the delegated VA is responsible for. So, even if an attacker gained access to an external VA responsible for an affected subCA, the attacker would not be able to sign anything associated with the affected subCA Certificate because access to the corresponding CA key resides on a separate system, located in a high security zone, behind a separate set of firewalls, as described in the diagram.

The HSMs protecting the affected CA keys are in an isolated network (separate L2 network from the delegated OCSP responders and RA software) and are only accessible from the CA Software which captures every signing activity as already described. Firewall rules allow access to specific APIs which accept only well-formed requests for “expected” and approved operations. Further Access Control rules are applied to the CA Software (we follow the least privilege principle) allowing access only to a specific set of operations and resources via Client Certificate authentication mechanism. Every such activity is captured at the CA’s tamper-proof audit log and continuously monitored by a host-based intrusion detection system.

Direct access to the HSMs require local privileged access to internal systems which are not accessible from the Internet or any of the RA or VA servers. This access is limited to a small number of HARICA’s Trusted Role Members. Local access to those critical internal systems is continuously monitored (with appropriate alerting) via technical and organizational controls (see below for more details).

Our trusted personnel performs regular checks to verify all the critical parts of the infrastructure, for example VLAN configuration, firewall rules (network level), user accounts, privileges and access history, changes and updates, audit logs and so on. We apply separation of duties and dual control (on top of background checks, confidentiality agreements, etc) to minimize the personnel-related risks.

In addition to our internal monitoring and evaluation functions, all the above are regularly audited by external CAB who meet the criteria specified in Root Store Programs for ETSI auditors (Mozilla, Microsoft). The audit team consists of people with complementary competencies in auditing, networks, systems, human resources, etc, who perform independent verification of these claims. Obviously, we cannot answer on behalf of our CAB but this information comes from our experience as the auditees.

We want to highlight that HARICA’s security team regularly monitors, verifies our PKI and doesn’t rely on external auditors to check for compliance. We use the external audit as an additional independent evaluation/confirmation.

Regarding the ETSI critique about attestations, we believe that ISO 17065 and ETSI EN 319 403 set the proper foundations for independent audit attestations for special issues covered in the ETSI EN 319 401 and 411-1, including CA Private Key management and all technical and procedural aspects of setting up and operating OCSP responders. Therefore, it should be expected for ETSI auditors to be in a position to perform an independent assessment of the issues at hand, produce attestation or opinion letters that the CA can provide to Relying Parties as an independently “witnessed” report. It is similar to the key generation reports and the key destruction reports that have been in existence. Special audits are tools in the ETSI scheme to check and verify corrective actions but also for evaluation of significant system changes including new services (see 7.10 in ISO 17065 and ETSI EN 319 403/403-1).

This is conflating several things, although I do appreciate your specific references. This conflates a key destruction report, which I agree in theory an ETSI-accredited auditor could provide, with a more general report about what controls the auditor is testing, the results of those tests, etc. You provide reference to how an auditor can offer their opinion, but as I mentioned, without understanding the underlying facts and basis of that opinion, it's not useful. This is especially true given the rather sizable differences of opinion with respect to ETSI auditors and browser expectations.

As already described, we use the external audit as an additional independent evaluation/confirmation of our own due diligence.

Since we don’t have experience about how external auditors approach CA key destruction activities, we can only draw conclusions from past auditing experience and remain in close collaboration with them in order to prepare this evaluation in the best possible way. We have been informed that our auditors monitor this set of bugs (ours and of other CAs) and any relevant discussion in m.d.s.p.

As I tried to highlight in Comment #8, this is the opportunity for HARICA to provide more detail about its architecture, and then be able to provide more detail about how the controls can test that nothing is amiss in that architecture. Obviously, this has to be holistic and multi-layered. For example, if the HSM will sign whatever an authenticated channel will send it, how do you ensure that whatever the authenticated channel is correct? If, for example, you've enabled arbitrary software to be automatically downloaded and installed (e.g. enabling OS updates), how are you managing and monitoring that software?

Only the software that has a need for specific keys has the credentials to establish the authenticated channel to the HSM. The software that has access to the HSM is only manually updated by an authorized administrator. All changes to the CA software (CA configuration, Certificate Profiles, CT Logs, etc) are applied using a two-person rule by authorized administrators.

We don’t consider the official security updates of software we previously reviewed and approved (e.g. Linux kernel) as “arbitrary” software. The integrity and authenticity of these signed updates are verified before installation using the digital signatures of the vendor. Detailed logs of all system updates are maintained and system logs are continuously monitored using log analysis tools and alerting the security administrators in case of suspicious activity.

I realize you've set your plan for October/November of this year, and that's quite encouraging, and better than several other CAs. However, what I'm trying to make sure we capture now, and think about going forward, is how we can document the "Here's how our system is designed, here's how we have multiple layers to detect anything goes wrong, here's how we'll be monitoring and getting independent confirmation", and going forward. In particular, I want to make sure that if, in the unfortunate likelihood, there's ever an incident "like" this (e.g. latent risk relying upon CA's correct operation, with a near-impossible ability to detect malfeasance by the public), we have better tools next time.

We tried to better describe how our CA-VA system is designed and how they interact with each other, and along with other information described in this bug, we believe that the possible “malicious” activity is well contained within HARICA. We are exploring additional controls that would even further minimize the risk, such as keeping the affected CA keys offline for the period until the key destruction ceremony (more details are provided in the progress report).

Our focus and commitment remains on overall improving our system controls, to be able to avoid such incidents and also to improve our response capability according to the expectations of the industry, in the unlikely event of a similar compliance issue. For this reason, we are implementing a series of improvements and better tools to handle such incidents in the future (more details will be shared in the delayed revocation bug).

Last week we continued testing our Certificate replacement tools (mainly the ones stored in FIPS devices) and changes to our new Registration Authority portal. We completed a special audit by our CAB to review these changes and confirm they are aligned with the applicable standards.

We had a conference call with our Supervisory Body explaining the need to revoke non-TLS Certificates that issue Qualified Certificates, our proposed timeline and how we will reach this goal.

We started a discussion at the ETSI ESI Working Group to get some clarity on the existing requirements regarding “Revocation Information” and the need to “publish” this information every 24 hours. We intend to continue this discussion with our Supervisory Body proposing the following practice:

The CRL must be updated and published:

  • for end-entity Certificates, every seven (7) days at the latest. The CRL will be in effect for a maximum time of seven (7) days. In case of revocation of an end-entity Certificate, a new CRL shall be issued and published within twenty-four (24) hours from the revocation timestamp.

This practice would allow us to further lower the residual risks by keeping the private keys of the affected subCAs offline, and bring them online only to sign CRLs until all end-entity certificates have been replaced and then revoked. Once all end-entity certificates are revoked, we would revoke the CAs, keep the keys constantly offline and would wait for the key destruction ceremony beginning of November.

We also did a little research on how to self-issue a CRL signing certificate (with a different key), and issue+publish a CRL with this new self-issued cert according to RFC5280. Our CA software doesn’t support this function so we are still investigating if it is feasible and secure to proceed with such a solution. We would also need to do some research about whether Relying Parties software supports this CRL validation method.

Finally, we re-evaluated the risks of immediate revocation based on discussions with Institutions that need more time to update their software implementations for critical Academic operations, and we concluded on the proposed timeline for revocation at the end of October 2020.

Based on the submitted information, we kindly ask setting the next-update to August 31, 2020.

Whiteboard: [ca-compliance] → [ca-compliance] Next update 31-Aug-2020

We would also need to do some research about whether Relying Parties software supports this CRL validation method.

Unfortunately, most RP software does not support this, and, if I recall correctly, relies on the CRL Distribution Point extension of existing certificates.

  • 2020-07-27 to 2020-08-31:
    • The Remote QSCD implementation was audited and certified by our CAB. Preliminary results were sent to the Supervisory Body according to the eIDAS Regulation
    • Changes to our RA system went live
    • The certificate replacement tool went live
    • Subscribers with the capability of reusing keys and issuing new Certificates from compliant Issuing CAs were notified and started replacing certificates remotely
    • A follow-up special audit was performed to verify the changes to the RA
    • Investigation about the need to issue daily CRLs from non-TLS Issuing CAs is still in progress
    • We started preparations for the CA key destruction ceremony of affected subCAs scheduled for the first week of November
    • We plan to revoke and bring the keys for 10 subCAs offline during the week of 2020-08-31. We plan to use this practice as we "evacuate" affected ICAs, meaning these subCAs will have either expired or revoked end-entity Certificates.

Our proposed plan is on track, therefore we would kindly ask to allow us to provide updates for this incident every two-weeks, unless our plan doesn't make progress as expected, in which case we will switch to weekly updates.

Whiteboard: [ca-compliance] Next update 31-Aug-2020 → [ca-compliance] Next update 22-Sept-2020
  • 2020-09-01 to 2020-09-21:
    • The following subCAs were revoked on 2020-09-03
      • HaricaAcademyOfAthensClientSubCAR2
      • HaricaAcademyOfAthensClientSubCAR3
      • HaricaAuthClientSubCAR1
      • HaricaECCAdministrationClientSubCAR2
      • HaricaEcclesiasticalAcademyofVellaClientSubCAR1
      • HaricaEcclesiasticalAcademyofVellaClientSubCAR2
      • HaricaEcclesiasticalAcademyofVellaClientSubCAR3
      • HaricaGrnetClientSubCAR1
      • HaricaGrnetClientSubCAR3
      • HaricaGunetClientSubCAR1
      • HaricaHealLinkClientSubCAR1
      • HaricaIasaClientSubCAR2
      • HaricaNtuaClientSubCAR1
      • HaricaOdeeSubCAR1
      • HaricaSchClientSubCAR1
  • The Private Keys of all affected subCAs were "isolated" in a separate HSM partition in an "offline" mode. This partition is manually set to "online" by authorized administrators solely for the purpose of signing CRLs, and once the CRLs are issued the partition is set to "offline".
  • Subscribers that had Certificates with Private Keys generated and stored in FIPS devices, started to replace their certificates using our new Remote QSCD service and have been alerted to stop using their previous certificates. This certificate replacement process is proceeding as expected in terms of volume of replaced certificates, and is inline with our expected deadline to have all certificates revoked by 2020-10-31.

Our proposed plan is still on track, therefore we kindly ask to set the next update on October 6th, 2020.

Whiteboard: [ca-compliance] Next update 22-Sept-2020 → [ca-compliance] Next update 9-Oct-2020

The Subscriber certificate replacement process is on track. Next week we plan on sending reminders to Subscribers that have not yet replaced their certificates.

The week of October 26th contains a Greek National Holiday (October 28th) and although our initial plan was to revoke all affected end-entity Certificates on October 31st (Friday), we are considering moving this revocation deadline to Monday November 2nd. We hope this is an acceptable change. If there are no objections, we will proceed with this change. CA revocation and key destruction is still scheduled for the week of November 2nd.

Whiteboard: [ca-compliance] Next update 9-Oct-2020 → [ca-compliance] Next update 2-Nov-2020

All remaining affected CAs were revoked today and all keys were destroyed along with associated backups and backup encryption keys. The key destruction ceremony was witnessed by a Qualified external auditor. We will provide more information later this week.

This is the final incident report which replaces https://bugzilla.mozilla.org/show_bug.cgi?id=1649945#c7

1. Incident Report Analysis

1.1 How HARICA first became aware of the problem

HARICA is monitoring the Mozilla dev-security-policy Forum. A post on 2020-07-02 reported a violation of the BRs for several CAs including HARICA. Later that day, bug 1649945 was created.

1.2 A timeline of the actions HARICA took in response

(A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.)

HARICA initiated an investigation to assess the issue and possible security implications on Relying Parties. Our top priority was to verify that these CA Certificates were not allowed to sign OCSP responses in our OCSP responder services. After reviewing configuration changes (all the way back to the issue date of the oldest affected CA Certificate) of our OCSP responder services, we determined that these CA Certificates were never enabled to sign OCSP responses.
As our investigation was progressing, we documented conflicting interpretations of the requirements by experts and a lot of contradictory messages in the discussion that followed in m.d.s.p. There has been a lot of information to process in a short period of time. We started working on a revocation/key destruction scenario and adjusted it based on the information that was analyzed.
Here is a detailed timeline:

  • 2020-07-02: Initial report of the incident
  • 2020-07-02: The team re-examined Baseline Requirements, RFC 5280 and RFC 6960 along with previous discussions in m.d.s.p. which acknowledged that some implementations would fail without “EKU chaining”, without calling out such specific security risks for a CA Certificate including this EKU. First analysis revealed two parts of the incident:
    • a compliance part (because of the lack of an extension which is mandatory for delegated OCSP responders and triggers a violation in the BRs despite the affected CA Certificates being Technically Constrained and out of BR scope) , and
    • a security part (because some OCSP clients accepted delegated OCSP responses regardless if the signing Certificate is an end-entity or a CA Certificate). We also started a separate investigation to discover how many popular Browsers are affected by this security issue. We quickly identified that Mozilla Firefox and implementations using NSS mozilla::pkix library are not affected.
  • 2020-07-02: We identified the full set of non-expired non-revoked CA Certificates affected by this incident. Since all HARICA’s SubCAs are operated internally, the risk of a third-party CA operating an affected SubCA that includes the id-kp-OCSPSigning EKU and sign OCSP responses on behalf of its issuing Root CA. As the CA keys were constantly protected inside HSMs and under strict management, the likelihood of this event taking place was very low. Regardless of this fact, we continued our investigation to collect evidence of our OCSP configurations since 2018-05-23 which is the notBefore date of the oldest affected CA Certificate.
  • 2020-07-03: Given the fact that Mozilla Firefox was not affected by this incident (and possibly other User Agents) and that HARICA had the affected CA keys under audited control (outside OCSP responder systems) at all times, the team started to draft a mitigation plan, assessing the damage of revoking all CA Certificates and associated end-entity Certificates within the 7 day period per section 4.9.1.2 of the BRs. The approach to create replacement CA Certificates without the id-kp-OCSPSigning EKU was considered (using same KeyPairs), in order to meet the 7-day revocation requirement of the BRs, but was dismissed by the team as it did address the compliance issue, but not the security one.
  • 2020-07-04: We searched the Certificate database for the exact number of affected non-expired, non-revoked non-TLS Subscriber Certificates from the affected subCAs, trying to assess the complexity and constraints of replacing them. Our investigation showed that a substantial percentage of these had private keys generated in FIPS crypto-devices which makes quick replacement a more difficult task compared to replacement of TLS certificates.
  • 2020-07-05: After considering all the compliance, security and business continuity implications, we agreed on main course of action and started drafting a complete mitigation plan. The first step of the plan was to stop issuing certificates from the affected SubCAs and plan for creating new Issuing CAs.
  • 2020-07-06: We stopped issuing Certificates from the affected subCAs. A preliminary revocation plan was created trying to balance the difficulty in replacing these non-TLS certificates with keys in FIPS devices and the risks associated with this incident. We contacted our Subscribers to evaluate the operational risks of revocation and determined that a large number of these Certificates are used in mission-critical Academic operations and their immediate revocation would interrupt important services of the Academic sector.
  • 2020-07-06: In parallel, we continued our investigation to identify affected clients as a means to assess the possible impact on relying parties. In particular, our analysis on Chromium code revealed that Chromium-based Browsers are vulnerable to this security issue. It also located a relevant comment in the source code: “Not all properties of the certificate are verified, only the signature and EKU. Can full RFC 5280 validation be used, or are there compatibility concerns?”
  • 2020-07-07: An audit log analysis was performed to collect evidence on our OCSP responder system configuration parameters, including all changes since 2018-05-23 (the notBefore date of the oldest affected SubCA). Our CA Software provides cryptographically verifiable audit logs, evaluated and certified by Common Criteria (Audit logging is specified in 6.1: Security Audit) (see more details from PrimeKey), which can be used both by us and by external qualified auditors to evaluate and confirm completeness (unbroken sequence of logs) and integrity. These audit logs confirmed that the Internal OCSP responder system was never configured to use any of the affected CA Certificates to sign OCSP responses. We also contacted our CAB to discuss possible special audit activities to provide public assurance on specific assertions regarding CA and OCSP Responder configuration history.
  • 2020-07-08: Since issuance of new Certificates has stopped for the affected subCAs, we continued exploring options to disable the affected CA keys in our HSMs and enable them before the end of the nextUpdate to sign new CRLs, until the day we revoke and destroy the keys. Fresh revocation information would primarily be delivered via our delegated OCSP responder service. We are still examining possible compliance issues with this plan.
  • 2020-07-09: We decided to speed up the design and implementation of tools, which were already part of our 2020 plan, to allow for faster replacement of non-TLS Certificates, especially those that require their associated private keys to be generated in crypto-devices. The updated plan is to deliver the first version of these tools by the end of July, test them thoroughly in August and roll out in September.
  • 2020-07-10: Discussions with affected Subscribers continued, especially those that had difficulties replacing Issuing CAs in production systems. HARICA analyzed some special use cases of these certificates in various applications, offered solutions and a replacement plan was agreed for every special use case. These special use cases dominated the final revocation deadline and key destruction of all affected subCAs.
  • 2020-07-13 to 2020-07-17:
    • Development effort increased on improving and testing Certificate replacement tools (mainly for certificates with private keys stored in FIPS devices) and parts of our Registration Authority.
    • We continue to engage with subscribers that have special uses of their non-TLS certificates associated with private keys in FIPS devices. We have proposed solutions that need to be thoroughly tested to ensure that no critical operations are disrupted
    • Notifications to affected Subscribers that had certificates with private keys not residing in FIPS devices were sent, allowing them to proceed with certificate replacement.
    • Reviewed standards applicable to the affected Issuing CAs to check compliance issues regarding issuing CRLs not every 24 hours
    • The set of existing controls to protect the delegated OCSP responders from malicious external attacks were re-evaluated in the light of the new vulnerability where a CA Certificate could issue a malicious OCSP response on behalf of its Root. The result of the evaluation did not change the already acceptable level, since the subCA keys were operated internally by HARICA along with the Root keys, systems are segregated so that the external VA services have no access to the private keys associated with the affected subCAs. More information is described in https://bugzilla.mozilla.org/show_bug.cgi?id=1649945#c15.
    • A special audit with our CAB was scheduled and performed on 2020-07-17 to cover the requirements for a self-hosted Qualified Signature Creation Device (QSCD) system which allows replacing FIPS devices of subscribers with remote signing.
  • 2020-07-20 to 2020-07-24:
    • Our CP/CPS was updated to incorporate changes related to CRL update frequency
    • Completed testing of the certificate replacement tool. The next phase was to use this tool in a small number of Subscribers first before announcing it to all remaining affected subscribers
    • Alternative docSigning solutions (remote eSignature) was offered to a large number of Subscribers as a replacement for EU Qualified Electronic Signatures.
    • We drafted a plan to move the affected CA keys to a separate HSM slot for additional monitoring and protection.
  • 2020-07-27 to 2020-08-31:
    • The Remote QSCD implementation was audited and certified by our CAB. Preliminary results were sent to the Supervisory Body according to the eIDAS Regulation requirements
    • Changes to our RA system went live
    • The certificate replacement tool went live
    • Subscribers with the capability of issuing new Certificates from compliant Issuing CAs by re-using the same keys already created in FIPS devices, were notified and started replacing certificates remotely
    • A follow-up special audit was performed to verify the changes to the RA
    • Investigation about the need to issue daily CRLs from non-TLS Issuing CAs was still in progress
    • We started preparations for the CA key destruction ceremony of affected subCAs scheduled for the first week of November.
    • We planed to revoke and bring the keys for 10 subCAs offline during the week of 2020-08-31. The plan was to revoke and bring the associated CA keys offline as soon as affected ICAs were "evacuated" (all of their issued end-entity certificate either expired or revoked).
  • *2020-09-01 to 2020-09-21:
    • The following subCAs were revoked on 2020-09-03
      • HaricaAcademyOfAthensClientSubCAR2
      • HaricaAcademyOfAthensClientSubCAR3
      • HaricaAuthClientSubCAR1
      • HaricaECCAdministrationClientSubCAR2
      • HaricaEcclesiasticalAcademyofVellaClientSubCAR1
      • HaricaEcclesiasticalAcademyofVellaClientSubCAR2
      • HaricaEcclesiasticalAcademyofVellaClientSubCAR3
      • HaricaGrnetClientSubCAR1
      • HaricaGrnetClientSubCAR3
      • HaricaGunetClientSubCAR1
      • HaricaHealLinkClientSubCAR1
      • HaricaIasaClientSubCAR2
      • HaricaNtuaClientSubCAR1
      • HaricaOdeeSubCAR1
      • HaricaSchClientSubCAR1
    • As an extra precaution, the Private Keys of all affected subCAs were "isolated" in a separate HSM partition in an "offline" mode. Temporary, short-term re-activation of the partition by authorized administrators was used each time signing CRLs was required.
    • Monitoring tools were added to check activity of this particular HSM partition, providing real-time alerting in case of any activity.
    • Subscribers that had Certificates with Private Keys generated and stored in FIPS devices, started to replace their certificates using the new Remote QSCD service and were alerted to stop using their previous certificates. This certificate replacement process was proceeding as expected in terms of volume of replaced certificates, and was inline with our expected deadline to have all certificates revoked by 2020-10-31.
  • *2020-09-22 to 2020-11-03:
    • All remaining unrevoked/unexpired end-entity certificates (with the exception of OCSP Responder certificates) from the affected subCAs were revoked on 2020-10-31.
    • The following subCAs were revoked on 2020-11-02 during a ceremony witnessed by a Qualified external auditor:
      • HaricaAegeanClientSubCAR1
      • HaricaAuebClientSubCAR1
      • HaricaAuthClientSubCAR2
      • HaricaAuthClientSubCAR3
      • HaricaCedefopClientSubCAR1
      • HaricaDuthClientSubCAR1
      • HaricaGrnetClientSubCAR2
      • HaricaGrnetClientSubCAR4
      • HaricaHuaClientSubCAR1
      • HaricaIhuClientSubCAR1
      • HaricaIonioClientSubCAR2
      • HaricaOdeeSubCAR2
      • HaricaOdeeSubCAR3
      • HaricaPanteionClientSubCAR1
      • HaricaTucClientSubCAR1
      • HaricaUnipiClientSubCAR1
      • HaricaUniwaClientSubCAR1
      • HaricaUoaClientSubCAR1
      • HaricaUocClientSubCAR1
      • HaricaUoiClientSubCAR1
      • HaricaUomClientSubCAR1
      • HaricaUopClientSubCAR1
      • HaricaUowmClientSubCAR1
      • HaricaUpatrasClientSubCAR1
    • The following Keys (SPKIs - SHA256) and associated encrypted backups containing these keys were destroyed on 2020-11-02 during a ceremony witnessed by a Qualified external auditor:
      • 777790FBECBF1D7812BBFDAE403BD5504749C6A2
      • 5352753E1C29BE1F06CC7C27CFF9EBA3AC4FA38F
      • CD0523FEFB603CE27E81E948FC9A43C12712115F
      • 2C52F3DFA7B7E37AABEB93ED4FD8A8FC3F4CF63D
      • BE8E6B084739CED16CC75EEBA98E4C6143C71017
      • 62357BF4B871F4BED880146BF5E0452CD79B7A27
      • 8B173D39872CFAF8833F318E363465653B5E2FEA
      • 63AADE3228756297C30BEB80FF84580B41813028
      • 62A8E94D35DD90CE75ABA13D6815CC98D05F2B0C
      • 9881B5E529A3887E9D0FBA55FE10DA206F38961B
      • D87BF38B2D04ABD232473D112060AAC198E927EB
      • F08CB232BFC744064704FC0696750DCBCB9977ED
      • EE3E364052E28BA4852202E0DB3803B379697C61
      • 4EEE9BEF783FB7C4AD062CB4DE0374413F5C3C62
      • F041BCE2D240DC98F3F423B53A7167ACC89DBCCC
      • 58BAB1976BC9E8B13028E0D82B36B3CAC601FC33
      • 945AA34F46A0F129E93D27E61436627CBB41B74C
      • 00E4496D610AD2E587ED0DAD32E287DD9F6FEFCA
      • BBCBF1C12A6F19BA56A35890CA29866A340603A1
      • 10E6F375A92AC54B11968B6968B448EE1C24487B
      • 9F061DB50ADEBC0D0C26CB40758CD909FD4D42C5
      • CBA77738A70A8B3FEA51A9EA7E94B5AFD4304926
      • E2AC23AC14B4D5CDABC156DBA1C1CE17EEA1380D
      • 52E00BB41B91047E4A7C5843A51E9B68F7CEB71D
      • 22991B8E0FD3EB63EC4B5A26C5E347A8992AD4F2
      • 6F0400FCD299F279539DCD939406C7D56BDDB9CC
      • ADF074B79F551C342A2C516D47A0A1DD8B3C7B12
      • 3B4E927739CB0ED64324C09906096B2E9B0013B1
      • D6CF5BB9E29A6F5DF0EA11C6BC18123E0D0E26DC
      • 2A894660B60AC4E402E8659E4E99E6469CA31449
      • 7A6EFAE67FBB9603DE9E3B1B4E42F04574399A28
      • 7CAA1D9ED61E717F2D14809217C1EA59FA071D22
      • 20DDBDE7F481A74A742A2FE879BCFE0B776EDD6B
      • 8E4BAC241E661F15F63C3625EA018DA02EFD8389
      • 980C3258F62B9A9E2ED79DB71C5AC164220062CA
      • A5AA25B64CD2B655C61CDEDC2782CB385222B675
      • ED95A6E1728DF63608458923653CF21772DB182A
      • C0860CAB88E80D704C127C515C5B295C4693A781
      • 280B99AA2FEE750517B60FD89EF96B26F4E80CBC
      • 70F2EDEF391CF0488CCD4D637F8B8E5B3AF14033
      • 4CDA90C6E631F93F9ECEB4ECAE4776FD0ABB3BB5

1.3 Has HARICA stopped, or has not yet stopped, certificate issuance or the process giving rise to the problem or incident?

(A statement that you have stopped will be considered a pledge to the community; a statement that you have not stopped requires an explanation).

Yes, issuance of new Certificates for Subscribers by the affected Issuing CAs has stopped since 2020-07-06.

The CA Certificate profiles were updated to remove the id-kp-OCSPSigning keyPurposeId.

1.4 In a case involving certificates, a summary of the problematic certificates

(For each problem: the number of certificates, and the date the first and last certificates with that problem were issued. In other incidents that do not involve enumerating the affected certificates (e.g. OCSP failures, audit findings, delayed responses, etc.), please provide other similar statistics, aggregates, and a summary for each type of problem identified. This will help us measure the severity of each problem.)

We counted 39 non-expired, non-revoked CA Certificates affected by this incident. These are non-TLS, Technically Constrained subCA Certificates. We found 2 additional revoked subCAs and scheduled their keys to be destroyed along with the non-revoked. A total of 41 CA Certificates were affected by this incident.

1.5 The complete certificate data for the problematic certificates

(In a case involving certificates, the complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem. In other cases not involving a review of affected certificates, please provide other similar, relevant specifics, if any)

The entire CA certificate database was examined. Most of these CA Certificates are disclosed in CCADB despite the fact that they are Technically Constrained. For greater transparency, HARICA had decided to add non-TLS CAs to CCADB once this became a requirement from Mozilla for unconstrained non-TLS CA Certificates. HARICA CA Certificates have been disclosed in https://repo.harica.gr since 2011.

This is the list of affected CA Certificates that contain the id-kp-OCSPSigning KeyPurposeId in their EKU extension:

  1. https://crt.sh/?id=2838197805
  2. https://crt.sh/?id=2838197797
  3. https://crt.sh/?id=2657659434
  4. https://crt.sh/?id=1222759679
  5. https://crt.sh/?id=1903248090
  6. https://crt.sh/?id=2838197806
  7. https://crt.sh/?id=2657662689
  8. https://crt.sh/?id=2657659332
  9. https://crt.sh/?id=900785759
  10. https://crt.sh/?id=2838197804
  11. https://crt.sh/?id=1222760114
  12. https://crt.sh/?id=2097908041
  13. https://crt.sh/?id=2838197796
  14. https://crt.sh/?id=2657659376
  15. https://crt.sh/?id=2657659279
  16. https://crt.sh/?id=2657659626
  17. https://crt.sh/?id=2657659294
  18. https://crt.sh/?id=2838197802
  19. https://crt.sh/?id=2838197795
  20. https://crt.sh/?id=2838197799
  21. https://crt.sh/?id=1222761078
  22. https://crt.sh/?id=2838197794
  23. https://crt.sh/?id=2657658699
  24. https://crt.sh/?id=2657659304
  25. https://crt.sh/?id=2657658676
  26. https://crt.sh/?id=2657659290
  27. https://crt.sh/?id=2858289782
  28. https://crt.sh/?id=2657662544
  29. https://crt.sh/?id=2657659170
  30. https://crt.sh/?id=2838197801
  31. https://crt.sh/?id=2657659681
  32. https://crt.sh/?id=2657657558
  33. https://crt.sh/?id=2657659465
  34. https://crt.sh/?id=2838197798
  35. https://crt.sh/?id=1265291634
  36. https://crt.sh/?id=909718586
  37. https://repo.harica.gr/certs/HaricaAcademyOfAthensClientSubCAR2.pem
  38. https://repo.harica.gr/certs/HaricaECCAdministrationClientSubCAR2.pem
  39. https://repo.harica.gr/certs/HaricaEcclesiasticalAcademyofVellaClientSubCAR2.pem
  40. https://repo.harica.gr/certs/HaricaGrnetClientSubCAR2.pem
  41. https://repo.harica.gr/certs/HaricaOdeeSubCAR2.pem

1.6 Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

The interpretation of the combination of applicable requirements (Root Store program requirements, BRs, the enforcement of both RFC 5280 and 6960) was that the inclusion of id-kp-OCSPsigning KeyPurposeId in technically constrained subCAs was allowed. In addition, according to BRs section 7.1.2.2, it would further protect Relying Parties. The protection of Relying Parties and the proper management/isolation of risks are the main reasons that HARICA extensively uses the nameConstraints and EKU extensions in Issuing CA Certificates, which is to limit the unintended use of associated CA Keys.

The issue was also discussed in November 2019 (during our monitoring of a relevant discussion on m.d.s.p. regarding interoperability issues of Microsoft’s CA software), but the result of that discussion, in our understanding, was inconclusive.

1.7 List of steps HARICA is taking to resolve the situation and ensure that such situation or incident will not be repeated in the future

(List of steps your CA is taking to resolve the situation and ensure that such situation or incident will not be repeated in the future, accompanied with a binding timeline of when your CA expects to accomplish each of these remediation steps)

HARICA makes extensive use of both EKU and nameConstraints extensions to technically constrain its ICAs in accordance with section 7.1.5 of Baseline Requirements and Mozilla Root Store Policy (for S/MIME Certificates), as a technically enforceable means of protecting Relying Parties and managing/isolating risks.

HARICA operates all its Root and Intermediate CA Certificates on its own; all affected Intermediate CA Certificates are operated internally under the same security environment and by the same trusted personnel as our Root CA Certificates. All our CAs, including the technically constrained ones, are regularly audited against the "baseline" ETSI standards (i.e. ETSI EN 319 401, ETSI EN 319 411-1 which apply to all certificate types and policies). All affected CAs are included in audit reports without gaps, as attested by the qualified external auditors.

After thorough investigation of our cryptographically verifiable audit logs (the audit mechanism has been evaluated/certified by Common Criteria (Audit logging is specified in 6.1: Security Audit), see more details from PrimeKey), we have confirmed that the affected CA Certificates have never been configured in our OCSP responder services that would enable them to issue a malicious OCSP response on behalf of their parent (RootCA). We have also contacted our CAB to explore the possibility of an external audit team to examine the above and to issue an independent letter of opinion (attestation). Our CAB confirmed that it is able to provide this independent assessment if requested, as an additional measure of assurance to Relying Parties.

We consider that the above facts significantly decrease the probability of a successful attack and result in very low security risk. Revocation within the BR required timeframe for our affected non-TLS certificates would cause significant operational risks for a large number of our subscribers and Relying Parties (who depend on those subscribers), causing more damage to the ecosystem than is warranted by the inherent risk. Thus, we designed a more balanced approach with regards to the revocation time-line of the affected CA certificates.

To further mitigate the threat, until all affected CA Certificates are revoked and their keys destroyed:

  1. The id-kp-OCSPSigning KeyPurposeId has been removed from all CA Certificate profiles.
  2. We switched issuance of new Subscriber Certificates to compliant ICAs. Only revocation information was signed using the keys of the affected CAs until their revocation.
  3. We contacted our Subscribers to replace certificates issued by the affected Issuing CAs, so that we revoke each ICA as soon as possible. We built tools and updated/improved our procedures to accommodate the large volume of operations.
  4. 2020-10-31 was set as the deadline for revoking all affected Issuing CA Certificates. We expected all affected subscribers to have replaced their non-TLS certificates by the end of October. The decision for this deadline was based on the difficulties of replacing some end-entity certificates. This was further explained in the corresponding delayed revocation bug.
  5. Keys of the affected CAs were destroyed during the first week of November as planned. The key destruction ceremony was physically witnessed by our CAB. Εxamination of the lifecycle of each key, audit logs (OCSP responder system configuration) for the period of time up to the destruction of keys is in progress for the completion of the attestation letter.

Learning from past experience with replacing TLS certificates (and also during the serialNumber entropy incident), HARICA had already drafted plans that would accelerate the replacement of non-TLS Certificates when necessary. These efforts were already underway before this incident, but they were not completed nor tested extensively. Some portions of these plans were used during the course of this incident. We were able to "fast track" the remaining pieces that will enable this replacement to happen timely. A significant number of affected certificates, with keys included in FIPS devices, will be able to be replaced in a relatively short period of time.

Considering the fact that different trust frameworks show increasingly different expectations, we have a plan to create separate hierarchies for different uses, especially hierarchies related to server TLS. This is scheduled for Q4 2020.

2. Incident Impact

This incident had unintended and unexpected consequences on a number of HARICA's non-TLS Subscribers, where certificates had to be replaced. No other security impact was detected for Subscribers or Relying Parties since existing controls already contained the new vulnerabilities introduced by this incident.

3. Conclusions and Recommendations

We consider the past and existing controls reasonable to assure Relying Parties that the affected CA Certificates were not used to sign any OCSP Responses on behalf of their Issuer (i.e. delegated OCSP responses). This new vulnerability makes these affected issuing CAs more "attractive" to attackers, because getting access to them would not only allow them to issue end-entity certificates but to also have the "unrevoke" capability if the Root CA were to revoke such an ICA. However, all of HARICA's affected ICAs are Technically Constrained to specific email domains and organizations via the nameConstraints extension, making them less "attractive" to attackers.

The mitigations applied due to this incident which resulted in the destruction of the Keys associated with the affected CA Certificates, provide further assurances going forward.

Separation of the TLS hierarchy from other usages will contain the TLS risks to the TLS hierarchy only.

This the first time that HARICA was forced to violate the revocation timelines of the Baseline Requirements. Even in the serialNumber entropy incident that affected a great number of TLS Certificates, HARICA managed to complete the revocations in the required timeline and already has the necessary controls to contact affected Subscribers and help them replace TLS Certificates timely.

This incident was an opportunity for HARICA to extend existing tools for certificate replacement in non-TLS environments, and communicate these tools to our Subscribers depending on the use cases. This will provide improved agility to our non-TLS services for our Subscribers and Relying Parties.

I will schedule this bug for closure on 18-November-2020 unless there are additional issues to discuss.

Flags: needinfo?(bwilson)
Flags: needinfo?(bwilson)
Whiteboard: [ca-compliance] Next update 2-Nov-2020 → [ca-compliance] Next update 2020-11-18

We just received the Key Destruction Attestation signed by our Accredited Conformity Assessment Body. It is now attached on this ticket.

This Key Destruction Attestation is also available from our CAB's web site.

Flags: needinfo?(jimmy)

I believe this bug can be closed and will do so sometime next week (Dec. 7-11).

Flags: needinfo?(bwilson)
Status: ASSIGNED → RESOLVED
Closed: 3 years ago
Flags: needinfo?(bwilson)
Resolution: --- → FIXED
Product: NSS → CA Program
Whiteboard: [ca-compliance] Next update 2020-11-18 → [ca-compliance] [ocsp-failure]
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: