1651465 - HARICA: Delayed revocation for non-BR-compliant CA Certificates within 7 days

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Dimitris Zacharopoulos]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Firefox/78.0

Steps to reproduce:

According to https://bugzilla.mozilla.org/show_bug.cgi?id=1649945, affected CA Certificates should have been revoked within 7 days according to section 4.9.1.2 of the Baseline Requirements.
Per https://groups.google.com/d/msg/mozilla.dev.security.policy/EzjIkNGfVEE/2WlxGVylBwAJ, we are creating this bug to provide more specific information about the challenges to replace our affected non-TLS Certificates in the timeframe designed for TLS Certificates.

Comment 1

[Dimitris Zacharopoulos]

We have prepared most of our report for reasons why we delayed revocation and expect to have it ready no later than Monday July 27.

Comment 2

[Dimitris Zacharopoulos]

Decision and rationale for delaying revocation

This is the first time that HARICA decided to violate the revocation timelines of the Baseline Requirements, in an effort to act responsibly to this security incident and balance the ecosystem damage from mass revocation of (non-TLS) Certificates. In our case, the private keys of a significant number of affected end-entity Certificates were stored in FIPS hardware crypto-devices which makes replacement a much more difficult and time consuming task compared to the replacement of TLS certificates, for which we already have the necessary organizational and technical measures to meet the revocation deadlines set in the Baseline Requirements. For this reason we are making some additional changes to our processes (see below) so that our revocation timelines are significantly improved for these non-TLS cases.

We identified several cases where these non-TLS Certificates are used for financial and organizational transactions within Universities (signing research contracts, grants, new hires, payment orders and other critical administrative activities). Greek Universities in general use limited personnel in July and shutdown almost all activities in August for at least 2 weeks. Summer vacation in the academic sector also takes place during this period. This means that for the aforementioned period, these Subscribers and IT administrators managing services that use these certificates, are unreachable for certificate replacement.

Some universities are using these Certificates not only for signing various documents but also in the following use cases:

• Strong Client Authentication (Certificate-based) where Issuing CAs are “pinned” as Authentication Trust Anchors in multiple systems. The suggested solution needs to be tested in collaboration with institutions using this. September is the beginning of the academic year and these activities are at their peak which makes software updates challenging.
• Student grade signing, where -similarly- Issuing CAs are “pinned” as Authentication Trust Anchors. Due to the extraordinary circumstances with COVID-19, several exams will take place at the end of September, which means Faculty members will need to submit grades in October.

Revoking these Certificates within the BRs specified timeline and for some of these cases issuing from new Issuing CAs without having performed some necessary changes, would cause significant service disruption in critical Academic activities.

Revocation Timeline

Based on the above rationale and the (low) risks analysed in the main incident bug, we set a deadline for automatic revocation of unexpired/unrevoked non-TLS Certificates from the affected subCAs which is Monday November 2, 2020. This deadline was decided based on these special use cases that need additional steps, while at the same time keeping the risks of not destroying these CA Keys under control. Our original plan was to revoke and destroy keys one month later (beginning of December) but we believe we can accomplish the November deadline because we have close collaboration with these Institutions. In the unlikely event we need further extension, we will update this bug with additional information.

Replacement rates of certificates will be monitored at regular intervals and targeted reminders will be sent accordingly to the special use case Subscribers mentioned above.

We plan on having one ceremony for key destruction, externally witnessed, during the first week of November.

Preventing future revocation delays

In addition to resolving this situation we remain focused on improving our policies and practices in order to meet possible future revocation timelines, even for non-TLS Issuing CAs that are in scope of the Baseline Requirements, like in this particular case.

• There is a plan to communicate with our non-TLS Subscribers explaining that they should avoid developing solutions based on publicly-trusted (in the wider sense) X.509 Certificates that "pin" to specific Issuing CAs, as these may need to be revoked or replaced in case of technical or other reasons. We expect doing that once we have all the pieces together from our collaboration with Institutions. These use cases and solutions would be useful information for our communication to Subscribers.
• For the specific use cases described above, we will be working with these Institutions in September helping them make the necessary changes to their software to avoid depending on specific Issuing CAs, as these may change by HARICA in case of incidents such as this.
• We accelerated the implementation of a remote signature solution (which was already in progress before the incident as part of our 2020 plan). This has now been deployed and is pending for approval by our CAB. Our National Supervisory Body also needs to be notified about this new service so we can switch some of our users with FIPS hardware crypto-devices to this new service. This new service will allow non-TLS Subscriber Certificates used for electronic signatures to be replaced in a timely fashion going forward.
• In addition, we developed tools to assist in the replacement of existing non-TLS Certificates with their associated keys stored in FIPS hardware crypto-devices. These tools are currently being thoroughly tested before being used in production.

Creating an independent TLS hierarchy (planned for Q4-2020) will fully mitigate potential revocation delays of non-TLS Issuing CAs, as we experienced in this incident.

Acceptance of remediation plan

For all of the above, we are in close collaboration with our CAB and Supervisory Body, providing updates as needed.

Even in the serialNumber entropy incident that affected a great number of Certificates, HARICA managed to complete the revocations in the required timeline and already has the necessary controls to contact affected Subscribers and help them replace TLS Certificates timely. However, in this particular case due to the factors explained above, we believe that following the BRs revocation timeline for these non-TLS Issuing CAs would cause greater harm than good.

Please let us know if we are good to proceed with this remediation plan or if any further information is needed.

Comment 3

[Ben Wilson]

This appears to be a well-considered plan of action. Please proceed. If there are any modifications please let us know. Similarly, if we have any requested modifications to your plan, we'll communicate them to you.

Comment 4

[Dimitris Zacharopoulos]

Ben, we provided an update to the main ticket. We will try to keep the two tickets separate, posting updates relating to the incident on the main ticket, and information related to the prevention of future delayed revocation, on this ticket.

As far as the delayed revocation improvements are concerned, we have already discussed with existing subscribers that had difficulties replacing their certificates within 7 days about the existing requirements, their Subscriber Agreement obligations and we still plan on sending a notification to all of our non-TLS Subscribers to avoid using solutions that "pin" to specific Issuing CAs. We expect to prepare this material beginning of October, thus we would like to set the next update of this ticket to October 5, 2020, if there are no objections.

Comment 5

[Dimitris Zacharopoulos]

We have been collecting information from some of our affected Subscribers that were able to share more details about some of their implemented services that relied on subjectDN information of the Issuing CA Certificate and other "pinning" practices. Most of these solutions were performing TLS Client Authentication and were expecting the Client Authentication Certificates from specific Issuing CAs.

We are still collecting feedback and will start drafting an article for our non-TLS Subscribers as planned. Our expectation is to have it ready by Oct 21.

Comment 6

[Dimitris Zacharopoulos]

We have completed the draft and after internal discussions, the team decided to make it public on this bug, asking for any additional feedback from the community before we create a final version and send it out.

The draft document is available at:

• https://docs.google.com/document/d/1rzGuIFum5yZnN0xtNrSq1CSXTWPjw_ir296XbIsMEf0

We would appreciate any comment or feedback and any useful information that we might have overlooked. We will keep this process until October 28.

Comment 7

[Dimitris Zacharopoulos]

This document is now considered approved and is being translated. It will be distributed to our Subscribers before the end of November.

We believe this remediation is sufficient for the particular incident. As mentioned in https://bugzilla.mozilla.org/show_bug.cgi?id=1651465#c2 we are making the necessary arrangements for a completely separate TLS hierarchy in Q4 2020.

Please let us know if there are any additional questions or concerns.

Comment 8

[Dimitris Zacharopoulos]

HARICA subscribers were notified today with the good and recommended practices for using publicly-trusted digital certificates as originally discussed in comment 6.

Comment 9

[Ben Wilson]

I intend to close this bug next week (Dec. 7-11).