1649964 - PKIoverheid: Incorrect OCSP Delegated Responder Certificate

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Ryan Sleevi]

The following was originally reported to m.d.s.p. at https://www.mail-archive.com/[e-mail]/msg13493.html

PKIoverheid has issued one or more OCSP Delegated Responders, as defined within RFC 6960, Section 2.6 and Section 4.2.2.2, without including the id-pkix-ocsp-nocheck response, as required by the Baseline Requirements, Version 1, Section 13.2.5 through Version 1.7.0, Section 4.9.9

Example certificate: https://crt.sh/?id=135970406

Please provide an incident report, including the timeline for revocation.

Comment 1

[David Weissenberg]

Logius (PKIoverheid) confirms receipt of this report and after investigating the issue together with its CA's recognizes the security issue and associated risk. Together with its TSP's and the rest of the PKIoverheid ecosystem the impact is being researched and Logius has begun working on a plan for a viable solution. More details will follow in the coming days.

Comment 2

[David Weissenberg]

Hello Ryan,

Our initial impact assessment (in conjunction with our TSPs) has showed that a few of the CAs listed can be revoked without issue due to the fact that no valid certificates issued by them exist (anymore) . Logius PKIoverheid is currently starting up the process of revoking these CA's.
The remaining CAs are responsible for issuing the bulk (if not all) of all currently valid PKIoverheid certificates and as such remediation will take quite a bit more planning and careful thought. We expect to give an update on those within the next few days.

Kind regards,
David

Comment 3

[Ryan Sleevi]

Thanks for the update.

Are there any currently revoked certificates from this hierarchy that would at at risk of “un”revocation? And is this the complete hierarchy or are there certificates not disclosed via CCADB or CT that might also pose risk?

I realize in previous incidents Logius had already begun work on moving to shorter-lived certificates. Does that apply to all certificates in this hierarchy, or only a subset?

What controls do you have existing to prevent misuse of these keys? For example, is there an audit log of all HSM based signings that can be matched to ensure no OCSP responses?

Comment 4

[David Weissenberg]

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

Logius PKIoverheid became aware of this issue due to an automated e-mail from Bugzilla (sent at 2020-07-02 03:52 CEST) indicating assignment of this bug to the responsible contact (Jorik van 't Hof; read at 2020-07-02 07:06 CEST)

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

(times are in UTC+2 (DST), dates are DD/MM/YYYY)
02/07/2020 07:06 e-mail read and first assessment made. Forwarded for internal discussion with PKIo team at Logius
02/07/2020 07:30 - 19:00 Analysis of the issue mentioned. Due to some confusion of the scope of the issue, at first deemed a non-issue until this being rectified later. Due to potential impact on the PKIoverheid ecosystem of resigning and revoking all currently valid issuing CAs Logius has designated this as a major incident and immediately involved the TSP's operating TLS-enabled issuing CAs and necessary management.
Discussions then arose over the next few days around the issue indicated and how remediation could take place while minimizing impact for subscribers. It became clear that several CAs could be revoked and the associated keys destroyed without additional risks due to the fact that no valid certificates were ever issued by them or no valid certificates remain. Previously revoked certificates that could be "unrevoked" could also have their associated key pairs destroyed without any risk.
What became clear quickly was that the revocation of all affected issuing CAs within the BR-mandated deadline of 7 days would result in large scale outages in many parts of the Dutch digital infrastructure (including health care) which especially in the current times is more vital than ever.
03/07/2020 11:00 Informed the other TSP's about the issue.
03/07/2020 17:00 Teleconference with all TSP's.
03/07/2020 21:40 First response to this bug acknowledging the issue.
04/07/2020 10:00 Meeting with parties involved (National Cyber Security Center (NCSC; the NL-GOV CERT), TSP's etc.).
04/07/2020 20:23 Quick update on this issue mentioning the first few CAs have been identified that could be revoked quickly and its associated key pairs destroyed without impact. Based on this, a key ceremony has been planned in which a few CAs (see listed CAs at bullet 5) will be revoked. Key destruction ceremonies, supervised by an external auditor will be planned as well, but due to the keys being in possession of different CAs, this process will take more time. Further analysis of the impact of revoking the other TSPs continued in co-operation with the TSPs involved and the NCSC.
As such, alternative plans to remediate the issue listed would have to be implemented . This proposal will be simultaneously offered to major trust stores (Microsoft, Google, Apple and Mozilla) to explain the motivation of PKIoverheid not to revoke the affected issuing CAs within the BR-mandated deadline of 7 days after receiving the initial report (for more details see bullet 5).

3. Whether your CA has stopped, or has not yet stopped, certificate issuance or the process giving rise to the problem or incident. A statement that you have stopped will be considered a pledge to the community; a statement that you have not stopped requires an explanation.

This issue concerns the CA certificates, of which issuance frequency is low and always requires a key ceremony to bring the intermediate CAs (TSP CAs) on-line. No current issuance of ICAs was planned. Any future issuance of ICAs will be with an amended certificate profile which doesn't contain the EKU id-kp-OCSPsigning.
Due to the existing and additional controls proposed under bullet 7 we consider the risk of exploiting of the vulnerability as very low and therefore we consider it responsible not to disrupt the availability of our customers while we plan the issuance of replacement CAs.

4. In a case involving certificates, a summary of the problematic certificates. For each problem: the number of certificates, and the date the first and last certificates with that problem were issued. In other incidents that do not involve enumerating the affected certificates (e.g. OCSP failures, audit findings, delayed responses, etc.), please provide other similar statistics, aggregates, and a summary for each type of problem identified. This will help us measure the severity of each problem.

This concerns all PKIoverheid TSP CAs (issuing CAs) issued between February 2015 and December 12, 2019 (no new CAs have been issued since that date), in total numbering 29 CAs. Due to the nature of the issue at hand (which opens the possibility of revoked CAs to "unrevoke" themselves via OCSP) an additional 16 CAs have been identified.
It should be noted that the ICAs are divided into five domains where the described threat can only occur on ICAs within the same domain. See the next bullet for the exact clustering.

5. In a case involving certificates, the complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem. In other cases not involving a review of affected certificates, please provide other similar, relevant specifics, if any.

(A) All currently valid CAs, sorted per domain:

1. Staat der Nederlanden Autonome Apparaten CA - G3 -- Government of The Netherlands, PKIoverheid (Logius) 1 ICA
   1.1 MinIenW PKIoverheid Autonome Apparaten CA - G3 https://crt.sh/?id=1838117644

2. Staat der Nederlanden Burger CA - G3 -- Government of The Netherlands, PKIoverheid (Logius) 4 ICA
   2.1 Cleverbase ID PKIoverheid Burger CA - G3 https://crt.sh/?id=318398866
   2.2 Cleverbase ID PKIoverheid Burger CA - G3 https://crt.sh/?id=1412330500
   2.3 Digidentity BV PKIoverheid Burger CA - G3 https://crt.sh/?id=1412330508 (Revocation planned)
   2.4 QuoVadis PKIoverheid Burger CA - G3 https://crt.sh/?id=135970406 (Revocation planned)

3. Staat der Nederlanden Organisatie Persoon CA - G3 -- Government of The Netherlands, PKIoverheid (Logius) 10 ICA
   3.1 KPN PKIoverheid Organisatie Persoon CA - G3 https://crt.sh/?id=19623446 (Revocation planned)
   3.2 QuoVadis PKIoverheid Organisatie Persoon CA - G3 https://crt.sh/?id=135970399
   3.3 KPN BV PKIoverheid Organisatie Persoon CA - G3 https://crt.sh/?id=147613373
   3.4 UZI-register Medewerker op naam CA G3 https://crt.sh/?id=164244353
   3.5 UZI-register Medewerker op naam CA G3 https://crt.sh/?id=1412330611
   3.6 UZI-register Zorgverlener CA G3 https://crt.sh/?id=164244352
   3.7 UZI-register Zorgverlener CA G3 https://crt.sh/?id=1412330520
   3.8 Digidentity BV PKIoverheid Organisatie Persoon CA - G3 https://crt.sh/?id=1412330543
   3.9 Ministerie van Defensie PKIoverheid Organisatie Persoon CA - G3 https://crt.sh/?id=1652799589
   3.10 MinIenW PKIoverheid Organisatie Persoon CA - G3 https://crt.sh/?id=1835407965

4. Staat der Nederlanden Organisatie Services CA - G3 -- Government of The Netherlands, PKIoverheid (Logius) 11 ICA
   4.1 UZI-register Medewerker niet op naam CA G3 https://crt.sh/?id=164244355
   4.2 UZI-register Medewerker niet op naam CA G3 https://crt.sh/?id=1412329219
   4.3 Digidentity BV PKIoverheid Organisatie Server CA - G3 https://crt.sh/?id=475894529
   4.4 Digidentity BV PKIoverheid Organisatie Server CA - G3 https://crt.sh/?id=1412330569
   4.5 KPN BV PKIoverheid Organisatie Server CA - G3 https://crt.sh/?id=63094369
   4.6 KPN BV PKIoverheid Organisatie Server CA - G3 https://crt.sh/?id=1412330519
   4.7 KPN BV PKIoverheid Organisatie Services CA - G3 https://crt.sh/?id=130486221
   4.8 MinIenW PKIoverheid Organisatie Services CA - G3 https://crt.sh/?id=1838117716
   4.9 QuoVadis PKIoverheid Organisatie Server CA - G3 https://crt.sh/?id=54959165
   4.10 QuoVadis PKIoverheid Organisatie Server CA - G3 https://crt.sh/?id=1412330560
   4.11 QuoVadis PKIoverheid Organisatie Services CA - G3 https://crt.sh/?id=130068222

5. Staat der Nederlanden EV Intermediair CA -- Government of The Netherlands, PKIoverheid (Logius) 3 ICA
   5.1 KPN PKIoverheid EV CA https://crt.sh/?id=130068222 (Revocation under consideration)
   5.2 KPN CM PKIoverheid EV CA https://crt.sh/?id=10646767 (Revocation planned)
   5.3 QuoVadis PKIoverheid EV CA https://crt.sh/?id=7846179 (Revocation assessed)

(B) All currently revoked CAs which could be "unrevoked", sorted by domain:

1. Staat der Nederlanden Organisatie Services CA - G3 -- Government of The Netherlands, PKIoverheid (Logius) 9 ICA
   1.1 KPN PKIoverheid Organisatie Server CA - G3 https://crt.sh/?id=18068225
   1.2 KPN PKIoverheid Organisatie Services CA - G3 https://crt.sh/?id=19623447
   1.3 KPN PKIoverheid Organisatie Server CA - G3 https://crt.sh/?id=18068237
   1.4 KPN PKIoverheid Organisatie Services CA - G3 https://crt.sh/?id=19623445
   1.5 QuoVadis PKIoverheid Organisatie Services CA - G3 https://crt.sh/?id=73706454
   1.6 Digidentity PKIoverheid Organisatie Services CA - G3 https://crt.sh/?id=363419312\
   1.7 Digidentity PKIoverheid Organisatie Server CA - G3 https://crt.sh/?id=363419317
   1.8 Digidentity BV PKIoverheid Organisatie Services CA - G3 https://crt.sh/?id=475894540
   1.9 MinIenW PKIoverheid Organisatie Services CA - G3 https://crt.sh/?id=1265291620

2. Staat der Nederlanden Burger CA - G3 -- Government of The Netherlands, PKIoverheid (Logius) 2 ICA
   2.1 Digidentity PKIoverheid Burger CA - G3 https://crt.sh/?id=363419311
   2.2 Digidentity BV PKIoverheid Burger CA - G3 https://crt.sh/?id=475894537

3. Staat der Nederlanden EV Intermediair CA -- Government of The Netherlands, PKIoverheid (Logius) 1 ICA
   3.1 KPN PKIoverheid EV CA https://crt.sh/?id=7801964

4. Staat der Nederlanden Organisatie Persoon CA - G3 -- Government of The Netherlands, PKIoverheid (Logius) 4 ICA
   4.1 KPN PKIoverheid Organisatie Persoon CA - G3 https://crt.sh/?id=19622894
   4.2 Digidentity PKIoverheid Organisatie Persoon CA - G3 https://crt.sh/?id=363419313
   4.3 Digidentity BV PKIoverheid Organisatie Persoon CA - G3 https://crt.sh/?id=475894533
   4.4 MinIenW PKIoverheid Organisatie Persoon CA - G3 https://crt.sh/?id=1265291511

5. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

Before the Microsoft requirements regarding separation of issuing CAs came into force, the PKIoverheid TSP CAs (issuing CAs) were not constrained (expect by policyIDs included in the intermediate CAs). When the requirements regarding issuing separation by EKUs came into force, the assumption was that the EKU id-kp-OCSPsigning was to be implemented in the issuing CA profile, because of chaining rules. As such, the CA profiles for all types of issuing CAs (including the EV hierarchy) had the EKU in question included. This policy was not questioned over the last few years, due to the incorrect assumption that continuing inclusion of the EKU for OCSP signing was required for OCSP signing, as it is (for instance) for ServerAuth. As the CAs were never intended to be used for OCSP responders, the id-pkix-ocsp-nocheck statement was not included in the CA profile.
A resigning in 2016 of a CA without the required ServerAuth EKU which led to issues a few weeks later reinforced this thinking. Even after reconsidering and reviewing all certificate profiles the ocsp-signing EKU was still considered a required and valid option.
Last year when PKIoverheid had to resign part of the G3 issuing CAs due to the serial number issue the focus was more on the prompt replacement of the issuing CAs and remediation of the matter at hand, and as such not enough emphasis was placed on other issues that might be present in the profile used (even though a few new issuing CAs were issued with technical constraints but still included the OCSP Signing EKU)

7. List of steps your CA is taking to resolve the situation and ensure that such situation or incident will not be repeated in the future, accompanied with a binding timeline of when your CA expects to accomplish each of these remediation steps.

Based on a risk analysis considering integrity, confidentiality and availability Logius comes to the following incident response plan. The architecture and the controls of PKIoverheid - as we will explain - are such that the inherent risk of occurrence of the threat is very low. The focus will be on immediately mitigating the non-conformity for the CAs where lack of availability is no problem. Where outage can lead to disruptions, a more balanced approach is taken. For these cases, however, additional measures will be taken immediately to reduce the already low risk to almost zero. Logius will accelerate the process it has already started to solve the underlying problem of the PKIo architecture in which different types of certificates with different risk profiles are issued under the same root. We will continue to communicate with the community about progress. The above approach leads to the next steps.
The revocation of the valid CAs 2.3, 2.4, 3.1 and 5.2 will be executed on Thursday July 9. Key destruction for those CAs as well as the CAs listed as currently revoked will be planned as soon as possible. We are also investigating if further continuation of the "Staat der Nederlanden EV Root CA" is desired.
The revocation of the remaining CAs, including key destruction, would lead to the subsequent effect that all 600.000 end-user certificates which chain to these CAs will become unusable overnight. These certificates are (depending on issuing CA) used for qualified signatures, S/MIME, TLS traffic between government parties and end-users (citizens and business), and as such represent a diverse field, especially the S/MIME and qualified part (of which all certificates have to be issued on a QSCD or other hardware-based SCD). Replacement of these end-user certificates within a short-time frame is impossible without causing large-scale outages and subsequent damage to society.
We acknowledge the security risk involved. The setup of PKIoverheid has several controls in place to reduce the probability of the risk to allow for time to replace the issued end-user certificates. All PKIoverheid CAs are housed in HSMs under dual control. All signing operations are strictly controlled and monitored. The current setup doesn't allow them to sign OCSP responses, either directly or as a delegated "responder". Changes to the system impact the trustworthy status of the CA system. All changes are both technically and procedurally strictly controlled within the TSPs and monitored and audited. The risk of any (attempt at) misuse of the TSP CA will be similar to any other "malicious CA" abuse cases, for which control measures have been implemented. Due to compartmentalization into domains in our PKI hierarchy the scope for possible misuse is limited to CA's within a single domain.
To limit the the current security risk to the PKI(overheid) ecosystem and taking the responses from both Mozilla and Microsoft into account, Logius PKIoverheid is going to implement the following short-term measures for a responsible phase-out of affected ICAs with a high outage risk:

1. As Ryan had already indicated in comment 3, we are going to build on the currently available HSM logs to further introduce procedural and audit controls in which CA signing logs will be supplied to an independent third party which will review these logs periodically and signs off on these. Furthermore, we are installing new requirements in our Program of Requirements.
2. After issuance of replacement CAs without the OCSP EKU, the impacted CAs for which revocation at this time would lead to large scale outages will be retired and as such will only issue OCSP responder certificates and CRLs for status information. This will be strictly controlled and supervised by an external auditor.
3. New issuing CAs will be issued (not necessarily on a one-to-one basis, see the (possible) retirement of a few CAs above) based on a new key pair, on which issuance of new certificates will take place. For signing and replacement of the CAs and underlying end-user certificates we will prioritize on TLS issuing CAs. Due to the fact that we already limited TLS certificate lifetimes, most certificates currently issued by these ICAs will expire during the course of 2020/early 2021.
4. For the S/MIME ICAs a timeline is being investigated, as these also depend on EUTL listing.

However, we acknowledge that the security risk must be fully remediated. Logius is still working on the analysis for further remediation and as such, intend to update this bug soon with more information. Our TSPs want to destroy the relevant keys as soon as possible to limit the risk as even the notion of mis-use will damage their reputation.

Comment 5

[Ryan Sleevi]

Thanks David. I think this is in line with the sort of incident reports we're looking for, and continues to show Logius' commitment to sharing information and building understanding about the tradeoffs, blockers, and challenges, which is the underlying goal with these reports.

I realize that PKIoverheid is, effectively, a form super-CA, and so I'm trying to understand how to square that with statements like:

All PKIoverheid CAs are housed in HSMs under dual control. All signing operations are strictly controlled and monitored. The current setup doesn't allow them to sign OCSP responses, either directly or as a delegated "responder".

Are these keys under third-party control? Or are they under the control of Logius/PKIoverheid? e.g. when I look at CAs like https://crt.sh/?id=1412330519 or https://crt.sh/?id=475894529 or https://crt.sh/?id=164244355 , CCADB shows they have distinct operators and audits, although they appear to share the same auditor.

Normally, the "malicious CA" risk is mitigated by revocation being an option, but we know this particular implementation poses challenges. What other mitigations may exist that could reduce this risk, for example, contractual and/or regulatory restrictions? Can you share more details about how those function?

You can see my questions are trying to understand the "verify" part of "trust but verify". Unlike certificates, which are logged to Certificate Transparency, OCSP responses aren't, and so there's limited opportunity to detect, especially potentially targeted issues. I'm also trying to understand the latency here between something going wrong and something being detected. Quarterly review means it may take up to 3 months to detect something, and so I'm trying to understand the holistic set of proposals here, both for insider risk and for external compromise.

Comment 6

[David Weissenberg]

To answer one of the questions in comment 5: The dual control being referred to is through segregation of duties within a single TSP. From a security point of view, dual control by both Logius and its TSP’s is preferable but in practice nearly impossible due to geographical location and availability of resources.

However, when assessing the risk it is good to get a clear picture of the likelihood that misuse can occur. To be able to do this it is essential to understand how the PKIoverheid system works. PKIoverheid is a governmental super-CA with all issuing CA’s under control of either governmental or commercial TSP’s. We have legally binding contracts with all the commercial TSP’s and covenants with governmental TSP’s (both will be referred to as contracts from now on). Although there are significant financial liability clauses in the contracts, the focus in the contracts, but also in the way we work together is on preventive controls, collaboration and transparency and therefore more principle-based. For example, the contract obliges all participating TSP’s to help each other and give openness in information in executing their mutual contracts. This culture of wanting to improve the system together has led to all participants working closely together to obtain informal mutual control and examination. Each TSP also mostly operates in its own market. If other questions exist on the legalities of the PKIoverheid system additional information can be supplied.

As pointed out above the insider risk (incentive for misuse) is limited. This does not mean we do not take the risk serious. When we provide an update we will elaborate more on other preventive measures which are in place or are considered as a mitigating control. For instance, we are investigating how we can implement monitoring in such a way that misuse will be detected real-time so that Logius will receive an alert as soon as the Issuing CA signs an OCSP response. We have to consult with all TSP’s and an independent oversight party on the feasibility of this. The long term solution will be shared as soon as possible.

Comment 7

[Ryan Sleevi]

Attachment: Old approach

Comment 8

[Ryan Sleevi]

Attachment: new hotness

Comment 9

[Ryan Sleevi]

Thanks. In the absence of a concrete date, I'll hold off setting a Needs-Update here and look forward to weekly updates.

I want to again commend Logius for the thoroughness of the incident reports and the opportunities to learn and improve. I think one remaining question is to ask whether you've considered segmenting your hierarchy into explicit purposes and use cases.

We see this reflected in Mozilla Root Store Policy v2.7, Section 5.3, where splitting out EKUs from S/MIME and TLS are required as of 2019-01-01.

This would apply to the "super-CA" hierarchy as well, but you may want to consider a step further to reduce the risk of issues like this in the future.

That is, consider the hierarchy from "old and busted" - a traditional approach, even for super-CAs, is to lump things under the root, or to split out by the operating organization, who then further splits out by purpose. Compare that with "new hotness" - which creates a "root" of trust for each purpose.

For your purposes like the EUTL, you can add at the "Root" to the EUTL and things just work. For clients that only care about specific purposes, rather than adding your "Root" (and thus running risk from, say, S/MIME), they can instead just add roots for the purposes they care about. As you meed to consider, long term, what the PKIoverheid will look like, I hope you give some consideration to the above; using "intermediates" as opportunities to anchor trust and exclude other purposes.

Comment 10

[David Weissenberg]

Hello Ryan,

Thanks for sharing the suggestions for future PKI designs. We are aware of the "new hotness" in PKI design, separating issuance by use case. PKIoverheid has, from its inception, always made a distinction between the different use cases (and as such, has never implemented the "old approach"), by introducing a level-2 domain CA only intended to issue TSP (issuing) CAs. However, this was historically only done by issuing several "domain CAs" which in turn only issued TSP CA's (issuing CA's). Both the level 2 and level 3 CAs didn’t include EKUs or specific Policy Identifiers.

As for the current PKIoverheid "Staat der Nederlanden Root CA - G3", it was created in 2013. As such, it predated any requirements separating the use cases by (issuing) intermediate CAs.

The requirement of separating use cases by including EKUs in intermediate CAs only happened after the introduction of the MS requirements mid-2015. The earliest PKIoverheid G3 TLS issuing CAs (now revoked) did not include any EKUs, nor did the intermediate domain CAs. At the time (in 2015) no active issuance was taking place from the G3 root CA due to lack of inclusion in several trust stores, so it was relatively painless to resign the TSP issuing CAs which included EKU's. This also applied to intermediate certificates under the EV Root, which, although created earlier, only was submitted for inclusion in 2014). The domain CAs were not resigned as they predated the MS requirements (being created in 2013 at the same time as the Root CA) and because they were not intended to issue end-user certificates. Hence, our current setup does not quite resemble the "new hotness" example as listed in comment 8.

The events that took place over the last year already prompted Logius to think about the future design of PKIoverheid. As indicated in our previous remediation plan, part of the proposed measures were focused on customer agility, for instance the voluntary reduction in certificate lifetimes last year. However, parallel to that, the design process of the next generation PKIoverheid certificates (G4) has also started, taking into account lessons learned.

At a bare minimum, the "new hotness" scenario (including EKU constraints) will be implemented. However, due to the recent events PKIoverheid is also considering other designs, which could potentially separate use cases even further (for instance, further separating use cases per root, use cases not requiring browser trust, ECC/RSA/quantum resistant PKIs, key rotation for intermediate CAs, shorter-lived root CAs etc.). All those considerations, however, something for the mid- to long-term solution since these require a new Root CA. For the short-to-medium term, please see our response in the next comment.

Comment 11

[David Weissenberg]

As indicated in the previous post, with this response we want to outline our remediation plan for this issue. Our remediation plan is:

1. The "Staat der Nederlanden EV Root CA" and its associated intermediate CAs will be retained. As such, the current "KPN PKIoverheid EV CA" and "QuoVadis PKIoverheid EV CA" will have to be revoked and its keys destroyed under external auditor supervision. They will be replaced by two new issuing CAs (one for each TSP) based on new key pairs. We plan to issue these TSP CA's on Wednesday July 29.

2. Before the revocation and key destruction of the issuing CAs listed above the currently valid EV certificates will first be re-issued under the newly created TSP EV CA's. Re-issuance of end-entity EV certificates will take place in August; revocation and audited key destruction of current TSP EV CA's will take place in September.

3. Over the past 2 years approximately 10.000 public certificates in various sectors have been replaced by end entity certificates from our Private Root. The approximately 37.000 remaining G3 TLS (OV) certificates will be replaced by certificates issued either under the newly minted EV CAs or under our Private Root ICA's. Replacement of these certificates will take longer than desirable. The reason for this is that many PKIoverheid TLS certificates are used by subscribers for securing machine-to-machine (M2M) connections, for which certificates under a private CA should have been used. Over the coming period PKIoverheid TSP's will have to contact all of their subscribers to disentangle both use cases and give them sufficient time to prepare their systems for the switch to the PKIoverheid Private Root in cases where public trust is not needed (like M2M). These are often complex cases with inflexible systems, hence the timeframe listed below. Therefore our schedule to replace and revoke TLS end-entity certificates under our G3 root is as follows:

   August 2020 20%
   September 2020 35%
   October 2020 50%
   November 2020 75%
   December 2020 90%
   January 2021 100%

4. After replacing all current EV and OV TLS certificates requiring public trust, the "Staat der Nederlanden Root CA - G3" can be withdrawn from the major trust stores. After this withdrawal the risk associated with the delegated OCSP responder vulnerability in the PKIoverheid hierarchy does no longer exist for the webPKI ecosystem. The remaining risk will only exist within the now private PKIoverheid G3 environment, which we will manage based on a risk-based approach.

The motivation for the remediation listed above is that revoking the affected CAs and destroying the keys would lead to massive disruption of critical systems which depend on them, and as such wasn't feasible within the time frame required. The remediation plan is designed to remove the risk to the webPKI ecosystem, but also balances out the continuity risk to PKIoverheid relying parties which do not (necessarily) need public trust to function properly. Seeing the short remaining lifespan of our EV Root we do acknowledge the risk that PKIoverheid might have to temporarily withdraw from the major trust stores (depending on the time needed for designing, building and admittance to the major trust store of our next PKIoverheid Root CA) for both the TLS and S/MIME use cases.

Part of the measures listed above are also intended as remediation for bug 1652604.

Comment 12

[Ryan Sleevi]

While I am normally be quite concerned over a plan that took 6 months to execute, I want to acknowledge that this plan involves working directly with customers to potentially migrate them to private CAs, if appropriate for their use case, and that's a migration plan that provides huge returns and benefit for the ecosystem. While revoking immediately is both expected (in the BRs) and desired (from the PoV of avoiding unnecessary compliance remediation delays), there is great risk that immediate revocation could cause either the CA or the relying party to view this issue as "fixed" and not touch on the systemic patterns.

I think one thing missing from Comment #11, and that likely better belongs on Bug 1649964, is explaining the motivation for how/why those percentages are, how they're prioritized, etc. How have you set the milestones, how are you prioritizing different cases, are you grouping by organization/validation type/some other property, etc.

Comment 13

[David Weissenberg]

Due to the circumstances already listed in comment 11 a fully detailed plan cannot be shared at this time. However, we can further expand on the reasons and motivations behind the previously shared numbers and dates. Do note that we intend to speed up the process where possible and that certificates can be replaced at an earlier date than listed below for certain use cases, if possible.

1. End of July: Although subscribers have been informed that changes were upcoming that might influence their operations, the exact details have not been shared yet because the remediation plan still needed more work. They also need to be instructed to contact their software vendors when custom software is in use employing PKIoverheid-based M2M connections. They need to check if moving to a private root impacts their software. If so, software needs to be adapted and tested.
2. August: Due to the earlier mentioned uncertainty regarding the exact remediation and time needed for CA configuration replacement issuance of the currently PKIoverheid certificates can't start before August. We are looking into ways to speed up this process, both for the current remediation but also for any future occurrences.
3. August and early September: All certificates under the G3 root solely used for web server authentication will be replaced by certificates under the new intermediate.
4. August up to end of September: For certificates which are used in both a web and M2M use case a new separate TLS certificate will have been issued for the web use case.
5. September up to end of October: The first batch of certificates for M2M uses, in which the connected systems do support private certificates but in which subscribers lack the agility needed for rapid replacement, will be replaced.
6. September up to end of December: Second batch of TLS certificates used for M2M purposes in which the supporting systems (e.g. receiving servers or clients) need software adaptions for support of private certificates are a blocking issue.
7. September up to end of January: Last batch of replacement for TLS certificates. This includes the most complex cases (e.g. hardcoded certificates or even possible embedded devices).

From our TSPs we demand bi-weekly updates on this directive. The outcomes of which we will also share in this Bugzilla ticket.

Comment 14

[David Weissenberg]

Because of operational constraints with subscribers we have decided to create a new OV intermediate CA with 3 Issuing CA's under the EV Root. These ICAs lack the EV policy OID and as such will only issue OV certificates. These new ICAs have been created Wednesday July 29. We will revoke the current EV ICAs issued by the "Staat der Nederlanden EV Root CA", and destroy the associated keypairs after the end-user (EV) certificates have been replaced. As previously noted this will take place by the first half of September due to the fact an auditor needs to be present for key destruction.

Comment 15

[Jorik van 't Hof]

Hello Ryan/Ben,

We wanted to give a quick update. As it stands, the PKIoverheid TSPs are busy replacing certificates under our G3 root with either new public OV certificates under our EV root (via the new CA Server 2020 intermediate CAs) or with certificates under our private root hierarchy. The transition of the EV certificates to OV certificates under de EV root has already been completed. The last subscriber certificates were replaced in early September and on September 10th, the EV issuing CAs under the "Staat der Nederlanden EV Intermediate CA" have been revoked.

With the revocation of:

• KPN CM PKIoverheid EV CA https://crt.sh/?id=10646767;
• QuoVadis PKIoverheid EV CA https://crt.sh/?id=7846179.

all remaining EV issuing CAs have now been revoked.

Key destruction for the affected “KPN PKIoverheid EV CA” and “QuoVadis PKIoverheid EV CA” is underway and will be witnessed by an external auditor. Once that has happened and the appropriate reports are available we will share that in this bug post.

During the current replacement process of G3 certificates we have noticed that many subscribers encounter problems with the transition to either the new CA 2020 certificates or switching to the PKIoverheid Private root. These subscribers often lack the knowledge required to resolve the encountered problems. This has resulted in Logius and the involved TSPs having to spend more time than initially estimated on communication plans and requires more guidance from our side for these subscribers to resolve their issues in a timely manner.

We reaffirm our decision to withdraw the G3 root from the browser root stores at the planned date. This has additional impact on several non-TLS (and non-S/MIME) use cases for which impact analysis is still ongoing. Although this is a separate issue from the one listed above, this does lead to similar issues for several subscribers and requires us to spend more time on additional guidance to resolve these issues.

Going forward, our next deadline is on October 1st, at which stage we want to have replaced all certificates used in a browser context with new certificates issued by a compliant issuing CA. Currently, only a small percentage of the G3 OV certificates have actually been revoked. This is caused by our specific situation. In many cases, certificates are used in multiple places at the same time. This means that, to avoid disruptions, we have to wait for confirmation from the subscriber after issuing a replacement certificate before revoking their original certificate. Sometimes the certificate needs to be replaced in places where this is less easy to realize. Because of this, the number of revoked certificates does not give a realistic picture of the activities that are ongoing. Our concern for the continuity of our critical processes forces us to take this approach. However, our deadline for withdrawal from the trust store is fixed.

Looking further ahead, December 31st is our deadline for replacing all remaining TLS certificates with either PKIoverheid private certificates, or with CA Server 2020 certificates, considering carefully if a certificate is elligible for a Server 2020 replacement or not. After replacement of the TLS certificates, January 31st 2021 is the deadline for subscribers using other (non-TLS) certificates issued by TSPs under the Staat der Nederlanden Root CA – G3 to adapt their systems for use after withdrawal of the PKIoverheid G3 Root CA from the major browser root stores.

We will keep sending regular updates on our progress up to January 31 (the foreseen withdrawal date).

Comment 16

[Ryan Sleevi]

this does lead to similar issues for several subscribers and requires us to spend more time on additional guidance to resolve these issues.

To the extent you can share, can you share more details here? This seems so important to understanding and assessing the challenges here with non-browser uses of browser PKIs.

our next deadline is on October 1st

Can you provide an update here?

Comment 17

[Jorik van 't Hof]

Hello Ryan,

The problems our subscribers encounter with non-TLS certificates mainly revolve around the usage of personal certificates which are widely in use in our government and relying parties. Not only the software that is used by subscribers needs to be altered, but there are a lot of relying parties who also have to check their software and adjust it if necessary. An obvious use case is signing of documents. Many vendors and integrators have written software to accomodate the digitization of inter- and extragovernmental processes involving signed documents. For the most part this software relies on trust stores on the OSs, which will be impacted by withdrawal from the Microsoft Windows trust store (mainly) and NSS (to a lesser extent). Not all organisations have a clear understanding on the usage of certificates and validation processes in their back-end systems, resulting in consultation with their integrators and vendors concerning the various implementation issues. Also, for some software it is easy to make changes and/or do a roll-out, for others this is far less so.

Regarding the October 1 deadline, all certificates have been re-issued and as far as we can see, most of the certificates have been replaced. Not all of these replaced certificates have and can be revoked yet. Because dual-use (both web and M2M) is more common than we had anticipated many parties use the certificates in several places and with different use cases. M2M usage suffers from the same problems as non-TLS usage described above. Therefore we would like to give our subscribers enough time to minimize the chance of outages.

The range of problems which can result from the removal of our G3 root from the publicly trusted root stores also depends on the different root store implementations. Will they just switch a trust bit, set a NotBefore, or do a complete removal? How will different OS versions cope with these different methods, and more importantly, how will software/libraries on these OSs?

Comment 18

[David Weissenberg]

Hello Ben,

Apologies for the delay in the update. The last couple of days we were focused on the key destruction of our EV intermediate CAs which were affected by this issue. A report about the key destruction will follow shortly.

Up to 95% of the TLS-certificates have been replaced. About 30% of these replaced TLS (G3) certificates have been revoked as well.
December 31st is our deadline for replacing all remaining TLS certificates with either PKIoverheid private certificates, or with CA Server 2020 certificates.

The remaining certificates will be revoked between 11 and 22 of January. This will be done in a phased manner, so that if any problems arise they can be dealt with accordingly.

We reaffirm our decision to withdraw the G3 root from the browser root stores as of 1 February. This means we will contact Mozilla Microsoft Google and Apple, asking them to remove trust as of 1 February 2021. Mozilla has stated removal of the website trust bit is sufficient in their case. Microsoft will set a NotBefore in the ServerAuth EKU in their trust store.

Comment 19

[Ben Wilson]

Thanks for the update.

Comment 20

[David Weissenberg]

Hello Ben,

We wanted to let you know we have filed bug1684158 PKIoverheid: Removal of websites trust bit for "Staat der Nederlanden Root CA – G3".

This request has also been sent to Microsoft Google and Apple, asking them to remove trust as of 1 February 2021

Comment 21

[Ben Wilson]

I'll close this on next Wednesday, 31-March-2021.