1609706 - PKIoverheid: Missing Intermediate CA from audit statement

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Jorik van 't Hof]

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

At the end of 2019 Logius was also reviewing other CCADB records (besides the one mentioned in bug 1605126) that have been flagged by ALV. The outcome was that only one other CA was flagged by ALV, specifically, the "UZI-register Medewerker Niet op Naam CA G21". Unlike the CAs mentioned in bug 1605126, it seems that in this case that the CA in question has been "skipped" in the yearly audit in 2017/2018 as well as 2018/2019. The cause for this seems to be a misinterpretation of Mozilla requirements by both Logius and CIBG.

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

(all dates are in the format dd/mm/yyyy)

25-11-2019 Logius was notified by CCADB that the fingerprint of the issuing CA with CN “UZI-register Medewerker niet op naam CA G21” was missing from the audit statement that CIBG supplied Logius as a result of their yearly audit concering the year 2017/2018. Logius then asked CIBG for a clarification
28-11-2019 CIBG gave a formal response in which they indicated that in June 2017 issuance of certificates by this CA was halted because of a new ETSI certificate profile requirement which their systems at the time couldn’t comply with (specifically, the inclusion of the Subject.Organizationidentifier field as required by ETSI EN 319 412-3 section 4.2.1). The intent was to do a system upgrade and resume issuance (but then under the new G3 TSP CA) at a later moment. At the time, the question of audits was discussed between CIBG and Logius, and Logius at that time gave the answer that a partial audit was still required (all areas of the certificate lifecyle minus certificate generation). These services were audited for the other CIBG (issuing) CA’s, but because CIBG was no longer issuing under this CA, they interpreted it as OK to place this CA out of scope of the audit in its entirety. CIBG argued that because certificate lifecycle processes are used for all (issuing) CAs of the TSP you can't partially audit an CA as Logius indicated. This wasn't communicated to or checked with Logius and due to human error wasn't spotted by Logius when the audit statement was handed over for the year 2017/2018. Notwithstanding the error in logic made at that time by Logius and CIBG, the majority of the areas involving the CA in question should have been audited and as such, the TSP CA should have been listed on the audit statements (with an asterisk/disclaimer). By the time this issue was discovered by Logius it was too late to amend the 2018/2019 audit to include the CA in scope (although admittedly that wouldn't have helped the earlier issue with the 2017/2018 audit).
15-1-2020 creation of this bug. Our original intent was to post at the same time as bug 1605126 but due to internal discussions (with CIBG) we didn't make that deadline. The delay was made worse by the holiday period with absence of key personell on both Logius and CIBG's side.

3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

See bullet 2. Issuance of certificates by the CA “UZI-register Medewerker niet op Naam CA G21” was stopped in June 2017

4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

The CA in question (https://crt.sh/?id=12715989), although not technically constrained, only issued S/MIME certificates (which, for obvious reasons, are not CT logged), The audit period concerned is from Q3 2017 to the present. Currently, only ~250 certificates are still valid.

5. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

See answer 4. Logging is not an option because of GDPR concerns, but we could provide a list of hashes if needed (working on that now).

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

See also our answer listed at question 2 and the reasons listed in bug 1578809. At the time Logius was too much focused on the eIDAS/ETSI domain and not taking into account browser requirements with regards to audit. As also indicated in bug 1586125 Logius made some logic errors in the analysis of the “technical” capabilities of a CA to issue certain kinds of certificates (the CA in question was decomissioned but not revoked). Also, the focus at that time(2017/2018) was too much on the then new G3 TSP CAs instead of the legacy CAs which were in most cases still issuing valid certificates.

7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

All remaining certificates will be revoked before February 17, 2020. The CA in question The CA in question will be revoked after that, but at the latest at February 24, 2020 . Unfortunately, this can't be done earlier mainly due to the fact that the certificates issued by the CA in question are issued on a SUD (smartcard) which takes some time to replace. Since the new certificates are issued by a new G3 CA for which the issuance process has changed slightly (mostly the forementioned Organizationidentifier subject field) this also causes some delay due to the way some healthcare organizations are set-up.

As indicated in bug 1578809, our Overview of Applicability template will contain in its next iteration an overview of the CAs in scope of an audit, including their details (fingerprint, validity dates etc). This way, Logius, the TSP and the auditor can all see if the audit has all the CAs in scope. Part of that overview will be as well for which type of audit the issuing CA in question will be audited (ETSI 411-1 or 411-2, but also for which policy, e.g. NCP, NCP+, EVCP etc).

Comment 1

[Ryan Sleevi]

Thanks for filing this, it's helpful to understand how things are being addressed.

A few follow-up questions, mostly around the procedures and prevention side:

1. Bug 1605126 was opened in 2019-12-19, and you mentioned you were aware of this since approximately 2019-11-28 in the report, yet this bug wasn't filed until 2020-1-15 "due to internal discussions (with CIBG)". That's a rather large time gap, and I'm trying to understand what happened here - both the delta between 2019-11-28 and 2019-12-19 and the subsequent delta from 2019-11-28 to 2020-1-15 - and what's being done to address that. This is because the discussion of Incident Reports states:

   Each incident should result in an incident report, written as soon as the problem is fully diagnosed and (temporary or permanent) measures have been put in place to make sure it will not re-occur. If the permanent fix is going to take significant time to implement, you should not wait until this is done before issuing the report. We expect to see incident reports as soon as possible, and certainly within two weeks of the initial issue report. While remediation work may still be ongoing, a satisfactory incident report will serve to resolve the issue from a Mozilla perspective.

2. You state "Since the new certificates are issued by a new G3 CA for which the issuance process has changed slightly (mostly the forementioned Organizationidentifier subject field) this also causes some delay due to the way some healthcare organizations are set-up.". It seems useful to understand more detail here, as well as how Logius is preventing delays in the future. Even if y'all are moving to a fully-private PKI, understanding the operational challenges that ecosystem participants face seems incredibly valuable, as well as possible mitigations for such issues.

3. You state "Logging is not an option because of GDPR concerns, but we could provide a list of hashes if needed (working on that now).". What steps are being taken to ensure that the TLS trusted hierarchy (up to and including the root) do not share these challenges regarding GDPR going forward?

Comment 2

[Jorik van 't Hof]

Hello Ryan,

1. The gaps are also deemed undesirable in our eyes, and we will change our internal policy for future events to switch to an approach that is more in line with the expectations that both Mozilla and the community have of CAs in the Mozilla trust programme. The specific delays in this case are due to multiple issues:

• The desire to come to full internal (within PKIoverheid) concensus about further steps to take in an incident before undertaking those steps
• The need of TSPs to first fully prepare commmunication strategies and inform stakeholders before any external communication (e.g. Bugzilla or MDSP posting) is made
• The mindset that compliance issues are less important than security or other operational issues (at multiple levels)

In this specific case steps were at first taken at the same time as in bug 1605126 but due to an error in priorities and thinking (see reasons above) other matters took precedence over this issue in December. After that, the holiday period started and the issue wasn't revisited before key stakeholders were back at work.

Because of this bug (and also earlier cases) we intend to institute the following measures:

1. Disclose future issues (of any kind) within the fastest possible time. If possible involved parties (like TSPs) will be asked for input, but there will be a fixed time window
2. Prepare both our internal and external stakeholders for the occurence of these kind of issues (increasing awarness)
3. No longer making a distinction between compliancy and security issues as far as (initial) reporting goes

Most of these issues concern a change in attittude for both Logius and it's TSPs, which will be dealt with by the new PA.

2. To explain our future and plans, firstly I want to elaborate on our current situation. This might be a bit redudant seeing our earlier bugs in 2019 but we rather want to over- than underexplain.

The issue discussed here and in a few related bugs mostly concern government-run TSPs. PKIoverheid has 3 government TSPs which used publicy trusted certificates:

• Ministry of Infrastructure, for use in the on-board taxi computers (BCT), which use PKIoverheid certificates to digitally sign transactions (qualified signature as described in eIDAS).
• Ministry of Defence, for use in digitally securing transactions and sign documents (qualified signatures, per eIDAS again)
• CIBG, for use in the healthcare system, but only S/MIME cases. TLS certificates were issued under our G2 Root CA (see bug XXXX) but have been revoked. New TLS certificates (used for machine-to-machine communication) are issued from a private PKI.

Although all three TSPs and their use cases are in principle a closed off system and as such wouldn't require the use of publicly trusted certificates there are several reasons why untill now these haven't been moved to a fully private PKI (which is our preferred solution):

• There are interactions with the outside world which neccessitates the use of publicly trusted certificates or at least a technically constrained intermediate under a publicly trusted root (because transactions of documents are validated by parties outside the closed off systems). This is mostly the case for the certificates used for digital (qualified) signatures.
• S/MIME (qualified) certificates issued by CIBG are used in the healthcare system which use a myriad of different software suites which usually come pre-loaded with a set of trusted root CAs, often populated by either Microsoft Trusted lists or NSS
• Historical precedence: Moving to a private PKI hasn't been official policy and only lightly encouraged untill recently. At the time the G2 hierarchy was created in 2008 there wasn't even a private PKIoverheid Root CA, and delays in switching to G3 issuing CAs was delayed by the long admission process (more specifically: update frequency) of a few Certificate Consumers

The issues and challenges faced by PKIoverheid in the last year have shown us that measures are needed to the reduce the impact of revocations due to issues and also to reduce the uninteded effects and/or uses of certificates other than the original intent. Due to the fact that the government TSPs have slightly different use cases and different subscriber population the approach for each TSP has been slightly different:

• IenW currently only issues certificates from intermediate CAs that have been technically constrained (see bug 1586125), the previous generation still has some valid certificates which all will expire in March 2020 (due to expiry of the G2 Root CA)
• CIBG issues S/MIME certificates (qualified signature and authentication use cases) from intermediate CAs that have been constrained by EKUs, but not by name constraints due to the diverse subscriber landscape. CIBG issues TLS certificates from an unconstrained private CA.
• The Ministry of Defence issues S/MIME certificates that have been constrained by EKUs, but not name constrained.

Seeing the current situation Logius still sees a few issues:

• Due to the way the ecosystem of each government CA is set-up they are (either by design or in practice) not suited for rapid changes which the modern (web)PKI requires.
• The risks associated with running a(n) (unconstrained) public CA are partly acknowledged by Logius and its TSPs, but due to reasons listed above (acceptance rate and planning transistion of moving off the G2 issuing hiearchy) until recently this hasn't been discussed in detail nor have policy changes been made or proposed to remedy this.

For the short term we are looking into the following remediation measures, in addition to the ones stated in the initial post:

• Re-evaluate the risks and trade-offs associated with the current set-up of issuing CAs under a public root.
• Regardless on the outcome, open talks with the concerned TSPs to discuss options limiting the issuing CAs mentioned above, by either including constraints in the CAs currently not constrained or moving off to PKIoverheid private certificates.

For the long-term, we're currently looking at the following options:

• Investigating steps into automating issuance of qualified / SMIME certificates.
• Re-evaluating the additional requirements imposed on PKIoverheid TSPs by our Programme of Requirements (PvE) which, although originally intended to enhance the PKIoverheid certificates, might be, by now, impending further changes and/or subscriber agillty.

Due to the complexity of both long-term goals we unfortunatly can't really give any more detail at the current time. We understand the potential value of more detailed solutions to other CAs and the PKI ecosystem as a whole, but unlike for TLS certificates (for which a lot of thinking has taken place already and a lot of measures/requirements have already been in place, parly also due to the wealth of knowledge gained over the years due to incident reports like this) it seems to us that for S/MIME there is still a long way to go, especially where eIDAS and qualified signatures are concerned. One easy way to limit impact and risk that we used for TLS certificates (reducing the certificate lifetime) for instance clashes with the wish for longer valid certificates for use in digital signatures. Etc. etc. We do hope that ideas and measures we will come up with in the future can be shared with a wider community to take advantage of, for instance by suggesting them for inclusion in the Baseline Requirements for S/MIME certificates (when the working group for that is in place).

3. Untill our G2 Root CA expires in March 2020 (March 25) there are several valid issuing CAs in existence that, although not designed or designated to issue TLS certificates, are still technically capable of issuing TLS certificates. The G3 and EV hiearchies only have intermediate (and issuing) CAs with EKU's included in them. Furthermore, ou CP will be amended to disallow the inclusing of certain information included in the subject.organizationName field in publicy trusted PKIoverheid TLS certificates. Only organization information of non-natural persons (e.g. organizations) will be allowed, therefore restricting the options that the BRG 7.1.4.2.2.b offers.

Comment 3

[Jorik van 't Hof]

Hello Ryan,

All certificates have been revoked. The CA in question will be revoked later this week.

Regards,

Jorik

Comment 4

[Jorik van 't Hof]

Hello Ryan,

We are glad to inform you that the CA “UZI-register Medewerker niet op naam CA G21” is now revoked and placed on CRL (http://www.csp.uzi-register.nl/cdp/zorg_csp_ca_g21.crl).

Regards,

Jorik

Comment 5

[Wayne Thayer]

I've confirmed that the CA is revoked. It appears that all questions have been answered and remediation is complete.