Closed Bug 1578809 Opened 5 years ago Closed 5 years ago

PKIoverheid: Compliance issues CIBG TLS certificates

Categories

(CA Program :: CA Certificate Compliance, defect)

Product:
CA Program
Component:
CA Certificate Compliance
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: jochem.vanden.berge, Assigned: jochem.vanden.berge)

Details

(Whiteboard: [ca-compliance] [ov-misissuance])

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Firefox/68.0

Steps to reproduce:

  1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

During a QA check on the certificates provided by CIBG for Bug 1573490

  1. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.
    2019-07-23 For disclosure in bug 1573490 PKIoverheid asked CIBG for a list of certificates in scope of the issue (insufficient serial number entropy)
    2019-08-27 CIBG provided Logius with a full list of certificates affected by the issue listed in bug 1573490 (earlier lists were incomplete).
    2019-08-28 Logius does a QA check on the certificates, found multiple compliance issues with them, and asks CIBG for a clarification.
    2019-08-29 Seeing that Logius found multiple compliance issues per certificate Logius orders CIBG to revoke the certificates involved before October 1, instead of letting them naturally expire as indicated in Bug 1573490
    2019-09-04 Publication of issue on Bugzilla. Unfortunately, this was not possible to do earlier due to the complex field and systems in which these certificates are used. We wanted to determine the impact of revocation and present a finalized approach to Mozilla for remediation of this issue.

  2. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

CIBG stopped normal issuance of publicy trusted TLS certificates to third parties under the “Staat der Nederlanden Root CA – G2” in December 2017. A few certificates were issued for tests in March 2018 but those have already been revoked.

  1. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.
    All affected certificates are issued by one of the following CAs: https://crt.sh/?caid=741 and https://crt.sh/?caid=754

  • 3311 certificates, issued to external parties between 1-9-2016 and 31-12-2017. They contain the “subject.altname.othername” field which is forbidden by both the BRs and the Logius PKIoverheid CP under which CIBG issues

  • All 3311 certificates are lacking the CA/Browser forum Policy OID for OV certificates 2.23.140.1.2.2. Although not mandated by the BRs (or Mozilla CA policy), they are listed as mandatory in our CP and in Microsoft Trusted Root Programme requirements.

  1. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

Because most of the certificates contain sensitive personal identifiable information (in the subject.organizationname field), we currently can’t share this list before any possible GDPR concerns have been addressed. We will work with our legal department about solutions to resolve this.

  1. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
    The reason for the inclusion of the subject.altname.othername field is that CIBG uses this field to convey a specific identifier for Dutch healthcare purposes. These certificates, although issued under a public root, are primarily meant for machine-to-machine communication.

Because all issuance predates the requirements of CT logging Logius and the vast majority of certificates is only used for machine-to-machine communication (and as such not picked up by crawlers and being logged to CT logs) Logius wasn’t aware of this issue. This only became clear because of information gathering for bug 1573490.

  1. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things
    The following actions have been formulated by Logius:

  • As indicated in the timeline, the intent is to revoke all affected certificates before October 1. Because of the same issues as listed in bug 1535871 revocation within the timespan defined in the BRs is not possible without massive disruption of the Dutch Healthcare system.

  • The revocation process has already started. (Almost) all certificates under the CA “ZOVAR Server CA G21” have already been revoked. For the “UZI-register Server CA G21” revocation is ongoing.

  • The measures as proposed in bug 153871 have been published and are in effect or will be in effect on November 1 at the latest, which will hopefully increase both TSP and customer agility.

  • CIBG has switched issuing TLS certificates under a private hierarchy (minus infrastructure certificates like OCSP etc. until expiration of the G2 Root CA and associated issuing CAs).

Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true
Assignee: wthayer → jochem.vanden.berge
Whiteboard: [ca-compliance]

A few questions, as I try to build a holistic view:

As best I can tell, issues with CIBG were discoverable via publicly available information sources.

Have you examined

  1. Why these were not detected by Logius' controls?
  2. Why these were not detected by BSI?

Similarly, as requested in https://bugzilla.mozilla.org/show_bug.cgi?id=1573490#c3 , can you make sure to explain in Bug 1573490 the reason for the delay between 2019-03-15 and 2019-08-13? It seems that if the window between CIBG informing Logius (2019-03-15) and Logius investigating (2019-08-13 - 2019-08-28) was smaller, this issue would have been caught sooner, and the response to both this and Bug 1573490 would have been substantially different?

I realize you've shared some of the plans in Bug 1535871, specifically Comment 14 on that bug, and those seem to address the responsiveness issues mentioned here. However, what's being done to improve the oversight and compliance issues revealed through this and Bug 1573490?

Flags: needinfo?(jochem.vanden.berge)

Hello Ryan,
The past weeks we’ve been looking into this matter and after extensive research (and talks with the involved parties) we think we can explain why these issues weren’t detected earlier.

  1. Why these were not detected by Logius' controls?

The subject.altname.othername field had been in use by PKIoverheid TSPs in the past (since at least 2006). Logius changed the PKIoverheid CP after it was pointed out in 2014 (during the application phase of our G3 Root CA) that this field is forbidden by the BR. Seeing that this would greatly impact CIBG, and they were only issuing TLS certificate for use in machine to machine communication, that TSP was given time to solve the problem. Unfortunately, Logius failed to administrate this correctly and as a consequence (worsened by personnel changes) failed to follow up on this issue.

This led, in time, to a false assumption within CIBG that inclusion of the subject.altname.othername was still allowed.
Because of the fact that CIBG has since moved to issue under a private root, Logius' focus was on supervision of TSPs which did still issue TLS certificates under one of our publicly trusted roots. Because the affected certificates are, as earlier indicated, mainly used for machine-to-machine communications in the healthcare system and were issued before the requirement of CT logging, Logius therefore received little to no indication this was happening which explains why this could continue.

  1. Why these were not detected by BSI?

We received the following response from BSI:

The subject.altname.othername extension has been in use by CIBG for a long time to specify information used for sectoral message services within the Dutch healthcare system. It was a design decision at the time to use publicly trusted SSL certificates for that purpose, but - in practice – these certificates were used to facilitate secure machine-to-machine communication within the healthcare ecosystem.
BSI has previously identified the deviation from the Baseline Requirements at CIBG. BSI was informed that CIBG was given time to find an adequate solution, whilst not risking impact on the continuity of operations of Dutch healthcare processes.

The mitigating control by CIBG was to transition to the Staat der Nederlanden PKIoverheid Private Root CA hierarchy for the machine-to-machine certificates. This hierarchy is not publicly trusted by the browsers and not subject to the Baseline Requirements. This was realized in January 2018 and certificate issuance resumed from these newly created Uzi-register and ZOVAR issuing CAs.
BSI accepted this mitigating control during its audits and monitored the transition to the private root hierarchy. Due the impact of the transition on the Dutch healthcare system, BSI may have accepted a longer than preferred transition period.

CIBG had previously chosen not to actively migrate/replace the issued SSL certificates under the public root to the private root, again due to the impact to the healthcare system and in accordance with CIBG’s risk assessment. BSI has indicated that in hindsight, this approach should not have been accepted. At this particular client, BSI focused too much on the client-specific environment and the specific context (related to the use of the server certificates and the subject.altname.othername extension).

CIBG is BSI’s only client that has been using publicly trusted server certificates in a specific infrastructure environment, and that contains specific certificate extensions in its design.
BSI identified the following improvements/corrective actions:
- The case situation is discussed with all our ETSI-auditors.
- At the date of publication of Bug 1578809 we were performing our annual audit activities at CIBG. We inspected the technical controls aimed at preventing issuance of certificates under the configured Uzi-Register Server CA G21 and Zovar Server CA G21 profiles. We also inspected the issuance of certificates from these CAs. Details on Bug 1578809 will be included in our report (conformity certificate).

  1. Why were we not triggered by crt log?

Although we have implemented CRT log scanning (as indicated in comment 12 in bug 1535871), for now the focus was on currently issued certificates and not on historical issuance. Measures will be taken to remediate this issue, see below.

  1. Explain delay between 2019-03-15 and 2019-08-13

Unfortunately, this was an oversight. In the bug 1535871 we did indicate that the intention was to file a separate bug for CIBG. Unfortunately, due to the fact that the original intent was to let all CIBG certificates expire naturally and that the (initial) focus for Logius was on replacement of the intermediate CAs and KPN end-user certs this didn’t have the appropriate level of attention of Logius.

  1. What's being done to improve the oversight and compliance issues revealed through this and Bug 1573490?
  1. As indicated in bug 1535871 measures have been implemented by Logius which will increase both subscriber awareness and agility, also for future changes in technical requirements for (web)PKI certificates.
  2. Logius is already reviewing OoA (Overview of Applicability) statements from TSPs which they need to submit before their annual audit. Audit statements will only be accepted by Logius if the OoA has been signed off beforehand. We intend to expand on this mechanism on two points:
    • The TSP will be required to list all changed requirements in applicable standards (including the Baseline Requirements) on a separate tab for Logius and the auditor to check
    • Inclusion of conformance statements to the BR in the OoA. The conformance statements are another requirement already in place in which Logius asks the TSPs after each applicable CAB/F ballot to indicate the impact of the changes listed in a ballot on their operations and if they will be compliant with the required changes on time. This has to be signed off by a TSP manager or director. The new measure is to include these statements in the OoA so that they will brought under the attention of the auditor.
  3. Logius will do a thorough check of historically issued, still valid PKIoverheid certificates. This will also involve TSPs having to supply Logius with full certificate data of certificates not previously submitted to CT logs so these can be checked (linting), and CT logged. For this (and in general) additional resources will be hired.
  4. The additional personnel listed above will also help to prevent future issues like listed in bug 1573490 about the slow follow-up.
  5. Regarding the statement by BSI as listed above, Logius will initiate a meeting at the management level with BSI about the audits they perform for PKIoverheid TSPs.
  6. Make post-issuance linting by the TSP mandatory, and where possible, also pre-issuance linting (due to TWS requirements the latter measure can be difficult to implement). Currently, both are encouraged but not required for our TSPs.

Regards,

Jochem

Flags: needinfo?(jochem.vanden.berge)

Thanks Jochem.

The level of detail and analysis that Logius/PKIOverheid provides it its responses continues to be of excellent detail and an example for other CAs to learn from.

I think this answers my questions, and I'm putting to Wayne to see if he has follow-ups.

Flags: needinfo?(wthayer)

It appears that all questions have been answered and remediation is complete.

Status: ASSIGNED → RESOLVED
Closed: 5 years ago
Flags: needinfo?(wthayer)
Resolution: --- → FIXED
Product: NSS → CA Program
Whiteboard: [ca-compliance] → [ca-compliance] [ov-misissuance]
You need to log in before you can comment on or make changes to this bug.