Closed Bug 1535871 Opened 5 years ago Closed 5 years ago

PKIoverheid: KPN Insufficient Serial Number Entropy

Categories

(CA Program :: CA Certificate Compliance, task)

task
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: wthayer, Assigned: jochem.vanden.berge)

Details

(Whiteboard: [ca-compliance] [ev-misissuance] [ov-misissuance])

Attachments

(2 files)

Jochem van den Berge posted the following incident report to the mozilla.dev.security.policy list on 13-March:

  1.    How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.
    

3/8/2019 12.30, due to reviewing discussions in mozilla.dev.security.policy.

  1.    A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.
    

30/9/2016 Ballot 364 came into effect. The CP of Logius PKIoverheid already stipulated the use of 64-bit serial numbers and as such, no change was deemed necessary to the CP. Our CP (Programme of Requirements) is a baseline document, stating the absolute minimum. This ballot predates the incident which PKIoverheid had about serial numbers with one of her other TSP's in 2017 [1]. Measures which were taken then didn't apply retroactively.

3/8/2019 12.30 While reading MSDP the Logius PKIoverheid started an investigation if it was possible that her TSP's had this implementation/interpretation issue

3/8/2019 13.15 Logius PKIoverheid suspects that this issue could potentially impact one or more of the TSPs under PKIoverheid. Logius PKIoverheid asked the TSP KPN to launch an investigation if said issue was applicable to certificates issued by KPN.

3/11/2019 09:53 Logius PKIoverheid asked KPN for an update following statements from both Google and Mozilla representatives stating that in their view the matter as reported by several other CAs violates the BRG.

3/11/2019 16:55 KPN answers that this issue is potentially impacting all of their issued TLS certificates issued between September 30, 2016 and March 5, 2019. On March 5, 2019 KPN switched to using 96 bit serial numbers (already planned a while ago, this was not related to the current issue at hand).

3/12/2019 10:30 Due to the potential impact of revoking (and replacing) the PKIoverheid certificates from KPN issued in the period an incident is raised within Logius. KPN PKIoverheid certificates are in use by many Dutch government parties including the national ID system (DigiD), the tax services and Dutch customs. Because of this a crisis team is formed (also due to the fact that March/April is the month in which most tax returns need to be filed and the ever increasing change of a no-deal Brexit, which would greatly impact Dutch Customs) .

3/13/2019 12:00 Logius PKIoverheid orders KPN to further investigate which certificates are exactly affected and order KPN to revoke the certificates in question.

  1.    Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.
    

All certificates issued by KPN after March 5 08:30 are using 96-bit serial numbers. As mentioned this was a change unrelated to the current issue. As far as we know there are no TSPs within PKIoverheid other than KPN were up to recently issuing certificates with this issue. Further investigation is ongoing to see if there are possible historic issuance that might be impacted by this issue. We will post an update when we have more information.

  1.    A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.
    

Potentially 22.000 TLS certificates issued by KPN CAs https://crt.sh/?id=63094369 and https://crt.sh/?id=16678400. Also potentially ~350 EV certificate issued by CA https://crt.sh/?id=15971988. Investigation is still ongoing to which certificates are exactly affected.

  1.    The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.
    

Still being collected. Will update when available.

  1.    Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
    

As stated in the timeline, the Programme of Requirements (PoR, CP) PKIoverheid already stipulated the use of a serial number with a 64-bit length. When ballot 264 went into effect, both the PA and the TSPs determined that PKIoverheid was already compliant. The conversations about the underlying thoughts or intent of the ballot were seen at the time but not taken into account when deciding the final impact. The final text of the ballot after it was passed was used to check if implementations were correct. In this case the TSP also relied on the configuration of EJBCA and assumed that this was the correct implementation (again, also based on their interpretation of the text).

  1.    List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.
    

Still being worked on. The intention is to revoke all affected certificates within 30 days. Will update when we have more information.

[1] https://groups.google.com/d/msg/mozilla.dev.security.policy/vl5eq0PoJxY/W1D4oZ__BwAJ

Summary: PKIoverheid: Insufficient Serial Number Entropy → PKIoverheid: KPN Insufficient Serial Number Entropy

Hello Wayne,

Short status update: In our investigation for historic issuance (as indicated in the pre-incident report) it became clear that one other TSP (CIBG) issued certificates which are possibly affected. Our estimation is that this will affect a maximum of 4600 certificates. CIBG is investigating which certificates are exactly affected. CIBG stopped issuing TLS certificates in December 2017. We’ll be filing a separate bug for CIBG.

Logius PKIoverheid is also investigating if some of the PKIoverheid intermediate certificates might be affected by this issue. Seeing that the systems for issuance of intermediate CA certificates are all offline, it’s estimated that this part of the investigation will take more time. Depending the outcomes, we will outline a renewed plan for remediation.

Jochem van den Berge posted the following update to the m.d.s.p. list on 22-March:

Firstly our apologies for the delay in our follow-up response. A joint
analysis of the 64-bit serial issue by Logius and its TSPs took more time
than expected. We would like to share an update to our initial post-mortem
of 03/13/2019 and our subsequent update of 03/19/2019.

In close consultation with Dutch CERT (NCSC) and the politically and
administratively responsible officials of the Dutch Ministry of Interior
Affairs and Kingdom Relations we have conceived the following blueprint:

We’ve formulated the following starting points:

  1.  The effort involved in replacing the end-user S/MIME certificates is
    

severe plus almost half of them will expire and be replaced in the coming
year by either private certificates or newly issued certificates which
comply with the Mozilla CA policy section 5.2.
2. Replacing non-compliant end-user certificates issued by a TSP CA
which needs to be replaced involves a lot of rework. First course of action
should be to replace the issuing (TSP) CAs.
3. We consider compliant end-user certificates issued by a
non-compliant TSP CA to be compliant. The non-compliant issuing TSP CA will
need to be replaced by a TSP CA that has a serial number with sufficient
entropy (128 or 159 bit). After all issued certificates from the previous
TSP CA have expired Logius will revoke the TSP CA in question.
4. We’re working on replacing part of the PKIoverheid certificates
(both TLS and S/MIME) by certificates issued by a Private CA (“Staat der
Nederlanden Private Root CA – G1). This will have to happen on a case by
case basis and will take, unfortunately, some time.
5. Logius PKIoverheid will reach out to other root store operators
about our intended remediation plan.
6. The rate at which TLS certificates will be replaced is dependent on
the importance of the certificates in question for the critical network
infrastructure of the Dutch Government.

To clarify our position, some more context about the setup of PKIoverheid
and it’s different involved parties

  1.   PKIoverheid has issued several domain (intermediate) and TSP
    

(issuing) CA’s under its “Staat der Nederlanden Root CA – G3” Root CA. The
domain CAs (managed by Logius) differentiate between “Burger”, “Organisatie
Persoon”, “Autonome Apparaten” and “Services” (including Server/TLS)
domains. Under the domain CAs Logius signs issuing CAs for TSPs. In case of
the “Services” domain further differentiation takes place between “Services”
and “Server” (TLS), per demands from Microsoft Trusted Root Progamme
Requirements.
2. The “Burger” and “Organisatie Persoon” TSP CAs can be used by
PKIoverheid TSPs to issue S/MIME certificates which can also be used for
placing Qualified Electronic Signatures per eIDAS (also known as Regulation
910/2014 issued by the EC). Services and “Autonome Apparaten” are
certificates used for several services in which the holder is not a natural
person but a legal person. All of these certificates need to be issued on a
SUD or an SSCD/QCSD (token/smartcard) according to ETSI EN 319 411-1/2
policies NCP+ or qcp-n-qscd/qcp-l-qscd.
3. PKIoverheid TLS certificates (issued under the “Server” TSP CAs)
don’t have this requirement (SUD/QSCD) in place and as such are more easily
replaceable. Issuance of PKIoverheid certificates is done manually.
Replacement of certificates takes a lot of effort because vetting needs to
be redone if and when old data isn’t usable anymore (either because of new
requirements or expiration of vetting data).
4. Replacement of these certificates will have a severe impact on Dutch
society. Among other implementations PKIoverheid S/MIME certificates are in
widespread use on smartcards within the Dutch healthcare system. They are
also used on the on-board computers of Dutch taxis.

Result of analysis:

  1.  We conclude that our current G3 TSP CA’s (Issuing CA certificates)
    

are not compliant with ballot 164 and/or the current valid section of the
Mozilla CA Policy. The G2 and EV intermediate CAs are NOT affected.
2. We also conclude that all end-user TLS certificates issued by our
TSPs KPN and CIBG issued after the effective date of ballot 164 are not
compliant. For KPN this concerns approximately 22.000 TLS certificates, for
CIBG approximately 5000 TLS certificates.
3. Furthermore we have to conclude that all personal (S/MIME)
certificates from our TSP CAs KPN, CIBG and IenW issued after the effective
date of Mozilla CA Policy 2.4 don’t conform to the entropy requirements from
the Mozilla CA Policy.

Considerations

  1.  The 64 bit serial issue is a compliancy issue, not a security issue.
    

It’s not acceptable to introduce security issues in our effort to remediate
a compliance issue.
2. The above means that because of the lack of a imminent security
risk, revoking and replacement within a short timeframe instead of following
a measured approach is not considered a viable option.
3. We consider it very important to be compliant with regulations and
requirements and to resolve any non-compliance as soon as possible with
maximum transparency.
4. The SSL/TLS certs affected by the 64-bit entropy issue are used in
the Dutch critical network infrastructure. S/MIME Certs are used on a large
scale in, for example, the Dutch Healthcare. Mass revoking and replacing
these certs in a timely manner has severe consequences for the public
interest.
5. The issuance of PKIoverheid certificates takes place in a manual,
non-automated manner. So replacing the certs involved will, in all cases,
amount to manual processes.
6. Specialist knowledge is required due to the manual procedures. Our
TSPs have small specialist teams that deal with these procedures. Mass
replacement of the certs involved, will therefore mean we will have to lean
heavily, and for a prolonged period, on the small teams currently in place.
This will lead to the situation that mistakes are made due to fatigue,
introducing additional (security) risks etc.
7. To mitigate the above risk, temporary, new teams can be formed and
interim specialists hired. However to find these specialists, with knowledge
of the specific processes and tooling used by the different TSP
administration/validation teams, will proof very difficult. Even then,
trying to accomplish this task will, without doubt, introduce faults because
these interim specialists lack experience, which will lead to additional
(security) risks.
8. The above compounds to: there is no impending security risk, “mass”
revocation is not viable, no leeway to reduce replacement effort, zero
tolerance on (security) risk caused by replacement efforts, zero tolerance
on impact on society caused by replacement efforts. This dictates any
replacement plan.

We are working hard on the details of our remediation plan, based on the
constraints, analysis and considerations as mentioned above. This includes
(but is not limited to) looking into measures to reduce future constraints
on mass revocation, to increase certificate replacement speed and looking
into more strict compliancy checks and balances to prevent these kind of
issues in the first place. These will be posted as soon as possible.

Status: NEW → ASSIGNED

Jochem van den Berge posted the following update to the m.d.s.p. list on 29-March:

Dear MDSP community,

We would like to share an update to our initial post-mortem of 03/13/2019
and our subsequent updates of 03/19/2019 and 03/22/2019

Following the discussions on MDSP Logius has determined that the following
G3 TSP CA’s (Issuing CA certificates) are not compliant with BR 7.1 and/or
the current valid section of the Mozilla CA Policy and will therefore be
replaced within 2 months:

Cleverbase ID PKIoverheid Burger CA - G3; Digidentity BV PKIoverheid
Organisatie Persoon CA - G3; Digidentity BV PKIoverheid Organisatie Server
CA - G3; Digidentity BV PKIoverheid Burger CA - G3; KPN BV PKIoverheid
Organisatie Server CA - G3; MinIenW PKIoverheid Organisatie Services CA -
G3; MinIenW PKIoverheid Organisatie Persoon CA - G3; QuoVadis PKIoverheid
Organisatie Server CA - G3; UZI-register Medewerker op naam CA G3;
UZI-register Zorgverlener CA G3; UZI-register Medewerker niet op naam CA G3.

The TSP CAs in question will be replaced by CAs with a serial number length
of 128 bits or higher.

The following roadmap will be used for remediation of the end user
certificates as outlined in our previous post:
• KPN has already switched to serial numbers with a 96-bit length on
March 5. New certificates issued by them after this date are not affected.
Of the 22.000 TLS certificates in scope ~3900 have been marked as being in
use for browser use cases (websites e.g.) and will be replaced after the new
issuing CA has been brought into operation. The replacement of the TLS
certificates will take 6 months so the total replacement will take 8 months.
The remainder of the certificates (~18K) will be regularly (re)checked to
confirm that they are in use for private systems and machine-to-machine
traffic. They will be replaced by compliant or private SSL certificates when
possible.
• Logius has decided not to replace the TLS certificates issued by
CIBG as these ~4500 certificates will expire before the end of March 2020
and will automatically be replaced by private certificates.
The following roadmap will be used for S/MIME certificates as outlined in
our previous post:
• We will let the current S/MIME certificates expire.
All other end-user certificates issued by PKIoverheid have a serial number
of sufficient length.

In addition, we have formulated the following new measures to ensure this
particular type of issue will not be repeated in the future and which will
improve the resilience of our PKI setup:

Technical constraints/use of Private CAs
PKIoverheid has found that for certain types of use, the supplied types of
certificates could be technically restrained without affecting its intended
use. We will further research this area and will strive to technically
restrain EE certificates as much as possible.

Increase in technical monitoring/oversight
PKIoverheid runs a program called VTT (verbeterd technisch toezicht;
“improved techical supervision”) which closely follows developing industry
practices concerning technical oversight tooling (e.g. linting tools). We
will increase resources for this program to enhance our insight into the
quality of certificates issued within the PKIoverheid scope and as such
catch potential issues earlier.

Self-service (re)provisioning/automated issuance
Although we cannot control how end users implement the certificates issued
by PKIoverheid TSPs, we think it prudent to distribute PKI implementation
best practises for end-users, using newly developed or existing self-service
(re)provisioning tooling.

Regards,

Jochem

Hello Wayne,

Our apologies for not following up earlier on this issue.

The first few actions of the remediation plan (as stated in comment #3) have been implemented. The new intermediate CAs have been signed with a serial number of 160 bits and some of them have been resigned with technical constraints (no emailProtection and serverAuth EKU) as to exclude them from the webPKI. KPN has started communication to end users about the replacement of the affected TLS certificates and intends to start issuance of new certificates from the new TSP CA later this week.

The scope has now been revised to ~3400 TLS certificates based on the earlier approach, but as indicated above monitoring is in place to check if the certificates that are out of scope for now, will remain so. If not, they will be replaced and revoked. PKIoverheid also has recruited additional personnel which will start on June 1 which allows the PA PKIoverheid to intensify its supervision task of the Trust Service Providers within
PKIoverheid, including technical controls.

Regards,

Jochem

Jochem: thank you for the update. Please continue to update this bug periodically as replacement of TLS certificates progresses.

Jochem: Can you provide an update on the progress? Including the June 1 activities?

Flags: needinfo?(jochem.vanden.berge)

Hello Ryan,

The current situation is as follows:

Since the last update a few more of the offending CA certificates (two from Digidentity) have been revoked (on the 27th of June). For now these will be the last revocations of the TSP CAs until we hit the moment that TLS and/or S/MIME certificates issued by the old intermediates will expire (which is in 2021). If anything changes in the meantime, I’ll post an update about that in this bug.

I'll have updated information about KPN replacement of TLS certificates. At the moment KPN is focussing on replacement before revoking certificates. As such, the current numbers are as follows:

  • Total certificates in scope: 3263
  • 2647 certificates have been reissued with a 96-bit serial number
  • 578 are in the process of domain/organisation (re)validation
  • 38 certificates have been revoked

The low number of revocation is due to the policy of asking for customer confirmation of installation of the new certificates before revoking certificates. For larger organizations this can take a while, especially if there are relying parties who need to update their own systems. We expect the number of revocations to increase of the coming months.

As for the intensified supervision: We have freed up capacity to make the The capacity has freed up time in the time to spin up our own instance of crt.sh (we have used the code graciously provided by Sectigo on Github). This internal monitor which for now pulls (pre)certificates from CT logs and filters out PKIoverheid certificates on which (post-issuance) linting is taking place. In case of any errors Logius will contact the CA in question to follow-up. Further steps are still being considered

Flags: needinfo?(jochem.vanden.berge)

Thanks for the update.

When you say "in scope", my understanding is that 22,000 TLS certificates are in scope of the BRs and Mozilla policy. I understood the 3,263 number to be a limited subset of those you knew targeted browsers specifically, but that the full set of certificates were in scope. Could you explain the discrepancy?

As to the issue, am I to understand correctly that of the 22,000 / 3,263 certificates (whichever number you prefer), only 38 of those certificates have been revoked since 2019-03-05, as per Comment #3? That's a serious and concerning lack of progress there, worse than any other CA responding to this matter, and I think is a matter of great concern, if I understand correctly.

As to the challenges, please pay careful attention and re-review https://wiki.mozilla.org/CA/Responding_To_An_Incident#Revocation regarding appropriate responses as well as expectations regarding discussion of challenges. Further, for each of the challenges identified, please include as part of your incident report the steps your CA is taking to ensure those are never challenges for your CA again, and thus never a reason your CA uses to delay revocation.

I'm thrilled to hear about spinning up your own instance of crt.sh. Provided it's regularly updated and maintained (along with the linters), this provides a good balance in terms of self-reliance while sharing community best practices.

Flags: needinfo?(jochem.vanden.berge)

Pointing Wayne at Comment #7 regarding the (lack of) progress.

Flags: needinfo?(wthayer)

Hello Ryan,

Regarding the scoping discussion: My intention was to highlight the subset that will be revoked and replaced and of course not the total set of affected certificates in scope of Mozilla Policy, which remains at the original number.

As detailed in our original remediation plan (see comment #3) the route we planned to take is to resign the affected intermediate CAs first , after which KPN uses the new intermediate CA to issue new, compliant end-entity TLS certificates with timelines as indicated earlier. The new intermediate CA wasn’t signed on 03-05-2019, so replacement couldn’t start at that date. After posting each update we haven’t received any comments from Mozilla or the community until recently. We took this as a sign of implicit agreement with the plan with the dates proposed therein.

The focus has been on replacing the certificates first, as indicated in my earlier posts. Some subscribers have opted to replace certificates slightly later as they will expire later this year. The switch to private certificates, where applicable, will be made at the same time, requiring some more lead time. We have confidence we will reach our goal according to our remediation plan as outlined earlier and we’re confident that the amount of revocations is going to go up quickly over the coming period.

As to measures to prevent future occurrences like these regarding slow revocation, concerning (critical) Dutch infrastructure and services, most of which have been already put forward in earlier responses like the switch to private CAs (where possible). We’re also thinking about reducing the maximum validity period of our (public) TLS/EV certificates to promote automation and faster replacement with subscribers.

Flags: needinfo?(jochem.vanden.berge)

Jochem: I have some questions and concerns:

  1. Nowhere has the precise number of affected certificates been identified, as required by incident report items #4 and (more specifically) #5. While I believe that the answer is effectively "all of them", the failure to provide a precise response has led to confusion that might have contributed to the unfortunate conclusion that Logius has "implicit agreement with the plan".
  2. A separate bug for CIBG was never created as proposed, further muddying the waters regarding the scope of the issue.
  3. Nowhere does Mozilla recognize the concept of TLS certificates that are not "in use for browser use cases." In fact, Mozilla has repeatedly made efforts to clarify the scope of our policies. Going forward, please do not redefine the scope of misissued certificates.
  4. When will all of the 3263 certificates that Logius plans to revoked be revoked? (apologies if there is a date that I'm missing somewhere in one of the comments)
  5. A number of steps have been proposed to prevent future failures to revoke certificates. That is good. However, it is not clear if some of these are ideas, or when any will be implemented. Considering the large extent to which Logius has chosen to violate the revocation requirement, please provide a roadmap with dates by which preventive measures will be completed. Also, please clarify when all un-revoked TLS certificates with low entropy serial numbers issued by Logius will have expired.
Flags: needinfo?(wthayer) → needinfo?(jochem.vanden.berge)

Hello Wayne,

Apologies for the confusion. Please find our responses to the issues you raised below.

(In reply to Wayne Thayer [:wayne] from comment #11)

Jochem: I have some questions and concerns:

  1. Nowhere has the precise number of affected certificates been identified, as required by incident report items #4 and (more specifically) #5. While I believe that the answer is effectively "all of them", the failure to provide a precise response has led to confusion that might have contributed to the unfortunate conclusion that Logius has "implicit agreement with the plan".

The number of certificates affected by this issue that are in scope of the Mozilla Policy are all TLS certifcates issued by KPN between October 1, 2016 and March 5, 2019 under several intermediate CAs that chain back up to the “Staat der Nederlanden Root CA – G2”, “Staat der Nederlanden Root CA – G3” and “Staat der Nederlanden EV Root CA”. This amounts to ~22.000 TLS certificates (as indicated in earlier posts), effectively all of the ones issued by KPN in that period. I should have CSVs with both the certificates in scope affected by this issue and the subset that we’re replacing shortly and share those in this bug.

  1. A separate bug for CIBG was never created as proposed, further muddying the waters regarding the scope of the issue.

Again, our apologies. This has been overlooked in the period after the first report. A separate bug will be filed with additional details regarding CIBG.

  1. Nowhere does Mozilla recognize the concept of TLS certificates that are not "in use for browser use cases." In fact, Mozilla has repeatedly made efforts to clarify the scope of our policies. Going forward, please do not redefine the scope of misissued certificates.

As described in my response to question #1 the intent was never to redefine the scope of misissued certificates, only to indicate the subset of certificates that will be replaced and revoked. This was something that was lost in translation I’m afraid. It was never our intend to deliberately mislead Mozilla (or other relying parties) about the amount of certificates that are in scope of the Mozilla policy.

  1. When will all of the 3263 certificates that Logius plans to revoked be revoked? (apologies if there is a date that I'm missing somewhere in one of the comments)

At the latest by December 1, 2019. As indicated in earlier posts a large percentage of the certificates have been reissued by now. We intend to follow-up with customers to confirm they’ve installed the new certificates so that we can revoke the old ones. Our expectation is that the amount of confirmations (and revocations) will rise over the coming weeks, but this might be hampered by the start of the holiday season in the Netherlands.

  1. A number of steps have been proposed to prevent future failures to revoke certificates. That is good. However, it is not clear if some of these are ideas, or when any will be implemented. Considering the large extent to which Logius has chosen to violate the revocation requirement, please provide a roadmap with dates by which preventive measures will be completed. Also, please clarify when all un-revoked TLS certificates with low entropy serial numbers issued by Logius will have expired.

Seeing that some of the proposals are related or connected, the dates listed next are approximate.

  • The own instance of crt.sh is already running, output is checked on a weekly basis
  • The switch to private certificates is connected to the idea of reducing validity, in sofar as customers who can deal with rapid replacement and/or automation can still use public certificates if needed. Those who can’t are forced to change their systems or switch to a private CA. However, the proposal to reduce the validity will have a rather large impact on many customers (both government and non-government) and is a rather significant policy change, which needs to be discussed with the Ministry of the Interior. If implemented, it is something that needs to be communicated in time to all affected parties (government and non-government). Seeing all this, we are aiming to get a formal decision whether to reduce the maximum validity to 397 days* at the beginning of December.

(397 days relates to ~13 months, accounting for leap years and maximum length of the last month of the lifetime of a certificate)

Flags: needinfo?(jochem.vanden.berge)

At the latest by December 1, 2019.

Jochem: I would strongly encourage you to read the exchanges with other CAs, such as starting with this one, to consider the timeline of the response and the level of detail provided. I cannot stress enough the significance of potentially needing to take 9 months to comply with Mozilla policy, and the serious concerns that would raise about the operation of the CA. Regardless of those concerns, however, such a significant delay must be accompanied with an equivalent level of significant detail, and this bug does not yet raise to that level.

The level of responses in Comment #12 are very similar to that, and I encourage you to carefully read that discussion. Mostly so I don't have to repeat the same concerns and admonitions here. That discussion was about a four month delay, and the proposal here for a nine month delay simply beggars belief.

After you've read that Bug 1534295, from Comment 13 onwards, I encourage you to revisit the reply here.

Flags: needinfo?(jochem.vanden.berge)

Hello Ryan,

During the holiday absence of Jochem I will try to provide you with the answers and information to the issues you raised.

Our apologies for the delay regarding our follow-up. The reason for this is that we implemented some policy changes and additional countermeasures to improve both the current status as well as to prevent a similar incident from happening in the future. See below for the details.

First, we would like to give you an update regarding our revocation timeframe. We are confident that most certificates will be revoked before October 1st. The original timeframe of December 1st as indicated in earlier comments, was based on the assessment of the crisis team combined with KPN estimates for certificate replacement speed at subscribers. Although the replacement speed and revocation speed has been low, we are confident that numbers will increase rapidly, because KPN is vigorously chasing their subscribers. A small group of certificates (roughly 20, see attachment) for the customs portal of the Dutch Government (Digipoort) need a longer timeline and are to be replaced in October.

Although we can’t provide a detailed report per subscriber, the earlier mentioned assessment by KPN and the crisis team indicates that in general most subscribers implement their certificates and corresponding processes in their critical infrastructures in such a way, that they cannot deal with rapid replacement.

We are concerned about the speed at which all certificates can effectively be reissued by a CA, as well as the agility of certificate management at the subscribers’ end. It is clear that they currently cannot deal with rapid replacement in case of a security or compliance issue.

We are going to implement the following structural solutions to prevent a similar incident from happening in the future:

First of all, we are going to reduce the maximum validity of TLS certificates to 397 days per the 1st of November this year. With the shortened lifespan of the certificates, we want to improve the certificate management agility of subscribers. For subscribers where this agility cannot be reached, this measure forces them to switch to the Private Root certificates.

Secondly, we are going to make sure that the CAs are able to rapidly reissue their certificates. Therefore, we made a policy change that states that CAs must be able to issue and distribute the total of issued TLS (including EV) certificates within 5 days. This change will be effective at the 1st of November this year. This will stimulate more efficiency, automation and therefore agility.

Thirdly, CAs will actively inform their subscribers at least twice a year that in conformity with the Terms of Use, certificates will be revoked under the conditions of and within the time periods mentioned in 4.9.1.1. BRG. Consequently, they should be able to replace certificates on a short notice. This subscriber awareness further contributes to the subscriber certificate management agility needed.

Attached file kpn_extension.txt

Hello Ryan (and Wayne),

We wanted to provide Mozilla with an update about the current status of the revocation effort made by Logius and KPN.

All 21.954 certificates in scope of the Mozilla Policy (as defined in comment 12) have been CT logged. For full disclosure, they were issued by (and as such can be found on crt.sh via) the following CAs:

KPN PKIoverheid Organisatie CA - G2 https://crt.sh/?id=16678400
KPN BV PKIoverheid Organisatie Server CA - G3 https://crt.sh/?id=63094369
KPN PKIoverheid EV CA https://crt.sh/?id=15971988

We’ve seen a speedup over the board since announcing the new revocation deadline, and KPN has been vigorously chasing subscribers, but it seems that some subscribers were not able to meet the new revocation deadline. At this moment 83% of the certificates in the subset has been revoked.

For the 17% of remaining certificates from the subset which haven’t been revoked yet the issue is that most of them are used in vital communication for the Dutch Government. Because of the often complex environments, subscribers have requested an extension of the revocation deadline. The 17% of remaining certificates amounts to 562 certificates (not including the certificates for the Dutch Customs which were mentioned in comment 15). These will be revoked before the following dates:

  • 474 before October 15
  • 78 before November 1
  • 10 before December 1

The 562 certificates mentioned above are included in a list attached to this comment.

The certificates for the Dutch Customs (mentioned in comment 15) will be replaced, where possible, by private certificates (not publicly trusted). Because of the complexity of the ecosystem which relies on this certificates, this will take extra time. They should be replaced (and the old certificates revoked) before December 1 as well.

Measures have been already mentioned above, most of which are unfortunately not in place yet.

Please let us know if you have any more questions.

Regards,

Jochem

Flags: needinfo?(jochem.vanden.berge)

Hello Ryan and Wayne,

We want to provide you with an update concerning the revocations as stated in comment 16. It was planned to revoke 474 certifcates before October 15.

We are happy to inform you that 507 certifcates of the 562 stated in comment 16 have been revoked.

We will revoke:

  • 45 before November 1
  • 10 before December 1

Regards,

Jorik

Leaving it to Wayne to set the Next Update, but I think Comment #17 sounds reasonable, given that the reason for the delay is to complete a meaningful and effective transition to a private PKI (as mentioned in Comment #16). That seems to provide assurance that it will not repeat in the future.

Flags: needinfo?(wthayer)

Jorik: were the 45 revocations noted in comment #17 completed on schedule?

Flags: needinfo?(wthayer) → needinfo?(jorik.vant.hof)

Hello Wayne,

We are still on schedule. The 45 certificates in question have been revoked.

Regards,

Jorik

Flags: needinfo?(jorik.vant.hof)
Whiteboard: [ca-compliance] → [ca-compliance] - Next Update - 01-December 2019

Hello Wayne,

All certificates have been revoked.

Regards,

Jorik

Flags: needinfo?(wthayer)

It appears that all questions have been answered and remediation is complete.

Status: ASSIGNED → RESOLVED
Closed: 5 years ago
Flags: needinfo?(wthayer)
Resolution: --- → FIXED
Whiteboard: [ca-compliance] - Next Update - 01-December 2019 → [ca-compliance]
Product: NSS → CA Program
Whiteboard: [ca-compliance] → [ca-compliance] [ev-misissuance] [ov-misissuance]
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: