1709872 - KIR S.A.: Delayed revocations of certificates

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Piotr Grabowski]

Bug report:

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

The issue is continuation of a bug https://bugzilla.mozilla.org/show_bug.cgi?id=1708965. 225 certificates (still valid ) were issued with a validity period 1 year + 1 second, where
CPS in the time of the bug was raised specified validity period equeal 1 year. The Certificates with an additional second were issued from 2020-09-17 to 2021-04-28.

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

2020-09-01 BR update - Certificates issued SHOULD NOT have a Validity Period greater than 397 days and MUST NOT have a Validity Period greater than 398 days.
2020-09-01 KIR CPS update specifying the validity period for SSL certifctaes for a maximum of 1 year. Until 01/09/2020, KIR issued certificates with a validity period of up to 2 years.
2021-04-20 08:30 am CEST CPS review was started. It was related to the bug https://bugzilla.mozilla.org/show_bug.cgi?id=1705904
2021-04-28 11:23 PDT KIR was informed by the third-party in the bug https://bugzilla.mozilla.org/show_bug.cgi?id=1705657#c35 about the issue.
2021-04-29 07:38 PDT New version of KIR CPS was announced to be published on the 1st of May 2021. In the new CPS the maximum validity period of SSL certificates is 398 days from the date of certificate generation.
2021-05-01 08:00 CEST New version of KIR CPS went live.
2021-05-01 08:00 am CEST Communication with end-users started to inform about the issue and to set up earliest replacement time.
2021-05-04 2:00 pm CEST Replacement plan was made and communicated.

**3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

KIR has stopped issuing certificates with the problem.

4. summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

180 certificates for banking system in Poland.
28 certificates for KIR systems
17 other customers

5. The complete certificate data for the problematic certificates.

We can provide full CN, SN list.

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

The issue started 01.09.2020 when we have published CPS v1.12 in which we have failed to consider new validity period definition:
Validity Period: Prior to 01.09.2020, the period of time measured from the date when the Certificate is issued until the Expiry Date. For Certificates issued on or after
01.09.2020, the validity period is as defined within RFC 5280, Section 4.1.2.5: the period of time from notBefore through notAfter, inclusive.
The error is due to wrong description of the validity period in the CSP. The validity period of the issued certificates falls within the validity period approved by the BR, however, it is imprecise and does not include inclusive.

7. List of steps CA is taking to resolve the situation and ensure it will not be repeated.

Due to the number of certificates and the importance of the systems, certificates cannot be revoked within 5 days. Revoking the certificates in such a short time could adversely affect the operation of banking systems.
The replacement of these certificates requires a lot of efforts from our clients.

New plan was prepared to split certificate replacement into subtasks:

180 certificates for banking system in Poland - will be replaced until 02.08.2021
28 certificates for KIR systems - will be replaced until 24.05.2021
17 other customers - will be replaced until 01.11.2021 - the certificates are generated for our very important customers. We are already involved in a replacement process and we will do our best to replace them as soon as possible, before 01.11.2021.

More information about remediation plan can be found here
https://bugzilla.mozilla.org/show_bug.cgi?id=1705904#c17

Comment 1

[Ryan Sleevi]

Thanks Piotr. I'd like to ask you to redo this report.

1. Please read https://wiki.mozilla.org/CA/Responding_To_An_Incident#Revocation for the expectations here.
2. Please read other revocation delay incident reports, so that you can both learn from them and so that you do not repeat the same mistakes.
3. When redoing this report, please actually address Question 5. Carefully review https://wiki.mozilla.org/CA/Responding_To_An_Incident#Incident_Report for the expectations here, which this clearly does not meet.
4. When redoing the report, please view Question 6 and 7 as the opportunity not to explain the reason why you need to revoke the certificates in the first place - that's what Bug 1708965 is specifically for. Instead, you need to be viewing them as explaining the issue with the revocation delay itself, and providing both sufficient details to explain (again, see https://wiki.mozilla.org/CA/Responding_To_An_Incident#Revocation ) and taking meaningful steps to ensure this does not happen again.

For example, it's completely unreasonable to believe that a CA needs 6 months to perform revocation of 17 certificates. That speaks to a critical failure of the CA to take compliance seriously, or to have designed their CA operations to comply with the Baseline Requirements, which from their very first version have required CAs to be capable of revoking within 24 hours. I understand that https://wiki.mozilla.org/CA/Responding_To_An_Incident#Revocation acknowledges there are tradeoffs, but it absolutely is not an exception to the BRs (which you will no doubt wish to speak of to multiple root programs about), and any CA that had been following m.d.s.p. or related Bugzilla incidents, as KIR S.A. has been required to do, should have long been aware of this fact.

So there's clearly a systemic failure here in how KIR S.A. has designed and operated its systems. For example, if the delay in revocation is because revoking will cause disruption, and because you cannot reissue because you need people to pick up their certificate in person, then a proper remediation would be the immediate and complete cessation of that requirement.

Similarly, if part of the reason in delays is because it will take considerable time to revalidate the non-domain related data, such as for OV or EV, then an appropriate and proper remediation would be for KIR S.A. to no longer issue OV and EV certificates, since it is demonstrably incapable of doing so while complying with the BRs' requirements.

Finally, if the issue is because your customers need more time to request and/or acquire certificates, then an example of an appropriate response would be to focus significant engineering resources to supporting automated methods of issuance (e.g. ACME), and requiring your customers to obtain your certificates that way.

Please understand: It's inexecusable for KIR S.A. to be in this position, given the industry trends and knowledge. While we are sensitive to the impact of revocation, the failure here - and this is both a choice and an egregious failure on KIR S.A.'s fault - is profoundly significant and troubling. Your response needs to show a deep and thorough, systemic, understanding of these issues and how to fix them.

I would also encourage you to carefully review https://wiki.mozilla.org/CA:Camerfirma_Issues , about a recently distrusted CA, and notice how many parallels there are to KIR S.A.'s ongoing series of incidents, so you can understand how significantly this calls into question KIR S.A.'s design and operations. If this is uncomfortable to hear, then perhaps deciding to no longer issue TLS certificates at all may be a more appropriate and meaningful resolution.

Comment 2

[Piotr Grabowski]

Thank you Ryan for all your comments. We really appreciate it. We will try to do our best to redo the report.
Responding to your comment, KIR can revoke and generate new certificates ASAP.
It’s not an issue at KIR availability, but actions which have to be taken at client site.
We assumed that these certificates MUST be replaced within 6 months, but we want to do it ASAP.
Believe us we don’t want to have an open issue for such a long time. We identified a group of clients with 17 certificates which need more time to generate new request and install certificates on their systems as the certificates are used in their important systems.
We will keep you informed about all changes in the issue.
I am enclosing the list of affected certicates.

Comment 3

[Piotr Grabowski]

Attachment: revocation-list.txt

Comment 4

[Ryan Sleevi]

Piotr: The goal is to understand what KIR S.A. specifically is doing to resolve these client issues.

Put differently: The choice to not revoke is a CA decision, and ultimately, the CA's responsibility. CAs are trusted on the basis of their commitment to follow the BRs, which means the CA, not the customer, needs to design their systems to support and ensure compliance. https://wiki.mozilla.org/CA/Responding_To_An_Incident#Revocation is meant to address the situation where the CA has failed to take reasonable steps to do so, but please be aware, each incident (from any CA) increases in severity, because every CA is expected to be learning from and improving their systems to ensure their customers are prepared.

That's why Comment #1 focuses significantly on KIR S.A.'s design and business decisions. KIR S.A. should have prepared for this situation, and designed their systems accordingly so that each and every one of their customers could promptly replace their certificates, or that KIR S.A. would reject requests from customers that were not prepared to do so. This is equally why the Baseline Requirements requires that each CA have a legally binding and enforcable agreement with their Subscribers to ensure that their Subscribers are aware of this.

Equally, comments like Comment #2 present it as if KIR S.A. simply cannot revoke, yet that's simply not true. KIR S.A. has both the technical capability and the legal authority (per the BRs) to revoke. So it boils down to a business decision by KIR S.A.'s management. https://wiki.mozilla.org/CA/Responding_To_An_Incident#Revocation is meant to capture that the CA is responsible not only for justifying that business decision, but in working to demonstrate they will never need to make such a decision again. Please understand, by making the business decision, this seriously calls into question KIR S.A.'s management, because it means KIR S.A., for years, failed to take proper steps to ensure they would never be in this place to begin with.

This is not a demand you revoke immediately, because it's clear either way, you're violating the BRs and the concerns now exist. But it is an expectation that KIR S.A. make it clear exactly why that business decision is justified, particularly the timelines (of which 6 months delay is impossible to justify for any CA), and what steps its taking to prevent that in the future. And, as mentioned, be aware that if KIR S.A. isn't able to demonstrate both why it's necessary (according to the guidelines in https://wiki.mozilla.org/CA/Responding_To_An_Incident#Revocation ), and demonstrate precisely how it will ensure it never happens again, then the inevitable outcome is a serious loss of confidence.

To put a comparison: This is no different than breaking a business contract, but expecting to still get paid. The difference in how long it takes you to remediate this is similar to the difference between showing up 5 minutes late to work vs not showing up at all, or having the cash register short $5 at the end of the day vs embezzling a million dollars. The burden is on KIR S.A. to demonstrate the factors, and to demonstrate the remediation plan to ensure those factors are never an issue again.

Resetting N-I for the incident report requested in Comment #1, which will hopefully closely examine both open and closed revocation delay issues, and look carefully at both what KIR S.A. should already have been doing, and what KIR S.A. can do to improve.

Comment 5

[Piotr Grabowski]

Bug report:

Delayed revocations of certificates issued by KIR

Bug report:

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

The issue is continuation of a bug https://bugzilla.mozilla.org/show_bug.cgi?id=1708965. 225 certificates were issued with a validity period 1 year + 1 second, where CPS in the time of the bug was raised specified validity period equal 1 year. The Certificates with an additional second were issued from 2020-09-17 to 2021-04-28.
We failed to meet the deadline of 5 days for revocations of these certificates because of the reasons stated below.

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

2020-09-01 BR update - Certificates issued SHOULD NOT have a Validity Period greater than 397 days and MUST NOT have a Validity Period greater than 398 days.
2020-09-01 KIR CPS update specifying the validity period for SSL certificates for a maximum of 1 year. Until 01/09/2020, KIR issued certificates with a validity period of up to 2 years.
2021-04-20 08:30 am CEST CPS review was started. It was related to the bug https://bugzilla.mozilla.org/show_bug.cgi?id=1705904
2021-04-28 11:23 PDT KIR was informed by the third-party in the bug https://bugzilla.mozilla.org/show_bug.cgi?id=1705657#c35 about the issue.
2021-04-29 07:38 PDT New version of KIR CPS was announced to be published on the 1st of May 2021. In the new CPS the maximum validity period of SSL certificates is 398 days from the date of certificate generation.
2021-05-01 08:00 CEST New version of KIR CPS went live.
2021-05-01 08:00 am CEST Communication with end-users started to inform about the issue and to set up earliest replacement time.
2021-05-04 2:00 pm CEST Replacement plan was made and communicated.
2021-05-12 2:00 pm CEST Updated replacement plan was made and communicated.

3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

KIR has stopped issuing certificates with the problem.

4. summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

180 certificates for banking system in Poland.
28 certificates for KIR systems
17 other customers

5. The complete certificate data for the problematic certificates.

The complete list of the affected certificates can be found here https://bugzilla.mozilla.org/attachment.cgi?id=9220972 (revocation-list.txt)

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

Short description about the reason why these certificates should be revoked:

The issue started 01.09.2020 when we have published CPS v1.12 in which we have failed to consider new validity period definition:
Validity Period: Prior to 01.09.2020, the period of time measured from the date when the Certificate is issued until the Expiry Date. For Certificates issued on or after 01.09.2020, the validity period is as defined within RFC 5280, Section 4.1.2.5: the period of time from notBefore through notAfter, inclusive.
The error is due to wrong description of the validity period in the CSP. The validity period of the issued certificates falls within the validity period approved by the BR, however, it is imprecise and does not include inclusive.

The delay in revocation is mainly caused by:

a) The disruption of some systems where the certificates are installed and propagated among the users. As we previously mentioned some of these certificates are installed in cross banking critical systems. Replacement of such a certificate requires a plan, prior notification and coordinated efforts of many banks.
b) Awaitng for the fix related to the full pre-linting https://bugzilla.mozilla.org/show_bug.cgi?id=1705187. Until we get the patch requested from the vendor the operators are not allowed to fix any typo in the requests provided. If a request contains a typo it is rejected.
c) Self-restriction imposed in https://bugzilla.mozilla.org/show_bug.cgi?id=1705187#c21 consisting mainly of:
-restricted access to the registration policies to the most trained 2 out of 100 operators,
-additional workload for these 2 operators in a form of another testing environment to pre-lint all P#10 request.
d) The time needed for domain re-validation for some of the affected certificates.
e) The number of the affected certificates.

7. List of steps CA is taking to resolve the situation and ensure it will not be repeated.

Due to the reasons described in point 6 the affected certificates cannot be revoked within 5 days.
Here we provide mitigation plan to resolve the situation and ensure it will not be repeated.

a) KIR has no direct effect on the systems in which the certificates are installed, they are completely independent. However, KIR used its all available communications channels to express the need of the review of the architecture of these systems in order to make such changes so that the certificate replacement was as short as possible. For this purpose, KIR offered the support of KIR's architects, analysts and experts.
b) Resolving the bug and applying the fix will significantly decrease the time needed for handling the request with full pre-linting feature. For the time being the operators are obliged to generate certificate in the tests environment and use the post-linting result as a pre-linting input.
c) Resolving the same bug https://bugzilla.mozilla.org/show_bug.cgi?id=1705187 will allow us to remove self imposed restrictions and assign the appropriate number of operators for handling similar issues.
d) We have already started the process of implementing ACME protocol in our system. This task is being analyzed and the architecture and integration options are considered. This component is a part of our "automation" project. We truly believe that it will minimize the time needed for validation, renewal or replacement cases.
e) same as d)

We decided to review our plan and focus all our efforts to set ourselves a target to replace all affected certificates until 15th of June 2021.
The replacement process is now in progress.

180 certificates for banking system in Poland
28 certificates for KIR systems
17 other customers

Comment 6

[Piotr Grabowski]

The replacement process is ongoing. The first part of certificates is already replaced. We will give more precise numbers by the end of May.
We keep the target date of replacement and revocations of all affected certificates until 15th of June 2021 still valid.

Comment 7

[Piotr Grabowski]

More than 90% new certifcates have beed generated. We keep the target date of replacement and revocations of all affected certificates until 15th of June 2021 still valid.

Comment 8

[Piotr Grabowski]

82% (186 of out 225) of affected certifcates were revoked.

Comment 9

[Piotr Grabowski]

90% (201 of 225) of affected certifcates were revoked.

Comment 10

[Piotr Grabowski]

All certificates were revoked except one:
https://crt.sh/?id=3502339085
Due to technical problems, the bank had to postpone the replacement. The new service window, due to the unavailability of the bank's systems, had to be agreed with the Polish Financial Supervision Authority and was set on June 18-20.

Comment 11

[Piotr Grabowski]

All affected certifcates were revoked.

Comment 12

[Ryan Sleevi]

It's unclear the timeline in Comment #5 for further improvements (e.g. ACME).

It's also unclear what steps KIR S.A. has taken to address and prevent issues like those in Comment #10. For example, it would appear that, at any time, the Polish Financial Supervision Authority may declare KIR S.A. should not revoke a certificate, and that KIR S.A. will agree to that, despite the commitments it has made to Mozilla and the broader community. It could be that you're trying to say that "Once we have ACME support, such postponement will be unacceptable", but if that's the case, it's good to be explicit about it.

Comment 13

[Piotr Grabowski]

Ryan, initially the ACME project was planned as one of the components of the automation project. Recently, we made the decision to try to extract ACME functionality to able to implement ACME independently and faster.
At this point in time, we are operating on two tracks. On the one hand, we are working to develop ACME ourselves, and on the other, we are putting pressure on the CA vendor, as we know that they are also planning to release this software soon, but in a completely new version and core architecure. It seems that we can develop the piece of software ourselves sooner. For today we are waiting for cost estimation and timeschedule from our development department.
We very tentatively assume that we will be able to deliver ACME in Q4 of this year.

Certainly the implementation of ACME will significantly speed up the replacement of certificates under one condition: if customers are really interested in using it. For a simple website the benefit is obvious. In case of complex banking or clearing systems this will not always could be the case. We know from the experience that replacement of simple certificate in complex system requires prior agreements, tests in pre-production environments and sometimes approval of regulatory institutions such as PFSA. As we said in comment 5: KIR has no direct effect on the systems in which the certificates are installed, they are completely independent. However, KIR used its all available communications channels to express the need of the review of the architecture of these systems in order to make such changes so that the certificate replacement was as short as possible. For this purpose, KIR offered the support of KIR's architects, analysts and experts. 90% of affected certificates in this bug were issued for banking. Replacement of every certificate was individually agreed with a given bank.

In this particular case, the bank did not presume during its arrangements with us that it would have to postpone the service window due to its own unforeseen problems. It appears that the additional 7-14 day margin for the replacment date declaration will allow to avoid similar cases.

Of course, if the revocation was due to a security risk, KIR would revoke the certificate without asking the customer's permission, in accordance with the terms that the customer accepts and that are implied by the BR.

Comment 14

[Piotr Grabowski]

No update here . Are there any additional questions?

Comment 15

[Ryan Sleevi]

Comment #13 stated

For today we are waiting for cost estimation and timeschedule from our development department.

Comment #14 provides no update there, or if Q4 2021 is still "tentative".

The remainder of Comment #13 appears to be a commitment by KIR S.A. that it cannot, and will not, abide by the BRs, because:

KIR has no direct effect on the systems in which the certificates are installed

I appreciate that KIR S.A. made progress, but does not appear to demonstrate an understanding of the expectations of a default-trusted CA. If KIR S.A. had employees who regularly showed up for work 2 hours late, or when processing payments consistently under-charged by 10%, KIR S.A. would not say "Well, they showed up for work" or "Well, they processed 90% of the payment."

There are few things more troubling for a CA to say then:

Of course, if the revocation was due to a security risk, KIR would revoke the certificate without asking the customer's permission, in accordance with the terms that the customer accepts and that are implied by the BR.

Because that shows a CA that has not been aware of the discussions for the past several years on precisely this point, and why CAs are explicitly not permitted to make this arbitrary distinction.

Comment 16

[Piotr Grabowski]

(In reply to Ryan Sleevi from comment #15)

Comment #13 stated

For today we are waiting for cost estimation and timeschedule from our development department.

Comment #14 provides no update there, or if Q4 2021 is still "tentative".

No update in https://bugzilla.mozilla.org/show_bug.cgi?id=1709872#c14 means that currently we have no progress with this issue.
Q4 2021 is still "tentative".

The remainder of Comment #13 appears to be a commitment by KIR S.A. that it cannot, and will not, abide by the BRs, because:

KIR has no direct effect on the systems in which the certificates are installed

I appreciate that KIR S.A. made progress, but does not appear to demonstrate an understanding of the expectations of a default-trusted CA. If KIR S.A. had employees who regularly showed up for work 2 hours late, or when processing payments consistently under-charged by 10%, KIR S.A. would not say "Well, they showed up for work" or "Well, they processed 90% of the payment."

We appreciate the kind words. Our goal is not just to show progress, but to be an example for other CAs to follow.
Speaking of understanding of the expectations of a default-trusted CA we really do our best to be 100% BR compliant. We are fully aware that even a minimal deviation from the standard should not occur. We are convinced that the remediation steps taken (ACME, technical support, awareness actions, additional margin etc) will definitely shorten the time needed for certificate replacement. We sincerely hope, that in case of similar incident, if any, we will be able to replace them within BR timeframe.

There are few things more troubling for a CA to say then:

Of course, if the revocation was due to a security risk, KIR would revoke the certificate without asking the customer's permission, in accordance with the terms that the customer accepts and that are implied by the BR.

Because that shows a CA that has not been aware of the discussions for the past several years on precisely this point, and why CAs are explicitly not permitted to make this arbitrary distinction.

All of our countermeasures are primarily focused on ensuring that similar incidents don't happen, but when they do, that the time it takes to replace certificates, even when dealing with important infrastructure, is minimal.
We will certainly subject each of our plans in comparable case to o public consultation in order to choose the optimal solution.

Comment 17

[Piotr Grabowski]

no update here

Comment 18

[Piotr Grabowski]

no update here

Comment 19

[Piotr Grabowski]

no update here

Comment 20

[Ryan Sleevi]

Piotr: Can you provide a more substantive update than Comment #16?

That is, it's slightly concerning that there's another month not only with no progress, but no explanation as to why no progress. I understand something is slated for Q4 2021, but it's clear that KIR S.A. doesn't view that as a hard committment, nor are we given any explanation as to what's being done in the interim that would prevent a more substantive response sooner. That is, from these updates alone, it would appear that KIR S.A. is no step closer to ACME than a month ago; if that is the case, that suggests Q4 2021 is unlikely. If that's not the case, then providing more meaningful updates here is expected.

I realize this might seem overly burdensome, but the goal here is to understand the progress being made to comply with the minimum requirements. To date, we're not seeing much progress being made, and that's understandably concerning. While I'm encouraged to at least see an awareness as to the expectation of weekly updates, the hope here is that we get more meaningful, concrete updates - or more meaningful, concrete plans of deliverables, to appropriately set NextUpdate to measure and assess progress.

Comment 21

[Piotr Grabowski]

Ryan. As I mentioned in earlier comments, the ACME project was separated from the larger project and pursued separately because it was prioritised.However, the lack of weekly substantive updates does not mean that no work is ongoing on it. At the moment, architectural and integration agreements are still being made at the business analysis stage, which emerged during the initial cost analysis and which will have a major impact on the estimation of labour intensity and, consequently, on the solution delivery time. I am referring here to integration with the purchasing system, billing system, internal CRM, CA and automation project itself. I think that this stage of work will be completed in mid-September and then we will be able to give a closer date for the ACME deployment. At this point, I still belive the Q4 deadline is not at risk.

Comment 22

[Ryan Sleevi]

Piotr: The issue here, as captured in Comment #12 and Comment #14, is that it appears that ACME is the only quantifiable change that KIR S.A. is proposing to ensure it can meet the revocation time in the future. This is why progress is so critical here. It does not appear, from the information provided, that KIR S.A. has any other meaningful plan to comply with the Baseline Requirements or those of root programs, short of ACME.

If that's not the case, then it would be good to have a clear commitment that, in no circumstances, KIR S.A. will violate the BRs regarding revocation delays going forward. This would make it easier to understand that the ACME is a way of adding value/reducing friction, but not a substitute for compliance.

The goal here is not ACME in itself; the goal is making sure that KIR S.A. has internalized the need for compliance, and that it is designing its systems, contracts, policies, and training to make sure that everything it does will be done so in a compliant manner.

Updates such as Comment #21 are useful, as are clear roadmaps of dates we can measure progress by. Providing weekly updates similar to Comment #21 about the activities going on are useful to ensure that activity is actually happening, as well as providing a signal early if anything may be delayed. However, again, the goal is making sure KIR S.A. never delays revocation, and comments such as Comment #13 and Comment #16 seem to suggest KIR S.A. does not view it as a "hard" obligation, nor even something they can control, which is definitely not true.

Comment 23

[Piotr Grabowski]

Thank you Ryan for your comment. We fully understand the obligation to report progress and do our best to fulfil it.
The overall objective here is also clear for us - it is not the ACME in itself but increasing our CA resilience to avoid such delays in the future.
In this case, indeed ACME is the only measurable value that we can implement and study its progress of execution. I think we will have more concretes in mid-September.

Comment 24

[Piotr Grabowski]

We have completed the analysis and architecture of the ACME server. Implementation work has already begun. This is a completely in-house solution.

Brief timeline (2021):

1. 15/09 - 30/11 - ACME Server core component.
2. 25/10 - 30/11 - Integrations with related components and systems.
3. 01/12 - 20/12 - Testing
4. 27/27 - 31/12 - Production deployment

In item 2, based on the current progress of the "automation project", we are going to extract its functionality https://bugzilla.mozilla.org/show_bug.cgi?id=1705647#c25 to modify the current certificate ordering processes to get rid of the CSR separation from the order as discussed here https://bugzilla.mozilla.org/show_bug.cgi?id=1705647#c21.

Comment 25

[Piotr Grabowski]

We completed the first 2 timeline items from https://bugzilla.mozilla.org/show_bug.cgi?id=1709872#c24

1. 15/09/2021 - 30/11/2021 - ACME Server core component.
2. 25/10/2021 - 30/11/2021 - Integrations with related components and systems.

The testing phase from point 3:
3. 01/12/2021 - 20/12/2021 - Testing

detected several minor bugs that need to be corrected and regression tested again. As a result, we plan to finish the patch implementation and regression testing on 10/01/2022 and the production deployment will be rescheduled to 21/01/2022 to include the changes from https://bugzilla.mozilla.org/show_bug.cgi?id=1705647#c27 at the same time.

Comment 26

[Piotr Grabowski]

The changes have successfully deployed to the production.

Comment 27

[Piotr Grabowski]

Ben, would you mind to shedule this issue to be closed?

Comment 28

[Ben Wilson]

I'll close this on Wed. 2-2-22 unless anyone has objections.