Closed Bug 1652610 Opened 4 years ago Closed 3 years ago

SECOM: Delayed Revocation of CA Certificate with OCSP EKU Issue

Categories

(CA Program :: CA Certificate Compliance, task)

Product:
CA Program
Component:
CA Certificate Compliance
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: bwilson, Assigned: h-kamo)

Details

(Whiteboard: [ca-compliance] [ca-revocation-delay])

Attachments

(1 file)

20210318_KeyDestruction_JPRS.pdf
53.86 KB, application/pdf
Details

Opening this bug to track incident report and discussion related to failure to revoke ICA within 7 days for OCSP EKU issue raised in bug 1649962.

Ben-san,

Thank you for the notice.

Please let us discuss this matter on the bug #1649962.
We will describe here the contents that converged as a result of the discussion on #1649962.

Best regards,
Hisashi Kamo

Note, as per https://wiki.mozilla.org/CA/Responding_To_An_Incident#Revocation , we want to use this bug to track the delays in revocation and what's being done to resolve it.

The following update was shared in Bug 1649962:

2020-07/29：will stop issuing from non-compliant CAs and get ready to issue from new compliant CAs.
2020-08-31：5% of the certificates will be renewed or reissued.
2020-09-30：25％ of the certificates will be renewed or reissued.
2020-10-31：30％ of the certificates will be renewed or reissued.
2020-11-30：35％ of the certificates will be renewed or reissued.
2020-12-31：45％ of the certificates will be renewed or reissued.
2021-01-31：70％ of the certificates will be renewed or reissued.
2021-02-28：85％ of the certificates will be renewed or reissued.
2021-03-31：100％ of the certificates will be renewed or reissued.
2021-04-30：Key destruction will be completed.

This is substantially longer than almost every other CA affected. Further, the CAs affected are seemingly primarily TLS issuing CAs (specifically https://crt.sh/?id=1671982906 and https://crt.sh/?id=1671982905 ), which are unquestionably expected to be able to be rotated quickly. Among the reasons given are:

we recognize that there is a hosting company who do not have enough automation to set up a certificate and will set up certificates of 10,000 or more, or a customer who manually setting the 5,000 certificates on a server. Our plan also includes prospects for the migration of these customers.

This issue is where I think it's quite important to understand those timings, how they were selected, and what's being done going forward. The idea that 15% of the TLS certificates issued by SECOM may take 7 months to revoke, rather than the 5 days required by the Baseline Requirements, is concerning, and I think we want to better understand the steps SECOM is taking to address that. In 2020, we shouldn't see CAs taking that long, especially given past incidents and the opportunity to learn from them.

Flags: needinfo?(h-kamo)

Dear Kamo-san,
As I explained in the other bug # 1649962, I'd like to use this bug going forward (after you have answered Ryan's question there). Particularly as Ryan implies in Comment #2 here above, we need you to move up your schedule to complete by the end of February, if possible. If not, for instance if the hosting company you are working with does not have automation, or has other barriers to rolling over to a new intermediate CA, then I need you to explain the problems in greater detail here.
Thanks,
Ben

Dear Ryan-san, Ben-san,

Our company is making adjustments by notifying the certificate recipient that we will be migrating CA in a prompt manner.
We would like to make a prompt changeover by fully considering the comments of Ryan, Ben and the contents of related Bugzilla and m.d.p.s, with taking care of the business impact on the certificate user.
At this point, however, we have difficulty to guarantee the completement by the end of February, so let us explain the details of the migration steps and issues as below:

The migration plan is generally performed in the following four steps:

  1. CA will explain the end user on this matter and confirm their intention to reissue the certificates in question.
  2. The end user shows their intension to CA to reissue.
  3. CA reissues the TLS certificate that are subjected.
  4. End user sets up TLS certificate on required device.
    4-1. Hosting/Cloud provider sets up the TLS certificates for the customer.
    4-2. The end user individually sets up the TLS certificates.

First of all, we are aiming to ensure the availability of end users (including important infrastructures by government agency or local public organizations), replace certificates as soon as possible, and destroy the keys by reissuing for this time.

This time, what we assume the time consuming factor especially is 4-1.
As mentioned earlier, we know that there are multiple hosting/cloud providers, each of them uses our TLS certificate that need to set up more than 10,000 certificates.

As a result of the dialogue with the business provider, the business provider that takes the longest time has the system currently held, that is, based on manual operation, 4-1, as the expected number of correspondence when setting up the TLS certificate, we expect 100 certificates per day, 2,000 per month, and a little more than 6 months until the end of support. In addition to this, it is also necessary to perform other related work such as preparation of the private key and CSR, so that we plan for a total of 8 months.

As a CA, we are aware of the need for measures to agile progress corporating with business providers. This incident give us more incentive for agility. Regarding the support for receiving large numbers of applications, increasing the number of employees and automation and etc, it goes without saying that we will continue to engage in dialogue, cooperation, and practice, and strive to shorten the plan.

In addition, we anticipate responding to such a situation, and are promoting efforts such as automation and encourage our client providers to do so, but it is in the middle of the road, and as a CA, TLS certificate settings for these providers have not fulfilled the direct support yet. We are fully aware of the deadlines suggested by everyone (Ryan, Ben), of course, and we will continue to support and dialogue with the providers to complete the plan earlier somehow.

Thank you for your consideration.

Best regards,
Hisashi Kamo

Flags: needinfo?(h-kamo)

I'm not sure I fully understand why 4-1 is the challenge, and hoping you can expand. In general, 4-1 seems to be the easiest way of scaling, as the cloud provider can automate things. I'm not sure I understand why 4-1 would require a lot of correspondence?

Ultimately, the CA is in the place of handling revocation. Previously, SECOM had committed that it would abide by the BRs and revoke according to the timelines specified. I understand that there are challenges, but I'm concerned that Comment #4 suggests that this problem will persist and isn't really being systemically addressed.

I understand there's a challenge here. If we ask you to abide by the BRs, it will be spun as "Mozilla and/or Google demands revocation", even if it's just asking to hold to the existing requirements. If you tell your customers you'll abide by the BRs, it'll be seen as "SECOM is breaking sites". Yet, if SECOM continues with the plan specified in https://bugzilla.mozilla.org/show_bug.cgi?id=1649962 , that's virtually indistinguishable from "SECOM is ignoring the Baseline Requirements", and that has lasting impact to the entire CA ecosystem. Similarly, if SECOM doesn't have systems in place to handle prompt revocation, then situations like this, or Heartbleed, or Shellshock, or the like, can all have real harm. One of the goals of having this as a requirement in the BRs is to make sure we're moving to more robust systems, and that's why it's important to be making meaningful strides here.

That's the aspirational part. But I think we need to understand more concrete committments from SECOM about the steps its taking to ensure that, going forward, there won't be delays like this. Even if SECOM decides to move to February, instead of April/May, that's still a significantly long time. As we've seen from situations like serial entropy, this wasn't unforeseen, and so it sounds like there are a number of systemic improvements that need to be made, and prioritized, ASAP. I'm hoping you can share more concrete plans and timelines?

Flags: needinfo?(h-kamo)

Dear Ryan-san,

We’d like to explain our current situation as follows:

  • How to improve the situation from now on
    In order to move to more robust systems to meet the revocation requirements of BR, we understand how important that, we as the CA and the hosting providers should promptly
    replace the certificates, and we consider that mission as urgent.
    As the improvement measure for speeding up the response to the mission, we are conducting a verification experiment on the issue/reissue of certificates and the automatic
    reconfiguration and revocation of certificates in cooperation with each hosting providers and CA.
    By August 31, we will review the plan and update the information in the Bugzilla.

  • Current issues
    Regarding the Step 4-1, we reconfirmed with the hosting providers, but it is necessary to set everything manually at the moment, so that we still need to have the time period
    as the plan presented previously.
    In order to shorten the time necessary for related activities, we are actively engaged in encouraging large hosting providers to develop automation tools to set server
    certificates on to the servers they are managing.
    And in the dialogue with the hosting providers, we figured out that the scale of the revocations which could occur at one time and the response period were beyond what
    they initially assumed.
    In that regard, we are talking to the hosting providers to update their awareness of the potential threats.

Thank you for your consideration.

Best regards,
Hisashi Kamo

Flags: needinfo?(h-kamo)

Dear Ryan-san,

Please let us update as below.

How to improve the situation from now on

We’d like to inform you of the updated status regarding the above.
Each of the hosting providers and CA are in cooperation with each other, and are planning to deal with the mechanism to automate the issue/reissue of the certificates, automatic resetting and revocation of the certificates.
At this point, we are in the stage to analyze the options and issues of the implementation method in the preliminary verification, and proceed to the concrete implementation phase.
The detailed plan will be as follows:
Finish designing and implementation at around by the end of September to December, 2020
Build a production environment at around from January to March, 2021
The automatic mechanism will be deployed sequentially from around April, 2021

We are proceeding the above automatic mechanism at the same time with replacing the certificates by manually setting.

Hope you understand our effort to complete the mission.

Best regards,
Hisashi Kamo

Ben: I'm curious your take. If I'm understanding Comment #6 right, it looks like the original timetable hasn't changed much.

I realize Comment #7 lays out some steps being taken to, eventually (circa 2021-04) get to a place where we've expected CAs to be since roughly 2015. But this still stands out as a remarkable outlier, and suggests that there may be multiple systemic issues at play here. Without wanting to berate SECOM here, I have to acknowledge that the scale of the challenges here makes me deeply suspicious about other possible areas of systems design and preparation where the stated objectives of the BRs and the Mozilla Root Program won't or can't be met, as expected, and just that we've gotten lucky so far.

For SECOM in particular, it seems like an important step here could be the adoption of a WebTrust Detailed Controls Report, to help us understanding exactly how the systems are designed, and better understanding how the current objectives are met, particularly in terms of preparation for future events (e.g. disasters, revocation, etc).

For CAs in general, I think it raises a question of needing to better understand what steps CAs do take to ensure they meet their BR obligations. It's stated in policy, for sure, but discussing what plans they have in place, particularly different scenarios, seems relevant.

However, I'm concerned about every example where a CA places the Root Store in the uncomfortable position of having to enforce compliance to the stated requirements, and presents that compliance enforcement as if it is the Root Store being unreasonable.

Flags: needinfo?(bwilson)

Dear Ryan-san,

Please let us update as below.

We’d like to inform you the issue of intermediate CA certificate revocation and the progress of the certificate replacement.
Please refer the list below showing the progress rate of the server certificate replacement (as of September 30, 2020):

  Year & month                        Progress rate
  Planned value  Actual value  Difference
  -----------------------------------------------------------------
  August 2020     5.0%            11.0%        6.0pt 
  September     25.0%            36.1%        11.1pt 
  October          30.0%   
  November      35.0%   
  December      45.0%   
  January 2021  70.0%   
  February        85.0%   
  March          100.0%

The next update will be in early November.

Thank you for your consideration.

Best regards,
Hisashi Kamo

Ben: Do you have anything to add (re Comment #8)?

As I mentioned, I don't really think the timeline is reasonable or sustainable, and I can't help but feel Comment #7 is too little, too late. I recognize SECOM is one of the older CAs in the Mozilla Root Store, but I can't help but feel this would a completely unacceptable response in the event of a DigiNotar or Heartbleed like scenario. If it is unacceptable in those cases, it should be unacceptable now, because all we're saying is when those events happen, we'll spend time haggling over whether or not they're truly serious (... as browsers had to do with DigiNotar, and with Heartbleed, and countless other CA incidents), and users are exposed to risk.

Minimally, it would seem like if the timetable is accepted, any future delays in revocation by SECOM should result in distrust, because it would establish things had not improved. However, I'd like to avoid that as much as possible, if only because that means more work for browsers to keep users safe, when this is clearly something the CA controls.

Dear Ryan-san, Ben-san,

Please let us tell you regarding the Comment #10 on the closed bug # 1649962.

The destruction of the key pair at one of the following CAs is planned to be done on 2020-10-31.

SECOM Passport for Member PUB CodeSigning CA G1
https://crt.sh/?id=2517735005

The revocation and destruction of the key pair at the CA above have been done with witness by auditor KPMG on 2020-10-30.

Thank you for your consideration.

Best regards,
Hisashi Kamo

Dear Ryan-san, Ben-san,

Please let us update as below.

Year & month Progress rate
Planned value Actual value Difference

August 2020 5.0% 11.0% 6.0pt
September 25.0% 36.1% 11.1pt
October 30.0% 49.4% 19.4pt
November 35.0%
December 45.0%
January 2021 70.0%
February 85.0%
March 100.0%

Thank you for your consideration.

Best regards,
Hisashi Kamo

Dear Ryan-san, Ben-san,

Please let us update as below.

We would like to inform you the issue of intermediate CA certificate revocation and the progress of the certificate replacement.

Please refer the list below showing the progress rate of the server certificate replacement (as of November 30, 2020):

                         Progress rate

Year & month Planned value Actual value Difference

August 2020 5.0% 11.0% 6.0pt
September 25.0% 36.1% 11.1pt
October 30.0% 49.4% 19.4pt
November 35.0% 58.4% 23.4pt
December 45.0%
January 2021 70.0%
February 85.0%
March 100.0%

Thank you for your consideration.

Best regards,
Hisashi Kamo

Dear Ryan-san, Ben-san,

A happy new year.
Please let us update as below.

We would like to inform you the issue of intermediate CA certificate revocation and the progress of the certificate replacement.

Please refer the list below showing the progress rate of the server certificate replacement (as of December 31, 2020):

                           Progress rate

Year & month Planned value Actual value Difference

August 2020 5.0% 11.0% 6.0pt
September 25.0% 36.1% 11.1pt
October 30.0% 49.4% 19.4pt
November 35.0% 58.4% 23.4pt
December 45.0% 76.4% 31.4pt
January 2021 70.0%
February 85.0%
March 100.0%

Thank you for your consideration.

Best regards,
Hisashi Kamo

Dear Ryan-san, Ben-san,

Please let us update as below.

We would like to inform you the issue of intermediate CA certificate revocation and the progress of the certificate replacement.

Please refer the list below showing the progress rate of the server certificate replacement (as of January 31, 2021):
Progress rate
Year & month Planned value / Actual value / Difference

August 2020 5.0% 11.0% 6.0pt
September 25.0% 36.1% 11.1pt
October 30.0% 49.4% 19.4pt
November 35.0% 58.4% 23.4pt
December 45.0% 76.4% 31.4pt
January 2021 70.0% 91.8% 21.8pt
February 85.0%
March 100.0%

Thank you for your consideration.

Best regards,
Hisashi Kamo

Dear Ryan-san, Ben-san,

Please let us update as below.

We would like to inform you the issue of intermediate CA certificate revocation and the progress of the certificate replacement.

Please refer the list below showing the progress rate of the server certificate replacement (as of February 28, 2021):

                          Progress rate

Year & month Planned value / Actual value / Difference

August 2020 5.0% 11.0% 6.0pt
September 25.0% 36.1% 11.1pt
October 30.0% 49.4% 19.4pt
November 35.0% 58.4% 23.4pt
December 45.0% 76.4% 31.4pt
January 2021 70.0% 91.8% 21.8pt
February 85.0% 93.7% 8.7pt
March 100.0%

Thank you for your consideration.

Best regards,
Hisashi Kamo

Thank you for the update. Regarding Comment #7:

Finish designing and implementation at around by the end of September to December, 2020
Build a production environment at around from January to March, 2021
The automatic mechanism will be deployed sequentially from around April, 2021

Have you made progress here?

Flags: needinfo?(bwilson)

Dear Ryan-san,

Have you made progress here?

We have completed the implementation of automatic mechanism based on ACME as scheduled.
In the event that a future revocation case occurs, we will take advantage of this for the prompt action.

Best regards,
Hisashi Kamo

Dear Ryan-san, Ben-san,

The destruction of the key pair for the old intermediate CAs have been completed, with witness by auditor KPMG on March 18, 2021.
The following intermediate CA certificates issued, which were based on the destructed keys, have been all revoked.

JPRS Organization Validation Authority - G3
1. https://crt.sh/?id=1671982905
2. https://crt.sh/?id=3067358278
3. https://crt.sh/?id=3068756691

JPRS Domain Validation Authority - G3
4. https://crt.sh/?id=1671982906
5. https://crt.sh/?id=3067358277
6. https://crt.sh/?id=3068756692

Thank you for your consideration.

Best regards,
Hisashi Kamo

Flags: needinfo?(bwilson)

Dear Kamo-san,
Are there other tasks that need to be completed? Do you have the letter from KPMG regarding key destruction?
Thanks,
Ben

Flags: needinfo?(h-kamo)

Dear Ben-san,

Thank you for your comment.

The witness with our auditor KPMG will be shared as soon as it is ready.
It will be completed the all tasks.

Best regards,
Hisashi Kamo

Flags: needinfo?(h-kamo)

Dear Ben-san,

Here is the witness with our auditor KPMG.
The minutes of key destruction authorized by 2 members of KPMG is attached.

Thank you for your consideration.

Best regards,
Hisashi Kamo

Status: ASSIGNED → RESOLVED
Closed: 3 years ago
Flags: needinfo?(bwilson)
Resolution: --- → FIXED
Product: NSS → CA Program
Summary: SECOM: Delayed Revocation of CA Certificate with OCSP EKU Issue → SECOM: Delayed Revocation of CA Certificate with OCSP EKU Issue
Whiteboard: [ca-compliance] [delayed-revocation-ca] → [ca-compliance] [ca-revocation-delay]
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: