Closed Bug 1707229 Opened 4 years ago Closed 3 years ago

SECOM: Delayed Revocation of non-technically constrained FUJIFILM Certificates

Categories

(CA Program :: CA Certificate Compliance, task)

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: h-kamo, Assigned: h-kamo)

Details

(Whiteboard: [ca-compliance] [leaf-revocation-delay])

Please let us provide our incident report as below.

  1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

In Bug 1695786, on April 16, 2021, Our CA became aware of that EE certificates issued prior to March 9, 2021 did not meet the requirements for the "Technically Constrained Sub-CA", so that recognized the necessity to revoke them. The revocation should be done within 5 days from April 16, 2021, but is getting delayed.
Applicable CA
https://crt.sh/?caid=80519

  1. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

March 1, 2021
Bug 1695786 (https://bugzilla.mozilla.org/show_bug.cgi?id=1695786) was introduced.

April 16, 2021
We recognized that the EE certificates of FUJIFILM CA issued by March 9, 2021 should be revoked. SECOM has begun coordinating the revocation plan with FUJIFILM.

  1. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

FUJIFILM CA has not stopped issuing certificates.

  1. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

The first certificate in question was issued on April 23, 2018 and has revoked.
The final certificate in question is the certificate issued on March 8, 2021.

  1. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

EE certificates issued from April 23, 2018 to March 8, 2021.
There are 127 certificates which are within the validity period.

We will write for section 6 and 7 later day.
Our current plan to revoke the applicable certificates will be completed on April 26, 2021.

Thank you for your consideration.

Best regards,
Hisashi Kamo

Assignee: bwilson → h-kamo
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true
Whiteboard: [ca-compliance] [delayed-revocation-leaf]

Ben-san,

Please let us tell you that all the relevant server certificates were revoked on April 26, 2021.

Here are our comments for 6 and 7 of the incident report.

  1. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

We have been adjusting the revocation plan since April 16th, and in the process that we are coordinating with FUJIFILM, they have informed us that the replacement of the server certificates will not be completed by April 21st.
If all the relevant certificates were revoked on April 21, we found that the system inside FUJIFILM would have a failure and the operation would be disrupted.
In addition, there was a system in which system maintenance work for replacing the server certificates could not be done except on holidays (April 24, Saturday, and April 25, Sunday,).

Since most of the server certificates in FUJIFILM were to be replaced, so we needed some time to figure out that the business impact and the background how so many server certificates had to be revoked.

  1. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

SECOM will instruct thoroughly that FUJIFILM and server certificate subscribers will understand the importance of BR-compliant revocation dates.
Currently, FUJIFILM's internal system uses a Public server certificates, but in the future, we will consider whether there is something that can be replaced by issuing a Private server certificates, so that if any similar incident occurs in the future, we will take effective measures to deal with it promptly

Thank you for your consideration.

Best regards,
Hisashi Kamo

(In reply to Hisashi Kamo from comment #1)

In addition, there was a system in which system maintenance work for replacing the server certificates could not be done except on holidays (April 24, Saturday, and April 25, Sunday,).

This policy makes it impossible to fulfill the revocation deadline in many cases. Why do you even continue to issue certificates to FUJIFILM if you know it will make you break the BR in case of revocations? This sounds like intentionally breaking the BR.

This incident report was also filled AFTER the deadline without any reason. As already stated in https://bugzilla.mozilla.org/show_bug.cgi?id=1695786#c18, SECOM's unresponsiveness in the original bug already accounted for more than 5 days.

Additionally, as already mentioned in https://bugzilla.mozilla.org/show_bug.cgi?id=1695786#c14, why do you think that you only have to replace certificates issued before March 9, 2021? And how did you check CAA records now? The applicable CP(S) does still NOT contain any exceptions for CAA checks (https://web.archive.org/web/20210428151634/http://repo1.secomtrust.net/sppca/xerox/fnetcas2/).

This whole "incident report" does not even come close to fulfill the expectation stated on https://wiki.mozilla.org/CA/Responding_To_An_Incident#Revocation and feels like a serious contempt for Mozilla's community. I think it shows that SECOM cannot be a trusted root CA anymore and it should be removed from Firefox.

Currently, FUJIFILM's internal system uses a Public server certificates, but in the future, we will consider whether there is something that can be replaced by issuing a Private server certificates, so that if any similar incident occurs in the future, we will take effective measures to deal with it promptly

Th9is is not really a step towards resolving the situation or ensuring it won't be repeated. This is a "Well, we'll look into things, and hope it doesn't happen again".

The fundamental issue here is that SECOM sold unauthorized, invalid, non-compliant certificates to FUJIFILM, and seemingly does not want to take responsibility for addressing the issue they created. The explanation provided here is no different than the explanations provided by past CAs, including SECOM, and that's a problem: SECOM has failed to take appropriate steps to learn from these past incidents and take steps to prevent this from happening in the future, which is how we got to where we are now.

The proportionate response to a revocation delay is dependent upon how well the CA, in this case, SECOM, demonstrates both that it has learned from past incidents and taken reasonable steps to prevent the same thing from happening for their own CA and customers, and that this situation is meaningfully new or different from these past situations. Yet that's clearly not the case here.

This puts root programs in an awkward place: SECOM has agreed to abide by the same rules all CAs are expected to, but is now declaring it won't, and for reasons that it reasonably could have known of and taken steps to prevent, but did not. This does not paint a picture of a CA actively prepared to deal with or is dealing with things.

This may seem a particularly harsh response for what, in effect, boils down to a 5 day delay, but the point is that SECOM can and should have been working to prevent any delays, so that it could meet the longstanding (nearly decade-long) requirements. On 2020-07-26, SECOM posted this comment, https://bugzilla.mozilla.org/show_bug.cgi?id=1649962#c10 , which acknowledged a lack of automation was a significant contributor to delays, which was further expanded upon on 2020-08-07 in https://bugzilla.mozilla.org/show_bug.cgi?id=1652610#c4 , and yet here we are, 9 months later, dealing with seemingly the same root issue.

SECOM needs to have a plan to address this, and also needs to have a plan for how it will remain accountable for the requirements it agreed to. For example, if SECOM hasn't prioritized automation since that discussion, that's entirely relevant and germane to this discussion. If SECOM supports automation, but FUJIFILM declined to adopt it, then it seems clear that any outage is, fundamentally, caused by FUJIFILM, and revocation would be entirely consistent with the Subscriber Agreement.

I realize all the certificates have now been revoked, as of 2020-04-26, and while it's good that there was only a 5 day delay, it's quite bad, especially given the context, that there was any delay at all. Unless and until SECOM has the tools in place to ensure it complies with industry requirements, it seems like this should be a top-level priority of SECOM to the exclusion of all other development. If that's not the case, SECOM should be candid and transparent about what's "more important" than this compliance (e.g. perhaps there are other areas of non-compliance being addressed first), and be clear about it's timeline to getting a point where, from both a technical implementation and a business policy, they're able to comply with the BRs and Root Program Requirements.

Flags: needinfo?(h-kamo)

Ryan-san, Paul-san,

Thank you for the comments.

We will take it very seriously and make our best effort to show improvements.
Regarding the delay in incident reports, we will definitely come up with the improvement plan.
In addition, we will persuade FUJIFILM, for us to take the initiative and make the effective plan to introduce automation.
Please let us report the details separately later.

Best regards,
Hisashi Kamo

Flags: needinfo?(h-kamo)

Re-setting NI since Comment #4 indicates there will be a follow-up soon.

Flags: needinfo?(h-kamo)

Ryan-san, Paul-san,

Please let us comment as below.

Comment2

This incident report was also filled AFTER the deadline without any reason. As already stated in https://bugzilla.mozilla.org/show_bug.cgi?id=1695786#c18, SECOM's unresponsiveness in the original bug already accounted for more than 5 days.

We started the revocation process on April 16, as the server-auth certificates to be revoked were clarified at https://bugzilla.mozilla.org/show_bug.cgi?id=1695786#c13.

Additionally, as already mentioned in https://bugzilla.mozilla.org/show_bug.cgi?id=1695786#c14, why do you think that you only have to replace certificates issued before March 9, 2021? And how did you check CAA records now? The applicable CP(S) does still NOT contain any exceptions for CAA checks (https://web.archive.org/web/20210428151634/http://repo1.secomtrust.net/sppca/xerox/fnetcas2/).

Because the server-auth certificates to be revoked were clarified at https://bugzilla.mozilla.org/show_bug.cgi?id=1695786#c13, we figured replacing the certificates would solve the problem. We referred to the following quote.

At the time this certificate was issued, NO CAA exception applied, because of the unrevoked, unconstrained intermediate that SECOM had failed to revoke prevent this from being a "Technically Constrained Sub-CA", and the BRs require CAA checking either during the issuance of a Pre-certificate or the Certificate (i.e. it MUST be checked)

It was not possible to issue server-auth certificates from domains other than the domain whose name was constrained on March 6. For that reason, there was no risk of issuing to a non-existent DNS. Therefore, we haven't revised the CP, because there was no problem ever happened with the CAA record check.

Comment3

In addition, we will persuade FUJIFILM, for us to take the initiative and make the effective plan to introduce automation.

We have begun coordinating with FUJIFILM on the implementation of automation. We will report the detailed plan later.

Thank you for your consideration.

Best regards,
Hisashi Kamo

Flags: needinfo?(h-kamo)

Ryan-san, Paul-san,

We are continuously talking with Fujifilm, and had a discussion regarding BR section 4.9.1 of revocation and necessity of prompt action for automation today.
Please let us report detailed plan later.

Thank you for your consideration.

Best regards,
Hisashi Kamo

Ryan-san, Paul-san,

In regards to the contents reported at the previous comment7, we had a discussion with the management of FUJIFILM this week.

We are continuously talking with Fujifilm, and had a discussion regarding BR section 4.9.1 of revocation and necessity of prompt action for automation today.

We are continuing discussions with FUJIFILM on prompt revocation in the event of a problem and effort to prevent the problem.
Please let us report detailed plan later.

Thank you for your consideration.

Best regards,
Hisashi Kamo

It's been a month since Comment #4, and we still don't have a concrete deadline for when we can expect more actionable results here.

To be clear: What we're looking for is a clear commitment from SECOM as to when it will have an update for us. For example, if it expects deliberations with Fujifilm to take another month, then that should be clearly stated. What we're looking for is a concrete commitment that we can measure progress by, while Comment #6 , Comment #7, and Comment #8 don't really give us insight into what's happening, and whether it's being met with the requisite priority. In particular, I want to avoid a situation where, say, we're 13 weeks from now, and still in the midst of discussions about steps that could be taken, and then take another 13 weeks to actually implement.

Flags: needinfo?(h-kamo)

Ryan-san,

Thank you for your comment.
Please let us comment as below.

Regarding with the automation plan, we are continuing discussions with Fujifilm, and will update it on July 15.

With including the management of Fujifilm (Board members), we shared the awareness that a large number of revocations could occur in a short period of time.
Then we are considering how to implement such numbers of revocations efficiently and effectively, including the means.
By deciding the grouping and priority of where to start with the implementation in Fujifilm, We will proceed our mission.

Thank you for your consideration.

Best regards,
Hisashi Kamo

Flags: needinfo?(h-kamo)

I will set a next update on this bug for Friday, 25-June-2021. I would like to know as soon as possible the result of the discussions and what the next steps are with Fujifilm.

Whiteboard: [ca-compliance] [delayed-revocation-leaf] → [ca-compliance] [delayed-revocation-leaf] Next update 2021-06-25

Ben-san,

Please let us comment as below.

As for the automation plan, SECOM will coordinate to carry out the test with Fujifilm by the end of September.
At the same time, we will figure out deployment plan for the production by the same deadline too.

Thank you for your consideration.

Best regards,
Hisashi Kamo

Kamo-san,
Please be prepared to provide us with an update on your automation progress on or about 15-July-2021.
Thanks,
Ben

Whiteboard: [ca-compliance] [delayed-revocation-leaf] Next update 2021-06-25 → [ca-compliance] [delayed-revocation-leaf] Next update 2021-07-15

Ben-san,

As a countermeasure and to improve cryptographic agility, we are proceeding with the following:

  1. Implementation of automation for issuance and management of certificates
  2. Migrating to issuance of non-Public TLS server certificates by non-Public CAs

1. Implementation of automation
As for the automation plan, SECOM will coordinate to carry out the test with Fujifilm by the end of September. At the same time, we will figure out the deployment plan for the production release by the same schedule, too.
After the development of automation for Public TLS server certificate has completed at SECOM, it is planned to be available to all CAs, not just FUJIFILM.

2. Issuing TLS server certificates from a non-Public CA
FUJIFILM is selecting the certificates that do not have to be Public from among the certificates issued by the CA.
Once the selection is completed, they will proceed with the transition to a private CA.
At this point, multiple CAs are already operating as non-Public CAs, so they are thinking of increasing such ones.

Thank you for your consideration.

Best regards,
Hisashi Kamo

Whiteboard: [ca-compliance] [delayed-revocation-leaf] Next update 2021-07-15 → [ca-compliance] [delayed-revocation-leaf] Next update 2021-08-01

Ben-san,

At the moment, we haven't made a noticeable progress on the following two points, but as we commented last time, we are working in cooperation with Fujifilm.

We will continue to respond based on the intention of BR.

  1. Implementation of automation for issuance and management of certificates
  2. Migrating to issuance of non-Public TLS server certificates by non-Public CAs

Thank you for your consideration.

Best regards,
Hisashi Kamo

Flags: needinfo?(bwilson)

Ben-san,

The noticeable progress has not been made, but we are continuously working hard on this issue.

1.Implementation of automation for issuance and management of certificates
2.Migrating to issuance of non-Public TLS server certificates by non-Public CAs

Thank you for your consideration.

Best regards,
Hisashi Kamo

Whiteboard: [ca-compliance] [delayed-revocation-leaf] Next update 2021-08-01 → [ca-compliance] [delayed-revocation-leaf] Next update 2021-08-16

Ben-san,

We are still in ongoing cooperation with Fujifilm for the implementation of automation and migrating to issuance by non-Public CAs.
Thank you for your consideration.

Best regards,
Hisashi Kamo

Flags: needinfo?(bwilson)
Whiteboard: [ca-compliance] [delayed-revocation-leaf] Next update 2021-08-16 → [ca-compliance] [delayed-revocation-leaf]

Kamo-san,
I've reset the next update to be blank - meaning that we will expect another status update next week. Please provide any timeframes if you have them.
Thanks,
Ben

Ben-san,

Please let us update as below.

As for the automation plan, we have constructed a CA(private environment) to carry out the test.
We are still working toward the test at the end of September, and the next update will be targeted on September 17.

Thank you for your consideration.

Best regards,
Hisashi Kamo

Whiteboard: [ca-compliance] [delayed-revocation-leaf] → [ca-compliance] [delayed-revocation-leaf] Next update 2021-10-01

Ben-san,

We are in the final discussion stage with FUJIFILM about which plan to go either (1) automation plan or (2) private certificate plan.
The prompt revocation plan in case of being compromised will be also provided shortly by October 1.

Thank you for your consideration.

Best regards,
Hisashi Kamo

Ben-san,

In consultation with FUJIFILM, SECOM has reached the following agreement with them regarding the future revocation plan.

  • Revocation of EE certificate in case of being compromised: within 24 hours
  • Revocation of EE certificate (BR violation) except for being compromised: within 5 days

By preparing the application data in advance for reissuance in case of emergency revocation, we expect that the TLS server certificate issuance application can be carried out in about one day in the future.
Previously, it took several days for the research work to identify the web server using the certificate and prepare the application data.
With this measure though, we will be able to reduce the number of days for revocation significantly.
(At the time of the previous revocation, 127 certificates were revoked in 10 days from April 16th to April 26th, 2021.)

SECOM also discussed with FUJIFILM about the CA transitions in the future.
We discussed whether to choose between automation plan and private TLS server certificates plan in the future.

As a result of such discussions, most of the certificates issued by "FUJIFILM Fnet CA - S2" are used in business systems, so that we are considering to utilize them as private TLS server certificates.

Thank you for your consideration.

Best regards,
Hisashi Kamo

Ben-san,

There is nothing new to report this week.
We will be continuing to watch this Bugzilla.

Thank you for your consideration.

Best regards,
Hisashi Kamo

Ben-san,

There is nothing new to report this week.
We will be continuing to watch this Bugzilla.

Thank you for your consideration.

Best regards,
Hisashi Kamo

Ben-san,

There is nothing new to report this week.
We will be continuing to watch this Bugzilla.

Thank you for your consideration.

Best regards,
Hisashi Kamo

Ben-san,

We had a meeting with FUJIFILM to discuss the specifications of non-Public CA on November 10th.
We will have an another meeting with FUJIFILM to discuss the specific implementation schedule of non-Public CA next week, and will be able to tell you the schedule by November 19th.

Thank you for your consideration.

Best regards,
Hisashi Kamo

Ben-san,

Please let us provide you our plan as below.

Non-Public CA Construction Plan

Date - YYYY/MM/DD Action
2021/11/30 Confirmed an Elements Definition
2021/12/20 Signed an agreement
2022/01/20 Constructed Non-Public CA
2022/01/21 Evaluation starts
2022/02/24 Operation (issuance) starts

Thank you for your consideration.

Best regards,
Hisashi Kamo

Hi Hisashi,

Can you please clarify the plans for FUJIFILM Fnet CA - S2 once the non-Public CA is online and issuing?

For example, are you intending to allow all currently valid end entity certificates issued by FUJIFILM Fnet CA - S2 to naturally expire - and at that point SECOM will be revoking the CA certificate?

Thank you,
Ryan

Ryan-san,

Thank you for your comment.

Please let us reply as below.

These end entity certificates are name-constrained by this CA certificate and comply with Baseline Requirements.
Although, we had delayed revocation of EE cert in the past, as we answered in comment # 21, it is possible to be revoked without delay now.

So we consider that we allow to wait until the end entity certificates naturally expire, as Ryan-san exemplifies in comment # 27.

Thank you again for your consideration.

Best regards,
Hisashi Kamo

Ben-san,

Please let us comment about our current status.
We have confirmed an Elements Definition on November 30th as scheduled.

Thank you for your consideration.

Best regards,
Hisashi Kamo

Whiteboard: [ca-compliance] [delayed-revocation-leaf] Next update 2021-10-01 → [ca-compliance] [delayed-revocation-leaf] Next update 2022-01-05

Ben-san,

There is nothing new to report this week.
We will be continuing to watch this Bugzilla.

Thank you for your consideration.

Best regards,
Hisashi Kamo

Ben-san,

There is nothing new to report this week.
We will be continuing to watch this Bugzilla.

Thank you for your consideration.

Best regards,
Hisashi Kamo

Ben-san,

According to our Non-Public CA Construction Plan, "Signed an agreement" was scheduled to be on December 20, 2021, but it takes days to complete the work for making the agreement, that means the agreement is taking longer than initially planned.
However, we'd like "Constructed Non-Public CA" to be completed on schedule, and we are in the process of working with our technical members on a detailed work plan.

Thank you for your consideration.

Best regards,
Hisashi Kamo

Ben-san,

we are still working for the arrangement of the agreement.
However, we would like "Constructed Non-Public CA" to be completed on schedule.

Thank you for your consideration.

Best regards,
Hisashi Kamo

Ben-san,

Due to the New Year's holidays in Japan, there is nothing new to report this week.
We will be continuing to watch this Bugzilla.

Thank you for your consideration.

Best regards,
Hisashi Kamo

Whiteboard: [ca-compliance] [delayed-revocation-leaf] Next update 2022-01-05 → [ca-compliance] [delayed-revocation-leaf] Next update 2022-01-17

Ben-san,

The "Signed an agreement" is delayed about for a month due to some clerical issue.
Although, we are now targeting "Operation (issuance) starts" as an originally planned schedule despite of delay of "Constructed Non-Public CA" and "Evaluation starts".

Thank you for your consideration.

Best regards,
Hisashi Kamo

Ben-san,

There is nothing new to report this week.
We will be continuing to watch this Bugzilla.

Thank you for your consideration.

Best regards,
Hisashi Kamo

Ben-san,

Please let us update as follows.

The "Signed an agreement" was delayed due to administrative issues, but was completed on January 31, 2022.
The "Constructed Non-Public CA" and "Evaluation starts" are now one month behind schedule, and the "Operation (issuance) starts" is now two weeks behind schedule.

We will revise the schedule and work on accordingly.

Date - YYYY/MM/DD Action
2022/01/31 Signed an agreement
2022/02/21 Constructed Non-Public CA
2022/02/24 Evaluation starts
2022/03/07 Operation (issuance) starts

Thank you for your consideration.

Best regards,
Hisashi Kamo

Ben-san,

We are working on the Constructed Non-Public CA for February 21.
The preparation work is going well.
Thank you for your consideration.

Best regards,
Hisashi Kamo

Whiteboard: [ca-compliance] [delayed-revocation-leaf] Next update 2022-01-17 → [ca-compliance] [delayed-revocation-leaf] Next update 2022-02-21

Ben-san,

Please let us update.

We constructed Non-Public CA as our revised schedule on comment 37.
2022/02/21 Constructed Non-Public CA

Thank you for your consideration.

Best regards,
Hisashi Kamo

Ben-san,

Please let us update.

The evaluation was started as our revised schedule on comment 37.
2022/02/24 Evaluation starts

Thank you for your consideration.

Best regards,
Hisashi Kamo

Ben-san,

Please let us update.

The operation was started as our revised schedule on comment 37.
2022/03/07 Operation (issuance) starts

Thank you for your consideration.

Best regards,
Hisashi Kamo

Hello Kamo-san,
Do you have a new set of timeline items and tasks so that we can monitor your progress?
Thank you,
Ben

Flags: needinfo?(h-kamo)
Whiteboard: [ca-compliance] [delayed-revocation-leaf] Next update 2022-02-21 → [ca-compliance] [delayed-revocation-leaf] Next update 2022-03-15

I'll take a look at this bug a little closer and decide what the next step should be.

Flags: needinfo?(h-kamo) → needinfo?(bwilson)
Whiteboard: [ca-compliance] [delayed-revocation-leaf] Next update 2022-03-15 → [ca-compliance] [delayed-revocation-leaf] Next update 2022-03-25

Just to reconfirm - per Comment #39, SECOM has created a non-public CA for Fujifilm, correct?
That CA is not chained to a root CA that is trusted in the root store, correct?
That CA has been tested and is now in operation?
If there are no other remediation steps related to this bug that still need to be addressed, then I believe this incident can be closed.

Flags: needinfo?(bwilson)
Whiteboard: [ca-compliance] [delayed-revocation-leaf] Next update 2022-03-25 → [ca-compliance] [delayed-revocation-leaf] Next update 2022-04-29
Flags: needinfo?(h-kamo)

Hello Ben-san,

Thank you for your comment.
Please let us answer for your question on Comment #44.

We have created a non-public CA for Fujifilm.
That CA is not chained to a root CA that is trusted in the root store.
That CA has been tested and is now in operation.

We believe there are no other remediation steps related to this bug.
Thank you for your consideration.

Best regards,
Hisashi Kamo

Flags: needinfo?(h-kamo)

I will close this bug on Wed. 27-April-2022, unless there are additional questions or issues to discuss.

Flags: needinfo?(bwilson)
Status: ASSIGNED → RESOLVED
Closed: 3 years ago
Flags: needinfo?(bwilson)
Resolution: --- → FIXED
Product: NSS → CA Program
Whiteboard: [ca-compliance] [delayed-revocation-leaf] Next update 2022-04-29 → [ca-compliance] [leaf-revocation-delay]
You need to log in before you can comment on or make changes to this bug.