1695938 - SECOM: FUJIFILM intermediate not listed in audit statement

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Andrew Ayer]

SECOM has disclosed the following intermediate certificate as having the same audit as parent:

https://crt.sh/?sha256=EDD4FBA40CD3ABAE14F175BBDF6706A6B728007A6B46D866CA07B76743AFF42C

However, the audit statement https://www.cpacanada.ca/generichandlers/CPACHandler.ashx?attachmentid=241990 does not list this intermediate as in scope.

Worryingly, the same public key appears in the following technically-constrained intermediate certificate:

https://crt.sh/?sha256=F0AFAE187046A431EDDCEFF19CE5AE306D52338F88A11EC11B95B81D521895A4

Note that SECOM previously issued an improperly-constrained intermediate certificate to Fuji Xerox (Bug 1563574).

I would ask SECOM to disclose as soon as possible who controls the private key for this intermediate.

Comment 1

[Hisashi Kamo]

Hello Andrew-san,

Let us answer for your question about the control of the private key.
The control of the private key is in Secom.

Best regards,
Hisashi Kamo

Comment 2

[Hisashi Kamo]

Andrew-san,

The FUJIFILM intermediate CA certificate was revoked on March 9th.
https://crt.sh/?id=405618312

Due to the impact of the major corporate split(technical and legal influence) of the subject company, we may need some time to scrutinize the information.

Thank you for your consideration.

Best regards,
Hisashi Kamo

Comment 3

[Ben Wilson]

We will be looking forward to an Incident Report on this matter. Thanks.

Comment 4

[Mathew Hodson]

I think the Assignee and QA Contact fields are reversed on this bug.

Comment 5

[Ryan Sleevi]

Doh! Thanks Matthew - the curse of editing bugs from mobile.

Comment 6

[Andrew Ayer]

It is very concerning that two weeks have passed and no incident report has been provided. Per https://wiki.mozilla.org/CA/Responding_To_An_Incident:

We expect to see incident reports as soon as possible, and certainly within two weeks of the initial issue report.

Even if the private key of this intermediate is controlled by SECOM, Bug 1695786 suggests that FUJIFILM had significant control over the operation of this intermediate, including the ability to bypass domain validation. That makes this a serious incident which severely undermines trust in SECOM.

It's unclear why "major corporate split(technical and legal influence) of the subject company" is delaying an incident report, as this was fundamentally a failure within SECOM. Even if full remediation takes longer than two weeks, Mozilla still expects to see an incident report. It is absolutely vital with an egregious failure like this that the community can understand what SECOM is doing to correct it. Absent an incident report, we must assume that SECOM is not taking this incident seriously, and Mozilla should begin taking steps to protect its users from future failures by SECOM.

Comment 7

[Hisashi Kamo]

Andrew-san,

Please let us provide you our incident report.

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

We became aware of the problem via this Bugzilla bug.

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

Issuance status of intermediate CA certificate (CN = FUJIFILM Fnet CA -S2, O = FUJIFILM, C = JP)
Apr 18 2018 GMT Intermediate CA certificate issued (without Name Constraints)
Dec 15 2020 GMT Intermediate CA certificate issued (with Name Constraints)

3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

The intermediate CA certificate without Name Constraints issued on Apr 18, 2018 has been revoked on Mar 9, 2021.

4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

Apr 18 2018 GMT Intermediate CA certificate issued (without Name Constraints)
https://crt.sh/?id=405618312
Dec 15 2020 GMT Intermediate CA certificate issued (with Name Constraints)
https://crt.sh/?id=3819070478

5. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

Apr 18 2018 GMT Intermediate CA certificate issued (without Name Constraints)
https://crt.sh/?id=405618312
Dec 15 2020 GMT Intermediate CA certificate issued (with Name Constraints)
https://crt.sh/?id=3819070478

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

The purpose of this CA is to issue certificates for limited use in Fujifilm's corporate domain.
We had the plan to migrate to a CA with name constraints, which was in progress.
The CA migration was initially scheduled for completion on March 31, 2021.
Fujifilm's system restrictions and operational rules, that are to issue certificates for the corporate domain, had been followed.

The migration to CAs with name constraints had been under consideration since July 2017, but due to lack of awareness of how to set name constraints (Bug 1563574), customer impact of CA migration (service outages and migration costs), customer-side organizational changes (corporate split), and SECOM personnel changes, it took some time, so that the completion scheduled date will be March 31, 2021.

Due to the part of the staff took an insufficient approach to promote the CA transition plan, and also insufficient handoff the work because of the retirement of the person in charge, we were unable to take effective measures for the CA certificate revocation plan. We will strengthen our system to make improvements.
Along with the change of plan, the intermediate CA certificate without name constraints was revoked on March 9, 2021.

7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

After re-reading "Cradle-to-Grave Contiguous Audits #153" (*1) thoroughly, we noticed that there was a violation against this purpose. As a reason for that, we also noticed the fact that we had not properly complied with "Audits are required even if no longer issuing #139" (*2).
Revocation of CAs that do not issue certificates or that are unnecessary, will be implemented sequentially after preparation.
CAs that can issue server certificates will undergo annual external audits or implement measures to include name restrictions in intermediate CA certificates.

We will strengthen our system, and improve the CA migration plan, so that we can share more useful and effective information with browser vendors.

(*1)
https://github.com/mozilla/pkipolicy/issues/153
(*2)
https://github.com/mozilla/pkipolicy/issues/139

It's unclear why "major corporate split(technical and legal influence) of the subject company" is delaying an incident report, as this was fundamentally a failure within SECOM.

There seemed to be a lack of explanation about the company split, so please allow us to explain for that in general.
Most of the information in the company split is basically a process protected by NDA, and the information about the time series in it should be also sensitive information.

In addition to that, for example, you can easily imagine a case where the corporate network is also split and newly constructed due to the corporate split, and the DNSname is changed or newly created accordingly, but when, how, and what range of people in charge are informed, all depends on a very strong NDA.
Even after the company split is over, there are still legal difficulties in disclosing information other than public information.

We also would like to report after understanding the structure of the company in as much detail as possible, but with the company split, it is technically difficult to organize information while grasping the split of the network, of course, which takes some time for that reason.

Corporate integration is also difficult, but we believe that there are unique difficulties due to deadlines and NDAs in corporate splits.
We consider the browser venders are fully aware of the challenge that accompanies the company split, but we would like to share as much information as you need.

Thank you for your consideration.

Best regards,
Hisashi Kamo

Comment 8

[Hisashi Kamo]

Sorry for miss-editing
Correct last sentence was following.

We consider the browser venders are fully aware of the challenge that
accompanies the company split, but we would like to share as much information as you need.

We would try to say “We believe the browser venders are aware of the challenge that accompanies the company split, but we would like to share as we can (if they need any).”

Best regards,
Hisashi Kamo

Comment 9

[Ryan Sleevi]

My understanding of "the split" was that it was discussing the "Subject company" (aka FUJIFILM), based on Comment #2. So I'm not sure how/why that delayed a response, since that seems mostly unrelated to the incident report in Comment #7. Perhaps you could explain more what part of Comment #7 took additional time?

I think the answer to Question 7 in Comment #7 misses a lot of root cause analysis. Question 6 at least acknowledges serious operational failures at SECOM ultimately contributed to this, namely:

Due to the part of the staff took an insufficient approach to promote the CA transition plan, and also insufficient handoff the work because of the retirement of the person in charge, we were unable to take effective measures for the CA certificate revocation plan. We will strengthen our system to make improvements.

I would have expected the response to Question 7 to more deeply dive into what's being done to ensure a sufficient approach, sufficient handoff, and robust systems that can handle retirement. Talking specifically about the concrete steps to "strengthen our system" is what we'd expect in Question 7, but I'm not really seeing that.

Comment 10

[Hisashi Kamo]

Ryan-san,

Thank you for your comment.

Perhaps you could explain more what part of Comment #7 took additional time?

The corporate split is relevant to 6 of comment 7, but as you commented, it was not compelling enough to be the single reason for the delay in incident reporting.

For this time, we had to sort out which information cannot be disclosed.
Some information can directly show or indirectly infers some sort of sensitive information regarding with NDA on company split (various timelines related with split, etc.), and SECOM needed to classify data..
As a result, it took some time to make a report.
In addition, due to the resignation of the person in charge, it took further time to confirm the facts and sort out the information.

I would have expected the response to Question 7 to more deeply dive into what's being done to ensure a sufficient approach, sufficient handoff, and robust systems that can handle retirement. Talking specifically about the concrete steps to "strengthen our system" is what we'd expect in Question 7, but I'm not really seeing that.

Along with following change on procedures, SECOM will review and change the approval flow and review flow to ensure that the issuance process is carried out correctly with several person.

As investigating this incident, we believe that a major cause of this incidents was that the shortage of time on the work of the person in charge.
The overview of circumstances that caused the shortage of time is as follows:

In corporate split, there is often a short period of time between the decision of split, and actual split.
It would be shorter (and can be even shorter than it was planned) period of time between the notification to the operator and actual split.

After operator was informed of the split, he needs to prepare, for instances, which can have very different from the one he had prepared.
We believe that such circumstances were a big factor in this process.

We believe that this incident would not have occurred unless SECOM received a possibly time-limited contract and tried to process with usual (non-time limited) procedures.
We will review the working hours and contract items at the time of contract more carefully. This is intended to reject contracts that are likely to be overly time-constrained.

In addition, if it is determined that the work time required for mutual check is insufficient, we believe we shall stop work approval and discuss the direction. We believe that there were not enough points to make such a judgment, so will improve it.

Along with review and ensure execution of procedures, the following changes on procedures are made.

Do not allow the postponement of name constraint (or configuring CA for enterprise) handling, even if the customer is unable to finalize the domain for its own reasons, and do it with right procedures.
Implement name constraints (or configure CA for enterprise) as soon as possible in the current domain (or architecture) even if a merger or split of the company is expected in the future.
After the merger or split, implement the name constraint (or configure CA for enterprise) again with the domain after the change.

Thank you for your consideration.

Best regards,
Hisashi Kamo

Comment 11

[Ryan Sleevi]

Thanks for further clarifications.

Going back to the original issue at hand now, with these added details, I'm trying to understand how the omission from the audit happened, and why it was not noticed. I understand from Comment #7 that there was a plan to migrate to name constraints, and from Comment #10, the implementation of those name constraints was rushed due to the corporate secrecy requirements. Yet it's not immediately clear to me why the nameConstraints were seen as time sensitive as having to be rushed (given the existence of the existing cert), or how the underlying audit issue happened.

The audit report mentioned in Comment #0 was dated 2020-09-04, for an audit period of 2019-06-07 to 2020-06-06. The (unconstrained) intermediate was issued 2018-04-18.

1. Who controlled the key material for the period from 2018-04-18 to 2020-06-06?
2. Who controlled the key material for the period from 2020-06-06 to 2020-09-04?
3. Who controlled the key material for the period from 2020-09-04 to 2020-12-15?
4. Who controlled the key material for the period from 2020-12-15 to 2021-03-09?

If I understand correctly, the answer to all of these is "FUJIFILM" (in some form), and not SECOM. And, if I'm understanding Comment #7 correctly, SECOM was unaware that they were in material non-compliance with Mozilla's requirements, and, for that matter, the Baseline Requirements. This was not detected until SECOM reread "Cradle-to-Grave contiguous audits" and realized the FUJIFILM intermediate needed to be audited.

The timeline is unclear when this realization was made, or when SECOM reportedly self-discovered this. It's also unclear to me how to interpret "so that the completion scheduled date will be March 31, 2021." (also from Comment #7), given that the intermediate was issued 2020-12-15, and bears the same subject and SPKI, so migration is effectively already done.

What this appears is that SECOM issued, and improperly disclosed, an unconstrained subordinate CA to a third party. Neither they, nor their auditor, discovered this fact, until this issue was created, which speaks very negatively for both SECOM (responsible for disclosing) and KPMG (responsible for auditing against the BRs, which includes the supervision of such sub-CAs). If this is not an accurate picture, I'm hoping you can clarify further.

Comment 12

[Hisashi Kamo]

Ryan-san,

Thank you for your comment.

1.Who controlled the key material for the period from 2018-04-18 to 2020-06-06?
2.Who controlled the key material for the period from 2020-06-06 to 2020-09-04?
3.Who controlled the key material for the period from 2020-09-04 to 2020-12-15?
4.Who controlled the key material for the period from 2020-12-15 to 2021-03-09?

We are afraid that the previous discussion had been based on miss-communication, so can we try to discuss again after confirming the following?

The key material for FUJIFILM's intermediate CA certificate was generated at SECOM’s facility and SECOM’s device. That facility is used for WebTrust, and that device is same standard device as for WebTrust certified CA. In addition, that non-exportable key is stored in that device with that facility since then.
Therefore, those key are managed by SECOM from April 18, 2018 until March 9, 2021.
We are afraid that there was some misunderstanding about situation, so please confirm that for further comment.

Thank you for your consideration.

Best regards,
Hisashi Kamo

Comment 13

[Ryan Sleevi]

So that answers one of the questions, but we still lack clarity over how, for three years, SECOM failed to have this CA audited, or even notice that it was not listed in scope. This is especially troubling, given the repeated m.d.s.p. discussions and because of Audit Letter Validation. The discussion about corporate secrecy makes even less sense as being relevant, because this was plain and simple "A CA that SECOM controlled which was not audited", and which we've also seen was not following the BRs.

Comment 14

[Hisashi Kamo]

Ryan-san,

Thank you for your comment.

The first intermediate CA certificate was issued on April 18, 2018, and the migration to a CA with name constraints was considered starting in July 2019, but there was a lack of awareness of how to set name constraints (bug 1563574), the impact of the CA migration on customers, and continuation of discussions for company split and other things of similar nature. As a result, the CA, which was responsible for issuing for FUJIFILM's own group based on name constraints and not for external customers, had not been audited for three years.
An information access control with some duties for a specific person in charge, had been put in place in discussions for the acquisition or absorption merger of a company.
There had been fallen behind a lot in communication from that company to the person. We presume that was the influence of continuing discussions of company merger, split and related lawsuits.
We believe that these were the reasons why the person in charge could not make the decisions that he should have taken earlier.
The problem was that the special information access control made it difficult for the person in charge to consult with other colleagues.
As a result, the CA became in an incomplete state of not being able to conduct audits or name constraints.
The person in charge needed to implement CA audits, or name constraints(or revoke Intermediate) promptly, but he kept holding action continuously.
He was the target person to an information access control, which may disturb the adequate escalation to the right direction.
Because of the above mentioned reasons, the company as a whole was not aware of the serious issues and the multi-person approval process was not functioning properly.
In the future, we will make certain that all CAs will be checked by multiple people on a regular basis, and intermediate CA certificates will be revoked or closed if the necessary procedures for BR have not been taken.
We will also inform the person in charge that he shall follow the principle of giving priority to BR over NDAs and contracts in special situations.
We will make sure that each person in charge is aware that the company will protect the individual in the above situations, excessive duty are not imposed on individuals, and make it possible to share information as long as it is necessary for works.

Thank you for your consideration.

Best regards,
Hisashi Kamo

Comment 15

[Ryan Sleevi]

Sending this to Ben.

I worry that Comment #14 gets to a bit of the "blameful post-mortem" that we'd like to avoid, by laying a lot of the responsibility on the individual, although towards the very end, it begins to recognize that the systems and design of SECOM played a part in that.

It seems clear that compliance needs to be integrated at the core, and must permeate every aspect of the business. This "could" have been addressed by ensuring multiple people (e.g. all of those responsible for compliance) were part of the group read in via the information access control, so that they could review. It also seems that procedures are necessary to ensure that management is careful when entering into such relationships that may need information access control, to make sure beforehand that everything is and will-be on the up-and-up.

Separately, there's a clear failing of audit controls here, if any of them rely on a single person. It seems carefully inventorying every key in SECOM's HSMs, and tying them to the issued certificates, could, at any point, have caught this. So there's a question when it comes to both the audit scope (prior to the auditor engagement) and as part of the controls provided to the auditor, to making sure that every key on every HSM can be tied to a purpose and that purpose is clearly in-scope or out -of-scope.

In https://bugzilla.mozilla.org/show_bug.cgi?id=1705480#c5 , SECOM reports that they're looking to establish a dedicated compliance team, which also seems like it can get to the heart of some of these systemic issues, if done properly.

While this is by no means a great report, and a particularly troubling incident to keep in mind with SECOM, I'm sending to Ben if he has further questions.

Comment 16

[Ben Wilson]

I will hold this open and consider this issue in conjunction with the broader ALV clean-up effort and Bug #1717044.

Comment 17

[Hisashi Kamo]

Ben-san,

There is nothing new to report this week.
We will be continuing to watch this Bugzilla.

Thank you for your consideration.

Best regards,
Hisashi Kamo

Comment 18

[Ben Wilson]

I will close this bug on or about Friday, 29-Oct-2021.