Bugzilla

Comment 3

•

3 years ago

(In reply to Hisashi Kamo from comment #2)

3/6 Fixed the setting to limit the FQDN that can be set in SAN.

I don't think this adequately provides the necessary information. Question 6 is where you might provide more detail, but even there, it's lacking.

FUJIFILM, which is in charge of issuing the certificate, was examining the permitted domain when issuing, but there happened to be an omission in the check at the time of examination.

It's deeply concerning that you had an improperly-disclosed externally operated CA (this is being tracked as Bug 1695938 ). I realize the CA is revoked now, but relevant to this incident, it suggests that FUJIFILM may have improperly validated any/all of its certificates, and the lack of (historic) audit records seem to suggest that, even with the existence of the newly created technically constrained sub-CA, it may be more appropriate to replace all of those certificates.

Although the automatic control function has been introduced, it did not operate normally due to an incorrect setting, so that the wrong certificate was issued.

This doesn't provide sufficient detail. Please understand, your goal should be to help us understand precisely how FUJIFILM's systems were operated, such that a reasonable practicioner in this field could understand. I have no idea what you mean by "automatic control function", both in general, but also how it's specifically implemented. A good incident report provides sufficient detail that an independent implementation of "the thing" (e.g. the automatic control validation) could be implemented by another CA. The reason this is the target level of detail is because it helps all CAs examine their systems to see if they've taken a similar design and suffer similar issues - but the only way to achieve that is by providing sufficient detail.

Except of this mistake, we already confirmed that the target certification authority has not issued any such certificates other than using the specified domain.

Please be more precise in exactly what you've done here to confirm and how you did it. This has been a frequent cause of CA issues through improper examination, and a full and detailed analysis of the steps performed are what reassure the community.

List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

We set to the correct value for the setting of the automatic control function, so that the issuance other than using the specified domain should be prevented for sure.

Please be aware this incident report fails to meet the minimum standard required. The goal of these incident reports is for the CA to systemically examine things, not merely "fix the immediate issue". For example, this incident report fails to understand how and why the "automatic control function" wasn't enabled, how it was not detected that it was not enabled, how it was not detected by SECOM that a certificate was misissued, or any of the other steps that could have prevented this.

Please understand, CAs have been distrusted over incidents like this, because this is a serious and egregious failure, and completely shakes the confidence in the CA. These incident reports are the opportunity for the CA to take a detailed examination about everything that went wrong, as well as what could have prevented this, and come up with a meaningful strategy to address that, and publicly and transparently help the community understand that they recognize the seriousness. Here's an example of an improved incident report from a CA that used to struggle, to give an idea, while the Incident Reporting page provides other good examples.

I realize there are a number of challenges here, but given that we rely on CAs to be candid, transparent, and above all, beyond reproach, I'd like to encourage you to provide a much more detailed report that looks into the systemic issues.

Flags: needinfo?(h-kamo)

Assignee

Comment 4

•

3 years ago

Ryan-san,

Please let us comment as below.

I don't think this adequately provides the necessary information. Question 6 is where you might provide more detail, but even there, it's lacking
It's deeply concerning that you had an improperly-disclosed externally operated CA (this is being tracked as Bug 1695938 ). I realize the CA is revoked now, but relevant to this incident, it suggests that FUJIFILM may have improperly validated any/all of its certificates, and the lack of (historic) audit records seem to suggest that, even with the existence of the newly created technically constrained sub-CA, it may be more appropriate to replace all of those certificates.

By as of March 12, 2021, we have extracted the issued EE certificate data (raw data) from the database of the target CA and have done inspection on it.
In addition to that, we confirmed the SAN request value from the application data of the certificate issuance request from the RA database, and performed multiple confirmation from two aspects. As a result, we confirmed that all the SAN values of the EE certificate are included in the domain name owned by Fujifilm, which has been examined and permitted in advance. The domain of the certificate should be restricted by the technically constrained sub-CA.

This doesn't provide sufficient detail. Please understand, your goal should be to help us understand precisely how FUJIFILM's systems were operated, such that a reasonable practicioner in this field could understand. I have no idea what you mean by "automatic control function", both in general, but also how it's specifically implemented. A good incident report provides sufficient detail that an independent implementation of "the thing" (e.g. the automatic control validation) could be implemented by another CA. The reason this is the target level of detail is because it helps all CAs examine their systems to see if they've taken a similar design and suffer similar issues - but the only way to achieve that is by providing sufficient detail.

The SAN value at the time of certificate issuance request was compared with the predetermined dNSName value that should not be set, and an error check function was implemented and set. In addition to this, for the CA, we implemented a check function that allows input by comparing it with the value of dNSName that can be set, but due to a setting error, the setting that allows input was not enabled.
Based on this incident, we inspected all the certificates issued in the past and confirmed that the certificates other than the permitted dNSName were not issued.
Although not a direct cause, we were using the CSR SAN for certificate issuance.
The basic idea is to use only the key information for CSR information, and SAN will be referred and entered from separated allowed SAN list for that entity, then impose system restrictions to that list also.

Please be more precise in exactly what you've done here to confirm and how you did it. This has been a frequent cause of CA issues through improper examination, and a full and detailed analysis of the steps performed are what reassure the community.
7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

We have inspected all EE certificates issued by the target CA as of March 12, 2021, and have confirmed that all SAN values in the EE certificate are included in the domain name owned by Fujifilm, which has been pre-examined and allowed.
We also extracted the issued EE certificate data (raw data) from the CA database and confirmed it. In addition, the SAN request value was confirmed from the application data of the certificate issuance request of the RA database, and multiple confirmations were made from two aspects.

Please be aware this incident report fails to meet the minimum standard required. The goal of these incident reports is for the CA to systemically examine things, not merely "fix the immediate issue". For example, this incident report fails to understand how and why the "automatic control function" wasn't enabled, how it was not detected that it was not enabled, how it was not detected by SECOM that a certificate was misissued, or any of the other steps that could have prevented this.

For domain control in the name constraint of the certificate issuing system provided by SECOM, there are multiple examination systems and input check systems, and we selected one of them depending on the time when the CA was built and the method of Enrollment of each service. We have done the expansion for functions of these multiple examination systems and input check systems. Due to the existence of multiple system implementations in the settings for the CA, there happened to have a flaw in the procedure of the person in charge at that time. We assume that one of the causes is that the management of multiple system implementations has become complicated.

how it was not detected by SECOM that a certificate was misissued

With the current system configuration, we were able to notice it in a quarterly self audit.
However, we couldn’t notice the certificate in question this time, because it had not yet entered the quarterly check period.

any of the other steps that could have prevented this.

Measures to prevent recurrence:
In the future, we will template or standardize the basic items for system restrictions that should be implemented for all CAs, and improve operations so that there will be no flaw happening.
We will implement two major system enforcements:
The first one is to add system restrictions when issuing certificates.
Although not a direct cause, we were using the CSR SAN for certificate issuance.
The basic idea is to use only the key information for CSR information, and SAN will be referred and entered from separated allowed SAN list for that entity, then impose system restrictions to that list also.
The second one is to centrally manage and systematize the check logic of subsystems.
This practice is not one-shot practice, but will be continuous attempts of more centralization.
We had a background that the addition of check logic accompanying the BR revision is scattered and implemented in a wide variety of subsystems, which makes complication.
To be able to manage without depending on human work, we will centrally manage the check logic of various subsystems and systematize them, so that setting values can be automatically generated.
We will improve to maintain quality while responding to the speed of rule changes.

As mentioned above, we will make efforts to implement improvements continuously to reduce and integrate the system and streamline operations.

Please understand, CAs have been distrusted over incidents like this, because this is a serious and egregious failure, and completely shakes the confidence in the CA. These incident reports are the opportunity for the CA to take a detailed examination about everything that went wrong, as well as what could have prevented this, and come up with a meaningful strategy to address that, and publicly and transparently help the community understand that they recognize the seriousness. Here's an example of an improved incident report from a CA that used to struggle, to give an idea, while the Incident Reporting page provides other good examples.I realize there are a number of challenges here, but given that we rely on CAs to be candid, transparent, and above all, beyond reproach, I'd like to encourage you to provide a much more detailed report that looks into the systemic issues.

Sorry for the lack of information in the previous incident report.
With referring to the good examples that you show us, we will try to make a better report in the future.

Thank you for your consideration.

Best regards,
Hisashi Kamo

Flags: needinfo?(h-kamo)

Comment 5

•

3 years ago

(In reply to Hisashi Kamo from comment #2)

How did you do CAA checks here? The root zone is signed using DNSSEC and there is no sgnwffw001., so there should have been a hard failure.

Assignee

Comment 6

•

3 years ago

Paul-san,

Thank you for your comment.

We’d like to answer your comment.
Our CAA check function for that enterprise CA works that
If relevant CAA set is empty or there exists non-empty relevant CAA set and it contains one (or more) CAA record(s) which allow(s) our CA to issue, then a certificate can be issued only for that domain name..
In the other word, issuance will be restricted only if there exists (relevant RRset, and) non-empty relevant CAA set and it do not allow our CA.
Therefore, even though the CAA check was done for the domain (sgnwffw001.) in this incident, the wrong issuance was not prevented because there was no record corresponding to the Relevant CAA Set.
We figure the problem with this incident is that there was a setting flaw for checking whether the submitted SAN has an appropriate DNS Name or not.
Regarding to the details of the cause and countermeasures for the problem, please refer as described previously in Comment 4.

Thank you for your consideration.

Best regards,
Hisashi Kamo

Comment 7

•

3 years ago

The BRs don't permit that, however.

CAs are permitted to treat a record lookup failure as permission to issue if:

the failure is outside the CA’s infrastructure; and

the lookup has been retried at least once; and

the domain’s zone does not have a DNSSEC validation chain to the ICANN root

Paul's point is highlighting that you didn't encounter an empty RRSet, you encountered a failure to lookup. You can't treat the failure to lookup as an empty RRSet when the domain's zone has a valid DNSSEC validation chain (which it would, saying the domain doesn't exist)

Flags: needinfo?(h-kamo)

Comment 8

•

3 years ago

Also, can you be more precise about this:

Due to the existence of multiple system implementations in the settings for the CA, there happened to have a flaw in the procedure of the person in charge at that time.

Am I understanding this correctly as "When the CA was created, the necessary pre-issuance lints/automated domain checks were not enabled. We believe this was not enabled because of an oversight when configuring CAs. This oversight happened because our system, and the checks we have, is complex, making it prone to human error. To reduce the risk of human error, we're working on centrally managing all of the necessary checks, to reduce the complexity and variance that can cause human errors."

If so, what's the timeline on that?

Assignee

Comment 9

•

3 years ago

Ryan-san , Paul-san,

Thank you for your comment.

domain's zone has a valid DNSSEC validation chain

We confirmed that it was indeed not met.
We also checked other certificates for that point.
As described in comment 2, we have confirmed that there are no other similar errors in the issued certificates. In addition, we are going to implement following countermeasures.

Prohibit TLDs on SAN. It will be done by CA subsystem (April 2021).
Implement CAA records lookup failure handling function into main system of CA (July 2021).

----------for Comment 8------
Many of the processes are mechanized rather than manual.
However, in this case, in addition to the interpretation in Ryan’s comment 8, there was an omission with time shortage of the work on the person in charge
(described as https://bugzilla.mozilla.org/show_bug.cgi?id=1695938)
In addition to changes on procedures described in above bug, following are timeline of countermeasures.
3) We implemented a restriction to issue certificates for domains other than those allowed (March 6, 2021).
4) In the process of issuing an intermediate CA certificate with name restrictions, we add an item to mutual check of domain names to be set, and confirmation of that.
This will prevent omission of the setting of the permitted domain name. (April 2021)
5) We are currently estimating implementation time and impact on performance of linting tools implementation into main CA system..

Thank you for your consideration.

Best regards,
Hisashi Kamo

Flags: needinfo?(h-kamo)

Comment 10

•

3 years ago

Implement CAA records lookup failure handling function into main system of CA (July 2021).

From your description of the issue, it seems like SECOM is actively in non-compliance with the BRs, and will not be compliant until July. Why has SECOM not halted all issuance until it can ensure the CAA checks meet the BR requirements?

Flags: needinfo?(h-kamo)

Comment 11

•

3 years ago

(In reply to Hisashi Kamo from comment #9)

domain's zone has a valid DNSSEC validation chain

We confirmed that it was indeed not met.
We also checked other certificates for that point.

How did you check this for existing certificates? Did you record DNS responses at the time of issuance and re-checked the recorded responses now or did you re-request CAA records now? In case of the latter, all of your existing certificates are misissued and have to be revoked within 5 days starting (at the latest) at 2021-04-07T11:45:00Z. Also, this (non-compliant CAA checking) is a separate issue. Where is the incident report for this issue?

Assignee

Comment 12

•

3 years ago

Ryan-san , Paul-san,

Thank you for your comment.
Please let us comment as below.

Up until now, the functionality of our CAA check system has been worked to determine whether or not to issue for Existing DNS.
If the DNS does not exist, an issuance error will occur before doing our CAA check, and the measures to detect such a certificate request. That function was implemented on March 6.

Let us clarify our understanding of BR compliance of CAA checking.

Certificate Transparency pre-certificates for all server-auth certificates for this CA have been created and have been logged in at more than 3 public trusted logs, including google log server.

This CA is currently technically constrained, in accordance with Baseline Requirements section 7.1.5, and name-constraint.

We believe CAA checking is optional as described in BR section 3.2.2.8 for above situation, and we do our automated CAA checking as the additional measure, but not as the permission of issuance.
.
Although our additional measure along with some other functions are working as the CAA check function, , we consider that some improvements are necessary for our CAA check function.
We shall plan to improve our CAA check function so that we can respond to future configuration changes effectively, and we are going to fix this point as described in the previous comment.

That was the intension of the first half of comment 9.

Thank you for your consideration.

Best regards,
Hisashi Kamo

Flags: needinfo?(h-kamo)

Comment 13

•

3 years ago

Let us clarify our understanding of BR compliance of CAA checking.

Certificate Transparency pre-certificates for all server-auth certificates for this CA have been created and have been logged in at more than 3 public trusted logs, including google log server.

This CA is currently technically constrained, in accordance with Baseline Requirements section 7.1.5, and name-constraint.

We believe CAA checking is optional as described in BR section 3.2.2.8 for above situation, and we do our automated CAA checking as the additional measure, but not as the permission of issuance.

This is a materially incorrect understanding of the Baseline Requirements, and seems to reflect a critical security failure that does suggest Comment #11 is accurate, and SECOM needs to immediately stop issuance, revoke all existing certificates, fix its CAA implementation, and not issue any new certificates until this is correct. Yes, this means no new certificates, period, and no replacement certificates, period.

Why?

Because, as described, SECOM has missed a critical requirement here. While I hope this is just a communication issue in the description, if it is, in fact, as SECOM has stated in Comment #12, I do not see another option that is consistent with the Baseline Requirements and the expectation of Root Stores that CAs adhere to the Baseline Requirements.

Here's the full clause from the Baseline Requirements, with emphasis added to highlight what SECOM is missing:

CAA checking is optional for certificates for which a Certificate Transparency pre‐certificate was created and logged in at least two public logs, and for which CAA was checked.

SECOM's second assertion is that CAA checking is not necessary for this subordinate, but I think this response in Comment #12 needs to take a full picture of the significant non-compliance revealed here.

Date and Time	Description	Reference(s)
2012-07-01 00:00:00	The Baseline Requirements are adopted, which include a sunset on Internal Server Name	BRs 1.0
2015-11-01 00:00:00	The Baseline Requirements forbid the issuance of certificates to an Internal name	BRs 1.0
2017-03-08 00:00:00	The CA/Browser Forum adopts Ballot 187, making CAA checking mandatory on 2017-09-08.	CA/B Forum Ballot 187
2017-08-11 00:00:00	The Baseline Requirements Ballot 204 is effective, forbidding the delegation of DNS validation to entities other than the operating CA.	CA/B Forum Ballot 204
2017-09-08 00:00:00	CAA checking is mandatory.	CA/B Forum Ballot 187
2018-04-18 06:44:14	SECOM issues the FUJIFILM CA, operated by SECOM	https://crt.sh/?id=405618312 Bug 1695938
2020-12-15 08:25:17	SECOM issues a revised version of this intermediate, still operated by SECOM, and which is a Technically Constrained Sub-CA. At this point, we are given to understand that SECOM has made CAA checking optional and disabled domain validation for this sub-CA, which is operated by SECOM.	https://crt.sh/?id=3819070478 https://bugzilla.mozilla.org/show_bug.cgi?id=1695938#c1 Comment #4
2021-02-22 00:38:11	SECOM issues an invalid certificate, containing an Internal Server Name, which it has not validated the CAA record for.	https://crt.sh/?q=1B0141338CA75750DF7A369674B6F55970A1496799FAD2DE78E8D4772283D779
2021-03-02 ??:??:??	SECOM is made aware that they have issued an invalid certificate, after failing to detect it themselves	Comment #0 Comment #2
2021-03-06 ??:??:??	SECOM updates their system to fail to issue certificates if the domain name does not exist. Previously, if the domain did not exist, it would succeed both CAA validation. For the FUJIFILM intermediate, SECOM was not validating DNS at all.	Comment #4 Comment #12
2021-03-09 03:07:57	SECOM revokes the non-technically constrained FUJIFILM CA.	https://crt.sh/?id=405618312 Bug 1695938

I put this timeline, in much greater detail than strictly necessary, to highlight a few things:

At the time this certificate was issued, NO CAA exception applied, because of the unrevoked, unconstrained intermediate that SECOM had failed to revoke prevent this from being a "Technically Constrained Sub-CA", and the BRs require CAA checking either during the issuance of a Pre-certificate or the Certificate (i.e. it MUST be checked)
SECOM's own admission is that its CAA implementation fails to correctly implement CAA, materially misreading the existing requirements about CAA record sets, even if it was doing CAA checking.
Internal Server Names had been forbidden for nearly 9 years at the time the incident occurred, and despite numerous m.d.s.p. incidents, CA communications, and community discussions, SECOM still had failed to implement basic safety checks to ensure the domain name actually existed
At all times, the failure was the direct responsibility of SECOM, on SECOM operated infrastructure, and which is otherwise shared with SECOMs existing (non-constrained) CAs

At this point, I think we have to conclude that SECOM does not know how to validate domain names, has not correctly validated domain names, and all certificates it has issued are misissued. This is not the first time SECOM has failed to correctly validate domain names (e.g. Bug 1524452 ), nor failed to implement necessary controls required by the BRs on the timeline specified (e.g. Bug 1524816 ). It's clear SECOM is not in a position to effectively and promptly replace certificates, as shown by Bug 1652610, and so raises the question if SECOM is even capable of restoring trust in their capability to operate a CA.

I'm aware Comment #12 is an attempt to provide assurance, but it yet again demonstrates SECOM's inability to adhere to the BRs, and I'm concerned about the technical acumen and ability to correctly understand the compliance obligations. This situation should not have been possible, especially given SECOM's longstanding participation in the CA/Browser Forum in which these issues, their motivations, and the context was fully available to them, both to ask questions and to propose clarifications.

I'm not confident, given the set of explanations we've seen on this and related bugs, that it's possible for SECOM to rise to the level of technical detail, as other CAs have, to provide a level of understanding and assurance about how their CA systems are designed and operated that would somehow show this is some misunderstanding. SECOM's own statements to date reveal critical misunderstandings, and so even the descriptions provided by SECOM will naturally and reasonably raise suspicion and doubt that SECOM has correctly understood the requirements. That said, if there is some critical detail missing, or some greater explanation of the validation flow that would demonstrate compliance with the BRs, I'd be curious, but it's very clear that at least at the time this certificate was issued, SECOM was wholly non-compliant, and all domain validations up through 2021-03-06, at the minimum, were non-compliant with the BRs.

Flags: needinfo?(h-kamo)

Comment 14

•

3 years ago

(In reply to Hisashi Kamo from comment #12)

We believe CAA checking is optional as described in BR section 3.2.2.8 for above situation, and we do our automated CAA checking as the additional measure, but not as the permission of issuance.

Everything else aside (and I think that trust in your statements has been completely undermined at this point and will be very hard to regain), BR 3.2.2.8 allows to specify and rely upon an exception in your CP/CPS (which would not apply in this case anyway as the sub-CA was not technically constrained). It does not provide an exception in itself. You fail to demonstrate that you have done so and your lack of correct interpretation of the BR is shocking, especially given the Microsoft incident around CAA checking which you are obligated to follow closely by the Mozilla Root Store Policy, at least via discussion on https://groups.google.com/g/mozilla.dev.security.policy/c/0TfT9BziQVc/m/gV3dqYWSAAAJ.

Comment 15

•

3 years ago

(In reply to Hisashi Kamo from comment #9)

domain's zone has a valid DNSSEC validation chain

We confirmed that it was indeed not met.
We also checked other certificates for that point.

From comment 12, it appears that this was a complete and utter lie. This is completely unacceptable for a CA.

Assignee

Comment 16

•

3 years ago

Ryan-san, Paul-san,

Thank you for your comments.
Please let us check again for your comments and BR.

We will consider countermeasures as soon as possible.
We have started the process to revoke EE certificates in which FUJIFILM CA was not "Technically Constrained Sub-CA".

Thank you for your consideration.

Best regards,
Hisashi Kamo

Flags: needinfo?(h-kamo)

George [:fozzie]

Reporter

Comment 17

•

3 years ago

As you have stated in bug 1695938, SECOM had full control over this intermediate. Is your domain validation and/or CAA methodology any different between FujiFilm and the rest of SECOM's infrastructure? If it isn't then surely all SECOM certificates should be revoked and issuance should be halted until you fix these issues?

Flags: needinfo?(h-kamo)

Comment 18

•

3 years ago

(In reply to Hisashi Kamo from comment #16)

We have started the process to revoke EE certificates in which FUJIFILM CA was not "Technically Constrained Sub-CA".

Where is the incident report about delayed revocation?

As stated in comment 11, the timer started (at the latest) at 2021-04-07T11:45:00Z. It took you more than 5 days to even respond to comment 11, so I do not see any justification for any delayed revocation.

Assignee

Comment 19

•

3 years ago

George-san, Paul-san,

Thank you for your comments.

FujiFilm and the rest of SECOM's infrastructure had different our domain validation and CAA methodologies.
An incident report on the revocation of Fujifilm's EE certificate is being prepared and will be posted on Bugzilla.

Best regards,
Hisashi Kamo

Flags: needinfo?(h-kamo)

Comment 20

•

3 years ago

(In reply to Hisashi Kamo from comment #19)

FujiFilm and the rest of SECOM's infrastructure had different our domain validation and CAA methodologies.

What about other sub CAs like https://crt.sh/?id=2961095380 (there is no exception in the CP/CPS on CAA checking either, how did you check CAA records here?)?

It seems like you do not take ANY opportunity to look at issues systemically. And if you are caught lying, you are constantly backtracking.

Assignee

Comment 21

•

3 years ago

Paul-san,

Thank you for your comment.

We apologize for that there were some parts which are difficult to understand because of the translated English sentences and expressions in the past comments. Since some discussions seem to be ongoing with misunderstandings, we would like to supplement the explanation of the past comments.
In the FUJIFILM CA, there was a flaw in the CAA checking.
That is, when the DNS Name does not exist, it was been issued without being an error.
As described in Comment2, on March 6, we took measures to ensure that intermediate CA certificates can only be issued from domains that have name constraint. As a result, issuance from inappropriate domains is now prevented.
In Comment 12, the description that CAA checking is optional is limited to FUJIFILM's CA, which is name constrained by Enterprise CA.
In the previous description, we wanted to confirm the interpretation of BR's provisions.
As an implementation, we believe that CAA checking is mandatory, and if it is not performed properly, issuance must be suppressed.
In Comment13, there was the following description on December 15, 2020.

“SECOM issues a revised version of this intermediate, still operated by SECOM, and which is a Technically Constrained Sub-CA. At this point, we are given to understand that SECOM has made CAA checking optional and disabled domain validation for this sub-CA, which is operated by SECOM.”
However, SECOM considered that it was a policy to continue CAA Checking and implemented it.
In the case of other CAs, such as the one shown in Comment 20, we have adopted a mechanism to check the CAA record before issuing, and if the corresponding DNS Name has a Lookup Error, to suppress the issuance. Also, when processing CAA records, if only other companies' CAs are allowed to issue them, the issuance is also suppressed.
In such case, when the fact that the issuance was suppressed is also confirmed, the certificate applicant is contacted, the CAA record is requested to be corrected, the CAA record is checked again when applying for issuance, and if there is no problem, and issuance is permitted. That makes our system.

Thank you for your consideration.

Best regards,
Hisashi Kamo

Comment 22

•

3 years ago

Trying to summarize where things are:

Bug 1695938 was filed for the improperly disclosed, improperly constrained CA that issued this certificate
Bug 1707229 tracks the delayed revocation for that CA's certificates
SECOM had a process to technically restrict which domains could be issued to (Comment #2), and has updated those domains for this CA
- Prior to this incident, these domains were configured, but an improper setting caused the check to be disabled (Comment #4), due to a buggy comparison check (See also Comment #8)
SECOM checked CAA, but treated the non-existent domain (NXDOMAIN) as a lookup failure, and thus permission to issue (Comment #6, Comment #7)
- SECOM expects to correct this by 2021-07 (Comment #9)
- SECOM also allowed CAA checking to be skipped for technically constrained sub-CAs (Comment #12), but did not disclose this via their CP/CPS (Comment #14)
- SECOM no longer allows CAA checking to be skipped in any circumstance (Comment #21)
SECOM prohibits TLDs on SANs since 2021-04 (Comment #9)

In terms of outstanding issues, I'd draw SECOM's attention that the sub-CA mentioned in Comment #20, was not technically constrained. I see it was revoked on 2021-04-27, approximately 8 days after Comment #20, but I don't see it revoked in OneCRL (although perhaps crt.sh is out of date in fetching/evaluating OneCRL). Hopefully, it has been updated in CCADB to indicate it's revoked, and Mozilla's attention is drawn to it (and other relevant CAs). It sounds like there may be a separate incident report regarding creating CAs with improper nameConstraints (i.e. that are not technically-constrained as per the BRs), but it's unclear based on this incident.

While it appears SECOM has taken steps for this direct issue, I think there are still problematic lessons to take away from this incident. As Comment #13 attempted to summarize, the prohibition on Internal Server Names, and the many Bugzilla incidents filed, meant that SECOM has had, for years, awareness about the need to validate the domain names in SANs and ensure that all domains are well-formed and registrable.

While it appears that technical controls (to enforce issuance to an allow-listed set of domains) existed, and failed, we expect controls to be layered. If SECOM had enforced that DNS names were registrable, for example, we might expect this issue to have been detected and prevented.

We similarly have learned SECOM improperly understood the BRs' requirements on CAA, and arguably may have misunderstood the requirements regarding Technically-Constrained Sub-CA as well. Similarly, when it came to reporting this incident, and incident response in general, we see a very narrow focus, rather than looking at systemic root causes.

Lastly, it's been over a month since any update from SECOM, which is in violation of https://wiki.mozilla.org/CA/Responding_To_An_Incident#Keeping_Us_Informed , specifically,

You should also provide updates at least every week giving your progress, and confirm when the remediation steps have been completed - unless Mozilla representatives agree to a different schedule by setting a “Next Update” date in the “Whiteboard” field of the bug. Such updates should be posted to the MDSP mailing list, if there is one, and the Bugzilla bug. The bug will be closed when remediation is completed.

This might seem like a pedantic point, but I raise it, because we've had CA incidents in the past explicitly reminding a CA (Sectigo) of this expectation, and we've had ongoing incidents reminding another CA of this expectation (Google Trust Services). Similarly, in the past few months, we've had multiple incidents from a variety of CAs emphasizing the expectations for incident reports. This suggests that SECOM does not have a holistic program in place to monitor issues and incidents and incorporate lessons from other CAs, which is not encouraging for how well SECOM will be able to prevent and detect issues before they become incidents.

That said, despite all of this incredibly sub-par response, I don't think there's any further follow-ups.

Ben: I think this could be Next-Update to 2021-07-01 to reflect the outstanding CAA implementation fixes (ensuring NXDOMAIN fails closed, rather than failing open), if I'm understanding the timeline right.

Flags: needinfo?(bwilson)

Updated

•

3 years ago

Whiteboard: [ca-compliance] → [ca-compliance] Next update 2021-07-01

Assignee

Comment 23

•

3 years ago

Ben-san,

Sorry for that we couldn't follow up.

Implement CAA records lookup failure handling function into main system of CA (July 2021).

In regards to CAA records lookup failure, we implemented it on April 25.
From now on, we will strengthen our action response so that we can track it much better.

Thank you for your consideration.

Best regards,
Hisashi Kamo

Comment 24

•

3 years ago

Hisashi-san,
Please provide an update with details for what SECOM is doing / has done to improve domain validation and check CAA records.
(1) specifics of what was done on April 25;
(2) specifics of what was accomplished between April 26 and June 1;
(3) specifics of what was done between June 1 and July 1; and
(4) specifics of what SECOM plans to work on in July and thereafter.
Thanks.
Sincerely yours,
Ben

Whiteboard: [ca-compliance] Next update 2021-07-01 → [ca-compliance] Next update 2021-July-09

Updated

•

3 years ago

Whiteboard: [ca-compliance] Next update 2021-July-09 → [ca-compliance] Next update 2021-07-15

Assignee

Comment 25

•

3 years ago

Ben-san,

We’d like to answer the questions as follows:

(1) specifics of what was done on April 25

Implement CAA records lookup failure handling function into main system of FUJIFILM CA.
This work was initially planned in July, but was done on April 25, 2021, ahead of the schedule.
All Work to improve domain validation and check CAA records has been completed on this day, so that there was no further work to do corresponding to (2) (3) (4).

Thank you for your consideration.

Best regards,
Hisashi Kamo

Comment 26

•

3 years ago

I believe this matter can be closed and will schedule it for closure on or about Friday, 16-July-2021.

Updated

•

3 years ago

Flags: needinfo?(bwilson)

Whiteboard: [ca-compliance] Next update 2021-07-15 → [ca-compliance] Next update 2021-08-01