1805866 - SECOM: One of the EV certificate was mis-issued with the incorrect Registration Number by Cybertrust Japan (CTJ)

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[ONO Fumiaki]

Cybertrust Japan (CTJ) operate subordinate CAs signed by Security Communication RootCA2 (SECOM).

Therefore, SECOM post an incident report prepared by CTJ on behalf of them.
CTJ undergo WebTrust audit by themselves so that their audit reports exist independently from SECOM.

Here is an incident report.

In a follow-up report, CTJ may reply with details on this matter and the audit depending on the contents.

The incident report

1.How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in the MDSP mailing list, a Bugzilla bug, or internal self-audit), and the time and date.

CTJ validation team received two EV certificate requests from an organization, and one of the certificate was mis-issued with the incorrect Registration Number included.
Secondary validation agent noticed the possibility of incorrect Registration number was included when processing the validation on the second certificate request so that the personnel re-checked the completed validation for the first certificate, the first certificate was mis-issued.

The problematic certificate was revoked within 5 days from the problem detection. The Revocation was completed at 2022/12/13 18:54 to be specific.

2.A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

All the date and time of following timelines are described in JST.

Time - YYYY/MM/DD HH:MM	Action
2022/12/12 16:17	The problematic certificate was mis-issued when secondary validation agent completed the secondary validation for the first certificate.
2022/12/12 16:18	The same secondary validation agent initiated another validation for the second certificate request (the other certificate for the same organization). During this validation process, the secondary validation agent realized there is a possibility for mis-issuance on the first certificate and then checked on the first certificate.
2022/12/12 16:19	The same secondary validation agent confirmed the Registration Number in the certificate issued at 16:17 and reported team supervisor that there was a mis-issuance occurred. The team supervisor confirmed and determined the certificate was mis-issued.
2022/12/12 16:30	The validation agent started contacting the subscriber for notifying the problem and explaining the revocation and reissuance is required for the problematic certificate.
2022/12/12 18:25	Issued a second certificate with the correct Registration Number.
2022/12/12 18:54	Reissued the problematic certificate with the correct Registration Number .
2022/12/13 14:19	Informed the problem to CTJ’s Policy Authority (described as “CTJ PA” herein after).
2022/12/13 18:54	The problematic certificate was revoked.
2022/12/13 20:43	Informed the problem to the root CA(SECOM).
2022/12/13 20:51	Informed the problem occurred to the Auditor.
2022/12/14 08:30	The analysis on the root cause and the considered remediation plan were reported to CTJ PA and has been discussed at the CTJ PA.
2022/12/14 15:32	An incident report was submitted to the CTJ PA for review.
2022/12/14 19:03	Submitted incident report (first report, Japanese version) to SECOM.

3.Whether your CA has stopped, or has not yet stopped, certificate issuance or the process giving rise to the problem or incident. A statement that you have stopped will be considered a pledge to the community; a statement that you have not stopped requires an explanation.

The problem found is one certificate so far.

CTJ essentially stopped issuing for non-Enterprise RA service, EV certificates in which the validation for the Registration Number was relied upon manual verification by the validation agents after one hour discussion held by CTJ PA from 2022/12/14 08:30.
This cessation will be continued until the technical control validation procedure on the Registration Number is implemented. However, in the case the certificate is immediately required to be issued, such as the expiration date of the current active certificate is getting too close, the validation team will issue the certificate after the tertiary validation processed in addition to usual operation of primary and secondary validations.

For the enterprise RA service, the Registration Number is fixed(locked) by its system for each subscriber and that fixed value is included in certificates.

At the same time, CTJ is currently inspecting all the previously issued active certificates to confirm if there is any other impacted certificate. The result will be updated as soon as the inspection is completed.

For now, CTJ has first confirmed there is no other EV certificates mis-issued with the same error (including certificates issued from Enterprise RA) since this incident discovered.

4.In a case involving certificates, a summary of the problematic certificates. For each problem: the number of certificates, and the date the first and last certificates with that problem were issued. In other incidents that do not involve enumerating the affected certificates (e.g. OCSP failures, audit findings, delayed responses, etc.), please provide other similar statistics, aggregates, and a summary for each type of problem identified. This will help us measure the severity of each problem.

The impacted certificate is listed in item 5.

5.In a case involving certificates, the complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem. In other cases not involving a review of affected certificates, please provide other similar, relevant specifics, if any.

There is one certificate mis-issued.

https://crt.sh/?id=8181190469

6.Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

The subscriber organization (described as“A Co.,Ltd.” herein after) has submitted the certificate requests with the Registration Number for its Parent Company, “A group Co.,Ltd.”
Note that both “A Co.,Ltd.” and “A group Co.,Ltd.” are located and registered at the exact same address.
The information verified on QGIS database is described following.
The (primary and secondary) validation agents misunderstood the organization names, “A” and “A group”, and the Registration Number, and mistakenly approved issuing one of certificates out of two.

Company Name : Registration Number : State : Locality
A Co.,Ltd. : XXXXXXXX2465 : Tokyo : Minato-ku
A group Co.,Ltd. : XXXXXXXX6795 : Tokyo : Minato-ku

The validation is processed by two different validation agents, primary validation agent and secondary validation agent, by manual verification (visual confirmation by those two validation agents), however both of validation agents misidentified it when verifying the first certificate out of two certificate requests from “A Co.,Ltd.”
The secondary validation agent realized and had doubt about the possibility of incorrect Registration Number when verifying the second certificate, then the problem was detected by reconfirmation.

CTJ has found, from the interview against the validation agents after the mis-issuance, both primary and secondary validation agents are aware that “A” and “A group” is completely different organizations, and their Registration Numbers should also not be mixed-up, however the mistake occurred.

7.List of steps your CA is taking to resolve the situation and ensure that such situation or incident will not be repeated in the future, accompanied with a binding timeline of when your CA expects to accomplish each of these remediation steps.

The RA system of CTJ validates the domain name, organization name, and location with the technical control, however the Registration Number was manually validated (visually verified by the validation agents).
CTJ will implement technical control on the Registration Number for the remediation on this matter.
CTJ will implement a tool to support the RA system within next few days as a temporal immediate remediation.
CTJ essentially stops issuing EV certificates as previously noted, till the implementation of a supporting tool. Afterwards, the technical control will be implemented on the RA system itself as an ultimate remediation.
Each schedule will be determined as soon as possible, and information will be updated in this incident report.

Comment 1

[Hisashi Kamo]

Attachment: follow up report

Comment 2

[ONO Fumiaki]

Here is our update report till 2022/12/19.

[Timeline]

Time - YYYY/MM/DD HH:MM	Action
2022/12/15 19:15	SECOM posted first report to Bugzilla.
2022/12/16 13:00	CTJ completed the confirmation of fixed values registered on Enterprise RA and kept working on inspection on other.
2022/12/18 18:00	CTJ has completed implementation of a support tool as a temporal immediate remediation.
2022/12/18 20:02	CTJ discovered two other mis-issued certificates out of certificates issued in the past. Reported to CTJPA. 55% of the inspection has been completed.
2022/12/19 09:00	Reported the above two certificates to SECOM.
2022/12/19 09:00	CTJ restarted issuance of EV certificates.
2022/12/19 15:04	CTJ discovered three other mis-issued certificates out of certificates issued in the past. Reported to CTJPA. 75% of the inspection has been completed.
2022/12/19 17:13	Reported the above three certificates to SECOM.

[Confirmed status on mis-issuance]
CTJ has completed the confirmation of the fixed(locked) values registered on Enterprise RA and discovered no problem.
Regarding EV certificates issued by non-Enterprise RAs, in addition to the first report, CTJ detected five additional mis-issuance (total of six certificates including the first report). The additional 5 certificates that were discovered are scheduled to be revoked around December 23rd(JST).

CTJ has currently completed 75% of the inspection on mis-issuance.

Following six which includes the one reported in the first are the problematic certificates detected so far.

Date of finding the mis-issued certificate	Revoked date	Status	crt.sh
2022/12/12	2022/12/13	Revoked	https://crt.sh/?id=8181190469
2022/12/18	-	Not revoked	https://crt.sh/?id=5963374840
2022/12/18	-	Not revoked	https://crt.sh/?id=5963129504
2022/12/19	-	Not revoked	https://crt.sh/?id=6570237511
2022/12/19	-	Not revoked	https://crt.sh/?id=7531523231
2022/12/19	-	Not revoked	https://crt.sh/?id=8158928388

CTJ is continuously confirming rest of the EV certificates issued from non-Enterprise RA.

[Cause of the additional five certificates mis-issued]
CTJ investigated the cause of the problem which includes the interview to the validation agents. As a result, the cause is the same as "6.Explanation about how and why the mistakes were made" reported to Bugzilla in previous report, both of two validation agents missed the Registration Number.

The Registration Number is manually entered by the customer and is manually(visual) confirmed by the validation agents. As reported in the previous report, two different validation agents are working on. As a result of the current investigation, we found the rate of incorrect issuance due to human error by both two agents was small (about 0.07%), but could not get it to zero. So far, we thought the double-checking was working well enough, but, in reality, the probability of human error of overlooking a slight difference in a numeric string could not be eliminated completely and it led to mis-issuance.

The manual validation method has been processed by using reusable previously obtained validation evidence less than 398 days from initial validation, or by retrieving the information from QGIS, if there is no reusable previously obtained evidence. Note that the information is not provided via API, but only PDF currently.
As a tendency of each mis-issuance found this time could be said that they all occurred based on the failure of visual confirmation on the reused validation evidence. However, this is only because there are many cases reusing previously obtained validation evidence, and also thinks the validation procedure of retrieving the information from QGIS could also lead the same problem as far as the validation is relied upon visual confirmation. Therefore, CTJ believes the implementation of technical control is vital, and took an action as following.

[Implementation of technical control]
As a temporal immediate remediation, the validation procedure of Registration Number check is now changed to a process with a validation support tool.
CTJ has additionally implemented the validation support tool which indicates to the CTJ validation agents the result of accuracy by loading each Registration Number and its organization name and matching them to the submitted information in certificate requests, during the primary and secondary validation continuously operated.
This validation support tool was implemented on December 18th(JST), and CTJ restarted the validation and issuance for EV certificates on December 19th(JST).

The implementation of technical control to the RA system itself as ultimate remediation is now continuously worked on.
We will update it in this incident report as soon as the schedule is determined.

Comment 3

[Hisashi Kamo]

Here is our update report till 2022/12/23(JST).

Timeline

Time - YYYY/MM/DD HH:MM	Action
2022/12/20 10:38	CTJ additionally discovered another mis-issued certificate out of certificates issued in the past. CTJ kept working on the inspection.
2022/12/22 15:00	CTJ completed the 100% of the inspection on certificates issued in the past and reported the inspection results to CTJPA. Total of 7 mis-issued certificates including 1 certificate in the first reports were discovered.
2022/12/22 16:00	Reported the above inspection results to SECOM.
2022/12/23 10:30	CTJ completed the revocation of additional 6 mis-issued certificates.
2022/12/23 13:30	CTJ checked that the above revoked certificates were listed on CRL and OCSP.
2022/12/23 13:30	CTJ reported to CTJPA that the revocation of all the mis-issued certificates and confirmation of CRL and OCSP had been completed.
2022/12/23 14:00	CTJ reported SECOM that the revocation of all the mis-issued certificates and confirmation of CRL and OCSP had been completed.

Confirmed status on mis-issuance

CTJ completed, on December 22nd (JST), the inspection of all the certificates issued in the past including being issued from non-Enterprise RA and found 7 total certificates
(1 certificate in the first report and additional 6 certificates) were mis-issued. The rate for mis-issuance occurs is approximately 0.07%.
6 certificates additionally discovered were revoked on December 23rd (JST) which is within 5 days from their detection, and with this revocation of 6 certificates, CTJ has completed revoking all the mis-issued certificates.

The problematic 7 all certificates (1 certificate in the first report and additional 6 certificates) are listed below.

Date of finding the mis-issued certificate	Revoked date	Status	crt.sh
2022/12/12	2022/12/13	Revoked	https://crt.sh/?id=8181190469
2022/12/18	2022/12/23	Revoked	https://crt.sh/?id=5963374840
2022/12/18	2022/12/23	Revoked	https://crt.sh/?id=5963129504
2022/12/19	2022/12/23	Revoked	https://crt.sh/?id=6570237511
2022/12/19	2022/12/23	Revoked	https://crt.sh/?id=7531523231
2022/12/19	2022/12/23	Revoked	https://crt.sh/?id=8158928388
2022/12/20	2022/12/23	Revoked	https://crt.sh/?id=6852218671

Cause of an additional mis-issued certificate of (id=6852218671)

CTJ found, based on a result of investigation, the cause of the problem was same as the one previously reported. Therefore, CTJ believes the validation support tool implemented on December 18th(JST) keeps working well for preventing the problem.

Status on a technical control on RA system as ultimate remediation

The implementation of technical control to the RA system itself as ultimate remediation is now continuously worked on. CTJ will implement it by January 22nd, 2023.
CTJ will updates it after its completion.

Comment 4

[ONO Fumiaki]

Here is our update report till January 23, 2023 (JST).

Status on a technical control on RA system as ultimate remediation

CTJ has completed implementing technical control of the RA system itself as the ultimate remediation on January 19, 2023 (JST).

The implemented technical control has gone through the normal testing process, and furthermore, CTJ has performed test operations on actual certificate requests and confirmed it functions with no problem.

Comment 5

[Ben Wilson]

What additional remediation items remain to be done?

Comment 6

[ONO Fumiaki]

There won't be any additional remediation items since CTJ has completed ultimate remediation.

Comment 7

[Ben Wilson]

I will then consider closing this on Wed. 1-Feb-2023.