Closed Bug 1548714 Opened 5 months ago Closed Last month

SECOM: "Default City" in Subject:localityName

Categories

(NSS :: CA Certificate Compliance, task)

task
Not set

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: wayne, Assigned: h-kamo)

Details

(Whiteboard: [ca-compliance])

A list of SECOM certificates containing a localityName of "Default City" was published at https://misissued.com/batch/51/ This is apparently the default placed in OpenSSL CSRs, indicating that this field was not validated. BR section 7.1.4.2.2(e) states: If present, the subject:localityName field MUST contain the Subject’s locality information as verified under Section 3.2.2.1. The EVGLs reference the BRs.

Please provide an incident report, as described at https://wiki.mozilla.org/CA/Responding_To_An_Incident#Incident_Report

Wayne-san,

We apologize for delay due to special long National holiday from April 27th to May 6th because of the new emperor's enthronement in Japan.

We realized this issue today by the email from Mr. Alex Cohn on dev-security-policy mailing list and bugzilla, and started for the arrangement to revoke with our customer.
An incident report will be posted as soon as it is ready.

Thank you for your consideration.

Best regards,
Hisashi Kamo

Wayne-san,

Please let us provide you our incident report as below.

  1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

We became aware of the certificates with problem via the mail notification by mozilla.dev-security-policy ML, and Bugzilla on 2019/05/07.
(We couldn’t check the mail from Apr. 27 to May 6 because of Japanese National holidays, so that we realized the incident this late. Very sorry about that.)

  1. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

2019/05/02 The 24 valid certificates with problem had been reported by misissued.com, but we couldn’t realize them because of the Japanese Holidays from Apr. 27 to May 6.
2019/05/07 We realized the incident via the mail notification by mozilla.dev-security-policy ML, and Bugzilla.
2019/05/07 We notified the incident to the customer, and started to make revocation.
2019/05/09 We did the research on the all CAs, and confirmed that other certificate were not affected except the 24 certificate which were reported this time.
2019/05/10 We revoked 20 certificates out of 24, which left 4 to be done.
We are planning to revoke 3 certificates out of 4 by May 12, and the final 1 will be revoked by May 17.

The one certificate that will be revoked by May 17, is to be used by the customer for the purpose of linking system with the important infrastructure of their system. Because of that reason, we concern that the revocation of the certificate makes some impact on the many end-users.
Continuously, we’ll consider the contents of the incident and these impacts thoroughly, and make our prompt and best effort to ease the situation.

  1. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

We made certain that RAs should strengthen the checking application. We try hard to work on when issuing the new certificates to prevent any problem. Also from now on, we’ll change the program into the system which reject the certificate application that makes L="Default City", which prevents the recurrence. That system will be completed by the end of June.

  1. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

https://misissued.com/batch/51/
24 certificates listed on the above site are subjects to the problem.

2018/08/21 The first problematic certificate was issued.
And 2019/04/18 the last problematic certificate was issued.
We did research on the all Certification Authorities (CAs), and found no certificates which makes L="Default City" except 24 mentioned above.

  1. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

https://misissued.com/batch/51/
Above page listed, 24 certificates below are subject to the problem:
https://crt.sh/?id=825576826
https://crt.sh/?id=671596799
https://crt.sh/?id=671596861
https://crt.sh/?id=671596836
https://crt.sh/?id=825576816
https://crt.sh/?id=883179685
https://crt.sh/?id=1436268856
https://crt.sh/?id=901932674
https://crt.sh/?id=901961819
https://crt.sh/?id=973167724
https://crt.sh/?id=911497749
https://crt.sh/?id=957570433
https://crt.sh/?id=982889580
https://crt.sh/?id=1157151014
https://crt.sh/?id=1227219817
https://crt.sh/?id=1229606496
https://crt.sh/?id=1218549172
https://crt.sh/?id=1377536304
https://crt.sh/?id=1336199339
https://crt.sh/?id=1339584547
https://crt.sh/?id=1339584894
https://crt.sh/?id=1339584560
https://crt.sh/?id=1436250208
https://crt.sh/?id=1436285481

  1. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

Our system was not capable of rejecting the certificate application which makes L="Default City".
The RAs needed to check it, but couldn’t reject the application which makes L="Default City" as an error.

  1. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

From now on, we’ll change the program into the system which reject the certificate application that makes L="Default City", which prevents the recurrence. That system will be completed by the end of June.

Thank you for your consideration.

Best regards,
Hisashi Kamo

Kamo-san,

Thank you for this incident report.

The RAs needed to check it, but couldn’t reject the application which makes L="Default City" as an error.

Could you please explain this statement? The BRs require information contained in the localityName field to be verified. Are you stating that SECOM RAs failed to validate this information, or that they unsuccessfully validated it but issued the certificate anyhow? Also, given the number of certificates issued with this error, why was it not caught by your internal audit processes?

Flags: needinfo?(h-kamo)

Wayne-san,

Please let us provide you our answer.

What we talked about the validation failure of the information, is as the former (RAs failed to validate this information) showed, RA administrator misjudged that the validation went successful when validated it in advance, so the certificate was issued. From now on, we’ll let RA administrator know the incident thoroughly, and make sure that such mistakes never happen again by implementing the system. Also regarding with the internal audit, we’ll check on the value of “L” as whether it is “Roman character” or not from now on, and be able to detect “Default City” which is not “Roman character”, that makes improvement. Our internal audit is not for the 3% sampling required by BR section 8.7, but all the certificates are subject to it, so that we can inspect comprehensively for the same issue.
We took this incident very seriously, and will try our best effort to work on the measures for preventing any recurrence.

Finally, as the update from the last time, We’d like to report that the revocation of the all problematic certificates has been completed.
The details are as follows:

 2019/05/10 We revoked 20 certificates out of 24, which left 4 to be done.
 2019/05/12 We revoked 3 certificates out of 4, which left 1 to be done.
 2019/05/16 We revoked the last 1 certificate.

Thank you for your consideration.

Best regards,
Hisashi Kamo

Flags: needinfo?(h-kamo)

As I find myself repeating in a number of bugs, can you confirm that SECOM has reviewed the (ongoing) responses to a similar issue in the following bugs:

This contains a number of discussions about systemic controls and practices, from the design of how CSRs are ingested to the user experience of the RA software. It contains discussions about how validation practices may change, specific requests for documentation about changes to training programs, and how systemic evaluation of root cause may include re-evaluating how and how often training is conducted.

My hope is that, after confirming the review of such bugs, SECOM may provide an updated analysis about the issue, taking into consideration the various systemic issues and concerns, and show how it plans to address those concerns.

Flags: needinfo?(h-kamo)

Ryan-san,
Thank you for your advice.
We are going to improve the system with reference to the information from the bugs and work hard for developing the system, which is completed by the end of June.

Thank you for your consideration.

Best regards,
Hisashi Kamo

Flags: needinfo?(h-kamo)
Whiteboard: [ca-compliance] → [ca-compliance] - Next Update - 01-July 2019

(In reply to Hisashi Kamo from comment #6)

We are going to improve the system with reference to the information from the bugs and work hard for developing the system, which is completed by the end of June.

Can you provide more specific details about the anticipated changes, so that we can understand progress by that date?

Flags: needinfo?(h-kamo)

Ryan-san,

We apologize that we couldn't report on our progress of the incident sooner.

Now we would like to report that we successfully completed the system modification for this matter. For the purpose of doing our best, we implemented the following modifications as two stages. Our modification shall prohibit the issuance of a certificate in which "Default City" is included in Subject L.

  • Controlled by the RA system of the CA targeted for the revocation this time
  • Controlled by CA systems of all CAs

Thank you for your consideration.

Best regards,
Hisashi Kamo

Flags: needinfo?(h-kamo)

Over a month elapsed since the requested feedback in Comment #7.

What steps is SECOM taking to ensure timely responsiveness? What caused the lack of timely details between Comment #7 and Comment #8?

As captured in https://wiki.mozilla.org/CA/Responding_To_An_Incident#Keeping_Us_Informed , CAs are expected to provide timely progress reports to assure the community that the CA is taking steps. Regular status updates help inform the challenges as well as demonstrate progress.

Understanding why timely updates weren't provided, and what changes are being made with respect to SECOM's Compliance/Management processes, is crucial to assuring trust.

Flags: needinfo?(h-kamo)
Flags: needinfo?(wthayer)

Ryan-san,

We apologize very much and we'd like to make some improvements to update the things more timely as follows:

  1. We'll assign more staff who can report to Bugzilla.
  2. We'll check the status on daily bases, and plan to send the report to Bugzilla at least once a week.
    Because we used to put the priority on the changes of the multiple systems and their releases, we tend to overlook the importance of making the report timely.
    From now on, we try our best efforts to review the operation rule, strengthen the reporting system of the person in charge, and check the status of daily questions, so that the whole situation will be improved accordingly.

Thank you for your consideration.

Best regards,
Hisashi Kamo

Flags: needinfo?(h-kamo)

Wayne-san,

We deeply apologize that we let the same problem happen again.

As the two new certificates that correspond to this problem are found, we’d like to submit the incident report.

  1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

2019/07/22 We found the two certificates with problem via the self-audit.

  1. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

2019/05/16 We revoked all of the 24 certificates including "Default city", issued at that time.
2019/05/27 We issued the one certificate with the problem.
2019/06/03 We issued the one certificate with the problem.
2019/06/10 The fully awareness of RA administrators for the incident, and the system improvement were completed.
2019/07/22 We found the two certificates with problem via the self-audit.
2019/07/26 We revoked the two certificates with the problem.

  1. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

Because the system improvement was completed on 2019/06/10, the certificate with the same problem will not be issued.

  1. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

The two certificates containing "Default City" in L are as follows:
https://crt.sh/?id=1513841945
https://crt.sh/?id=1542951120

2019/05/27 The first certificate with the problem was issued.
And 2019/06/03 the last certificate with the problem was issued.

  1. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

The two certificates with problem are as follows:
https://crt.sh/?id=1513841945
https://crt.sh/?id=1542951120

  1. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

The all certificates with the problems revoked as follows:
(https://bugzilla.mozilla.org/show_bug.cgi?id=1548714#c2).
However, the two certificates found this time were newly issued during the system improvement after the all certificates with problem had been revoked.
Therefore, before we completed the system improvement and let the RA administrator know the incident thoroughly, the two certificates were issued.

  1. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

From now on, we will conduct the self-audit after doing the system improvement completed, and also will do the system improvement after confirming that "The status of all certificates with the similar problems will be checked, and make sure whether all are in a revoked state or not, and whether there is no newly issued certificate or not".

Best regards,
Jinta Nakamura

It appears that all questions have been answered and remediation is complete.

Status: ASSIGNED → RESOLVED
Closed: Last month
Flags: needinfo?(wthayer)
Resolution: --- → FIXED
Whiteboard: [ca-compliance] - Next Update - 01-July 2019 → [ca-compliance]
You need to log in before you can comment on or make changes to this bug.