1622539 - Microsec: Issuance of 2 IVCP precertificates without givenName, surName, localityName fields

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Wayne Thayer]

Dr Sandor Szoke posted the following incident report to the mozilla.dev.security.policy mailing list:

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.
   Microsec became aware of the problem from the new discussion at the mozilla.dev.security.policy
   https://groups.google.com/forum/#!topic/mozilla.dev.security.policy/jRKOr4nvOfY

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.
   2019-10-02 2 precertificates were issued for internal testing purposes with short validity
   2020-03-10 Microsec was informed about the faulty precertificates
   2020-03-10 Microsec checked the faulty precertificates. They were already expired, so revocation was not possible.
   2020-03-10 Microsec decided to immediately stop issuance of IVCP certificates until all corrective measures are in place, to prevent similar mistakes in the future
   2020-03-10 Microsec decided to develop the CA software by adding further controls regarding the required fields of IVCP certificates to guarantee compliance with the certificate profile in the future
   2020-03-13 Microsec deactivated all the IVCP profiles in the CA software to prevent issuance of IVCP certificates until the controls in the CA software are in place
   2020-03-13 Microsec opened this incident report

3.Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.
Two subordinate CA were affected. They haven’t been stopped, but the issuance of IVCP certificates has been suspended by informing the RA operators about the decision and by deactivating all the IVCP certificate profiles.

4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.
   There are 2 certificates involved.
   They were issued on the same day which was 2019-10-02

5. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.
   The problematic certificates are:
   https://crt.sh/?id=1947655126&opt=zlint,ocsp
   https://crt.sh/?id=1947655112&opt=zlint,ocsp

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
   Microsec made an investigation and could find the following:

• the certification policy and practice statement contain the proper requirements regarding the content of the IVCP certificates
• the problem was caused by human mistakes, the RA operators could not recognize the missing data
• the CA software presently does not have automatic controls to check for the existence of the required fields in case of IVCP certificates
• the certificates have been checked by cablint, but cablint did not indicate this fault
• the certificates have never been published and used, so the fault has not been discovered until now

7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.
   Microsec decided the following measures to avoid having the same problem in the future.

• the problematic precertificates were issued for short period and have already expired, so revocation is not necessary
• suspended the issuance of the IVCP certificates with immediate effect
• decided to develop the CA software to add further controls and checks in case of IVCP certificates - no deadline has been set due to the COVID-19 situation
• decided to hold a training about the required IVCP profiles for the RA operators before enabling the issuance of IVCP certificates again
• the issuance of IVCP certificates may be continued only after the successful testing of the upgraded CA software
• Microsec will check all the issued IVCP certificates looking for similar issues - deadline 2020-03-20
• Microsec will review the CA software looking for possible similar problems - deadline 2020-03-31

Comment 1

[Wayne Thayer]

The following update was posted to the m.d.s.p. thread on 19-March:

Microsec has finished the detailed investigation on the issued TLS IVCP certificates looking for similar issues. The findings are the following:

Microsec issued altogether 9 test certificates on 2019-10-01 and 2019-10-02 with 30 days validity.
The purpose of the test was to issue DV and IV certificates from each subordinate CA which is used to issue these types of certificates.
The test covered both the RSA-based hierarchy and the ECC-based new CA hierarchy.
The test certificates were issued directly from the CA software by using the operator interface. The CA software forces the use of dual control.
The RSA-based system is configured to use Certificate Transparency to fully comply with the present requirements. 4 of the 9 test certificates were issued in the RSA-based system.
The ECC-based system was configured to issue the test certificates without Certificate Transparency, because this root is not trusted yet and none of the log servers issues SCT for this root. 5 of the 9 test certificates were issued in the ECC-based system.
The Subject DN fields were configured according to the DV profile requirements, this way the issuance of the DV certificates was successful. The 2 successfully issued RSA-based DV certificates can be found in the crt.sh as
https://crt.sh/?q=1947651733
https://crt.sh/?q=1944631156
The issuance of the test IV certificates was done using the same Subject DN fields by mistake.
None of the operators identified the missing fields in case of IV certificates.

The following RSA-based IV certificates were issued with missing fields in the Subject name:
https://crt.sh/?q=1947655126
https://crt.sh/?q=1947655112

They are already included in the incident report and there are no other IV certificates issued under the RSA root with this problem.

A similar set of 3 DV and 2 IV test certificates were issued under the ECC root, but without SCT, as explained above. Because of this, they can’t be found in the crt.sh. The 3 DV test certificates were correct. The 2 IV test certificates have the same problem: missing givenName, surName, localityName fields in the Subject DN. These certificates are already expired, so revocation is not possible.
Microsec has only issued test TLS certificates under the ECC root so far.
The cause of the problem was the same as for the RSA-based test certificates, so the remediation and preventive measures are the same too.

Comment 2

[dr. Sándor SZŐKE]

The following update was posted to the m.d.s.p. thread on 31-March:

• Microsec will review the CA software looking for possible similar problems - deadline 2020-03-31

Microsec has completed a detailed review of the automatic controls built into the CA software. The review covered all SSL/TLS certificate types and focused on the presence of required fields in the Subject DN.

Microsec first created a table with all possible Subject DN fields based on the current version of the CABF BR, EVG, and Microsec CPS documents. The following certification policies are included in the table: DVCP, IVCP, OVCP, EVCP/QWAC, EVCP/PSD2. Microsec has collected rules for each field and policy combination, which may include:
 mandatory
 forbidden
 optional

For optional fields, Microsec also identified dependencies.

After completing the complete list of requirements, Microsec reviewed the source code for the CA software. As a result of this review, Microsec determined that the scope of automated checks should be expanded for the IVCP profile, but they are complete for all other certificate profiles.

Microsec has created a work item and has already begun to upgrade the CA program with additional controls. There is no specific deadline for completing the development, but Microsec plans to do the development and test the new software version by 2020-04-15.

Comment 3

[dr. Sándor SZŐKE]

Attachment: SSL_Certificates_SN-DN_rules.xlsx

I attach the excel sheet which contains the rules for each possible SN/DN field and policy combination.

Comment 4

[Ryan Sleevi]

Comment #2 states:

Microsec plans to do the development and test the new software version by 2020-04-15.

Yet it's now 2020-05-17 and there's been no updates, here or on the thread, about the progress.

Please review https://wiki.mozilla.org/CA/Responding_To_An_Incident#Keeping_Us_Informed

Comment 5

[dr. Sándor SZŐKE]

(In reply to Ryan Sleevi from comment #4)

Comment #2 states:

Microsec plans to do the development and test the new software version by 2020-04-15.

Yet it's now 2020-05-17 and there's been no updates, here or on the thread, about the progress.

Please review https://wiki.mozilla.org/CA/Responding_To_An_Incident#Keeping_Us_Informed

I give you a brief status report on CA software development.

Microsec has already completed the development of CA software and introduced additional automatic checks for the proper configuration of the SN fields in case of different certificate types.
A number of internal tests have already been performed on the test system, but the new release is still awaiting final acceptance tests.
The new release of the CA software has not been activated in the live system yet, the scheduled activation date is 2020-05-20.
IVCP profiles are still inactive, Microsec does not issue IVCP certificates.

As Microsec informed the community in its first comment (2020-03-13), there was no deadline for CA software development due to the COVID-19 emergency situation. This information was confirmed in comment 2020-03-19, and the date 2020-04-15 was given as a plan only and not as a strict deadline.

Microsec will immediately notify the community in case of any further change in this thread.

Comment 6

[Ryan Sleevi]

Sure, but the CA is expected to provide regular updates, in part to help inform us of such delays and what's being done to address them. Unless Next-Update is set, the CA is expected to be providing regular updates.

Comment 7

[dr. Sándor SZŐKE]

I inform you that as planned two days ago, Microsec today activated the new CA software release in the live system.

The CA sofware as been improved to support more automatic checking for the presence of SN fields for different certificate profiles.

As part of the project, Microsec has developed an automated testing tool that allows for more efficient testing of the CA program.
Test inputs and expected CA responses can be given in a configuration file, and the test runs automatically in batch mode.
The test was passed without error.
The development was approved yesterday and the installation of the new release on the live system was approved.

Despite the software upgrade, IVCP profiles are still not active, and Microsec does not issue IVCP certificates.
There is no official deadline for activation, but Microsec plans to activate IVCP profiles together with the other profile changes due to the release of new public documents on 2020-05-26.

Prior to activation, Microsec will organize a training for Registration Officiers on the proper configuration of different TSL profiles.

Microsec will immediately notify the community as soon as any changes occur.

Comment 8

[dr. Sándor SZŐKE]

I inform you about the latest changes:

Microsec organized the training for the Registration Officers regarding the proper configuration of different TSL profiles yesterday.

The IVCP profiles were reactivated today morning.

It is possible to issue IVCP certificates again.

Comment 9

[Ben Wilson]

I am requesting another update prior to July 15.

Comment 10

[dr. Sándor SZŐKE]

There is no new information regarding this incident.

The CA software runs without any problem and Microsec is ready to issue IVCP certificates.

The same problem did not happen again.

Comment 11

[Ryan Sleevi]

I believe this is ready to be closed.