Further investigation indicates that there were 5 miss-issued certificates from the same incident with the same root cause. I have updated the miss-issue items below to cover the full incident.
- How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.
On June 21, 2019 at approximately 18:30 UTC, Entrust Datacard discovered a miss-issue of 2 certificates with a validity period greater than 825-days through online testing of certificates for the root embedding test sites.
On June 25, 2019 at approximately 14:30 UTC, Entrust Datacard discovered the miss-issue of 5 EV SSL certificates with OV subject data. These certificates include the 2 certificates which were issued for more than 825-days. This data was discovered by reviewing zlint data. The zlint data showed that the certificates were missing the business category and the serial number from the subject. Further review also indicated that the registration jurisdiction information was was also missing from the subject of all certificates.
- A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.
June 21, 16:00 UTC - Issued three certificates to support valid, expired and revoked test sites
June 21, 18:30 UTC - Issue was discovered that 2 certificates were issued for 3 years which exceeds the maximum allowed of 825-days
June 21, 19:46 UTC - Both certificates were revoked and replaced by this time.
June 25, 14:30 UTC - Zlint data indicated that all 5 certificates issued on 21 June 2019 were issued with OV validated data, but included the EV certificate policy OID. Zlint indicated an error of the serial number and business type were missing from the subject name.
June 25, 20:59 UTC - One certificate was not expired or revoked, so was revoked at this time.
- Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.
The CA has stopped issuing EV SSL certificates with incorrect data in the subject and with an incorrect validity period.
- A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.
Two EV SSL certificates were issued with a validity period greater than 825-days on 21 June 2019.
Five EV SSL certificates with incorrect data in the subject were issued on 21 June 2019.
- The complete certificate data for the problematic certificates.
- Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
The issuing CA is not set up for production as there are no third party certificates currently being issued. There is no automated issuance nor is there any pre-issuance linting set for a non-production CA. The certificates from the issuing CA are issued manually. There is a process for manually issuing certificates. This process defines the certificate type, the validity period, and the subject name. It also provides the CSR. The process requires verification to confirm the subject name information has been verified along with checking CAA.
In this incident the trusted role which issued the certificates made errors where 1) 2 certificates were issued using the default validity of 36 months instead of the requested 26 months and 2) all 5 certificates were issued as EV SSL certificate type, but included only OV validated information.
- List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.
The certificates were revoked or expired as follows:
- 2 certificates were issued, then revoked on June 21, 2019, as they were issued to support a revoked test site.
- 1 certificate was issued for a 1 day validity, as it was issued to support an expired test site. This certificate expired on June 22, 2019.
- 1 certificate was revoked on June 21, 2019 as it was discovered that the validity period exceeded 825-days.
- 1 certificate was revoked June 25, 2019, after the EV data miss-issue was discovered.
In order to avoid this issue in the future, it will be addressed in 2 phases.
Phase 1 - The existing manual process will be updated to include the following:
- Manual certificate issuance must be justified and approved
- Manual certificate issuance request must include certificate type, validity period, subject name, SANs and certificate profile
- CSR must be verified to meet policy requirements which includes key size and no use of weak keys
- Verification team must ensure that the subject information has been verified and is correct to use with the certificate request
- CAA must be checked
- Certificate issuance must be checked against the approved certificate profile, the approved subject and also checked against a publicly available linting tool
Phase 2 - The process will be moved to a managed account with automates checks:
- All subject data in the account will be pre-verified
- All certificate requests will 1) validate the quality of the CSR, 2) move through our "policy engine" to ensure BR and EV requirements are met, and 3) check CAA
- All certificate requests will have pre-issuance linting performed
- All EV pre-certificates will be published to CT logs
We think that Phase 2 will eliminate the errors from manual certificate issuance.