This is the final incident report.
- How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.
2019-07-05, 04:29 UTC: Internal quality assurance noticed the error
- A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.
2019-07-05, 04:29 UTC: Internal quality assurance noticed the error / Start Incident
2019-07-05, 04:45 UTC: Issuing stopped
2019-07-05, 06:30 UTC: Start investigating the error
2019-07-05, 09:00 UTC: Short-term measures are identified to prevent further errors. Bringing the measures into effect and training the validation team. Correction of certificate request.
2019-07-05, 10:10 UTC: Issuing restarted
2019-07-05, 10:30 UTC: Production of the certificate with correct OU field length
2019-07-10, 12:30 UTC: Start of thorough analysis according to internal problem management procedures
2019-07-12, 09:30 UTC: Management decision to shut down the affected application processing website
2019-07-12, 10:00 UTC: Shut down of affected application processing website
2019-07-12, 12:40 UTC: Informing Conformity Assessment Body about the issue
2019-07-15, 06:54 UTC: Revocation of defective pre-certificates
2019-07-18, 16:00 UTC: Management decision to terminally shut down the application processing system for PTC retail certificates
2019-07-19, 07:10 UTC: Shut down of affected application processing system
2019-07-23, 14:00 UTC: End of thorough analysis according to internal problem management procedures
2019-07-24: Final incident report
- Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.
CA has stopped the production after detecting the error. Production was resumed after corrective action was taken. As a result of our thorough analysis, we shut down the affected application processing website. New applications are no longer accepted. In this context, as part of the ongoing thorough analysis, it was later decided that we shut down the application processing system for PTC retail certificates as well. Certificates are no longer produced via this system. This application processing system is considered a legacy system and remains shut down for good.
- A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.
Problem: Precertificate OU > 64 characters
Number of affected certificates: 2
Issuing date of first certificate: 2019-07-04
Issuing date of last certificate: 2019-07-04
- The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.
- Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
We have determined that in our application processing system for retail certificates the X.509 control for pre-certificates has actually failed. The OU our customer requested from registration personnel exceeded the maximum length for the field. The field length check was not effective in this special use case. This was determined to be the root cause of the failure. The production of a final TLS certificate with the error was successfully prevented.
This affected application processing system predates the issuing of pre-certificates to ct logs. This functionality was added at a later stage. The last quality gate which caught the error acted right before the issuing of the final certificate but did not cover the newly implemented issuing of pre-certificates.
We have not valued the role of pre-certificates to the degree which is consensus here.
- List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.
We shut down the affected application processing website. New applications are no longer accepted. We shut down the application processing system as well. Certificates are no longer produced via this system. The application processing system is considered as a legacy system and remains shut down for good.
We reassigned reading duty to follow incidents on Bugzilla and m.d.s.p.