Here is the incident report requested:
Incident Report – Mozilla Policy Violation
1.How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.
On May 11, 2019, DigiCert was informed of an issue with certificates being issued with values containing a stateOrProvinceName of "Some-State” via an MDSP discussion (subject: Certificates with subject stateOrProvinceName "Some-State"). The issue indicates that the State/Province field was not validated, and is therefore, a baseline requirement violation.
2.A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.
May 11, 2019 – DigiCert was informed via an mdsp discussion of certs we issued with the “Some-State” value. Investigation began and concluded 8 certs required revocation.
May 13, 2019 – DigiCert was informed again via this bug that certificates were identified with a “some-state” value in the stateorProvinceName field of the certificate record.
May 13, 2019 – DigiCert added “Some-State” and “some state” as flags for order processing. This will require the standard first and second checks as well as built in CA blockers and a required manager review.
May 15, 2019 – All identified problem certs were revoked.
3.Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.
Our CA has stopped issuing these certificates. Please see below for our get well plan (section 7). All certificates identified with invalid values related to this incident have now been revoked and blockers have been added at a CA level to prevent additional signing of certificates with these invalid values.
- A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.
8 certificates were identified (see below).
The first cert was issued: May 31, 2016
The last cert was issued: April 16, 2017
5.The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.
6.Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
The root issue was a combination of the auto-populator inserting incorrect data into the certificate request and improper identification of the invalid values during our verification process. While reviews were in place, unfortunately these values were missed.
7.List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.
Following our incident in 2017, we added city and state field checking system that flags potentially incorrect city and state field values. When a potentially incorrect field is flagged it forces Validation staff to do an additional review and checking process. Additionally, on 13-May-2019, we have added a blacklist check for the terms: “default”, “city”, and “some” to ensure this triggers an additional review to clear before allowed to issue.
May 16, 2019 - We completed running scans to search for any more instances of some-state across our cert database and are reviewing the results. We will post an update once we have completed our analysis.
Additionally, by the end of May 2019, we have scheduled additional training (to be completed no later than months end), on soft-blocks, flags and triggers to prevent this issue in the future.