- How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.
We received a report to our problem reporting mailbox firstname.lastname@example.org from Alex Cohn at 18:16 BST on 2nd May 2019. The same email was sent to mozilla.dev.security.policy.
Of the 8 Sectigo certificates reported in that batch, 1 was already revoked at the time of reporting.
- A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.
2019-05-02 18:16 BST Initial Report received. 1 previously revoked, 7 to revoke.
2019-05-02 23:06 BST We started revoking the identified certificates.
2019-05-03 00:38 BST The 7 reported certificates were revoked.
2019-05-03 00:51 BST This bug was created by Wayne
2019-05-03 13:20 BST Investigation commences to identify any further certificates containing ‘Default City’ in the subject.
2019-05-03 17:26 BST Investigation concludes. 3 additional certificates found and revoked.
- Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.
We have stopped issuing certificates with the problem.
- A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.
There were 11 time-valid certificates that contained Default City in the Subject.
The earliest such certificate was issued on 2016-12-19.
The latest such certificate was issued on 2019-01-15.
- The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.
- Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
This is yet another case where visual comparison by human validators has failed because they see what they expect to see. Even with training and (since some of these are EV certificates) even with two people looking over the data sometimes they miss spurious values such as this.
- List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.
We have two programs of work underway that we believe will address these visual comparison errors.
We have had the first under development for some time and it aims to verify the addresses in OV and EV certificates with bulk address records. Subscriber generated addresses that do not match any of the bulk data sources will require a higher level of scrutiny by validators before issuance.
This system is now undergoing testing with production data. We anticipate a few more weeks development before it is fully live.
The second has come out of suggestions made to (and by) CAs in the incident reports around these ‘default’ values appearing in certificate subjects, such as using GeoNames data. We had not previously considered that source and are evaluating how best we can use it.