- How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.
By problem reports to firstname.lastname@example.org:
Times are in BST (= EDT + 5) (dd/mm/yyyy hh24:mi)
#certs is the number of distinct Sectigo issued certificates in each report.
||Fri 16/08/2019 23:18
||subject:serialNumber wrong. JOI=SE, but subject:serialNumber is for a DK entity.
||Sat 17/08/2019 21:53
||subject:serialNumber wrong. JOI:country=SE should have subject:serialNumber of 10 digits, but subject:serialNumber is not 10 digits
||Mon 19/08/2019 01:45
||businessCategory is Private Organization, but should be Government Entity. Also serialNumber=0
||Mon 19/08/2019 02:00
||Mon 19/08/2019 02:05
||Mon 19/08/2019 02:26
||serialNumber seems to be 0
||Tue 20/08/2019 03:41
||Tue 20/08/2019 18:46
||JOI:locality & JOI:stateOrProvince wrong.
||Tue 27/08/2019 16:53
||Subject:serialNumber of '0' or some combination of '0's
- A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.
||Code deployed which, as a side effect, blocks all-zero entries in subject:serialNumbers
|16/08/2019 23:18 thru 27/08/2019 16:53
||Problem reports received
||Analysis of problem reports begins
||Revocation of identified certificates begins
||A distinct 2nd approval team was established
||Code deployed to automatically check stateOrProvinceName for the most common Jurisdictions.
||Revocation of identified certificates concludes
Not all of the certificates were revoked within 24 hours of being reported. Our goal and intent was that all of the certificates were to have been revoked within 5 days of receiving the reports however, regrettably, the 6 certificates with Invalid JOI:stateOrProvince field values reported to us at 20/08/2019 03:41 took nearly 8 days to revoke due to an administrative error. All of the other reported certificates were revoked (or expired) within 5 days of receiving the relevant report.
- Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.
We have stopped issuing certificates with these problems.
- A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.
Counts of affected certificates for which we have received reports.
|JOI (jurisdiction of incorporation) contains a state or a locality when it should not because the registration scheme is at the country or state level.
||30-Aug-17 to 30-Oct-18
|Subject:serialNumber omits digits
||28-Nov-17 to 23-Nov-18
|Subject:serialNumber is a date when it should be a registration number
||05-Jan-18 to 05-Dec-18
|JOI is wrong. Subject is registered in both SE and DK. The organizationName and the subject:SerialNumber are for DK, but the JOI says SE
||23-May-18 to 05-Oct-18
|JOI is wrong. subject:Country and JOI:Country are both SE instead of NL
|JOI is wrong. should be SK, not US
|businessCategory is wrong. It shows Private Organization when it should be Government Entity
|Subject:serialNumber consists of only zeroes
||30-May-17 to 21-Jun-19
|Subject:State wrong - Confusion over the status of Puerto Rico & US Virgin Islands
||16-Jun-17 to 03-May-19
|Subject:State wrong - City listed as state. (e.g. Manhattan, Las Vegas..)
||13-Sep-17 to 16-Jan-18
|Subject:State wrong - State misspelled. (e.g. Deleware, Marryland, NewYork..)
||08-Jan-18 to 08-Aug-19
|Subject:State wrong - irrelevant text present
||27-Oct-18 to 17-Dec-18
|Subject:State wrong - State and locality reversed
|Subject:Country wrong: US instead of OM
|jurisdictionStateOrProvinceName wrong - misspelled. (e.g. NewYork, Deleware)
||09-Aug-17 to 25-Jun-19
|jurisdictionStateOrProvinceName wrong - irrelevant text present
||30-Aug-17 to 12-Sep-18
|jurisdictionLocality - irrelevant text present
10 certificates have 2 of the above problems, so the total number of certificates affected is 2090.
152 of the above certificates had been revoked before the reports were received.
- The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.
Links to problematic certificates
- Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
There are a number of separate issues occurring here and they do not split out distinctly across the problem reports listed above which is why we are dealing with them all in one incident report.
Sectigo Certificate Manager (SCM).
SCM is an Enterprise RA platform written and hosted by Sectigo. It allows Enterprise customers to enter Organization Details and certificate requests for domain names and, after an initial validation of the Organizational Details and domain names to the requirements of our CPS (and the BRs and the EVGLs), the Enterprise customer may then order further certificates for the same details without repeating the validation steps for as long as the guidelines permit the validation information to be re-used.
Validators and 2nd Approval validators
We still rely substantially on human validators for the validation of OV and EV subject details, and also for the EV 2nd approval step.
Sectigo Information Manager (SIM).
SIM is a data matching tool and Jurisdiction database that we have been working on for the last 18 months. We hoped to have integrated this tool fully into our issuance pipeline before now, but it is still in a Beta test.
- Zero serial numbers.
At some time previously there was a bug in SCM that required a company registration field to be non-null when entered. This led to a pattern of behaviour where customers entering data to be validated would put zeroes into this field to mean both ‘there is no registration number’ and ‘I don’t know the registration number’. Furthermore, our validators had picked up the concept that this field could not be blank and were used to seeing ‘0’ entries there for Government Entities. This led over time to the validators becoming essentially blind to the presence of zeroes in the company number field.
We do not present this information to blame our subscribers or our validators for these errors, but merely to explain the sequence of events that led to their occurrence. We have automated internal policy checks and we use the external lint checkers and at least our internal policy checks should have been updated to catch this error of zeroes appearing in subject:serialNumbers long ago.
There were 1762 certificates for 91 different organization names in 8 enterprise accounts in SCM that contain a subject:serialNumber containing only zeroes.
There were also 209 certificates for 148 organization names in 57 accounts not in SCM that contain a subject:serialNumber containing only zeroes.
- Misspellings of State names, invalid state names, and incorrect use of abbreviations in JOI:states
These exhibit the difficulty of precise text comparison and text rule implementation (e.g. no abbreviations) by human validators. The fact that these errors made it through to our issued certificates is due to us not yet having incorporated automated checking and provision of state names into the data entry and policy checking modules. We had planned to have been using the SIM tool by now but the tool remains in a Beta test state.
- Incorrect JOI
These are data entry errors by validators after the QGIS lookups are done. A validator looks for data in one QGIS but does not find it, then they look in another QGIS and find it. The enter the registration number from the 2nd QGIS but the JOI from the 1st.
2nd approval should have caught this every time.
- Invalid JOI
Some of these are data entry errors and others are misunderstandings of the level at which the JoI operated.
- subject:serialNumber is a date instead of a number
There was a misunderstanding by some of our validators over the precedence rules around the use of registration numbers and dates with private organizations.
- Business Category wrong
This certificate should not have been issued. The category is wrong, but the registration number is also absent (0).
- Company Registration Numbers wrong (other than zero)
There were a number of cases where digits had been omitted from registration numbers. These are generally copied and pasted or provided automatically so there should be no scope for such errors.
- JOI wrong, subject registered in two jurisdictions.
The validation of these 4 certificates for one organization became confused and the subject ends up being a mixture of the details from the two registrations.
We were initially puzzled by why our regular self-audits of the issued EV certificates had not found these problems before, especially the issue with subjectSerialNumbers containing only zeroes.
On investigation we found that self-audit samples had revealed single certificates with subject:serialNumber=0 on more than one occasion. The auditor had followed up with the individual validators responsible in each case for further training around the issue, but neither the systems problem that led to the zeroes nor the wider acceptance of the zeroes amongst other validators were identified as an issue.
- List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.
We have made a number of personnel changes among our Validators and 2nd Approval Validators. Although the tools we use are being improved, there are a number of cases where the performance of individual validators fell below what was required. We do not require perfection of our validators, and we have training and retraining schemes in place.
We have introduced an adversarial scheme of 2nd approval where the validator and the 2nd approval validator do not meet and may not discuss the certificate subject data except through official channels (tickets & validation notes). We have tightened up the 2nd approval process and increased the resource available for 2nd approval so that more careful and precise checks will be made.
On 23/06/2019 we introduced a code change that improved the treatment of meta data in subject fields and this also blocked the issuance of certificates with subject:serialNumbers containing only zeroes.
As a result of these reports we instigated an urgent development task to introduce an automated system of checks on states in EV subjects and JOI that catches the state name errors here. The first phase of this which, for the most common jurisdictions, blocks invalid state names in our policy checking module went into production at 30/08/2019 13:24. We have since extended and will further extend the list of jurisdictions over which this is effective but this is a halting process since the ISO 3166-2 data is not always a perfect fit for real-world requirements as has been discussed elsewhere.
We will redouble our efforts to integrate the SIM tool with our issuance and validation flow so that we may reap the benefit of the considerable investment we have made in SIM.
As an immediate term step, we are improving our documentation and training over JoI structure and selection. We will develop as a matter of urgency a rule-based JoI policy checker that will enforce correct usage and seek additional human approval when a rule-based approach is inappropriate. This will remain useful even after the integration of our SIM tool is complete.
We have not yet completed scanning our systems and data for other instances of these same failures. We will post to this ticket when we have done so.
We thought our self audit process had been working OK, but we have evidently been failing to capture vital information. We will have a formal process around the output of the self-audit where a report of any variance from the ideal for subject information will be noted in the report and that report will be discussed in a meeting with validation, compliance, and development with recorded minutes where the action points will be enumerated and allocated to executives for implementation.