- How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.
At approximately 17:00 UTC on August 10 2020, we received a report from a third party indicating that Entrust had issued certificates with invalid state/province data.
After further examination, we were able to confirm that the third party assessment was accurate and that there were multiple OV SSL certificates issued with incorrect state/province information for a total of 3 organizations. A total of 397 certificates are impacted, with 395/397 certificates issued to one large international organization.
- A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.
We are still gathering information and developing action plans. This is what we have so far:
10 August 2020: A notification was received from a third party with a list of certificates with potentially invalid state/province data.
10 August 2020: An investigation is started and the issue is confirmed. A preliminary root cause is determined (detailed in section 6) and it appears that there are no other certificates being issued with invalid state/province at this time
11 August 2020: Certificate population is checked against the list of certificates that were provided and we confirm that a number of certificates that were in the initial third party report have already been revoked as part of regular certificate lifecycle and replacement. It is confirmed that a total of 397 certificates are impacted, 395 of which are for a single organization, and 2 other certificates associated with 1 organization each.
11 August 2020: The 3 impacted organizations are notified that their certificates must be revoked within 5 days of the incident being reported to us. A deadline of 15 August 2020 17:00 UTC has been set.
11 August 2020: After a discussion with the large customer who has 395 out of 397 certificates to revoke, they informed us that they will need a day or two to communicate the issue internally across their global IT organization and that it was unlikely that they would be able to replace 395 certificates by the deadline and that forced revocation would have a serious impact on their critical infrastructure. We will file a separate incident that details their situation and the plan they are committing to replacing these certificates as quickly as possible.
- Whether your CA has stopped or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.
As part of our initial investigation, it appears that the last certificate issued with incorrect state/province data was on 6 September 2019. As described in 6, the main root cause for the large customer issue was fixed in June 2019. We will be taking additional steps to remove the future possibility of human error that caused these issues(section 7).
- A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.
A list of problematic certificates is attached. The first certificate in the list was issued on 7 August 2018. The last certificate was issued 17 June 2019.
A total of 397 OV SSL certificates were impacted.
- The complete certificate data for the problematic certificates.
See the attached certificate list with corresponding thumbprints and crt.sh links.
- Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
In our vetting system, the state field is currently a text input field where the vetting agent can enter the state.
The first problematic retail certificate used st=NA, where the vetting agent entered “NA” (Not Applicable) for a country where there are no states. We have strict procedures that require that the state field be left blank when a country does not have states or provinces and our regular 2 person verification check would normally catch this.
The second problematic retail certificate used st=Slovakia, where the vetting agent appears to have accidentally copied the full country name from the source verification document that was found during the business verification process. One thing to note that we have received multiple requests from this organization over time and the same agent did not make this mistake in the past.
The 3rd case involves a large Enterprise customer who issues many certificates based on a pre-verified organization profile. To explain what happened, we need to go back to August 2018. At this time, we used a different verification system with a different user interface. At the time, the organization contact “State” field was set to “NA”, as the contact country does not have any state/provinces. In our older verification system, the contact state, locality, country, and organization were what set the certificate profile. A year later in June 2019, the organization profile was updated and the certificate state field was left blank and fixed. 6 months earlier in January 2019, we switched to the new verification system where there is a clear interface that shows the organization profile and we longer pull the organization name state, locality, and country from the organization contact fields. In June 2019, the state field was left blank by the vetting agent in the organization profile section, which corrected the state field on all certificates for that organization profile moving forward. Please note that there is another incident that we filed back in 2018 that was related to the organization's contact fields being updated and inadvertently changing the certificate profile - https://bugzilla.mozilla.org/show_bug.cgi?id=1535735 .
The system changes we have made in January 2019 to update our Enterprise verification interfaces will prevent these types of issues from happening again in the future as it relates to making inadvertent changes to the organization profiles that are set for our customers based on the pre-verified data. In 7 section, we will discuss ways that we can eliminate issues with human error when entering state/provinces for our customers.
- List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.
In addition to the system interface changes that have already been made in 2019, we are going to implement checks and system changes based on what other CAs reported who have experienced similar issues:
- Enhance our verification system auto-populate the state field based on the selected country and restrict any other values from being entered by validation agents or customers.
- We will be doing a wider check of our certificate population to check for other uses of “NA”, “N/A”, “Not Applicable and we will also check state/province and country combinations based on the current ISO state/province country lists that are available. This will also allow us to check for any other potential issues, such as spelling mistakes.
- Add checks to our linting software to check for st and l values for “na”, “n/a”, “not applicable”
We will update the timelines to run the scan and implement the text field changes as we continue to investigate and plan our releases.