Entrust: Invalid data in State/Province Field
Categories
(CA Program :: CA Certificate Compliance, task)
Tracking
(Not tracked)
People
(Reporter: dathan.demone, Assigned: dathan.demone)
Details
(Whiteboard: [ca-compliance] [ov-misissuance])
Attachments
(2 files)
User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:79.0) Gecko/20100101 Firefox/79.0
Actual results:
It was discovered that a number of certificates included invalidate state/province data.
|
Assignee | |
Comment 1•4 years ago
|
||
- How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.
At approximately 17:00 UTC on August 10 2020, we received a report from a third party indicating that Entrust had issued certificates with invalid state/province data.
After further examination, we were able to confirm that the third party assessment was accurate and that there were multiple OV SSL certificates issued with incorrect state/province information for a total of 3 organizations. A total of 397 certificates are impacted, with 395/397 certificates issued to one large international organization.
- A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.
We are still gathering information and developing action plans. This is what we have so far:
10 August 2020: A notification was received from a third party with a list of certificates with potentially invalid state/province data.
10 August 2020: An investigation is started and the issue is confirmed. A preliminary root cause is determined (detailed in section 6) and it appears that there are no other certificates being issued with invalid state/province at this time
11 August 2020: Certificate population is checked against the list of certificates that were provided and we confirm that a number of certificates that were in the initial third party report have already been revoked as part of regular certificate lifecycle and replacement. It is confirmed that a total of 397 certificates are impacted, 395 of which are for a single organization, and 2 other certificates associated with 1 organization each.
11 August 2020: The 3 impacted organizations are notified that their certificates must be revoked within 5 days of the incident being reported to us. A deadline of 15 August 2020 17:00 UTC has been set.
11 August 2020: After a discussion with the large customer who has 395 out of 397 certificates to revoke, they informed us that they will need a day or two to communicate the issue internally across their global IT organization and that it was unlikely that they would be able to replace 395 certificates by the deadline and that forced revocation would have a serious impact on their critical infrastructure. We will file a separate incident that details their situation and the plan they are committing to replacing these certificates as quickly as possible.
- Whether your CA has stopped or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.
As part of our initial investigation, it appears that the last certificate issued with incorrect state/province data was on 6 September 2019. As described in 6, the main root cause for the large customer issue was fixed in June 2019. We will be taking additional steps to remove the future possibility of human error that caused these issues(section 7).
- A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.
A list of problematic certificates is attached. The first certificate in the list was issued on 7 August 2018. The last certificate was issued 17 June 2019.
A total of 397 OV SSL certificates were impacted.
- The complete certificate data for the problematic certificates.
See the attached certificate list with corresponding thumbprints and crt.sh links.
- Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
In our vetting system, the state field is currently a text input field where the vetting agent can enter the state.
The first problematic retail certificate used st=NA, where the vetting agent entered “NA” (Not Applicable) for a country where there are no states. We have strict procedures that require that the state field be left blank when a country does not have states or provinces and our regular 2 person verification check would normally catch this.
The second problematic retail certificate used st=Slovakia, where the vetting agent appears to have accidentally copied the full country name from the source verification document that was found during the business verification process. One thing to note that we have received multiple requests from this organization over time and the same agent did not make this mistake in the past.
The 3rd case involves a large Enterprise customer who issues many certificates based on a pre-verified organization profile. To explain what happened, we need to go back to August 2018. At this time, we used a different verification system with a different user interface. At the time, the organization contact “State” field was set to “NA”, as the contact country does not have any state/provinces. In our older verification system, the contact state, locality, country, and organization were what set the certificate profile. A year later in June 2019, the organization profile was updated and the certificate state field was left blank and fixed. 6 months earlier in January 2019, we switched to the new verification system where there is a clear interface that shows the organization profile and we longer pull the organization name state, locality, and country from the organization contact fields. In June 2019, the state field was left blank by the vetting agent in the organization profile section, which corrected the state field on all certificates for that organization profile moving forward. Please note that there is another incident that we filed back in 2018 that was related to the organization's contact fields being updated and inadvertently changing the certificate profile - https://bugzilla.mozilla.org/show_bug.cgi?id=1535735 .
The system changes we have made in January 2019 to update our Enterprise verification interfaces will prevent these types of issues from happening again in the future as it relates to making inadvertent changes to the organization profiles that are set for our customers based on the pre-verified data. In 7 section, we will discuss ways that we can eliminate issues with human error when entering state/provinces for our customers.
- List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.
In addition to the system interface changes that have already been made in 2019, we are going to implement checks and system changes based on what other CAs reported who have experienced similar issues:
- Enhance our verification system auto-populate the state field based on the selected country and restrict any other values from being entered by validation agents or customers.
- We will be doing a wider check of our certificate population to check for other uses of “NA”, “N/A”, “Not Applicable and we will also check state/province and country combinations based on the current ISO state/province country lists that are available. This will also allow us to check for any other potential issues, such as spelling mistakes.
- Add checks to our linting software to check for st and l values for “na”, “n/a”, “not applicable”
We will update the timelines to run the scan and implement the text field changes as we continue to investigate and plan our releases.
|
||
Updated•4 years ago
|
|
|
Assignee | |
Comment 2•4 years ago
|
||
We ran a full scan on our certificate population and did not find any other certificates where the state was set to "NA", N/A", or Not Applicable.
We will continue to provide updates as we work through revocation and the items listed in section 7.
|
|
Assignee | |
Comment 3•4 years ago
|
||
The 2 certificates that did not belong to the large organization were revoked today. There are still 395 certificates to be revoked. A separate incident (https://bugzilla.mozilla.org/show_bug.cgi?id=1658794) has been filed to track certificates that will not be revoked on time, as stated in our previous comment in section 2 of our initial post.
|
|
Assignee | |
Comment 4•4 years ago
|
||
As of 18 August 19:00 UTC, 383 certificates have not yet been revoked. We have set a deadline of 5 September 2020 to revoke the remaining certificates.
We are currently exploring different designs to ensure that the correct state value is set based on the country. We are also working with our development team to make additional changes to our linting software that was mentioned in section 7 of the initial incident report. We will provide updated details and timelines as we work through our design and release process.
|
|
Assignee | |
Comment 5•4 years ago
|
||
There are currently 328 certificates to be revoked, as per the late revocation incident.
We are still working on finalizing a design for our software changes. We should be able to provide this plan along with a release date in the next couple of weeks.
|
|
Assignee | |
Comment 6•4 years ago
|
||
Our development team has begun design on software changes that will replace the current state/province text input field in our verification system with a drop-down based on the country that is selected. We will pull a list of states/provinces from a third party geographical database. This drop-down selection will ultimately control the ‘ST’ value that is set in the certificate DN. If there are no states/provinces available for the selected country, the state field drop-down will not provide an option to the verification user and the issued certificate will not include an ‘ST’ value. These changes will ensure that verification users do not accidentally enter a state when the country does not have any states/provinces or enter a state that does not correspond with the selected country. These software changes will be applied to both our retail and Enterprise customer verification screens.
In addition, we will also be making changes to our certificate linting too to check for state values such as ‘na’, ‘n/a’, and ‘not applicable’.
Based on our current release schedule, these changes will go into effect in our next release that is scheduled for February 2021.
There are 21 certificates left to be revoked. These will be revoked on or before 5 September 2020.
|
|
Assignee | |
Comment 7•4 years ago
|
||
Just a quick update that the 21 remaining certificates were revoked on 4 September 2020, as mentioned our last comment in Bug 1658794
There are no other updates at this time.
|
||
Comment 8•4 years ago
|
||
We have strict procedures that require that the state field be left blank when a country does not have states or provinces and our regular 2 person verification check would normally catch this.
Did I overlook a more detailed analysis here as to the root cause? I see the third case discussed somewhat at length, but this first case, as best I can tell, is mostly attributed to human error? Have I overlooked anything?
Related, can you discuss whether Entrust performs any false-positive testing of secondary reviewers? That is, send approvals to a secondary set of eyes that have known issues, to simulate them making it past the first reviewer, to ensure that they're detected, or to test the effectiveness of existing controls?
|
|
Assignee | |
Comment 9•4 years ago
|
||
Ryan - The root cause for the first case is a combination of human error and our system allowing agents to enter any value for the state/province text field. As mentioned in our earlier posts, we are planning to update our vetting systems to allow agents to select a state/province value from a drop-down where the values are populated based on the country that is selected.
You also asked if Entrust does any false-positive testing of secondary reviewers. At this time, Entrust does not perform this type of testing on a regular basis. We only perform this type of testing when a secondary reviewer is being onboarded into their role to make sure that they are able to properly detect issues with the verification process. After further internal discussion, we like your suggestion of conducting false-positive tests on a regular basis and will implement a process in the near future to support this. We will target January 2021 to have this process fully implemented across our global teams. We will implement tests based on previous incidents that Entrust and other CAs have reported that relate to the validation process.
One other item that we wanted to call out is our monthly review process that is conducted by our secondary reviewers with our first level approvers. The purpose of these reviews is to help our teams learn from any mistakes that were detected by the secondary reviewers and to continuously improve our knowledge of the BRs and EV Guidelines. Note that our secondary approvers are tenured members of our verification organization who have spent years as first-level approvers before they are promoted to secondary approvers and they are well versed in CA/Browser forum policies. When a secondary approver notices an issue with the vetting information provided by the first level approver, it is sent back to the first level approver so they can address the problems that were found. At the end of each month, the secondary approvers conduct monthly one on one sessions with each first level approvers to review the verification requests that were sent back to make sure they are learning from their mistakes. This is a big part of our compliance program and I wanted to make sure that this was called out as part of our current best practices.
|
|
Assignee | |
Comment 10•4 years ago
|
||
We received a report from a third party via our incident reporting system yesterday morning of a single certificate that was issued with incorrect state data - https://crt.sh/?id=2663086686 . The report was received on 26 September at 10:04 am UTC. After further review, it appears that the state and city information was entered into our system in reverse order. The ST field was set to “North Sydney” instead of “New South Wales” and the L field was set to “New South Wales” instead of “North Sydney”. The vetting information that was collected clearly showed that the state should have been set to New South Wales and that this was the intended value. After further analysis, this particular case appears to the same root cause as the other cases reported in this incident, and future mis-issuances of this type should be prevented by the system updates we are planning to make. The certificate is scheduled to be revoked before the 5-day deadline.
|
|
||
Comment 11•4 years ago
|
||
Dathan: It sounds like this new incident is completely unrelated to those explanations offered in Comment #1. It also suggests that a holistic evaluation of existing certificates wasn't done (per your commitment to do exactly that on Comment #1). Given that commitment was made on 2020-08-12 to systemically evaluate and address this, it seems like it both could and would have been accomplished by now, so I'm concerned that Comment #10 highlights third-parties are still finding issues.
Specifically, the commitment made in Comment #1 was
we will also check state/province and country combinations based on the current ISO state/province country lists that are available
I understand that Comment #2 stated:
We ran a full scan on our certificate population and did not find any other certificates where the state was set to "NA", N/A", or Not Applicable.
But that's not all that you committed to.
|
|
Assignee | |
Comment 12•4 years ago
|
||
We have not yet run the scan that I previously committed to using an ISO list to check our current certificate population. We recently purchased a list of ISO state/province values and plan to use that in our system enhancements that have already been detailed. Our development team has been working on a query that will check our entire certificate population against this same list. If there is a certificate in our population where the ST value does not match the values included in the ISO list, they will be included in a report that will be reviewed by the verification and compliance teams to check the results and confirm any additional problem certificates. Unfortunately, we have not been able to make the required changes to our production databases to run the query due to a planned database upgrade. I just received an update that we should be able to run the query in the next 3 weeks (at most) unless there is some unexpected issue. I will provide weekly updates as we work through our scan and review process.
|
|
Assignee | |
Comment 13•4 years ago
|
||
We are targeting to be able to complete the required database changes to run our scans against the ISO list early in the week of October 19th. We will provide an update on the results at that time.
|
|
Assignee | |
Comment 14•4 years ago
|
||
|
|
Assignee | |
Comment 15•4 years ago
|
||
We completed a scan of our certificate population and a full review of all of the certificates that were flagged as part of the scan. We started by comparing our certificate population against the state/province values included in the ISO 3166-2 list. As expected, our report included many false positives that we had to review. For example, the ISO list includes many translated state/province names such as “Roma” or “Genève” where we had used the English translation of the value based on the business registration documents we had collected.
Based upon this methodology, our scan and subsequent review process found 145 certificates with what we considered to be invalid states. I have attached a list of crt.sh links to this incident. We will work to have all of these certificates revoked within the 5 day period, as per the BRs. We completed our review on 20 October 2020 at approximately 20:00 UTC. We will set a deadline to revoke all of the certificates included in the newly attached list by 25 October 2020 20:00 UTC.
In summary, most of the issues were typos where the intended value was spelled incorrectly (Maharastra instead of Maharashtra / Guateng instead of Gauteng were two common examples). There were some other issues with state/province and country mismatches, abbreviated state/province values, and other copy/paste errors.
One thing we would like to note based on the review we just completed is that it does not appear that using an ISO list exclusively for state/province values will solve all of our problems. We found many examples where the proper business registration data for a Subscriber does not use ISO state/province values. We are going to further analyze the results of this scan to develop a solution where we use the ISO values as a foundation, but also have the ability to include other properly reviewed or whitelisted state/province values as well based on what we are verifying and the data we obtain.
|
|
Assignee | |
Comment 16•4 years ago
|
||
All 145 certificates that were identified last week to have invalid state/province values were revoked before the 25 Octoboer 2020 deadline.
We will provide further updates on our plan to address the issue systematically with a state/province drop down list.
|
|
Assignee | |
Comment 17•4 years ago
|
||
We ran another scan/review today for certificates that were verified over the last 3 weeks (since our last review) to make sure that no new certificates have been issued with incorrect stateOrProvince values. We did not find any new certificates with incorrect values. We will continue to run bi-weekly scans on certificates that have been issued since our previous scan/review until we fully implement the drop-down list, which is still targeted for the new year.
|
|
Assignee | |
Comment 18•4 years ago
|
||
We are continuing to run biweekly scans to check for any new issues. Our last 2 scans/reviews were conducted on 23 November 2020 and 8 December 2020. We have not found any new certificates that have been issued with incorrect state/province values. We will continue to run these scans/reviews until we have implemented our drop-down solution for state/province values in our vetting systems.
|
|
Assignee | |
Comment 19•4 years ago
|
||
I just wanted to provide an update that we continue to run biweekly scans and have not found any new issues.
I also wanted to provide an update on the use of ISO state values in our vetting system to control the ST field in the business profile. This system change was originally expected to hit our production systems in February, however, this release has been delayed until April.
|
|
||
Comment 20•4 years ago
|
||
Ben: I think I'm OK closing this, unless you wanted to keep open through April. It sounds like most of the substantive changes have been implemented, with the expectation that Entrust will continue making regular improvements.
|
|
||
Comment 21•4 years ago
|
||
I will leave this open with a Next Update set for 1-Apr-2021 for a status on the system update.
|
|
||
Updated•4 years ago
|
|
|
Assignee | |
Comment 22•3 years ago
|
||
We found 6 more certificates with incorrect stateOrProvince values during one of our scans. These certificates were discovered on April 12 16:00 UTC and will be revoked before April 17 16:00 UTC.
https://crt.sh?sha256=047da9ade705610f73fc3f03a830e554ebe1188c017d6b15e42932a9b9783b74
https://crt.sh?sha256=e873907b6159d0a7c5fb64d986515f68234df058c64361c35b9329960d75197e
https://crt.sh?sha256=cdcad674f83fd6fdda3701b848520bdfc8c136ecf925ffb3f759d93f7d758305
https://crt.sh?sha256=674a25b80785dd587ebf95a30ef572176ad36c6982f7f3c55ccde1be5958e0b8
https://crt.sh?sha256=3113533b69d94e261cd3b5b5eae2ef602ff432fb706d358b23868ffae4611b23
https://crt.sh/?sha256=3525e1d93b3852411990f8651c470b320d292da5a8b2849efd7cfac666843b40
The system update to implement ISO 3166-2 subdivisions as drop-down values in our system has been delayed and is now scheduled for release on May 12. I will provide additional updates if anything changes.
|
||
Comment 23•3 years ago
|
||
At least two of these certificates have been issued after this bug was opened, so what has Entrust done to prevent further misissuance before the drop-down menu is shipped? Surely if Entrust cannot ensure compliance it should have stopped issuance?
https://crt.sh/?sha256=cdcad674f83fd6fdda3701b848520bdfc8c136ecf925ffb3f759d93f7d758305
https://crt.sh/?sha256=3113533b69d94e261cd3b5b5eae2ef602ff432fb706d358b23868ffae4611b23
|
|
Assignee | |
Comment 24•3 years ago
|
||
To prevent possible compliance issues ahead of our ability to deliver the drop-down driven by ISO 3166-2, we use a two-person verification and final approval process even when this is not required by the BRs for OV SSL. However, since this is a human-based/manual process, there can always be mistakes (which we are going to solve through system enhancements in the near future). To mitigate the risk of human error and to find potential issues ourselves, we have been running biweekly scans to flag any verifications over that two-week period that resulted in state/province values in the vetting record that were not part of the ISO 3166-2 list. These scans always result in many false positives, as there are mismatches due to translations, accents, punctuation, and other factors. As part of the review process, someone has to look at the verification evidence files to ensure there is evidence that supports our decision to use a particular value, as the BRs require us to verify the address information but do not currently require that state/province values be based on ISO 3166-2 or any other official list.
In this particular case, we found 6 certificates that had invalid state/province values that were actually based on 4 total mistakes. 3 of these were spelling errors resulting in 5 miss-issued certificates. In these cases, the state/province values were verified correctly but just misspelled.
In the lead-up to standardizing our state/province values based on ISO 3166-2, we have spent a significant amount of time and resources running biweekly scans and reviews on everything that is being flagged as non-ISO compliant. We believe that we are making continuous improvements and demonstrating transparency as we do so, as is expected by any CA.
|
|
Assignee | |
Comment 25•3 years ago
|
||
We are still on track to deliver the system changes on May 12th
|
|
Assignee | |
Comment 26•3 years ago
|
||
The ISO 3166-2 state/province dropdowns are now live in our system and will be used for all verifications moving forward.
|
|
Assignee | |
Comment 27•3 years ago
|
||
Two more certificates were reported to us today that have typos in the stateOrProvince field:
https://crt.sh/?id=3365653722
https://crt.sh/?id=3365650261
Note that these two certificates were issued as a result of a single mistake that was made in the certificate profile that was verified in September 2020.
These certificates will be revoked on or before Monday, May 24 15:32 UTC in accordance with the BRs.
|
|
||
Comment 28•3 years ago
|
||
Didn't Entrust retroactively scan all certificates against this new list? So is this value included in this new allowlist system or was there an issue in scanning that missed these certificates and ultimately led to a third party reporting them?
|
|
Assignee | |
Comment 29•3 years ago
|
||
The value that is included in our allow list for this state is "Ciudad de México".
These two certificates were missed during our previous reviews. We had many false positives that were flagged for stateOrProvince = Ciudad de Mexico (missing the accent in Mexico) that likely resulted in us missing these two certificates during our review.
|
|
Assignee | |
Comment 30•3 years ago
|
||
I wanted to provide an update that we also added a new check to our post certificate issuance linting software on May 13 that will check all of our certificate stateOrProvince values against the same list of values we implemented in our drop-down menu based on ISO 3166-2. Any certificate with a stateOrProvince value that does not match our allow list will be flagged for review. Since this was implemented on May 13, we have not found any certificates with an invalid stateOrProvince value.
|
|
||
Comment 31•3 years ago
|
||
I intend to close this bug on or about Friday, 4-June-2021, unless there are unresolved issues that remain.
|
|
||
Updated•3 years ago
|
|
||
Updated•2 years ago
|
|
||
Updated•1 year ago
|
Description
•