1575022 - Sectigo: EV SSL Certificates with incorrect subject details.

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Robin Alden]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36

Steps to reproduce:

We have received over the last few days a number of problem reports identifying EV SSL certificates issued by Sectigo where either the Jurisdiction of Incorporation (JoI) or the subject:serialNumber (i.e. the registration number for the subject organization) appear to be wrong.
We are dealing with these reports and revoking affected certificates as soon as is practicable.
We will follow up here with a full incident report shortly.

Comment 1

[Robin Alden]

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

By problem reports to [e-mail]:
Times are in BST (= EDT + 5) (dd/mm/yyyy hh24:mi)
#certs is the number of distinct Sectigo issued certificates in each report.

Report#	Received at	#certs	Issue
1	Fri 16/08/2019 23:18	1	subject:serialNumber wrong. JOI=SE, but subject:serialNumber is for a DK entity.
2	Sat 17/08/2019 21:53	15	subject:serialNumber wrong. JOI:country=SE should have subject:serialNumber of 10 digits, but subject:serialNumber is not 10 digits
3	Mon 19/08/2019 01:45	1	businessCategory is Private Organization, but should be Government Entity. Also serialNumber=0
4	Mon 19/08/2019 02:00	1	serialNumber=00
5	Mon 19/08/2019 02:05	1	serialNumber=0000
6	Mon 19/08/2019 02:26	11	serialNumber seems to be 0
7	Tue 20/08/2019 03:41		2 problems:
		6	Invalid JOI:stateOrProvince
		102	Invalid subject:stateOrProvince
8	Tue 20/08/2019 18:46	1	JOI:locality & JOI:stateOrProvince wrong.
9	Tue 27/08/2019 16:53	1963	Subject:serialNumber of '0' or some combination of '0's

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

date	event
23/06/2019	Code deployed which, as a side effect, blocks all-zero entries in subject:serialNumbers
16/08/2019 23:18 thru 27/08/2019 16:53	Problem reports received
17/08/2019 13:30	Analysis of problem reports begins
18/08/2019 21:45	Revocation of identified certificates begins
22/08/2019 17:00	A distinct 2nd approval team was established
30/08/2019 13:24	Code deployed to automatically check stateOrProvinceName for the most common Jurisdictions.
01/09/2019 13:00	Revocation of identified certificates concludes

Not all of the certificates were revoked within 24 hours of being reported. Our goal and intent was that all of the certificates were to have been revoked within 5 days of receiving the reports however, regrettably, the 6 certificates with Invalid JOI:stateOrProvince field values reported to us at 20/08/2019 03:41 took nearly 8 days to revoke due to an administrative error. All of the other reported certificates were revoked (or expired) within 5 days of receiving the relevant report.

3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

We have stopped issuing certificates with these problems.

4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

Counts of affected certificates for which we have received reports.

Problem	#certs	dates
JOI (jurisdiction of incorporation) contains a state or a locality when it should not because the registration scheme is at the country or state level.	8	30-Aug-17 to 30-Oct-18
Subject:serialNumber omits digits	7	28-Nov-17 to 23-Nov-18
Subject:serialNumber is a date when it should be a registration number	3	05-Jan-18 to 05-Dec-18
JOI is wrong. Subject is registered in both SE and DK. The organizationName and the subject:SerialNumber are for DK, but the JOI says SE	4	23-May-18 to 05-Oct-18
JOI is wrong. subject:Country and JOI:Country are both SE instead of NL	1	04-Jan-19
JOI is wrong. should be SK, not US	1	29-Nov-17
businessCategory is wrong. It shows Private Organization when it should be Government Entity	1	20-Feb-18
Subject:serialNumber consists of only zeroes	1970	30-May-17 to 21-Jun-19
Subject:State wrong - Confusion over the status of Puerto Rico & US Virgin Islands	2	16-Jun-17 to 03-May-19
Subject:State wrong - City listed as state. (e.g. Manhattan, Las Vegas..)	2	13-Sep-17 to 16-Jan-18
Subject:State wrong - State misspelled. (e.g. Deleware, Marryland, NewYork..)	87	08-Jan-18 to 08-Aug-19
Subject:State wrong - irrelevant text present	2	27-Oct-18 to 17-Dec-18
Subject:State wrong - State and locality reversed	1	23-Jul-19
Subject:Country wrong: US instead of OM	1	29-Jun-18
jurisdictionStateOrProvinceName wrong - misspelled. (e.g. NewYork, Deleware)	4	09-Aug-17 to 25-Jun-19
jurisdictionStateOrProvinceName wrong - irrelevant text present	2	30-Aug-17 to 12-Sep-18
jurisdictionLocality - irrelevant text present	1	12-Sep-18

10 certificates have 2 of the above problems, so the total number of certificates affected is 2090.

152 of the above certificates had been revoked before the reports were received.

5. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

Links to problematic certificates

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

There are a number of separate issues occurring here and they do not split out distinctly across the problem reports listed above which is why we are dealing with them all in one incident report.

Systems involved:

Sectigo Certificate Manager (SCM).

SCM is an Enterprise RA platform written and hosted by Sectigo. It allows Enterprise customers to enter Organization Details and certificate requests for domain names and, after an initial validation of the Organizational Details and domain names to the requirements of our CPS (and the BRs and the EVGLs), the Enterprise customer may then order further certificates for the same details without repeating the validation steps for as long as the guidelines permit the validation information to be re-used.

Validators and 2nd Approval validators

We still rely substantially on human validators for the validation of OV and EV subject details, and also for the EV 2nd approval step.

Sectigo Information Manager (SIM).

SIM is a data matching tool and Jurisdiction database that we have been working on for the last 18 months. We hoped to have integrated this tool fully into our issuance pipeline before now, but it is still in a Beta test.

Mistakes/bugs:

1. Zero serial numbers.

At some time previously there was a bug in SCM that required a company registration field to be non-null when entered. This led to a pattern of behaviour where customers entering data to be validated would put zeroes into this field to mean both ‘there is no registration number’ and ‘I don’t know the registration number’. Furthermore, our validators had picked up the concept that this field could not be blank and were used to seeing ‘0’ entries there for Government Entities. This led over time to the validators becoming essentially blind to the presence of zeroes in the company number field.
We do not present this information to blame our subscribers or our validators for these errors, but merely to explain the sequence of events that led to their occurrence. We have automated internal policy checks and we use the external lint checkers and at least our internal policy checks should have been updated to catch this error of zeroes appearing in subject:serialNumbers long ago.

There were 1762 certificates for 91 different organization names in 8 enterprise accounts in SCM that contain a subject:serialNumber containing only zeroes.

There were also 209 certificates for 148 organization names in 57 accounts not in SCM that contain a subject:serialNumber containing only zeroes.

2. Misspellings of State names, invalid state names, and incorrect use of abbreviations in JOI:states

These exhibit the difficulty of precise text comparison and text rule implementation (e.g. no abbreviations) by human validators. The fact that these errors made it through to our issued certificates is due to us not yet having incorporated automated checking and provision of state names into the data entry and policy checking modules. We had planned to have been using the SIM tool by now but the tool remains in a Beta test state.

3. Incorrect JOI

These are data entry errors by validators after the QGIS lookups are done. A validator looks for data in one QGIS but does not find it, then they look in another QGIS and find it. The enter the registration number from the 2nd QGIS but the JOI from the 1st.
2nd approval should have caught this every time.

3. Invalid JOI

Some of these are data entry errors and others are misunderstandings of the level at which the JoI operated.

4. subject:serialNumber is a date instead of a number

There was a misunderstanding by some of our validators over the precedence rules around the use of registration numbers and dates with private organizations.

5. Business Category wrong

This certificate should not have been issued. The category is wrong, but the registration number is also absent (0).

6. Company Registration Numbers wrong (other than zero)

There were a number of cases where digits had been omitted from registration numbers. These are generally copied and pasted or provided automatically so there should be no scope for such errors.

7. JOI wrong, subject registered in two jurisdictions.

The validation of these 4 certificates for one organization became confused and the subject ends up being a mixture of the details from the two registrations.

8. Self-Audit

We were initially puzzled by why our regular self-audits of the issued EV certificates had not found these problems before, especially the issue with subjectSerialNumbers containing only zeroes.

On investigation we found that self-audit samples had revealed single certificates with subject:serialNumber=0 on more than one occasion. The auditor had followed up with the individual validators responsible in each case for further training around the issue, but neither the systems problem that led to the zeroes nor the wider acceptance of the zeroes amongst other validators were identified as an issue.

7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

We have made a number of personnel changes among our Validators and 2nd Approval Validators. Although the tools we use are being improved, there are a number of cases where the performance of individual validators fell below what was required. We do not require perfection of our validators, and we have training and retraining schemes in place.

We have introduced an adversarial scheme of 2nd approval where the validator and the 2nd approval validator do not meet and may not discuss the certificate subject data except through official channels (tickets & validation notes). We have tightened up the 2nd approval process and increased the resource available for 2nd approval so that more careful and precise checks will be made.

On 23/06/2019 we introduced a code change that improved the treatment of meta data in subject fields and this also blocked the issuance of certificates with subject:serialNumbers containing only zeroes.

As a result of these reports we instigated an urgent development task to introduce an automated system of checks on states in EV subjects and JOI that catches the state name errors here. The first phase of this which, for the most common jurisdictions, blocks invalid state names in our policy checking module went into production at 30/08/2019 13:24. We have since extended and will further extend the list of jurisdictions over which this is effective but this is a halting process since the ISO 3166-2 data is not always a perfect fit for real-world requirements as has been discussed elsewhere.

We will redouble our efforts to integrate the SIM tool with our issuance and validation flow so that we may reap the benefit of the considerable investment we have made in SIM.

As an immediate term step, we are improving our documentation and training over JoI structure and selection. We will develop as a matter of urgency a rule-based JoI policy checker that will enforce correct usage and seek additional human approval when a rule-based approach is inappropriate. This will remain useful even after the integration of our SIM tool is complete.

We have not yet completed scanning our systems and data for other instances of these same failures. We will post to this ticket when we have done so.

Self audit.

We thought our self audit process had been working OK, but we have evidently been failing to capture vital information. We will have a formal process around the output of the self-audit where a report of any variance from the ideal for subject information will be noted in the report and that report will be discussed in a meeting with validation, compliance, and development with recorded minutes where the action points will be enumerated and allocated to executives for implementation.

Comment 2

[Wayne Thayer]

Robin: thank you for providing this comprehensive incident report. It describes a number of remediation steps, but no timeline for completing them. That is particularly concerning because Sectigo has a history of not providing regular updates to incident bugs, even when an update is specifically requested. Please provide a remediation timeline here and update this bug at least weekly unless a "Next Update" date is added to the "whiteboard" field.

Also, please explain the "administrative error" that delayed revocation of some certificates, and the remediation for that problem.

Comment 3

[Jeremy Rowley]

I have a question about this as well. Does this cover where there is an abbreviated state field (https://censys.io/certificates/f0d1ca1c2a7c705bd791487484a7921351d93170b5c955734501e02cc1999df5)? I couldn't tell if those errors were captured in the above. Per section 9.2.4 the full name of the state/locality is required.

Comment 4

[Ryan Sleevi]

Under Mistakes/Bugs, this was listed as #2 "incorrect use of abbreviations in JOI:states"

Comment #1 also stated:

We have not yet completed scanning our systems and data for other instances of these same failures. We will post to this ticket when we have done so.

That said, I am concerned that more than a week has elapsed since Comment #2. This is deeply troubling, and I've now flagged this issue as another repeat of Bug 1563579, which is deeply troubling with respect to the future of Sectigo continuing as a trusted CA.

Comment 5

[Robin Alden]

I will provide an update on this bug by the end of this week, i.e. by 20-sep-2019, and no less than weekly thereafter.

Comment 6

[Robin Alden]

We will redouble our efforts to integrate the SIM tool with our issuance and validation flow so that we may reap the benefit of the considerable investment we have made in SIM.

An address matching system is operational as part of our validation workflow and document processing system. It compares requested names and addresses with those from the bulk address records and uses colour coding to indicate matching and non-matching data to the validator.
We will continue to improve this system as it is only as good as its data sources. We have started with the domains of data that get us the most benefit, but achieving the capability of doing this matching for every type of applicant in every jurisdiction of incorporation is our goal, although perhaps a distant one.

As an immediate term step, we are improving our documentation and training over JoI structure and selection. We will develop as a matter of urgency a rule-based JoI policy checker that will enforce correct usage and seek additional human approval when a rule-based approach is inappropriate. This will remain useful even after the integration of our SIM tool is complete.

Our JoI policy checker module is code complete and scheduled to go into production on September 29th.
We have an initial rule set for it to work on. Again this initial set is docussed on the JoIs that give us the most benefit. We will have further expanded this ruleset before the checker module goes live on the 29th and will continue to expand it afterwards.

We have not yet completed scanning our systems and data for other instances of these same failures. We will post to this ticket when we have done so.

We have found a number of other cases where there are errors in JoI data. We are preparing a report of these that we plan to attach to this bug next week.
As we identify and add new rules for our JoI policy checking module we will apply these to our body of issued certificates to determine if more such errors exist.

We identified a software error that caused our database of issued certificate data to have had some data fields overwritten after issuance in a some cases. The nature of the error has been identified and a fix is being worked upon. This has caused some of the fault analysis work to need repeating using fresh copies of certificate data to fill gaps in our earlier testing.

We will have a formal process around the output of the self-audit

Our next regular self-audit sample for EV certificates has been taken and the results are undergoing an initial examination.
When the report is ready, we will have our first self-audit review meeting under the new regime, with minutes taken and actions noted and carried through.

Comment 7

[Robin Alden]

There are two deliverables that we are tracking, but which we do not yet have ready to include in this report.

• we will publish in this bug the list of JoI mismatches that we have found in our issued certificate data. Although this will be an ongoing activity as our JoI policy checking module is further extended, but we will aim to public an initial list next week.

• We do not yet have a firm release date for the code fix that ensures that prevents the overwriting of some data fields in out database. Although this problem does not cause or lead to misissuance, it complicates the process of post-issuance policy checking as the policy checking modules are improved, and is therefore relatively urgent for us to deliver. I will aim to have a deployment date for this next week.

We continue to work on fleshing out our spelling and validity checking data for the JoI and subject state fields for more countries. Although strongly based on ISO3166 we find that a number of other additions are prudent and we will be looking towards a means to share the lists of acceptable values that we take for these fields so that other CAs may make use of them and so that other interested parties may also see what we accept. We hope for, and will work towards, the existence of a single location or a systematic means to determine the location in which CAs can agree and provide this information.

Comment 8

[Robin Alden]

The code fix that prevents the overwriting of some data fields in our database is scheduled for release to our live systems is scheduled for next weekend, 13th October.

The JoI policy checking module was released to our live systems last weekend (29th September) as scheduled. We are working to extend the JoI 'rules' table that it uses for more countries. This remains a work in progress.

I apologize that we do not yet have a list of JoI mismatches to publish. I had hoped we would have had it this week, but I now anticipate we will have an initial list to publish during next week.

We have still not had the self-audit review meeting under the new regime (minuted, with actions noted and tracked). We anticipate that happening during the next week.

Comment 9

[Robin Alden]

The code fix for the overwriting of data fields (post issuance) in our database has undergone further work and is now going through an extensive QA process which is estimated to take a further 2 weeks. In the mean time we are extending our JoI checking of previously issued certificates to use other data sources. Obviously we retain a complete internal record of all issued certificates and since the advent of CT these are also available in public data sources such as https://crt.sh and the database behind it.

We have identifed a further batch of EV certificates that may have incorrect Subject:JoI information and are refining and cleaning that list as we action the required revocations and reissuances and we will publish that list next week.

We are still tracking the formal EV self-audit review meeting and will report on its occurrence in this bug.

Comment 10

[Ryan Sleevi]

Robin: I want to highlight, that's two weeks now that a deliverable has slipped (the JOI mismatches mentioned in Comment #7, and then deferred in Comment #8). Comment #9 proposes yet another week.

However, it's not clear to me why. I don't really see much explanation for the delay, and that's concerning. This is exactly where transparency is meant to help. I don't want to discourage CAs from giving clear direction about next steps, nor do I want to encourage CAs to always say "Take however long you think it'll take, double it, and adding 10" (the ol software estimation game), but I do think that if there are delays, it's important to be clear about why, so the community can help balance the risks to the benefits, and build systems and understanding about how to prevent those delays going forward.

I'm not sure that delaying another week is good, and so I'd really appreciate the sooner response explaining, at the minimum, why, but failing a comprehensive explanation there, the actual list.

Comment 11

[Robin Alden]

(In reply to Ryan Sleevi from comment #10)
I agree that further delay of the resolution is inappropriate and we've split out the task of fixing our database overwriting bug from the identification of past issuance with the incorrect JoI information. The latter is no longer dependent on the former.

The JoI policy checker is effective for newly issued certificates (within the scope of the rules so far defined), and we have replicated the logic of the JoI checker outside our production database so that we can proceed with the analysis of prior issuance.

We will publish an initial list on Monday 21st. There will be further phases to come, and I apologize for the delay in getting this first one out.

Comment 12

[Robin Alden]

We have identified 313 Sectigo issued EV SSL certificates whose subjects include jurisdictionCountryName=GB and businessCategory="Private Organization" but which also include jurisdictionStateOrProvinceName or jurisdictionLocality attributes. Registration of such UK entities is at the national level, so the only jurisdiction attribute that should be present is jurisdictionCountryName=GB.

These are all certificates issued prior to our fixes to this bug being implemented.

I have published the list of crt.sh IDs of the certificates here.

I have also posted the list to the revocation tracker at https://misissued.com/batch/60/

There will be at least 2 more lists to come, and I shall aim to have at least the next one out before the end of this week.

Comment 13

[Robin Alden]

We are tracking the preparation of another batch of EV SSL certificates whose subjects include JoI errors. We anticipate having that batch published by the end of next week.

Comment 14

[Robin Alden]

I apologize for not yet having this next list ready. I will provide the next list or a substantial progress report by Wednesday 20th November.

We have added further rules to our pre-issuance JoI policy checker module which gives us the opportunity to re-scan our body of issued certificates looking for EV SSL Certificates with incorrect subject details. This should be complete by Friday this week (22-Nov-2019).

Comment 15

[Robin Alden]

The list we had been preparing turned out not to be correct. That wasted a chunk of work for our guys, but if there is a positive it has improved some wider understanding of the JoI issue within our organization.

I now anticipate having a small list of EV SSL certificates whose subjects include JoI errors to publish later this week.

Comment 16

[Ryan Sleevi]

Robin: It'd be super helpful, both for this incident and for the broader community, to capture a bit about why the list was incorrect?

I'm also trying to understand why Comment #14 promised an update by 20-11-2019, with a list by 22-11-2019, and the update in Comment #15 wasn't until 2019-12-09. It sounds like the procedures instituted in Bug 1563579 really aren't helping address these? It would certainly warrant an update there, but I think Comment #15 needs more substantive detail in order to address the concerns about Sectigo's operations here. More transparency is the mitigation for repeated systemic failures like this.

Comment 17

[Robin Alden]

We have identified 5799 Sectigo issued EV SSL certificates whose subjects include jurisdictionCountryName=NL and businessCategory="Private Organization" but which also include jurisdictionStateOrProvinceName or jurisdictionLocality attributes. Registration of such NL entities is at the national level, so the only jurisdiction attribute that should be present is jurisdictionCountryName=NL.

These are all certificates issued prior to our fixes to this bug being implemented.

I have published the list of crt.sh IDs of the certificates here.

I have also posted the list to the revocation tracker at https://misissued.com/batch/63/

(In reply to Ryan Sleevi from comment #16)

Robin: It'd be super helpful, both for this incident and for the broader community, to capture a bit about why the list was incorrect?

The list we were originally preparing in comment #15 was of EV certificates with jurisdictionCountryName=DE.
We had found that some EV SSL certificates include the jurisdictionStateOrProvinceName or jurisdictionLocality and some don't, and that registration is widely documented as occurring at the city or regional level. We were preparing a list on that basis but then we spoke with our legal team and they said that for German 'Private Organizations', the Jurisdiction of Incorporation is at the national level, i.e. that despite you being able to go to your local courthouse to register a company, the Jurisdiction of Incorporation is not your local courthouse because they did not pass the law under which the 'Private Organization' is registered. That is consistent with our practice, but at odds with what we observe from other CAs. At that point we were unable to proceed with that list.
I intend to seek clarification on how to proceed further, but I suspect that is probably over at the CA/B forum. They have the face to face meeting soon and that could provide a useful venue for the discussion.

(In reply to Ryan Sleevi from comment #16)

I'm also trying to understand why Comment #14 promised an update by 20-11-2019, with a list by 22-11-2019, and the update in Comment #15 wasn't until 2019-12-09. It sounds like the procedures instituted in Bug 1563579 really aren't helping address these? It would certainly warrant an update there

I will follow up in Bug 1563579.

Comment 18

[Robin Alden]

We apologize for the delay in providing this response.

At the February 2020 face to face meeting of the CA/B forum we made a presentation and led a discussion around the challenges of identifying a single 'correct' selection of JoI information in some cases. Rather than trying to address the general problem we picked a single country, Germany, for discussion since we observed that it has a relatively elaborate registration scheme, is one of the more common sources of subject organizations for EV certificates and, we felt, on close consideration does not have an excellent match between the JoI requirements of the EV Guidelines and the structure of the German regulation on the registration of private legal entities.

This is a contentious issue and the presentation was deliberately tongue-in-cheek in places while having a serious intent overall. There was a good natured but occasionally exasperated exchange of views around the clarity of the requirements of the EVGLs in respect of the JoI fields and how it related to the legal structure around private organization registration.

Our presentation and the minutes of the session (search for "Jurisdiction Of Incorporation in Private Organizations") including a summary of the discussion are shared from the CA/B forum's website.

We are supportive of the EV Guidelines Agency disclosure as a step towards resolving the problems of inconsistency of JoI recording. Enforced transparency provides a means of discovery of current practice that does not rely on voluntary disclosure that will always be partial or on analysis of historic issuance that will always suffer from time lags. It does not provide the complete solution though since we feel that, for some countries at least, a sensible and complete representation of the Jurisdiction of Incorporation is not possible within the structure defined by the EV Guidelines.

The CA/B Forum's strength was that it came out early with a unilateral scheme that more-or-less works worldwide. GLEIF and EIDAS have both addressed the Jurisdiction issue in ways that are more sympathetic to National schemes and structures and, while both imperfect in their own ways, the CA/B Forum could improve its scheme by using ideas from those other two systems.

Comment 19

[Ben Wilson]

Are there any other questions or issues anyone would like to raise concerning this bug / incident report?

Comment 20

[Ryan Sleevi]

I don't think we have a clear timeline from Sectigo, even still, which is something Wayne raised in Comment #2. This bug details a rather large set of systemic failures, and Sectigo has proposed a variety of responses for mitigating these, but without clear timelines or a holistic picture about how validation works. There's promises of SIM, but that's in Beta, and it's not clear that it handles for the jurisdictions Sectigo issues for.

I'd encourage Sectigo to revisit the answer to Question 7 on the incident report, and provide a rather complete and comprehensive timeline of the changes made and changes still pending, chronologically ordered, and tying back to the root cause that they're trying to address. Some of these are still rather forward looking, and given Sectigo's rather complete and continual failure to provide timely updates, I have zero confidence at this point in their ability to meet that timeline without encountering yet another delay. I think having the timelines clearly stated makes it easiest if it becomes necessary to discuss removing EV and/or trust in Sectigo in the future, either to show Sectigo continuing to fail or to document how their management was able to execute a turn-around from an otherwise disappointing slide.

Comment 21

[Robin Alden]

We are working now on the requested timeline and will post that in this bug as soon as it is prepared.

Comment 22

[Robin Alden]

Responding to the request for a clear timeline of actions taken, and to be taken, framing the answer to #7 of the topics to reported actions required by https://wiki.mozilla.org/CA/Responding_To_An_Incident .

In Comment #1, I said:

We will redouble our efforts to integrate the SIM tool with our issuance and validation flow so that we may reap the benefit of the considerable investment we have made in SIM.

but our plans changed.
SIM remains as a tool for our validators but it has not been integrated into the issuance flow. We have learned useful lessons from the development of SIM and it is a tool we will make further use of, but we have addressed the issues raised in this ticket in other ways as set out below.

7. List of steps your CA is taking to resolve the situation and ensure that such situation or incident will not be repeated in the future, accompanied with a binding timeline of when your CA expects to accomplish each of these remediation steps.

Date (Time UTC)	Action
22/06/2019 23:04	Code deployed that improved the treatment of meta data in subject fields and this also blocked the issuance of certificates with subject:serialNumbers containing only zeroes.
22/08/2019 17:00	A distinct 2nd approval team was first established to undertake the Final Cross-Correlation and Due Diligence as per EVGL 11.13. Whereas previously we had complied with the EVGL in terms of using a different staff member to do the 2nd approval step they were part of the same team as had done the first parts of the approval. Separating the teams helped to ensure that all communication between first and second approval teams is available for training and quality control purposes and that any responses to queries raised by 2nd approval were evidence based.
30/08/2019 13:24	Code deployed to check EV stateOrProvinceName and joi:stateOrProvinceName for the most common Jurisdictions (US, CA, GB) at issuance time and block if an error found.
06/09/2019 16:11	Additional data deployed to check EV stateOrProvinceName and joi:stateOrProvinceName for Switzerland (CH) issuance time and block if an error found.
September 2019	We made a number of personnel changes among our Validators and 2nd Approval Validators. Better metrics allowed us to identify a number of cases where the performance of individual validators fell below what was required.
14/09/2019 23:59	Code deployed to flag mismatches for stateOrProvinceName and joi:stateOrProvinceName (for configured countries) during validation.
17/09/2019 07:47	Code deployed to improve mixed case tolerance in stateOrProvinceName and joi:stateOrProvinceName.
28/09/2019 23:04	Code deployed to add EV JoI scope checking for Business entities in (GB, DE, NL, US) to block issuance if the JoI:stateOrProvince was populated when it should not have been (and vice-versa)
12/10/2019 23:01	Code deployed to automatically expand abbreviations in stateOrProvinceName and improve country-matching (for configured countries), also to block stateOrProvince values detected as being wrong at the point of EV certificate request.
15/03/2020 00:05	Metadata removal improved to prevent a single quote being permitted as a valid field value.
05/04/2020 23:04	Code deployed to assist validation team by checking address fields from certificate requests by passing them to Google's GeoCoding API and using the returned information to indicate doubtful address values and improvement suggestions.
22/06/2020 16:45	Additional data deployed to check EV stateOrProvinceName and joi:stateOrProvinceName for another 196 countries at issuance time and block if an error found.
25/06/2020 12:00	Code deployed to provide option to blank stateOrProvince when Locality is also present and supplied stateOrProvince value does not check out.
12/07/2020 23:11	Code deployed to increase performance of stateOrProvinceName checking.
25/07/2020 23:06	Code deployed to improve stateOrProvinceName and joi:stateOrProvinceName checking by making country name matching always case-insensitive.

Although we will continue to improve our validation processes and systems, there are are no further system changes planned in response of this ticket.

Comment 23

[Robin Alden]

We have no further update.

Comment 24

[Robin Alden]

We have no further update.

Comment 25

[Robin Alden]

Ben, we have no further planned actions on this bug. May we ask that it is closed?

Comment 26

[Ben Wilson]

I will close this bug on or about 1-Sept-2020 unless there are other important issues or questions raised.

Comment 27

[Rob Stradling]

We have no further update.

Comment 28

[Rob Stradling]

(In reply to Ben Wilson from comment #26)

I will close this bug on or about 1-Sept-2020 unless there are other important issues or questions raised.

We have no further update. Ben, can this bug be closed now?