Open Bug 1575022 Opened 4 months ago Updated 17 days ago

Sectigo: EV SSL Certificates with incorrect subject details.

Categories

(NSS :: CA Certificate Compliance, task)

task
Not set

Tracking

(Not tracked)

ASSIGNED

People

(Reporter: Robin.Alden, Assigned: Robin.Alden, NeedInfo)

References

(Blocks 1 open bug)

Details

(Whiteboard: [ca-compliance])

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36

Steps to reproduce:

We have received over the last few days a number of problem reports identifying EV SSL certificates issued by Sectigo where either the Jurisdiction of Incorporation (JoI) or the subject:serialNumber (i.e. the registration number for the subject organization) appear to be wrong.
We are dealing with these reports and revoking affected certificates as soon as is practicable.
We will follow up here with a full incident report shortly.

Flags: needinfo?(Robin.Alden)
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true
Assignee: wthayer → Robin.Alden
Type: defect → task
Whiteboard: [ca-compliance]
Summary: Sectigo: EV SSL Certificate errors in Jurisdiction and registration number → Sectigo: EV SSL Certificates with incorrect subject details.
  1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

By problem reports to sslabuse@sectigo.com:
Times are in BST (= EDT + 5) (dd/mm/yyyy hh24:mi)
#certs is the number of distinct Sectigo issued certificates in each report.

Report# Received at #certs Issue
1 Fri 16/08/2019 23:18 1 subject:serialNumber wrong. JOI=SE, but subject:serialNumber is for a DK entity.
2 Sat 17/08/2019 21:53 15 subject:serialNumber wrong. JOI:country=SE should have subject:serialNumber of 10 digits, but subject:serialNumber is not 10 digits
3 Mon 19/08/2019 01:45 1 businessCategory is Private Organization, but should be Government Entity. Also serialNumber=0
4 Mon 19/08/2019 02:00 1 serialNumber=00
5 Mon 19/08/2019 02:05 1 serialNumber=0000
6 Mon 19/08/2019 02:26 11 serialNumber seems to be 0
7 Tue 20/08/2019 03:41 2 problems:
6 Invalid JOI:stateOrProvince
102 Invalid subject:stateOrProvince
8 Tue 20/08/2019 18:46 1 JOI:locality & JOI:stateOrProvince wrong.
9 Tue 27/08/2019 16:53 1963 Subject:serialNumber of '0' or some combination of '0's
  1. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.
date event
23/06/2019 Code deployed which, as a side effect, blocks all-zero entries in subject:serialNumbers
16/08/2019 23:18 thru 27/08/2019 16:53 Problem reports received
17/08/2019 13:30 Analysis of problem reports begins
18/08/2019 21:45 Revocation of identified certificates begins
22/08/2019 17:00 A distinct 2nd approval team was established
30/08/2019 13:24 Code deployed to automatically check stateOrProvinceName for the most common Jurisdictions.
01/09/2019 13:00 Revocation of identified certificates concludes

Not all of the certificates were revoked within 24 hours of being reported. Our goal and intent was that all of the certificates were to have been revoked within 5 days of receiving the reports however, regrettably, the 6 certificates with Invalid JOI:stateOrProvince field values reported to us at 20/08/2019 03:41 took nearly 8 days to revoke due to an administrative error. All of the other reported certificates were revoked (or expired) within 5 days of receiving the relevant report.

  1. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

We have stopped issuing certificates with these problems.

  1. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

Counts of affected certificates for which we have received reports.

Problem #certs dates
JOI (jurisdiction of incorporation) contains a state or a locality when it should not because the registration scheme is at the country or state level. 8 30-Aug-17 to 30-Oct-18
Subject:serialNumber omits digits 7 28-Nov-17 to 23-Nov-18
Subject:serialNumber is a date when it should be a registration number 3 05-Jan-18 to 05-Dec-18
JOI is wrong. Subject is registered in both SE and DK. The organizationName and the subject:SerialNumber are for DK, but the JOI says SE 4 23-May-18 to 05-Oct-18
JOI is wrong. subject:Country and JOI:Country are both SE instead of NL 1 04-Jan-19
JOI is wrong. should be SK, not US 1 29-Nov-17
businessCategory is wrong. It shows Private Organization when it should be Government Entity 1 20-Feb-18
Subject:serialNumber consists of only zeroes 1970 30-May-17 to 21-Jun-19
Subject:State wrong - Confusion over the status of Puerto Rico & US Virgin Islands 2 16-Jun-17 to 03-May-19
Subject:State wrong - City listed as state. (e.g. Manhattan, Las Vegas..) 2 13-Sep-17 to 16-Jan-18
Subject:State wrong - State misspelled. (e.g. Deleware, Marryland, NewYork..) 87 08-Jan-18 to 08-Aug-19
Subject:State wrong - irrelevant text present 2 27-Oct-18 to 17-Dec-18
Subject:State wrong - State and locality reversed 1 23-Jul-19
Subject:Country wrong: US instead of OM 1 29-Jun-18
jurisdictionStateOrProvinceName wrong - misspelled. (e.g. NewYork, Deleware) 4 09-Aug-17 to 25-Jun-19
jurisdictionStateOrProvinceName wrong - irrelevant text present 2 30-Aug-17 to 12-Sep-18
jurisdictionLocality - irrelevant text present 1 12-Sep-18

10 certificates have 2 of the above problems, so the total number of certificates affected is 2090.

152 of the above certificates had been revoked before the reports were received.

  1. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

Links to problematic certificates

  1. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

There are a number of separate issues occurring here and they do not split out distinctly across the problem reports listed above which is why we are dealing with them all in one incident report.

Systems involved:

Sectigo Certificate Manager (SCM).

SCM is an Enterprise RA platform written and hosted by Sectigo. It allows Enterprise customers to enter Organization Details and certificate requests for domain names and, after an initial validation of the Organizational Details and domain names to the requirements of our CPS (and the BRs and the EVGLs), the Enterprise customer may then order further certificates for the same details without repeating the validation steps for as long as the guidelines permit the validation information to be re-used.

Validators and 2nd Approval validators

We still rely substantially on human validators for the validation of OV and EV subject details, and also for the EV 2nd approval step.

Sectigo Information Manager (SIM).

SIM is a data matching tool and Jurisdiction database that we have been working on for the last 18 months. We hoped to have integrated this tool fully into our issuance pipeline before now, but it is still in a Beta test.

Mistakes/bugs:

  1. Zero serial numbers.

At some time previously there was a bug in SCM that required a company registration field to be non-null when entered. This led to a pattern of behaviour where customers entering data to be validated would put zeroes into this field to mean both ‘there is no registration number’ and ‘I don’t know the registration number’. Furthermore, our validators had picked up the concept that this field could not be blank and were used to seeing ‘0’ entries there for Government Entities. This led over time to the validators becoming essentially blind to the presence of zeroes in the company number field.
We do not present this information to blame our subscribers or our validators for these errors, but merely to explain the sequence of events that led to their occurrence. We have automated internal policy checks and we use the external lint checkers and at least our internal policy checks should have been updated to catch this error of zeroes appearing in subject:serialNumbers long ago.

There were 1762 certificates for 91 different organization names in 8 enterprise accounts in SCM that contain a subject:serialNumber containing only zeroes.

There were also 209 certificates for 148 organization names in 57 accounts not in SCM that contain a subject:serialNumber containing only zeroes.

  1. Misspellings of State names, invalid state names, and incorrect use of abbreviations in JOI:states

These exhibit the difficulty of precise text comparison and text rule implementation (e.g. no abbreviations) by human validators. The fact that these errors made it through to our issued certificates is due to us not yet having incorporated automated checking and provision of state names into the data entry and policy checking modules. We had planned to have been using the SIM tool by now but the tool remains in a Beta test state.

  1. Incorrect JOI

These are data entry errors by validators after the QGIS lookups are done. A validator looks for data in one QGIS but does not find it, then they look in another QGIS and find it. The enter the registration number from the 2nd QGIS but the JOI from the 1st.
2nd approval should have caught this every time.

  1. Invalid JOI

Some of these are data entry errors and others are misunderstandings of the level at which the JoI operated.

  1. subject:serialNumber is a date instead of a number

There was a misunderstanding by some of our validators over the precedence rules around the use of registration numbers and dates with private organizations.

  1. Business Category wrong

This certificate should not have been issued. The category is wrong, but the registration number is also absent (0).

  1. Company Registration Numbers wrong (other than zero)

There were a number of cases where digits had been omitted from registration numbers. These are generally copied and pasted or provided automatically so there should be no scope for such errors.

  1. JOI wrong, subject registered in two jurisdictions.

The validation of these 4 certificates for one organization became confused and the subject ends up being a mixture of the details from the two registrations.

  1. Self-Audit

We were initially puzzled by why our regular self-audits of the issued EV certificates had not found these problems before, especially the issue with subjectSerialNumbers containing only zeroes.

On investigation we found that self-audit samples had revealed single certificates with subject:serialNumber=0 on more than one occasion. The auditor had followed up with the individual validators responsible in each case for further training around the issue, but neither the systems problem that led to the zeroes nor the wider acceptance of the zeroes amongst other validators were identified as an issue.

  1. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

We have made a number of personnel changes among our Validators and 2nd Approval Validators. Although the tools we use are being improved, there are a number of cases where the performance of individual validators fell below what was required. We do not require perfection of our validators, and we have training and retraining schemes in place.

We have introduced an adversarial scheme of 2nd approval where the validator and the 2nd approval validator do not meet and may not discuss the certificate subject data except through official channels (tickets & validation notes). We have tightened up the 2nd approval process and increased the resource available for 2nd approval so that more careful and precise checks will be made.

On 23/06/2019 we introduced a code change that improved the treatment of meta data in subject fields and this also blocked the issuance of certificates with subject:serialNumbers containing only zeroes.

As a result of these reports we instigated an urgent development task to introduce an automated system of checks on states in EV subjects and JOI that catches the state name errors here. The first phase of this which, for the most common jurisdictions, blocks invalid state names in our policy checking module went into production at 30/08/2019 13:24. We have since extended and will further extend the list of jurisdictions over which this is effective but this is a halting process since the ISO 3166-2 data is not always a perfect fit for real-world requirements as has been discussed elsewhere.

We will redouble our efforts to integrate the SIM tool with our issuance and validation flow so that we may reap the benefit of the considerable investment we have made in SIM.

As an immediate term step, we are improving our documentation and training over JoI structure and selection. We will develop as a matter of urgency a rule-based JoI policy checker that will enforce correct usage and seek additional human approval when a rule-based approach is inappropriate. This will remain useful even after the integration of our SIM tool is complete.

We have not yet completed scanning our systems and data for other instances of these same failures. We will post to this ticket when we have done so.

Self audit.

We thought our self audit process had been working OK, but we have evidently been failing to capture vital information. We will have a formal process around the output of the self-audit where a report of any variance from the ideal for subject information will be noted in the report and that report will be discussed in a meeting with validation, compliance, and development with recorded minutes where the action points will be enumerated and allocated to executives for implementation.

Robin: thank you for providing this comprehensive incident report. It describes a number of remediation steps, but no timeline for completing them. That is particularly concerning because Sectigo has a history of not providing regular updates to incident bugs, even when an update is specifically requested. Please provide a remediation timeline here and update this bug at least weekly unless a "Next Update" date is added to the "whiteboard" field.

Also, please explain the "administrative error" that delayed revocation of some certificates, and the remediation for that problem.

I have a question about this as well. Does this cover where there is an abbreviated state field (https://censys.io/certificates/f0d1ca1c2a7c705bd791487484a7921351d93170b5c955734501e02cc1999df5)? I couldn't tell if those errors were captured in the above. Per section 9.2.4 the full name of the state/locality is required.

Under Mistakes/Bugs, this was listed as #2 "incorrect use of abbreviations in JOI:states"

Comment #1 also stated:

We have not yet completed scanning our systems and data for other instances of these same failures. We will post to this ticket when we have done so.

That said, I am concerned that more than a week has elapsed since Comment #2. This is deeply troubling, and I've now flagged this issue as another repeat of Bug 1563579, which is deeply troubling with respect to the future of Sectigo continuing as a trusted CA.

Blocks: 1563579

I will provide an update on this bug by the end of this week, i.e. by 20-sep-2019, and no less than weekly thereafter.

We will redouble our efforts to integrate the SIM tool with our issuance and validation flow so that we may reap the benefit of the considerable investment we have made in SIM.

An address matching system is operational as part of our validation workflow and document processing system. It compares requested names and addresses with those from the bulk address records and uses colour coding to indicate matching and non-matching data to the validator.
We will continue to improve this system as it is only as good as its data sources. We have started with the domains of data that get us the most benefit, but achieving the capability of doing this matching for every type of applicant in every jurisdiction of incorporation is our goal, although perhaps a distant one.

As an immediate term step, we are improving our documentation and training over JoI structure and selection. We will develop as a matter of urgency a rule-based JoI policy checker that will enforce correct usage and seek additional human approval when a rule-based approach is inappropriate. This will remain useful even after the integration of our SIM tool is complete.

Our JoI policy checker module is code complete and scheduled to go into production on September 29th.
We have an initial rule set for it to work on. Again this initial set is docussed on the JoIs that give us the most benefit. We will have further expanded this ruleset before the checker module goes live on the 29th and will continue to expand it afterwards.

We have not yet completed scanning our systems and data for other instances of these same failures. We will post to this ticket when we have done so.

We have found a number of other cases where there are errors in JoI data. We are preparing a report of these that we plan to attach to this bug next week.
As we identify and add new rules for our JoI policy checking module we will apply these to our body of issued certificates to determine if more such errors exist.

We identified a software error that caused our database of issued certificate data to have had some data fields overwritten after issuance in a some cases. The nature of the error has been identified and a fix is being worked upon. This has caused some of the fault analysis work to need repeating using fresh copies of certificate data to fill gaps in our earlier testing.

We will have a formal process around the output of the self-audit

Our next regular self-audit sample for EV certificates has been taken and the results are undergoing an initial examination.
When the report is ready, we will have our first self-audit review meeting under the new regime, with minutes taken and actions noted and carried through.

There are two deliverables that we are tracking, but which we do not yet have ready to include in this report.

  • we will publish in this bug the list of JoI mismatches that we have found in our issued certificate data. Although this will be an ongoing activity as our JoI policy checking module is further extended, but we will aim to public an initial list next week.

  • We do not yet have a firm release date for the code fix that ensures that prevents the overwriting of some data fields in out database. Although this problem does not cause or lead to misissuance, it complicates the process of post-issuance policy checking as the policy checking modules are improved, and is therefore relatively urgent for us to deliver. I will aim to have a deployment date for this next week.

We continue to work on fleshing out our spelling and validity checking data for the JoI and subject state fields for more countries. Although strongly based on ISO3166 we find that a number of other additions are prudent and we will be looking towards a means to share the lists of acceptable values that we take for these fields so that other CAs may make use of them and so that other interested parties may also see what we accept. We hope for, and will work towards, the existence of a single location or a systematic means to determine the location in which CAs can agree and provide this information.

The code fix that prevents the overwriting of some data fields in our database is scheduled for release to our live systems is scheduled for next weekend, 13th October.

The JoI policy checking module was released to our live systems last weekend (29th September) as scheduled. We are working to extend the JoI 'rules' table that it uses for more countries. This remains a work in progress.

I apologize that we do not yet have a list of JoI mismatches to publish. I had hoped we would have had it this week, but I now anticipate we will have an initial list to publish during next week.

We have still not had the self-audit review meeting under the new regime (minuted, with actions noted and tracked). We anticipate that happening during the next week.

The code fix for the overwriting of data fields (post issuance) in our database has undergone further work and is now going through an extensive QA process which is estimated to take a further 2 weeks. In the mean time we are extending our JoI checking of previously issued certificates to use other data sources. Obviously we retain a complete internal record of all issued certificates and since the advent of CT these are also available in public data sources such as https://crt.sh and the database behind it.

We have identifed a further batch of EV certificates that may have incorrect Subject:JoI information and are refining and cleaning that list as we action the required revocations and reissuances and we will publish that list next week.

We are still tracking the formal EV self-audit review meeting and will report on its occurrence in this bug.

Robin: I want to highlight, that's two weeks now that a deliverable has slipped (the JOI mismatches mentioned in Comment #7, and then deferred in Comment #8). Comment #9 proposes yet another week.

However, it's not clear to me why. I don't really see much explanation for the delay, and that's concerning. This is exactly where transparency is meant to help. I don't want to discourage CAs from giving clear direction about next steps, nor do I want to encourage CAs to always say "Take however long you think it'll take, double it, and adding 10" (the ol software estimation game), but I do think that if there are delays, it's important to be clear about why, so the community can help balance the risks to the benefits, and build systems and understanding about how to prevent those delays going forward.

I'm not sure that delaying another week is good, and so I'd really appreciate the sooner response explaining, at the minimum, why, but failing a comprehensive explanation there, the actual list.

(In reply to Ryan Sleevi from comment #10)
I agree that further delay of the resolution is inappropriate and we've split out the task of fixing our database overwriting bug from the identification of past issuance with the incorrect JoI information. The latter is no longer dependent on the former.

The JoI policy checker is effective for newly issued certificates (within the scope of the rules so far defined), and we have replicated the logic of the JoI checker outside our production database so that we can proceed with the analysis of prior issuance.

We will publish an initial list on Monday 21st. There will be further phases to come, and I apologize for the delay in getting this first one out.

We have identified 313 Sectigo issued EV SSL certificates whose subjects include jurisdictionCountryName=GB and businessCategory="Private Organization" but which also include jurisdictionStateOrProvinceName or jurisdictionLocality attributes. Registration of such UK entities is at the national level, so the only jurisdiction attribute that should be present is jurisdictionCountryName=GB.

These are all certificates issued prior to our fixes to this bug being implemented.

I have published the list of crt.sh IDs of the certificates here.

I have also posted the list to the revocation tracker at https://misissued.com/batch/60/

There will be at least 2 more lists to come, and I shall aim to have at least the next one out before the end of this week.

We are tracking the preparation of another batch of EV SSL certificates whose subjects include JoI errors. We anticipate having that batch published by the end of next week.

I apologize for not yet having this next list ready. I will provide the next list or a substantial progress report by Wednesday 20th November.

We have added further rules to our pre-issuance JoI policy checker module which gives us the opportunity to re-scan our body of issued certificates looking for EV SSL Certificates with incorrect subject details. This should be complete by Friday this week (22-Nov-2019).

You need to log in before you can comment on or make changes to this bug.