Closed Bug 1710243 Opened 5 years ago Closed 5 years ago

Sectigo: Invalid stateOrProvinceName

Categories

(CA Program :: CA Certificate Compliance, task)

Product:
CA Program
Component:
CA Certificate Compliance
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: michel, Assigned: rob)

References

Details

(Whiteboard: [ca-compliance] [ov-misissuance])

Attachments

(1 file)

Additional Malta state.xlsx
19.75 KB, application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
Details

Hello,
I found the certificate https://crt.sh/?id=1442539123 that has stateOrProvinceName: Moldova. That's basically the same issue as https://bugzilla.mozilla.org/show_bug.cgi?id=1709392, but with a different CA.

Assignee: bwilson → rob
Status: NEW → ASSIGNED
Whiteboard: [ca-compliance]

The six certificates mentioned in this report were all issued in 2019, predating bug #1645686 (Sectigo: Lack of input validation in stateOrProvinceName). In bug #1645686 comment #82 we posted a list "of all ST field values which were contained in the Subject information of issued EV or OV certificates at the start of this bug and the final determination as to whether or not that value required revocation: ReviewedSTFinal.csv.”

ReviewedSTFinal.csv contains:
subject:stateOrProvinceName = Moldova
subject:country = MD
Require Revocation? = No

and

subject:stateOrProvinceName = Malta
subject:country = MT
Require Revocation? = No

The community had the opportunity to question our decision not to revoke these certificates when bug #1645686 was active. We received no objections to these published intentions at the time.

We believe this bug should be closed as DUPLICATE or INVALID.

Flags: needinfo?(bwilson)

Although this was not challenged in the original bug 1645686, it has now been challenged, so instead of closing this as INVALID out of the gate can we have Sectigo's response on why they think this value is valid for the ST field?

Flags: needinfo?(tim.callan)

Also, https://crt.sh/?id=1337930177 has stateOrProvinceName: Warminsko-Wazurskie (that's a typo: https://en.wikipedia.org/wiki/Voivodeships_of_Poland) and organizationalUnitName: COMODO EV SSL I really doubt that they have such a unit.

(In reply to George [:fozzie] from comment #3)
Let's have this dialog on bug #1645686. This is a clear continuation of that matter. It can be highly disruptive when topics are unnecessarily split up, not only for the sake of the conversation at the time but also in case there is a point in the future when we or others want to look back at the discussion that took place.

Ben, we are requesting to reopen #1645686 and close this bug as DUPLICATE to that one. We can move these points across and discuss them there.

At present there are four items to be discussed.

  1. Revocation of Moldova
  2. Revocation of Malta
  3. Warminsko-Wazurskie
  4. OU field containing Comodo
Flags: needinfo?(tim.callan)

I also found https://crt.sh/?id=2208084200 that has stateOrProvinceName: Malopolskia. That's a typo.

(In reply to Michel Le Bihan from comment #4)

Also, https://crt.sh/?id=1337930177 has [...] organizationalUnitName: COMODO EV SSL I really doubt that they have such a unit.

That issue was previously discussed at Bug 1593776 and Bug 1620561

I have added a capsule summary to bug #1645686 comment #92. We will begin addressing the open items in that thread.

I'd like to mention that bug #1645686 comment #92 calls out my Comment 7 as "addressing" the OU section in Comment 4. As I'm a non-native speaker and "addressing" (to the best of my knowledge) has a plurality of meanings, I'd like to make the following clear:

Tim, I do not think that my comment adresses (in the sense of resolves / closes the discusson) the OU section, but merely provides context on the issue that was raised.

The problem of invalid OU usage (and lack of revocation) still remains until the last certificate with the problem either expires or is revoked, and although the bugzilla issues may be closed, the issue will likely keep popping up each time someone looks at the details of the problematic certificates and cross-references the BR.

(In reply to Tim Callan from comment #5)

(In reply to George [:fozzie] from comment #3)
Let's have this dialog on bug #1645686. This is a clear continuation of that matter. It can be highly disruptive when topics are unnecessarily split up, not only for the sake of the conversation at the time but also in case there is a point in the future when we or others want to look back at the discussion that took place.

While I appreciate this appeal, I think it's best to continue the discussion here. Fundamentally, we're discussing both issues with that resolution (with respect to Moldova and Malta) and the failure to detect (with respect to Warminsko-Wazurskie).

I agree with you, however, that the OU issue raised in Comment #4 would have been better reported as a separate issue, which would then have duplicated into Bug 1593776, including Sectigo's decision to not revoke as required by 4.9.1.1.

Contrary to Comment #5, let's continue discussion here.

Flags: needinfo?(bwilson)

(In reply to Ben Wilson from comment #12)

Contrary to Comment #5, let's continue discussion here.

Will do.

On the other bug we have stated that Warminsko-Wazurskie and Malopolskia are misissuances. We're slating them for revocation within the specified time period. I will provide a write up for the both together since they're variants on a theme.

I also owe a reply on Malta and Moldova, which is in the works as well.

This write up is for the two reported certificates with the incorrect Polish state names of Warminsko-Wazurskie and Malopolskia.

  1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

As part of the discussion on this bug, a poster called our attention to two certificates with misspellings of Polish states. After brief investigation we concurred that these two strings were wrong.
The original postings were comment #4 on May 10 at 2:28 pm EDT and comment #6 on May 11 at 5:31 EDT. We saw each shortly thereafter.

  1. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

May 10, 2:28 pm (all times EDT)
Original post with crt.sh record reporting Warminsko-Wazurskie.

3:07 pm
Sectigo acknowledges Warminsko-Wazurskie as an error.

2:28 pm
Comment #6 is posted reporting Malopolskia. Sectigo notices it shortly thereafter.

3:45 pm
Sectigo acknowledges Malopolskia as an error.

May 14, 3:07 pm.
https://crt.sh/?id=1337930177 (Warminsko-Wazurskie) is revoked.

May 15, 3:10 pm.
https://crt.sh/?id=2208084200 (Malopolskia) is revoked.

  1. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

Today we use a discrete list of allowed State field values based on ISO 3166-2. At the time of issuance for both of these certs that was not the case. The misspellings Warminsko-Wazurskie and Malopolskia are not on that list, and this error won’t recur.

  1. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

Two certificates, issued April 1, 2019 and December 17, 2019.

  1. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

https://crt.sh/?id=1337930177
https://crt.sh/?id=2208084200

  1. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

At the time of issuance for both these certs the State field had no checks on its content. These were both human errors by validation representatives who were not Polish speakers. These reps did not spot the misspellings of Warminsko-Wazurskie for Warminsko-Mazurskie and Malopolskia for Małopolskie.

Following up on our Q4 revocation event (as detailed in bug #1645686), we created a list of all State field entries in our corpus of active certificates and reviewed this list for accuracy. This was a massive task requiring review of nearly 15,000 individual strings in many languages. On January 14, 2021 we published a list of these and our disposition of each (bug #1645686 comment #82). This list included Malopolskia and Warminsko-Wazurskie as permissible values, clearly both errors. These errors came about during the one-time task of making sense of state and province records in our issuance practices and certificate base, which was massive in scope and complexity.

  1. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

Early in 2021 we implemented a lookup table for allowable State field values based on the ISO 3166-2 list. Bug #1645686 detailed this process as it developed. The table has correct spellings for these two Polish states, and had it been in place at the time, it would have prevented these issuance errors. This is an improvement over the previous open-text functionality for this field in that it greatly reduces error potential in going-forward certificates.

Just as we failed to discover these two misspellings occurred during our manual cleanup of State fields in Q4 and Q1, it is possible that other misspellings in the stateOrProvinceName field slipped through as well. We presently are working on a plan to further scrub our certificate base for similar misspellings and are actively investigating potential ideas. We will update this bug as we arrive at an action plan. I expect to know more in the next few weeks.

We will revoke the six certificates identified in this bug with either Malta or Moldova in the stateOrProvince field (which for ease of typing I’ll refer to as the State field). This message will answer an earlier question about why we chose not to revoke them originally and also why we are doing so now.

As stated in comment #14, Sectigo recently changed our approach to populating the State field. There is no definitive, authoritative list of states or provinces globally, so CAs traditionally have had to come up with strategies for building lists of State field values they would use. Sectigo used to use a curated list of state names on a country-by-country basis. Some countries, like the US, Canada, and Australia, were obvious to us. Others less so, including countries where we had no employees in residence. For these countries we relied on the advice of our local partners to tell us what the norms and expectations were in those regions. This hand-built list evolved over the course of years as we filled in gaps in our knowledge, corrected errors, and adjusted to new information. This fundamental approach is fraught with risk for error. Human error can occur while creating and updating the list, especially when dealing with languages and regions not familiar to the CA’s employees.

It also relies too much on opinion. Our local partner in a country tells us, “These are the provinces and everyone knows it.” Later on someone on Bugzilla, who’s expertise we are in no way able to judge, tells us, “This isn’t really a province and everyone knows it.” Now suddenly we have to determine which of these two opinions is right, with very little to go on.

We greatly prefer reliable, unambiguous sources. While ISO 3166-2 is not perfect, it is the closest to that ideal we can find. So late in 2020 we decided to rely very heavily on this source moving forward. We continue to believe it will prove to be superior to our previous state of affairs.

Early this year we tackled our existing certificate base. These certificates had gone through authentication using the previous, hand-built list of values, which would not always align with our new source. So we went through the exercise of looking at nearly 15,000 values to determine if they met our criteria for going-forward issuance and, if not, if they therefore required replacement at the time of the change.

Note that the two are not necessarily the same. As stated above, there is no single, definitive authority on state and province names throughout the world. To add predictability and reduce human judgement, we chose to restrict going-forward authentication to a state names based on ISO 3166-2. Those are our internal authentication rules that we will hold ourselves to. But it still could be that a state name absent from this list is nonetheless in alignment with the BRs. In fact, we believe there are arguably valid state or province names not included in ISO 3166-2 and that the presence of one of these names in a certificate would not by itself constitute misissuance, even if we’ve chosen not to use that name ourselves as we move forward. The exercise I refer to in the previous paragraph was to disposition the values in existing certificates according to that principle.

In the case of Malta and Moldova, our previous authentication procedure had allowed these two stings as State names in their respective countries. It is not out of the question that a country name may also be the name of a province or city in that country (for example, the capital of Belize is Belize City), so the repetition of the country name in the State field is not an automatic disqualifier. While we’re unable to go back and trace the provenance of every value in our previous list of allowable state names (most of which go back to the 2000s), most commonly these we based on the attestation of a local source that had credibility with Comodo at the time.

Therefore, in the case of these two State field values, we judged that they were allowable for existing certificates even though we would not use them in the future. As stated earlier, bug #1645686 comment #82 included these decisions for the community’s information.

We have been following others CAs’ analysis of Malta as a State name. We will align ourselves with the prevailing outcome and revoke the five identified certificates with Malta in the State field and the one with Moldova. We will follow up when that’s complete.

(In reply to Tim Callan from comment #15)

We will revoke the six certificates identified in this bug with either Malta or Moldova in the stateOrProvince field...
These six certificates are all revoked. We'll provide a more thorough update soon.

(In reply to Tim Callan from comment #15)

Note that the two are not necessarily the same. As stated above, there is no single, definitive authority on state and province names throughout the world. To add predictability and reduce human judgement, we chose to restrict going-forward authentication to a state names based on ISO 3166-2. Those are our internal authentication rules that we will hold ourselves to. But it still could be that a state name absent from this list is nonetheless in alignment with the BRs. In fact, we believe there are arguably valid state or province names not included in ISO 3166-2 and that the presence of one of these names in a certificate would not by itself constitute misissuance, even if we’ve chosen not to use that name ourselves as we move forward. The exercise I refer to in the previous paragraph was to disposition the values in existing certificates according to that principle.

I think it's interesting to make sure we're surfacing and highlighting these, with a mind to "If the BRs explicitly referenced and referred to ISO 3166-2, what of value would be lost". It seems to be a recurring them in this response that there are values outside the ken of 3166-2, and thus this allows mistakes, and in the effort to prevent mistakes, Sectigo has since chosen to align to 3166-2.

If Bug 1645686 Comment #82 is meant to be that list, it doesn't seem obvious from that, so it'd be useful to clarify if so.

Flags: needinfo?(tim.callan)

(In reply to Ryan Sleevi from comment #17)

I think it's interesting to make sure we're surfacing and highlighting these, with a mind to "If the BRs explicitly referenced and referred to ISO 3166-2, what of value would be lost". It seems to be a recurring them in this response that there are values outside the ken of 3166-2, and thus this allows mistakes, and in the effort to prevent mistakes, Sectigo has since chosen to align to 3166-2.

The biggest one is that while the ISO list is not limited to English names, it is limited to what appears to be an extended ASCII character set. Our list includes Chinese, Korean, and Cyrillic. To obtain these state names, we tasked our local sales representatives or trusted local partners to provide the equivalent of the English names and populated them into our own list. There is no obvious reason this principle couldn’t extend to additional languages like Japanese, Arabic, or Hebrew. In our case we deemed the benefit not to be worth the effort.

The second one for us was that some of the names in the ISO list are not particularly user friendly. For instance, 3166-2 includes “London, City of” and “Bruxelles-Capitale, Région de”. In addition to the original user-hostile names, which we will keep as options, we likely will allow “City of London” and “Région de Bruxelles-Capitale” as valid names. This idea applies to a few more listed regions, but not many.

These decisions come with their downsides. As a general rule we dislike hand curated lists. They are prone to error. They can get out of date. They require ongoing upkeep. In this case we decided to accept these disadvantages for the benefit provided. The total additional work and risk is small. In general, moving to 3166-2 is a big plus as it reduces the amount of curation vastly.

We have published our current list at https://github.com/sectigo/subject_address_fields to provide transparency on our practices and to help the community with similar exercises. As I said above, we may make a few small changes to this list.

(In reply to Tim Callan from comment #18)

These decisions come with their downsides. As a general rule we dislike hand curated lists. They are prone to error. They can get out of date. They require ongoing upkeep. In this case we decided to accept these disadvantages for the benefit provided. The total additional work and risk is small. In general, moving to 3166-2 is a big plus as it reduces the amount of curation vastly.

In noting your CSV, do you keep track of what you consider the 'canonical' name (e.g. ISO 3166-2 form), and what are acceptable variations, such that should ISO 3166-2 change, you'd receive notice/alert of the change and the opportunity to re-review? That seems like it might help reduce the risk of out-of-dateness.

Similarly, tracking when regional variations were approved, such that you can have recurring processes to re-vet such variations as allowed on a period basis (e.g. annually) to ensure they're still relevant and accurate.

This detailed writeup covers the reported certificates issued with Moldova and Malta state names.

  1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

This bug originally appeared on May 8, 2021 reporting a certificate with Moldova in the stateOrProvince field, followed two days later by five reported certificates with Malta in the same field.

We tracked other CAs’ investigation of this same issue and concluded that we should follow the prevailing behavior and revoke the certificates in question.

  1. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

All times Eastern Daylight Time

May 8, 2021, 4:03 pm
Original posting identifies one certificate in Moldova.

May 10, 11:58 am
Poster adds five certificates in Malta.

May 10, 1:38 pm
Sectigo acknowledges receipt of these notifications.

May 17, 4:26 pm
After following public discussion on this same issue with other CAs, Sectigo announces that we will revoke these certificates.

May 21, 12:52 pm
The six reported certificates are revoked.

  1. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

In the fall of 2020 we implemented new lookup functionality for stateOrProvince to limit the risk of incorrect information in this field. Malta and Moldova are not present as values for stateOrProvince in that table. Therefore, certificates with these names in that field have not been possible for more than six months.

  1. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

Six certificates.
The first was issued June 5, 2019.
The last was issued December 13, 2019.

  1. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

https://crt.sh/?id=1442539123
https://crt.sh/?id=2148277587
https://crt.sh/?id=2185099679
https://crt.sh/?id=1544924747
https://crt.sh/?id=2147496653
https://crt.sh/?id=1748170187

  1. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

Some of the information here will be very similar to what you see in comment #14, as many of the factors are the same.

At the time of issuance for these certificates the stateOrProvince field had no checks on its content. We have no documentation of the rationale for including these names in this field. We can best surmise that the representatives involved at the time made the judgement that this name was acceptable for the stateOrProvince field.

Note that there is no reason to expect that the employees involved had any particular cultural or geographical knowledge of either Moldova or Malta. Ultimately we have to attribute this misissuance to human error.

  1. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

In 2020 we implemented a lookup table for allowable State field values based on the ISO 3166-2 list. We have discussed this decision in detail earlier in this thread. This change will prevent misissuance of this sort in the future.

Flags: needinfo?(tim.callan)
Flags: needinfo?(tim.callan)

(In reply to Ryan Sleevi from comment #19)

In noting your CSV, do you keep track of what you consider the 'canonical' name (e.g. ISO 3166-2 form), and what are acceptable variations, such that should ISO 3166-2 change, you'd receive notice/alert of the change and the opportunity to re-review? That seems like it might help reduce the risk of out-of-dateness.

That's right. Changes to 3166-2 trigger a review of derived names associated with that change. If a new name is added that warrants a derived name (most don't), we can add that derived name at any time. For example, the "friendly" derived named I mention in comment #18 are not in our list today, but we can add them whenever we get through other items we consider higher priority right now. Adding derived names need not depend on changes to the ISO names. If a canonical name disappears, we remove any associated derived name.

Similarly, tracking when regional variations were approved, such that you can have recurring processes to re-vet such variations as allowed on a period basis (e.g. annually) to ensure they're still relevant and accurate.

That will be governed by the status of 3166-2. Reviews of derived names are triggered by changes in the canonical names. If the canonical name remains static, the derived name can remain static as well. It's hard to see the scenario where the written Simplified Chinese for Zhejiang Sheng is going to change. Likewise, we don't foresee "City of London" ceasing to be the way people talk any time in the near future.

Flags: needinfo?(tim.callan)

As part of the decision to revoke certificates with Malta in the stateOrProvince field, we have searched our own certificate base and found 37 additional certificates. The attached file “Additional Malta state” lists these certificates.

We revoked them on June 12 at 5:15 pm Eastern.

(In reply to Tim Callan from comment #18)
Another example of how we have to deviate our own list from 3166-2 is the removal of countries where the US State Department does not allow us to do business. If you look at our list on Github, you won’t find the Republic of Cuba or the Democratic People's Republic of Korea, even though they are both present on ISO’s list. Since we cannot issue certificates to those regions, leaving the country codes in the table would introduce error potential. As the list of embargo countries changes over time (which it does, albeit slowly), we need to push these updates back into our available countries list as well. In fact, we recently added the Sudan back to our list based on the change in its embargo status.

Are there any other questions or comments on this issue?

As there appear to be no other questions or comments on this issue, is it time to close this bug?

Flags: needinfo?(bwilson)

I will schedule this bug to be closed on or about Friday, 9-July-2021.

Status: ASSIGNED → RESOLVED
Closed: 5 years ago
Flags: needinfo?(bwilson)
Resolution: --- → FIXED
Product: NSS → CA Program
Whiteboard: [ca-compliance] → [ca-compliance] [ov-misissuance]
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: