Closed Bug 1708934 Opened 4 years ago Closed 3 years ago

Sectigo: Invalid postalCode field

Categories

(CA Program :: CA Certificate Compliance, task)

Product:
CA Program
Component:
CA Certificate Compliance
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: fozzie, Assigned: tim.callan)

Details

(Whiteboard: [ca-compliance] [ev-misissuance])

Attachments

(3 files)

Sectigo-Postal-Fingerprints.txt
10.31 KB, text/plain
Details
Hong Kong 00000 crt records.xlsx
15.31 KB, application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
Details
Misc postal code revocations.xlsx
18.00 KB, application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
Details

Sectigo has issued a number of certificates with a combination of 0's in the postalCode field.

Assignee: bwilson → tim.callan
Status: NEW → ASSIGNED
Whiteboard: [ca-compliance]

We acknowledge receipt of this notification and will post a full report soon.

There are also hundreds of certificates which include only a set of digits (i.e 1, 1111 or 9999), some of these may be valid but as Sectigo is the only CA with some of these combinations of numbers then I suspect most are invalid.

(In reply to George [:fozzie] from comment #2)

There are also hundreds of certificates which include only a set of digits (i.e 1, 1111 or 9999), some of these may be valid but as Sectigo is the only CA with some of these combinations of numbers then I suspect most are invalid.

We will look into these also and get back to you on this thread.

The 160 signatures submitted equate to 80 leaf certificates and their 80 corresponding precertificates.

Among the leaf certificates, 30 are either revoked or expired.

Of the 50 remaining certificates, 45 of them are in Hong Kong. Hong Kong Post’s official web site directs those seeking to include a postal code for any address inside Hong Kong to use 000, 0000, 00000, or HKG.

Of the five remaining certs, one is in Dubai. Dubai does not use postal codes, but local sources say 00000 is acceptable for mail delivery. Out of an abundance of caution, we will go ahead and revoke this certificate.

Of the four remaining certs, one is in Beirut, Lebanon. Beirut does not use postal codes. Local sources say 00000 is acceptable for mail delivery. Out of an abundance of caution, we will revoke this certificate also.

The remaining three certificates have incorrect postal code information. Including the two discussed above, these five certificates are scheduled for revocation on May 6.

Our investigation and response are ongoing. We will continue to post developments and will of course offer a full incident report.

(In reply to Tim Callan from comment #4)

The 160 signatures submitted equate to 80 leaf certificates and their 80 corresponding precertificates.

Among the leaf certificates, 30 are either revoked or expired.

Of the 50 remaining certificates, 45 of them are in Hong Kong. Hong Kong Post’s official web site directs those seeking to include a postal code for any address inside Hong Kong to use 000, 0000, 00000, or HKG.

Are you referring to this remark at https://www.hongkongpost.hk/en/about_us/tips/correct_address/index.html

Remark: Postcode system is not in place in Hong Kong. If it is required to input postcode of Hong Kong at website(s), you may leave the field blank or try to fill in with "000", "0000", "000000" or "HKG" which are frequently used by others nowadays.

How does issuing certificates with these placeholder values meet the requirement in Section 7.1.4.2 of the BRs?

Subject attributes MUST NOT contain only metadata such as '.', '-', and ' ' (i.e. space) characters, and/or any other indication that the value is absent, incomplete, or not applicable.

(In reply to Mathew Hodson from comment #5)

Are you referring to this remark at https://www.hongkongpost.hk/en/about_us/tips/correct_address/index.html

Remark: Postcode system is not in place in Hong Kong. If it is required to input postcode of Hong Kong at website(s), you may leave the field blank or try to fill in with "000", "0000", "000000" or "HKG" which are frequently used by others nowadays.

Yes.

How does issuing certificates with these placeholder values meet the requirement in Section 7.1.4.2 of the BRs?

Subject attributes MUST NOT contain only metadata such as '.', '-', and ' ' (i.e. space) characters, and/or any other indication that the value is absent, incomplete, or not applicable.

They are unrelated. BR 7.1.4.2 refers to placeholders inside certificates to indicate the lack of field data. Hong Kong Post has no interest in CABF or digital certificates and is not talking about them.

Rather, Hong Kong Post has offered a set of valid strings that can be included on the address label of posted mail while still enabling successful delivery. They are authorized postal codes that may be used successfully, even though they are not required.

(In reply to Tim Callan from comment #6)

(In reply to Mathew Hodson from comment #5)

Remark: Postcode system is not in place in Hong Kong. If it is required to input postcode of Hong Kong at website(s), you may leave the field blank or try to fill in with "000", "0000", "000000" or "HKG" which are frequently used by others nowadays.
…
Subject attributes MUST NOT contain only metadata such as '.', '-', and ' ' (i.e. space) characters, and/or any other indication that the value is absent, incomplete, or not applicable.

They are unrelated. BR 7.1.4.2 refers to placeholders inside certificates to indicate the lack of field data.

I’m having trouble understanding the conclusion here.

Hong Kong Post clearly indicates postcodes are not used. The BRs prohibit including characters indicating a field is not applicable, which seems to be what has happened here. Why was the field included, if not applicable, given the BRs do not require it?

Flags: needinfo?(tim.callan)

(In reply to Ryan Sleevi from comment #7)

Hong Kong Post clearly indicates postcodes are not used. The BRs prohibit including characters indicating a field is not applicable, which seems to be what has happened here.

After the passage from Hong Kong Post stating that a postal code system is not “in place,” it goes on to acknowledge that postal codes may be included as a fact of life in posted mail and lists a set of acceptable codes for those circumstances. Note that Hong Kong Post didn’t write, “Put in any old thing. We really don’t care.” Because the post office does care. It has given the public a specific set of acceptable postal codes that can be included without disrupting its established delivery process. The implication is that other strings may not fare equally well. We have viewed that as an official list of acceptable postal codes from the post office.

Note that nearly two years ago we chose to change our practice (see below for more detail), and we can’t foresee going back to the old one for any reason. So while it’s valuable to iron out subtleties like this, we’re not issuing new certificates with just a string of 0s as a postal code in any region in the world. Nonetheless, it may be useful to establish what is permissible in this case, as we aren’t the only CA who will ever operate in Hong Kong.

So with that said, our conclusion has been based on two pillars:

  1. The local mail delivery service as established by the government is the ultimate authority on acceptable postal codes. While there are norms and pragmatic considerations, no hard and fast universal rules exist around postal codes and the local authority may establish these codes however it likes, up to and including having variable formats or otherwise doing things that may not seem optimal to an outsider. We must look to information from the local authority about which postal codes are acceptable.
  2. Hong Kong Post has given an unambiguous list of strings that may be included in the Postal Code field. This is not coming from some third party offering tips for how to get your mail delivered in Hong Kong. This is the official post office telling users which values to include in scenarios where they prefer to include something. Even if a blank entry is better, these other values are explicitly allowed.

Why was the field included, if not applicable, given the BRs do not require it?

The last of these certificates was issued in June of 2019. Comodo had legacy systems going back many years that in some circumstances would require fields to be populated for certificates to be processed. As a result the team had added steps to the codified authentication process to include a set of 0s in certain fields if they were otherwise without content. In June of 2019 as part of the company’s ongoing program to review and clean up various aspects of our issuance and operations, we updated that process to what we have now, which is to keep these fields blank. As mentioned above, Sectigo does not issue new public certificates with a string of only 0s in the postal code field.

(In reply to Tim Callan from comment #8)

(In reply to Ryan Sleevi from comment #7)

Hong Kong Post clearly indicates postcodes are not used. The BRs prohibit including characters indicating a field is not applicable, which seems to be what has happened here.

After the passage from Hong Kong Post stating that a postal code system is not “in place,” it goes on to acknowledge that postal codes may be included as a fact of life in posted mail and lists a set of acceptable codes for those circumstances. Note that Hong Kong Post didn’t write, “Put in any old thing. We really don’t care.” Because the post office does care. It has given the public a specific set of acceptable postal codes that can be included without disrupting its established delivery process. The implication is that other strings may not fare equally well. We have viewed that as an official list of acceptable postal codes from the post office.

I disagree with your wording here. Hong Kong Post is quite explicit about when to input these pre-specified strings:

Postcode system is not in place in Hong Kong. If it is required to input postcode of Hong Kong at website(s) , you may leave the field blank or try to fill in with "000", "0000", "000000" or "HKG" which are frequently used by others nowadays.

(emphasis mine)

As specified in BR section 7.1.4.2.2 (g), the postalCode field is optional. Combining those two requirements, the values "000", "0000", "000000" and "HKG" are thus not explicitly condoned by HKP (it is an optional field, thus the "you can use these values" clause doesn't apply). Additionally, as Hong Kong does not have a notion of postalCode this field should not be included in certificates that have subject:countryName:HK.

Note that nearly two years ago we chose to change our practice (see below for more detail), and we can’t foresee going back to the old one for any reason. So while it’s valuable to iron out subtleties like this, we’re not issuing new certificates with just a string of 0s as a postal code in any region in the world. Nonetheless, it may be useful to establish what is permissible in this case, as we aren’t the only CA who will ever operate in Hong Kong.

So with that said, our conclusion has been based on two pillars:

  1. The local mail delivery service as established by the government is the ultimate authority on acceptable postal codes. While there are norms and pragmatic considerations, no hard and fast universal rules exist around postal codes and the local authority may establish these codes however it likes, up to and including having variable formats or otherwise doing things that may not seem optimal to an outsider. We must look to information from the local authority about which postal codes are acceptable.
  2. Hong Kong Post has given an unambiguous list of strings that may be included in the Postal Code field. This is not coming from some third party offering tips for how to get your mail delivered in Hong Kong. This is the official post office telling users which values to include in scenarios where they prefer to include something. Even if a blank entry is better, these other values are explicitly allowed.

Correction: Hong Kong Post has given us an unambiguous list of strings that may be included in the Postal Code field if the field is required.

Why was the field included, if not applicable, given the BRs do not require it?

The last of these certificates was issued in June of 2019. Comodo had legacy systems going back many years that in some circumstances would require fields to be populated for certificates to be processed. As a result the team had added steps to the codified authentication process to include a set of 0s in certain fields if they were otherwise without content. In June of 2019 as part of the company’s ongoing program to review and clean up various aspects of our issuance and operations, we updated that process to what we have now, which is to keep these fields blank. As mentioned above, Sectigo does not issue new public certificates with a string of only 0s in the postal code field.

This certificate [0] was recently issued on 2020-Sept-01, and contains subject:postalCode=HKG. Although indeed not "only 0s", this is still problematic because it asserts an postalCode in a region where the concept of postalCode is not applicable. Several others like that one also exist [1].

[0] https://crt.sh/?id=3321869123&opt=ocsp
[1] https://crt.sh/?id=2827285313, https://crt.sh/?id=1701347885, https://crt.sh/?id=1517606714

This discussion in exemplary of the ambiguity that still exists in the BRs and EVGs. Like so many online debates, it doesn't look like either party changed its opinion based on the other's points.

As I mentioned in comment #8, we no longer allow postal code fields containing only the numeral 0. The discussion on this thread has been motivated by our belief in Hong Kong Post's ability to declare which postal codes are legitimate for its territory. Beyond the desire to come to the right conclusion, we don't really have a horse in this race. So maybe it makes the most sense to step away and let someone else worry about it at a later date.

To make this resolution truly clean, we will proceed with revocation of all Hong Kong certificates with only a series of 0s or HKG as a postal code. We also discovered a few other Hong Kong certs with wonky postal codes like HK or NT. We will revoke these as well. These last certificates mentioned are unambiguously incorrect, so we'll provide a full write up on this thread, including a list of certificates, when the revocation event is complete.

Tim: With all due respect, I don’t believe this is an ambiguity here. As the language highlights, we’re talking about something where it’s required, when clearly it isn’t here. This isn’t a debate about how many angels dance on the head of a pin, but one that tries to understand how Sectigo’s processes for compliance are implemented, because this does seem an error in judgment/interpretation.

That said, I appreciate your comment #10 as being reflective of exactly what a CA should do when presented with these challenges, which is avoid the appearance of doubt or impropriety.

I’m looking deeper at this incident, I think we’d both agree it highlights an opportunity to build deeper knowledge of the requirements or, alternatively, change requirements. Obviously, this issue wholly disappears when postalCode isn’t included at all. If that isn’t desirable, figuring out when, where, and how postalCode is used and seen as necessary can also help spark discussion on how to build a shared understanding about its appropriateness and rules. I think the example you raised, of Sectigo’s reforms to how they process such data, is both a positive example and one that may benefit from further standardizing, if it’s been working.

I appreciate that Sectigo readily had a government source to cite, but in considering this incident, it would have been useful to explicitly state that source, rather than leave it to the community (in Comment #5) to guess at it. This suggests there is also opportunity for greater disclosure and transparency, ideally on an ongoing basis, but certainly within incident reports. I appreciate that this was not raised via the standard problem reporting mechanisms, and so this may have been an understandable oversight, but it is likely something worth considering.

Naturally, the question that remains is largely one of understanding the decision making process, and whether there exists further opportunities to improve or clarify. One example is whether adversarial review is incorporated, and whether such “worst case interpretations” are encouraged (even if they may compete with specific product/marketing goals), and whether there is a process in place to seek further community clarification on jurisdictional issues such as this. It would be remiss to say “Sectigo made a bad call” without looking for further opportunities to set Sectigo - and all CAs - up for success in the future.

I have been working on a substantive response to your last comment, Ryan. As you may imagine, bug #1712188 consumed the end of my week and is still ongoing, and so I haven't been able to finish and post the response here. Please be assured that it is on the way.

With the goal of working through the multiple issues covered in this bug, this writeup will discuss the five reported certificates that were not from Hong Kong and which were all revoked on May 6. The Hong Kong certificates have a different timeline and some different issues covered. So once the Hong Kong revocation is complete, we’ll follow up with a new writeup to cover those certificates.

We’re also aware that there is at least one outstanding question posted to this bug. This write up is not an attempt to answer that question, and we’re working on that response. Therefore, I am leaving the needinfo request open for now.

  1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

This bug report originally listed 160 signatures, which ultimately translated to 50 active certificates. Of those, 45 were in Hong Kong and will be covered in a separate write up. This write up covers the remaining five.

The bug was filed May 1 at 5:49 EDT. We acknowledged it at 8:33 am EDT the following morning.

  1. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

Most of the actions for the five certificate outside of Hong Kong can be seen on this thread, as they occurred in real time. Here is a capsule summary.

All times Eastern Daylight Time

June 21, 2019
As part of our ongoing process improvement we discontinue by policy to issue certificates with a series of 0s where a blank field is considered acceptable. This policy originally owed itself to software limitations in certain parts of the old Comodo workflow that go back many years. Even after those limitations were no longer a factor, the policy had continued until this date.

March 6, 2021
Sectigo ceases issuing public TLS certificates with zip/postal codes included.

May 1, 5:49 pm
This bug filed.

May 2, 8:33 am
Bug acknowledged. Compliance team begins investigation.

May 2 to 3.
List of 160 signatures is examined. Half are precertificates for the other half. Of the remainder, most are expired, revoked, or issued to organizations in Hong Kong. We will discuss this last group in a second write up.

Five certificates are found to be active and from countries other than Hong Kong. Three of these are unambiguous misissuance. The two others are from countries where our local associates told us at one point in the past that a postal code full of zeros was considered acceptable practice.

As part of our response to this bug we look into published postal codes from these three countries. While we find online sources repeating this same claim, no official sources are among them. We decide to take the conservative approach and revoke these certificates.

May 4, 3:18 pm Eastern.
We publish the results of our investigation.

May 6, 2:03 pm Eastern.
The five mississued or suspect certificates are revoked.

  1. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

We ceased issuing certificates with all 0s in postal code fields in 2019. We ceased issuing public TLS certificates with postal codes on March 6.

  1. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

The earliest of these certificates was issued 5/17/2019.

The last was issued 6/4/2019.

  1. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

Five certificates

https://crt.sh/?q=eb9abfea0664b93deb705f80b44ce257bcf4531674707049ecea9fd2e397b484
https://crt.sh/?q=68e6b0d398f76355032147af00a29823c7baff4fc8e339acbb3ccfcd67d9bc62
https://crt.sh/?q=8269dee53b211a4a68d219900144a1522dcb1e39cf540256f94cfc3a1253729f
https://crt.sh/?q=d2151a147b4c5e6cf69cac6fbf438691ee46716940bfc5b577b0927eae911792
https://crt.sh/?q=ccab2684861df54c383e8865ef4447a9cfcf1f9f7cf7d5569d15719b126b48dd

  1. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

Comodo had legacy systems going back many years that under certain workflows required various fields to be populated for certificates to be processed. As a result the team had included steps in the codified authentication process to add a set of 0s to these fields if they were otherwise without content. Even after this software limitation no longer existed, the policy had remained in place.

The last of the revoked certificates was issued June 4, 2019. At that time we had no software checks on postal code field information. Assurance of the quality of this field came through human checks. It appears that for these three certificates this process failed.

It is difficult to create and maintain a list of acceptable postal code values for use in a software check. The number of values is vast. They can change at any time with no credible, centralized source for acceptable values.

  1. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

In June 2019 Sectigo discontinued its policy of issuing certificates with a series of 0s where a blank field is considered acceptable.

In March 2021 Sectigo removed the postalCode field from public certificates.

Flags: needinfo?(tim.callan)
Flags: needinfo?(tim.callan)

Thanks for the update, Tim. I think one thing to capture on this bug is something similar to what I expressed in Bug 1712120 Comment #2 - what sort of steps Sectigo is taking to reduce the complexity of its systems and the flow of data. I realize that Comment #13 makes it clear you've transitioned off including postalCode in the profile, which along with Sectigo's other changes (such as streetAddress), so a much better, more reliable approach to compliance than what we've seen in other CAs' similar incident reports, and that's quite commendable. However, it also seems that the Comodo/Sectigo split continues to turn up new complexity and surprises, and I'm curious at what point Sectigo expects to have those tamped down. I suspect and hope that Bug 1712188, which seems conceptually similar, has highlighted to Sectigo management the critical importance of auditing-and-replacing these code paths and complexity, and I'm hoping that, between these three bugs, you can share a bit of what a roadmap to improvement looks like, incorporating these lessons.

(In reply to Ryan Sleevi from comment #11)

Naturally, the question that remains is largely one of understanding the decision making process, and whether there exists further opportunities to improve or clarify. One example is whether adversarial review is incorporated, and whether such “worst case interpretations” are encouraged (even if they may compete with specific product/marketing goals), and whether there is a process in place to seek further community clarification on jurisdictional issues such as this. It would be remiss to say “Sectigo made a bad call” without looking for further opportunities to set Sectigo - and all CAs - up for success in the future.

The original policy decision to include 0s in the case of a missing postalCode field occurred in the deep past, and understanding the thought process behind that decision is likely not possible. Let’s focus instead on how Sectigo operates in the present. We can discuss in detail how we evaluated and responded to this report when it came in.

When presented with this objection to a set of certificates, the first task was to refresh our understanding of the actual rules in the local region. The fact that a policy decision occurred in the past does not necessarily mean the current compliance team would support that same decision, particularly if at decision time it was a different decision maker at a different company with different ownership and a different philosophy.

We recognized that while a postal code of all 0s appears to be wrong, that doesn’t necessarily mean it is. As I stated in comment #8, there sometimes exist practices that appear strange to our eyes but are nonetheless the official, “correct” behavior in other regions. So we felt it was worthwhile to investigate. That investigation quickly left us believing that the certificates for the five countries apart from Hong Kong required revocation.

In the case of Hong Kong, we discovered the passage from the Hong Kong Post web site mentioned a few times in this thread, which spells out a specific set of codes that are available to be used. The argument against allowing these codes seems to hinge on two passages from the web site. Let’s address them individually.

The first is this sentence, “Postcode system is not in place in Hong Kong.” The belief seems to be that the statement “Postcode system is not in place in Hong Kong” is definitive proof of a second statement: “There are no allowable postal codes for Hong Kong.” That may seem sensible, but we have direct evidence that this second statement is not true. China Post, the official post office of mainland China, has designated 999077 as the postal code for Hong Kong addresses. Despite Hong Kong’s unusual status, China is of course the country governing that region.

So for sure there is at least one allowed postal code in Hong Kong, despite the published statement that postal codes are not used. Based on that we are of the belief that the postal code 999077 in a certificate issued to a Hong Kong address would be compliant with the BRs.

This is an important point because we concluded that “Postcode system is not in place in Hong Kong” definitively did not prove that “There are no allowable postal codes for Hong Kong,” as in fact we had established that there was at least one. At this point it was worth asking if there are other postal codes that are valid in the region as well.

Here we returned to the list given on the web site. Hong Kong post selected and published a specific list of discrete postal codes to use, at least one of which would have obvious communicative value to a human reading it (HKG). Since the permissible nature of 999077 demonstrates that the post office does not categorically disallow all postal codes, these others deserve evaluation on their merits. As discussed earlier on this thread, we felt it was significant that Hong Kong Post chose a discrete and specific list of codes to publish. Clearly these codes have status in the eyes of the post office superior to that of simply any random string. That imprimatur, along with the refutation of this idea that no postal code could ever be allowed, let us to conclude that these codes were permissible.

The second objection has been focused on this word “required.” We brought that up in our own internal review of these Hong Kong certificates. With the permission of the other participant, I am posting verbatim and unedited a portion of our thread on this exact point.

Rob Stradling:
I presume Mathew is arguing that the statement "Postcode system is not in place in Hong Kong" equates to (BR 7.1.4.2) "not applicable", and that (emphasis mine) 'If it is required to input postcode of Hong Kong at website(s), you may leave the field blank or try to fill in with "000", "0000", "000000" or "HKG"...' is not relevant to this bug because the postalCode field is not required in certificates.

 My response, unless you guys convince me otherwise, is that this is a permitted postal code

Having thought about this some more, I think I would argue that, at best, it's a "permitted placeholder" in cases where a website demands that a postal code field must have a value; but being a "permitted placeholder" doesn't make it a "permitted postal code" in certificates.

It's far from black and white in terms of obviousness though.

Tim Callan:
I’m going to push back on that argument.  The interdiction in the BRs is to prevent the population of certificate fields with placeholders that have no actual, valid meaning in the system we use for physical addresses.  That is not the case here.  Hong Kong Post is stating that the values given below will be honored if they appear in the address on actual posted mail.  They explicitly didn’t say, “Just stick in any old thing because we really don’t care.”  Rather, they stated that these strings are the ones the system will handle correctly.  While you don’t need to have any of them included for your mail to be delivered, these values are indeed valid for the postal code position on sent mail.

That’s entirely different from the examples given in the BRs, which are not deemed to be valid strings by the local mail delivery authorities.

Thank you for pushing on my ideas.  That’s the point.  In this case I remain convinced in the original position.

You ask specifically if we incorporate adversarial review and encourage consideration of “worst case interpretations.” I have long believed the answer to that is yes. That does not necessarily mean that we will ultimately decide we believe every worst case interpretation, of course. Some such interpretations are simply absurd. Take, for example, the apparent prohibition on CAs issuing certificates for their own sites. Yes, we discussed the worst case interpretation there, but we rejected it on the consensus that that prohibition was not the intention behind the rules.

Now, as we dig into the root causes of bug #1712188 and certificates ordered by our QA department, it appears that not all employees in all parts of the company feel this same “obligation to dissent.” That has been an eye opener for me and I think a number of others in leadership here. We are still working on our understanding and our more sweeping response to that revelation, and I expect much discussion of this point on that bug. But I can’t truly address this question here without acknowledging bug #1712188 and the work that has to go on over there.

Flags: needinfo?(tim.callan)

This write up covers the certificates as part of this bug that were issued to Hong Kong addresses.

  1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

This bug report originally listed 160 signatures, which ultimately translated to 50 active certificates. Of those, 45 were in Hong Kong. Those are covered in this write up.

The bug was filed May 1 at 5:49 EDT. We acknowledged it at 8:33 am EDT the following morning.

  1. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

Most of the actions for these can be seen on this thread, as they occurred in real time. Here is a capsule summary.

All times Eastern Daylight Time

June 21, 2019
As part of our ongoing process improvement we discontinue by policy to issue certificates with a series of 0s where a blank field is considered acceptable. This policy originally owed itself to software limitations in certain parts of the old Comodo workflow that go back many years. Even after those limitations were no longer a factor, the policy had continued until this date.

March 6, 2021
Sectigo ceases issuing public TLS certificates with zip/postal codes included.

May 1, 5:49 pm
This bug filed.

May 2, 8:33 am
Bug acknowledged. Compliance team begins investigation.

May 2 to 3
List of 160 signatures is examined. Half are precertificates for the other half. Of the remainder, 30 are expired or revoked. 45 are issued to organizations in Hong Kong.

May 4, 3:18 pm
We publish the results of our investigation.

May 4 to May 17
Public discussion of whether or not these postal codes are allowed in Hong Kong.

May 17, 6:01
We announce that we will revoke the Hong Kong certificates.

May 21, 1:22
The Hong Kong certificates are revoked.

  1. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

We ceased issuing certificates with all 0s in postal code fields in 2019. We ceased issuing public TLS certificates with postal codes on March 6.

  1. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

45 certificates.
The earliest was issued 4/2/2019.
The last was issued 6/21/2019.

  1. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

The list is attached as the Excel file “Hong Kong 00000 crt records.”

  1. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

Comodo had legacy systems going back many years that under certain workflows required various fields to be populated for certificates to be processed. This requirement led to the creation of rules for populating the postalCode field even in places where it could have been omitted. The established rule for Hong Kong appears to have been a series of zeroes.

The affected certs were all issued in the first half of 2019, during which time these rules had not yet been scrutinized. Starting in June of 2019 Sectigo changed the policy to leave the postalCode field blank in Hong Kong.

  1. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

In June 2019 Sectigo discontinued its policy of issuing certificates with a series of 0s where a blank field is considered acceptable.

In March 2021 Sectigo removed the postalCode field from public certificates.

Thanks Tim.

I appreciate the deeper look at the systemic issues here, and showing how they're being addressed. As mentioned in Comment #14, it definitely sounds like there's still opportunity for simplification and de-complexifying things, which Sectigo has already done here with postalCode. I don't think I have outstanding questions; I'm setting N-I for Ben here for close out, although if others (Matthew, Matthias, George) have questions or believe I've missed stuff, they should definitely chime in.

Flags: needinfo?(bwilson)

Tim, thank you for providing the information for the 0s-certificates, but could you also provide a response for any subject:postalCode = HKG certificates that you have found (and/or any other Hong Kong certificates with subject:postalCode)? I mentioned at least 4 of those in Comment 9, but haven't seen those addressed afterwards. The data from your Comment 17 also doesn't seem to contain those, seeing that your dates are from 2019.

Flags: needinfo?(tim.callan)

Thanks Tim & Ryan.

I'm really impressed with Sectigo's response to this bug, it's great to see Sectigo improving their handling of incidents.

Tim: Did you take a look at any of the cases I mentioned in comment #2? While I haven't looked at any specific certificates in depth, it seems like there may be further misissuances with these as the Censys queries I used for them returned hundreds of certificates issued by Sectigo and only tens issued by other CAs.

That said, I don't think comment #2 blocks this bug from being closed out as it is simply just checking any possible further misissuances and now that Sectigo has remedied this issue, if it feels like it's too much work to analyse then I don't think they're compelled to do so, as I haven't found any misissued certificates with those cases.

(In reply to George [:fozzie] from comment #20)

Tim: Did you take a look at any of the cases I mentioned in comment #2?
In fact, we have an ongoing investigation into these cases and others like them right now. It is one of the many things we're trading off at the moment as we've had a sudden flood of compliance-related matters to deal with. (We went from zero open bugs to six in just a few weeks.)

We have, I hope understandably, given higher priority to other items like ensuring our DCV issuance is correct. At the same time we don't want to let these other issues simply languish, so there is a balancing act and more or less constant prioritization that we must perform. We are working these issues as fast as we can to put them to bed and keep the community up to date on what we are doing. I hope that as we work our way through this perfect storm of events we'll be able to return to the high level of responsiveness we desire for all our open Bugzilla threads.

I do expect our research to reveal more certificates that need to be revoked for the same reason as the earlier certificates described in comment #13. We simply haven't been able to put the elbow grease into it, but we will. We of course will report what we find.

(In reply to Matthias from comment #19)

[C]ould you also provide a response for any subject:postalCode = HKG certificates that you have found (and/or any other Hong Kong certificates with subject:postalCode)?

That was my error. We revoked those certificates simultaneous with the 45 discussed in comment #17, and I just missed them in my write up. Let me amend part 5 of that comment to include:
https://crt.sh/?id=3321869123
https://crt.sh/?id=1701347885
https://crt.sh/?id=1517606714

The certificate described in https://crt.sh/?id=2827285313 actually expired before revocation occurred.

See my comment #21 for what we're doing regarding other certificates that may have questionable postal codes.

(In reply to Tim Callan from comment #21)
Excuse the formatting error in this comment. The first paragraph of my reply was included in the quoted section. It reads:

In fact, we have an ongoing investigation into these cases and others like them right now. It is one of the many things we're trading off at the moment as we've had a sudden flood of compliance-related matters to deal with. (We went from zero open bugs to six in just a few weeks.)

(In reply to Tim Callan from comment #23)

In fact, we have an ongoing investigation into these cases and others like them right now.

Here's a result from that investigation. We have identified 27 certificates in Sri Lanka with single-digit postal codes, which are not correct in that country. We issued a revocation order today. Once they're revoked next week we will provide full details.

This investigation continues, and we will report what we find.

Flags: needinfo?(tim.callan)

(In reply to George [:fozzie] from comment #2)

There are also hundreds of certificates which include only a set of digits (i.e 1, 1111 or 9999), some of these may be valid but as Sectigo is the only CA with some of these combinations of numbers then I suspect most are invalid.

As a follow on to this comment we have conducted a search of our public SSL certificate base for postal codes with suspicious content. We ran a query to find active certificates with postal codes meeting at least one of these qualities:

  • Consisting entirely of a single character repeated any number of times, from one through the maximum length of the field
  • Consisting of two or fewer characters

This query gave us more than 500 suspect certificates. Most turned out to be okay, as the above criteria do not necessarily constitute a bad postal code. For example, 2222 is a valid postal code in Sydney, 1111 is valid in Budapest, and 111111 is valid in Bogota. Some cities like Dublin use one- or two-digit postal codes for local delivery, and all postal codes in Jamaica are one or two digits long.

In addition to the certificates from Sri Lanka listed in comment #25, we revoked a total of 51 additional certificates, which you can see in the attached file "Misc postal code revocations." We revoked the certificates today, June 22, at 3:13 pm EDT. This action concludes our investigation.

I believe this matter can be closed - I'll call this up again next Wed. 2021-07-07 for closure.

Status: ASSIGNED → RESOLVED
Closed: 3 years ago
Flags: needinfo?(bwilson)
Resolution: --- → FIXED
Product: NSS → CA Program
Whiteboard: [ca-compliance] → [ca-compliance] [ev-misissuance]
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: