Closed Bug 1714193 Opened 3 years ago Closed 3 years ago

Sectigo: Incorrect locality information

Categories

(CA Program :: CA Certificate Compliance, task)

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: tim.callan, Assigned: tim.callan)

Details

(Whiteboard: [ca-compliance] [ov-misissuance])

  1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

Our internal audit team discovered two certificates with incorrect localityName and in one case postalCode information on May 6, 2021.

Both certificates were issued to the same organization three months apart, and both contained the same identical error in the localityName field. localityName contained the organization’s correct postal code in both cases.

  1. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

All times Eastern Daylight Time

May 6, 5:20 am
Internal audit discovers a certificate with incorrect locality and postal code information. The auditor quickly discovers a second certificate from the same RA partner issued to the same organization with the same error in its localityName field. Internal audit reports these certificates to Compliance.

10:20 am
Certificates are acknowledged by Compliance as incorrect and scheduled for revocation.

May 10, 3:36 pm
Both certificates revoked.

  1. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

These appear to be two elements of a single, isolated incident. We can find no evidence of other certificates issued with this same problem.

Nonetheless, we are removing this partner’s RA privileges. Our internal authentication team will perform authentication for certificate orders coming from this partner.

  1. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

The two certificates were issued January 4, 2021 and March 17, 2021.

  1. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

https://crt.sh/?q=bd8d0cdf5d1568da7f3adbbffe4d22562a45aeb4
https://crt.sh/?q=a64e03411d4f3fe4ecc27d5ea19a240768f2e562

  1. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

Though they were issued three months apart, both certificates were issued to the same organization, and they had identical erroneous information in the localityName field. Both certificates contained the organization’s correct postal code in the localityName field. The earlier certificate also had a postal code of a single digit (5), which is the first digit of the correct postal code. The later certificate has no postal code in keeping with Sectigo’s policy beginning in March 2021.

These two certificates were processed by one of our few remaining RA partners. OV certificates from RA partners do not undergo validation from Sectigo’s internal validation department. Rather, we police RA partner practices using internal audit and WebTrust.

We have reviewed the RA’s documentation for these two certificates, and it is correct for this organization and the other information available in these two certificates. It appears that the RA accidentally made some kind of copy-and-paste error while populating information for the first of these two certificates from the correctly authenticated organization and then failed to detect it. It appears that the RA somehow reused the same, erroneous information in the second certificate.

This RA has a strong track record over the past fifteen years. It has a current WebTrust RA Principle Report in good standing. We have gone back and reviewed all internal audit results for this RA going back to the beginning of 2017, and we found a single, subtle authentication error in that time period. In short, we had no reason to doubt this RA’s effectiveness prior to this incident.

  1. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

We are removing this partner’s RA privileges. We have reiterated our instruction to remaining RAs that they are not to copy information from previous orders when authenticating new ones.

In Q4 of 2020 and Q1 of 2021 we removed RA privileges from ten partners, mostly because we were unconvinced that they would be able to maintain the quality standards we require. For those partners we now authenticate all public certificate orders. That had left us with four active RAs, and with this change we will be down to three.

As a consequence of these two certificates taken together, we will remove RA privileges from this partner on June 15. The period between now and then is a transition period for the partner. We are reminding our remaining RAs that they must follow the authentication rules for each certificate individually.

We are open to finding a path to reinstating this partner’s RA privileges. As this is untrodden ground for us, we don’t have an established policy or procedure for how to become convinced that a partner has made the necessary changes to return to RA status. We have invited the partner to think on that and get back to us with potential evidence that could be compelling, and we will be keeping an eye on that same question on our end.

Thank you for the report.

(In reply to Tim Callan from comment #0)

  1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

Our internal audit team discovered two certificates with incorrect localityName and in one case postalCode information on May 6, 2021.

Both certificates were issued to the same organization three months apart, and both contained the same identical error in the localityName field. localityName contained the organization’s correct postal code in both cases.

How did you discover the certificates? Were they pulled at random (part of the n-% of the BR self-audit requirement), or were they flagged in some other way?

  1. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

All times Eastern Daylight Time

May 6, 5:20 am
Internal audit discovers a certificate with incorrect locality and postal code information. The auditor quickly discovers a second certificate from the same RA partner issued to the same organization with the same error in its localityName field. Internal audit reports these certificates to Compliance.

Seeing that it's currently June 2nd, could you add a comment as to why it took so long to report this issue?

  1. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

We are removing this partner’s RA privileges. We have reiterated our instruction to remaining RAs that they are not to copy information from previous orders when authenticating new ones.

In Q4 of 2020 and Q1 of 2021 we removed RA privileges from ten partners, mostly because we were unconvinced that they would be able to maintain the quality standards we require. For those partners we now authenticate all public certificate orders. That had left us with four active RAs, and with this change we will be down to three.

As a consequence of these two certificates taken together, we will remove RA privileges from this partner on June 15. The period between now and then is a transition period for the partner. We are reminding our remaining RAs that they must follow the authentication rules for each certificate individually.

We are open to finding a path to reinstating this partner’s RA privileges. As this is untrodden ground for us, we don’t have an established policy or procedure for how to become convinced that a partner has made the necessary changes to return to RA status. We have invited the partner to think on that and get back to us with potential evidence that could be compelling, and we will be keeping an eye on that same question on our end.

Have you considered adding a validator that checks non-postalCode fields for a postalCode-formatted value (if this can be done based on the subject:countryName field)?

Assignee: bwilson → tim.callan
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true
Whiteboard: [ca-compliance]

(In reply to Matthias from comment #1)

How did you discover the certificates? Were they pulled at random (part of the n-% of the BR self-audit requirement), or were they flagged in some other way?

If you are asking whether or not we were targeting this RA for special scrutiny, we were not. As stated above, this RA had a sterling record with no previous indication of potential problems.

It also was not part of our pull for the BR self-audit requirement. When our internal compliance team has spare cycles, it spends the time looking through our active certificate base for potential problems. In this case the rep noticed what clearly appeared to be wrong data in this field and then quickly found the other cert as well.

Seeing that it's currently June 2nd, could you add a comment as to why it took so long to report this issue?

Let me quote myself from bug #1694233 comment #24.

Since comment #28 we have posted more than 7500 words across six bugs. Most of these are complex in the investigation and response they require. Even the one I expected to be straightforward (bug #1712120) has left us with a significant homework assignment.

There’s a lot going on. We’re active on all of it. Where necessary we have prioritized things like responses (finding and fixing flaws; revoking affected certificates) over writing up details for this forum. That doesn’t mean we think that reporting our activities isn’t important. It is. Rather, the benefit the community gains from the information we post will still be there next week or the week after while other urgent matters get our focused effort right now.

And lastly,

Have you considered adding a validator that checks non-postalCode fields for a postalCode-formatted value (if this can be done based on the subject:countryName field)?

We're seeking to do a bunch of things like that. See the last two paragraphs of bug #1712120 comment #0 for a description of our "Guard Rails" project.

Yesterday we removed RA privileges as scheduled.

From Comment #0

We are open to finding a path to reinstating this partner’s RA privileges. As this is untrodden ground for us, we don’t have an established policy or procedure for how to become convinced that a partner has made the necessary changes to return to RA status. We have invited the partner to think on that and get back to us with potential evidence that could be compelling, and we will be keeping an eye on that same question on our end.

Do you have any updates here? I think this seems important and generalizable for how all CAs deal with their RA partners.

For example, has Sectigo investigated the WebTrust for RAs audit reports? If so, could you share your thinking there: is it something you'd allow an RA to simply provide a report on, or would you (as other CAs have done for sub-CAs), require Sectigo be a named recipient of the full, detailed audit package, to allow Sectigo to review?

What's Sectigo's current process for supervising RAs, and ensuring RAs systems, processes, and employees meet Sectigo's standards?

In Q4 of 2020 and Q1 of 2021 we removed RA privileges from ten partners, mostly because we were unconvinced that they would be able to maintain the quality standards we require. F

Similarly, what are the factors Sectigo presently uses? Issued certificates alone - or are there other processes?

I realize this probably feels like a very general question, and while it may seem unrelated to the actual misissuance event, it does seem to speak to root causes, and understanding what tools and processes can exist to prevent or reduce the risk of such systemic events.

Flags: needinfo?(tim.callan)

Despite my intentions, I haven't been able to make the cycles to compose a considered and informative reply to this question. Bug #1712188 has been greedy with my time in the past week. This post is to acknowledge the question and make it clear that we will be getting back to you in a matter of days (not weeks). Thanks again for your patience.

I'm leaving the needinfo open until we post that reply.

(In reply to Ryan Sleevi from comment #4)

Do you have any updates here?

Hi. This is Eric Kramer. I was in charge of compliance for Xolphin until our acquisition by Sectigo last year. Now I’m on the compliance team here, and among other things since April this year I own the compliance part of the RA program.

Concerning this partner, we only turned off the RA privileges on the 15th of this month, so it’s a little early to say, but so far, we haven’t received any complaints from the local market. We have the ability to give partners a variety of permissions such as read-only access to their customers’ order status and the ability to upload documents. And of course, the partner is still able to hold the subscriber’s hand through the process with guidance, answers to questions, and the like.

So, there is plenty of opportunity for the partner to continue adding value, and we have found that after an initial adjustment period almost all former RAs are sanguine with the new process they’ve been handed. That might be the outcome here as well.

[H]as Sectigo investigated the WebTrust for RAs audit reports? If so, could you share your thinking there: is it something you'd allow an RA to simply provide a report on, or would you (as other CAs have done for sub-CAs), require Sectigo be a named recipient of the full, detailed audit package, to allow Sectigo to review?

What's Sectigo's current process for supervising RAs, and ensuring RAs systems, processes, and employees meet Sectigo's standards?

Yes, we require an annual WebTrust for RAs audit report from every RA. We have not gone beyond that as internal audit has been our primary method of checking RA performance.

We have used our internal audit resources to watch orders from our RAs. We ensure we are auditing 3% of each RA’s certificates. If our general sample falls short of that goal for an individual RA, we will add random orders from that RA until the sample size meets our threshold. If we discover errors, we share them and discuss them with the RA so they can take action to prevent the same error in the future. Note that internal audit goes beyond whether or not the details in the certificate are correct. We are reviewing the RA’s full execution of our defined process and its recording of documentation and other evidence. Most errors we find are in the area of procedure or record keeping and do not constitute misissuance.

We also pay close attention to other interactions with the RA partner. An RA is not simply an order spigot. We are in regular contact with them as they ask questions and seek to resolve customer orders. Our validation and compliance teams have regular working contact with all current RAs, and we will form an impression of the quality of that operation. If an RA doesn’t seem to learn from its mistakes or internalize the updated instructions we give it, that undermines our confidence in its ability to meet quality requirements. It’s not possible for a good impression to overcome a bad metrical track record, but a bad impression is very important even when facing a good track record.

This removal of privileges is an example of that. This RA partner, as mentioned in comment #0, had a very good metrical track record. Two misissued certificates on their own likely would not have led to the removal of privileges. It was the nature of the misissuance that concerned us. Two certificates, issued to the same organization at two very different times, had the same, highly specific and rather esoteric problem. That is troubling as it invites us to question if the RA somehow copied and pasted info from earlier orders without actually going through the verification process. We can’t demonstrate that’s what happened, but we can’t come up with another explanation that doesn’t depend on an astounding level of coincidence.

So that qualitative judgement was an important part of our assessment, not just the raw numbers.

Are there any other questions or comments on this bug?

Flags: needinfo?(tim.callan)

Thanks Eric.

Ben: I don't have any questions. I'm interested to see if Sectigo's RA partner has any suggestions, but I don't think this needs to block resolution of this issue. Bug 1715024 has some overlap in terms of systemic controls being looked at on the technical side, but it seems this is primarily an area where, to the extent Sectigo is delegating their reputation (and risk to it) to third-parties, the controls are going to be non-technical, such as the option they took: removing the RA's access.

Flags: needinfo?(bwilson)

Was the RA partner required to provide you with its own incident report? If so, can that be shared here?
What remediation steps / improvements will you require the RA partner to perform before its privileges can be reinstated?

(In reply to Ben Wilson from comment #9)

Was the RA partner required to provide you with its own incident report? If so, can that be shared here?

No, it was not. After we discovered the mis-issued certificates, we conducted our own investigation, which included questioning the RA partner.

What remediation steps / improvements will you require the RA partner to perform before its privileges can be reinstated?

Since we shut down the privileges of this RA Partner, we became confident that in this case those privileges maybe should remain “turned off”. This partner hasn’t proposed any such steps, and we haven’t created a path to reinstating these privileges.

And, as I explained in comment #6, we have the ability to give partners a variety of permissions such as read-only access to their customers’ order status and the ability to upload documents and to offer additional guidance on the ordering process. We believe that these permissions open the door to a very good customer experience without requiring RA privileges.

If you have any other questions, let me know.

(In reply to Ben Wilson from comment #9)

Ben, we are aware that you are on PTO until the 28th, so after your return please let us know if you have any further questions, or if this bug can be closed.

I intend to close this matter on Friday, 30-July-2021.

Status: ASSIGNED → RESOLVED
Closed: 3 years ago
Flags: needinfo?(bwilson)
Resolution: --- → FIXED
Product: NSS → CA Program
Whiteboard: [ca-compliance] → [ca-compliance] [ov-misissuance]
You need to log in before you can comment on or make changes to this bug.