Open Bug 1575880 Opened 2 months ago Updated 14 days ago

GlobalSign: SSL Certificates with US country code and invalid State/Prov

Categories

(NSS :: CA Certificate Compliance, task)

task
Not set

Tracking

(Not tracked)

ASSIGNED

People

(Reporter: douglas.beattie, Assigned: douglas.beattie)

Details

(Whiteboard: [ca-compliance])

Attachments

(2 files)

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36

Steps to reproduce:

On Tuesday August 20th 2019, GlobalSign was notified by a third party through the report abuse email address that two certificates were discovered which contained wrong State information, either in the stateOrProvinceName field or in the jurisdictionStateOrProvinceName field.

The two certificates in question were:
https://crt.sh/?id=1285639832
https://crt.sh/?id=413247173

GlobalSign started and concluded the investigation within 24 hours. Within this timeframe GlobalSign reached out to the Certificate owners that these certificates needed to be replaced because revocation would need to happen within 5 days, following the Baseline Requirements. As of the moment of reporting, these certificates have not yet been replaced, and the offending certificates have not been revoked. The revocation will happen at the latest on the 25th of August.

Following this report, GlobalSign initiated an additional internal review for this problem specifically (unexpected values for US states in values in the stateOrProvinceName or jurisdictionStateOrProvinceName fields). Expected values included the full name of the States, or their official abbreviation. We reviewed all certificates, valid on or after the 21st of August, that weren't revoked for other unrelated reasons.

To accommodate our customers globally, the stateOrProvinceName field or in the jurisdictionStateOrProvinceName are text fields during our ordering process. The unexpected values were not spotted or not properly corrected. We have put additional flagging in place to highlight unexpected values in both of these fields, and are looking at other remedial actions. None of these certificates were previously flagged for internal audit, which is completely randomized.

We will update with a full incident report for this and also disclose all other certificates found based on our research.

Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true

Doug: Thanks. As part of your update for the full incident report, it'd be great to have more detail about what "additional flagging" is and what you're examining for other remedial actions. You can see some recent discussion about this on m.d.s.p.

One thing I'd think that is useful to carefully examine is the other "Some-State" issues reported for a number of CAs. Was GlobalSign aware of these and following the discussion? The mitigations that came about, including one being listed as a good example, are both worth looking at, in the context of remedial actions, but also in understanding why GlobalSign management may not have evaluated or detected the same issue.

Assignee: wthayer → douglas.beattie
Whiteboard: [ca-compliance]

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, via a discussion in mozilla.dev.security.policy, or via a Bugzilla bug), and the time and date.

On Tuesday August 20th 2019 03:47 BST, GlobalSign was notified by a third party through the report abuse email address that two certificates were discovered which contained invalid State information, either in the stateOrProvinceName field or in the jurisdictionStateOrProvinceName field.

The two certificates in question were: 
https://crt.sh/?id=1285639832
https://crt.sh/?id=413247173

Additional reporting came in from the mdsp list [1]  on August 22 that indicated censys reported 130 GlobalSign certificates with abbreviated state in the jurisdictionStateOrProvinceName field.  We looked at this also and found that most of those were test certificates issued from a private root for testing purposes and posted to Google Test Tube log and then subsequently included in the report.  All publicly trusted SSL certificates from this report [2] are included in this incident report.

  1. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

GlobalSign started and concluded the investigation of these 2 certificates within 24 hours, at 16:51 BST. Within this timeframe GlobalSign reached out to the Certificate owners that these certificates needed to be replaced and revoked.   The certificates were revoked on 2019-08-23 12:59:04 UTC.

Following this report, GlobalSign conducted additional internal reviews for this problem. We searched for or invalid values for US states in values in the stateOrProvinceName field, or for invalid values (including abbreviations) in the  jurisdictionStateOrProvinceName fields. We reviewed all certificates, that had a notAfter date of 21st of August or later that weren't revoked for other unrelated reasons.  All of the misissued certificates were revoked on the 28th of August at the latest.  See attachment for detailed dates.

In total, we found 31 unique certificates with invalid values in the stateOrProvinceName or jurisdictionStateOrProvinceName fields. The oldest certificate was issued in October of 2016, the most recent one in July 2019.

  1. Whether your CA has stopped, or has not yet stopped, issuing TLS/SSL certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

Yes, we have stopped issuing certificates with this problem.  We added warning indicators for our validation staff so that when either of these fields contain invalid values for the US they will be highlighted.  We have also added email alerts to the Validation managers when a certificate request contains an invalid value so they can also be aware and double-check these requests.

  1. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

Number of SSL certs affected: 31
First one issued: October 2016
Most recent one issued: July 2019
Detailed dates and certificate contents are in the attached.

  1. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

We've included the details for the 31 certificates in the attached.

  1. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

This was a breakdown in our manual vetting process for Organizational details.  The unexpected values were not spotted or not properly corrected during the validation process, nor were any of these certificates selected as part of our monthly random audit sample set.  GlobalSign was aware of the "Some-State" issue, but since we were not affected by this particular issue we didn't pursue looking beyond the "some-state" value.

  1. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

We have updated our Validation system to flag stateOrProvinceName  that are not valid full or abbreviated statues, and to flag jurisdictionStateOrProvinceName values that are not full and accurate State names. We also email validation managers of received requests with these issues for secondary notifications.

We have launched an additional awareness campaign to our validation specialists, which will be followed up by targeted training and testing. Increased audits will be performed on the validation specialists associated with these misissued certificates. 

This additional awareness and training isn't limited to US State values only as there are many manually verified values in OV and EV certificates.  The awareness and importance of 100% accuracy is being re-trained into the entire global validation team.

We've increased the exposure of the MSDP list topics to our Validation Agents and their management and these topics will be discussed in monthly meetings to be sure the issues spotted are disseminated to a larger group which will help educate the whole team on industry issues.

We are examining other possibilities such as an automated address validation solution.

This file contains the details of the 31 misissued certificates.

Wayne: Do you have further questions or additional suggestions for remediation?

In particular, compared to issues like Bug 1576013 this seems like much less, and thus much more likelier to result in repeat issues going forward. One thought is for GlobalSign to share its monthly analysis of the industry issues on a recurring basis, to better understand how GlobalSign management is approaching and communicating these compliance issues, to build confidence in the remediation strategy.

Flags: needinfo?(wthayer)

Doug: your investigation into this issue and the flagging that should help to prevent it in the future is limited to certificates with US address and/or jurisdiction information - is that correct? If so, what are your plans for the rest of the world? I acknowledge that there are certain regions where the proper way to represent this information is ambiguous, but there are lots of places and situations where a CA can identify with certainty that one of these fields is incorrect, or inconsistent with the other.

Flags: needinfo?(wthayer) → needinfo?(douglas.beattie)

At the moment, it is true that the preventative measures are limited to US address or jurisdiction information. We will address this during this week (starting the 16th of September). Because in many jurisdictions, the meaning of State (and even Locality) is ambiguous, we decided to prioritize countries with clear definitions of States. In the next few days, we will review orders from Canada, Argentina, Mexico, Germany, Switzerland, Brazil, India and Austria. We will have the results available by Friday the 20th at the latest, and we will publish the results here. For these countries, we will put the same preventative measures in place as we did for the United States at first instance. If any revocations are required, we will offer our customers the opportunity to replace the certificate prior to revocations. This means that revocations will be processed on Wednesday 25th of September at the latest (depending on when the final results are in). We will publish the results in the same manner as before.

In the meantime, we are looking at other solutions for the rest of the world, and will keep you updated on the progress.

We continue to follow the different reports published by other Certificate Authorities so we can review our own processes if required.

Eva: Thanks for the update.

If I understand the appropriate to compliance, it appears to be that GlobalSign is assuming things are correct, and thus continuing issuance (potentially with incorrect or misleading information), and then on a country-by-country basis, evaluating after-the-fact whether the results were correct and/or what they should be.

That seems to be inverted from what a risk management approach might take, which would be to assume that things are not correct, and on a case-by-case basis develop controls to ensure things are correct. Among other things, for example, it might mean making sure that a centralized compliance team reviews and discusses every proposed value and, once accepted, allow-lists them. This would then be augmented by your country-by-country approach, allowing you to comprehensively address that country, while also ensuring that any localities which represent trickier approaches are reviewed by the team tasked with compliance.

Could you explain why you didn't take this approach to minimize risk? Does GlobalSign separate our validation and compliance, or are all validation agents empowered to make interpretations of compliance as well?

Flags: needinfo?(douglas.beattie) → needinfo?(eva.vansteenberge)

Hello Ryan (and all). I will answer your questions in depth by Monday, but in the meantime I wanted to confirm that we are reaching out to the 76 customers to replace their affected certificates in the lead up to the revocation by the end of Wednesday, and to publish the certificates that are currently not included in the CT logs. We will publish our results in the same manner as before (PDF with the links and the details) on Wednesday, alongside a summary of the issues we have encountered in this investigation.

Flags: needinfo?(eva.vansteenberge)

Hey Ryan, to reply to your other questions: at GlobalSign, Validation and Compliance are two separate departments with distinct functions: The validation department performs the subscriber validation based on the information provided by the applicant and applies corrections where needed - following the policies and guidance delivered by Compliance. Compliance writes the policies, reviews and approves sources of information, develops and implements the training for validation staff, performs the regular testing of the validation staff's knowledge as well as the internal audit, developing and applying appropriate controls to ensure compliance with these policies, and serving as an escalation point for questions by the Validation specialists and their management.

Customers may have a perfectly good reason not to want the exact value that our Validation Specialists can find in a QGIS, but a synonymous value, or a functional equivalent. For example "'s Gravenshage" doesn't mean much outside the Netherlands, but "The Hague" does. In ISO 3166-2, two subdivisions are listed in Belgium (Regions and Provinces), but in reality there are more (including Communities). There are municipalities and also subdivisions of these, which can be verified via the postcode listed in the QGIS (GlobalSign is listed as Leuven, but if you check the postcode, it's actually in Kessel-Lo). Additionally, the localities (and states/provinces) can be in different languages.

This is a complex subject as the discussion on the mail lists have demonstrated [1]. We support the effort to standardize on a common set of whitelist values for each country, including the supported synonymous values and those with language variations. By starting with those with minimal different sets of values we should be able to focus the majority of the countries in the first set of recommendations. This goes back to including States and Localities in the first place, depending on the country and we should specify what is necessary on a country by country basis. Saying you must have either State or Locality in the subject isn't accurate and providing a rule for which is needed for each country will standardize subject DNs across all CAs.

[1] https://cabforum.org/pipermail/servercert-wg/2019-September/001045.html

Hey all - just to confirm that we've completed the last revocations today following the investigation which was completed on Friday. In total, we revoked 481 certificates. We are completing the publication of the affected certificates on the CT logs, which will be finished on Friday, when we will publish the result here.

Type: defect → task
Attached file list_20190927.pdf

Hey all - I have added the list, which still misses a few crt.sh links. As soon as they become available I will update the document and re-upload.

(In reply to Eva Van Steenberge from comment #9)

Customers may have a perfectly good reason not to want the exact value that our Validation Specialists can find in a QGIS, but a synonymous value, or a functional equivalent.

So, I think one of the challenges with GlobalSign's responses, and why I'm pushing back, is I think we (browsers, relying parties) want more information to understand why this "may". The example you included of "The Hague" is useful, and I think meaningfully including more of these examples, or categorizing these problems, is how we move forward to build a better understanding.

I think it's important to not lose sight of the context: GlobalSign was actively issuing certificates with clearly invalid data. GlobalSign wasn't alone in this. So when it comes to trusting CAs' judgement as to what are "perfectly good reasons", the trust in CAs to make good judgments here, both individually and collectively, has been quite eroded. The way to rebuild that trust is through better transparency, and that's why I'm trying to build a picture here about how GlobalSign does this, and how and why it can or should be trusted to do this going forward.

For example "'s Gravenshage" doesn't mean much outside the Netherlands, but "The Hague" does. In ISO 3166-2, two subdivisions are listed in Belgium (Regions and Provinces), but in reality there are more (including Communities). There are municipalities and also subdivisions of these, which can be verified via the postcode listed in the QGIS (GlobalSign is listed as Leuven, but if you check the postcode, it's actually in Kessel-Lo). Additionally, the localities (and states/provinces) can be in different languages.

This, of course, opens more questions. How does GlobalSign determine the different languages are equivalent? Does it maintain a list? Is this left to the Compliance team or the Validation team to decide? What's the process for making these decisions, and if/how are they periodically reviewed to make sure the right decisions were made?

This is all presuming there is a formalized process. Some CAs have left incredible discretion to their agents, some have required a confab, a decision, and the memorialization of that decision. That's why I'm pushing GlobalSign to be much more forthcoming here - there's no such thing as "Too much information" on these incidents, especially when trust is on the line.

We support the effort to standardize on a common set of whitelist values for each country, including the supported synonymous values and those with language variations. By starting with those with minimal different sets of values we should be able to focus the majority of the countries in the first set of recommendations. This goes back to including States and Localities in the first place, depending on the country and we should specify what is necessary on a country by country basis. Saying you must have either State or Locality in the subject isn't accurate and providing a rule for which is needed for each country will standardize subject DNs across all CAs.

Concretely, talk is cheap :) It's easy to be supportive of the idea, but without actually contributing to make it real. That's part of that transition going from "describing the problem" to "fixing the problem". I appreciate that, slowly, through repeated questions, we've built a better picture about what's going on at GlobalSign. There's still a lot of opacity here, and I would hope GlobalSign would see it as an opportunity to lead, by being as transparent and proactive as some of their competitors like SecureTrust and DigiCert, to help show how they're on top of this. Similarly, it's an opportunity here to be a leader, by helping define better rules, and not just support the idea that better rules should be developed.

So are there opportunities here where GlobalSign can commit to contributing or helping? Or is this something that will need to be imposed, even if you're supportive of the general idea?

Hopefully this was read more as a call to action than an admonition, as that's what these incidents are intended to be: Stepping beyond just explaining where things went wrong, but committing to develop solutions that prevent mistakes like this.

Flags: needinfo?(eva.vansteenberge)

Hey Ryan, thank you for your feedback, which was received as a call to action as intended.

We have generally taken the view that it us up to the Validation team to validate the information provided by the Applicant, by following the policies and guidance delivered by Compliance. This means that Compliance does not maintain a list of approved values, but instead answers the questions of the Validation team by providing procedures and sources to validate the information. As mentioned previously, we have started to build “whitelists” for jurisdictions where certain information is clearly defined, but this is not the case for the majority of the jurisdictions in which we operate. Obviously it is also Compliance who checks the application of those procedures and the use of the correct sources as part of our monthly internal audit.

The way the equivalence is established depends on a number of factors, including but not limited to the broad category and the jurisdiction. We understand the need to share more information, and we have started with a categorization and quantification exercise of “synonymous values (or functional equivalents)” on a relevant sample of requests. It will hopefully bring clarity in understanding GlobalSign’s methods and sources by having both an overview of categories as well as examples of those categories together with our way of establishing equivalences in these circumstances, backed up with quantifiable data. This is definitely an area in which GlobalSign is committed to contributing and helping, and we hope that this exercise is a first step in that direction. We aim to have a draft of this exercise completed and presented in two weeks time, but we will keep you in the loop of the progress.

Flags: needinfo?(eva.vansteenberge)
You need to log in before you can comment on or make changes to this bug.