(In reply to Ryan Sleevi from comment #8)
(In reply to douglas.beattie from comment #7)
How was this issue missed?
- The AEG product codes were initially designed for issuing non-public SSL certificates, but over time the need arose to support publicly trusted SSL certificates. When this happened, our testing did not uncover this missing check.
- Normally, all API requests using this product code originate from GlobalSign on-site application and that application generates valid requests; however, as was evident as part of this incident, there are also situations when different end entity applications submit certificate requests. It's evident that those clients don't always generate requests with valid CN and SAN values, and that identified this deficiency in our CA checks.
Thanks, Doug. This sounds a bit like saying you relied on client-side checks/validation (via the on-site application), rather than server-side checks.
Actually, no, we didn't rely on client side checks. It turned out that the specific scenario which generated these non-conformant certificates was one that we hadn't specifically tested (clearly). The typical AEG flow originates from the AEG client application and uses the AEG product codes; however, this request originated from a different client and we hadn't tested that specific (unexpected) scenario.
I also totally understand that products evolve over time, and features grow and accumulate. With respect to this incident, and preventing future incidents, what sort of steps have been taken (either in response to this, or in the past, or planned) to help ensure that this API and the AEG product codes cover all the necessary requirements?
Looking back, we've completed an exhaustive review of the processing of AEG certificate requests and we've verified that all of the proper checks are in place. Going forward, we've certainly learned that we need to pay even closer attention to product changes when we add new features. We're also discussing if/how we could completely remove the AEG variant of OV from OV into it's own dedicated product line so the processing of OV doesn't include any conditional logic for private trust. It's certainly the safest solution, but also has a much larger impact on our dev team and customers who wold all need to be upgraded.
I don't think linting is necessarily a sufficient answer here, in as much as there are plenty of procedural opportunities for issues (e.g. the Some-City/Some-State discussions going on).
Yes, agree. We include necessary checks into the CA system to prevent this, and we use linting as a secondary, independent check.
I'm largely unaware of how divergent this API and product code are from the existing GlobalSign issuance pipeline and practices, and so I'm trying to get a picture about what the different systems in play are, where they're similar, and where they're different, so I can better understand how this issue has been both contained and remediated.
It's unclear exactly what procedural concerns are related to this incident. The enterprise system is based on previously verified organizational details and domains, then when requests are placed using this data, certificates are immediately issued. There weren't any procedural breakdowns as it relates to this issue. The bug here was that he system didn't include a proper check for the format of the SANs, which we've addressed.
As far as this API vs. others and how they relate, this is THE core GlobalSign MSSL API for the issuance of all SSL certificates from our managed platform, so this isn't divergent, it's our core platform. As discussed, we certainly missed an important check when AEG product code was used for publicly trusted SSL, and we've resolved that.
In this regard, it strikes me as a bit similar to the guidance given to DigiCert in Bug 1550645, both in the request for information and in understanding the procedural approach being taken. Can you help build a similar detailed understanding for how GlobalSign's systems are architected, how the failure happened, and what sort of steps have been taken?
Have you reviewed the other product codes (not sure what this refers to? API endpoint? specific customer?)?
- Yes, we have completed an exhaustive review and have not found any other related deficiencies for the validation of certificate requests
Great. Coupled with the above question, it might be useful to highlight or indicate what of those systems in place you reviewed. This is the general desire for incident reports - trying to build an understanding about how the system(s) operate (and all of the system(s) in play), and how the CA is responding or addressing them.
We reviewed our GCC ordering system and especially our RA system which enforces all of the required checks, domain validations, etc. It's separate from the GCC code and databases and is developed by a different set of engineers who focus on compliance enforcement checks. So in summary, we reviewed both GCC and RA systems.
Has there been any timeline to deprecating the AEG interface, as originally implied?
- Yes, the V1 API will be restricted to only a small set of customers and those customers will be moved from this API. We're targeting the end of July for this. The AEG product code will continue to be used via the V2 API, and with the additional unit tests and the use of zlint preissuance checks, we don't expect issues like this to reoccur.
I'm still a bit confused between the relationship between the V1 API and the AEG product code. I had thought AEG meant the V1 API, but this response makes me think they're different systems. Can you help build an understanding here about how GlobalSign's systems are (perhaps this is already documented in your CP/CPS?)
The versioning of APIs isn't documented in the CPS, but let me try to explain it here. Several years ago we rolled out an updated API (V2) with some additional features and fields that would have otherwise broken current V1 implementations. We provide customers a transition time to move over and update their code, then we were planning to disable the prior API. In this case, V1 remained in use longer than it should have. Ordering via this API will be disabled by the end of June, but we'll permit a limited set of customers to continue using the query features in the API (some still use it)
V2 of the API supports all of our enterprise SSL products as well as the AEG (a variant of our MSSL OV product). As discussed above, we've completed an extensive review of the code and have determined that when any required BR checks are omitted, it's for either privately trusted SSL certificates, or it's for certificates that don't have the serverAuth EKU.