SecureTrust: Unconstrained Intermediate CA Certificate not included in WTBR audit report
Categories
(CA Program :: CA Certificate Compliance, task)
Tracking
(Not tracked)
People
(Reporter: cbonnell, Assigned: cbonnell)
Details
(Whiteboard: [ca-compliance] [audit-failure])
User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:67.0) Gecko/20100101 Firefox/67.0
Steps to reproduce:
On November 26 at approximately 9 AM EST, we observed 22 Intermediate Certs with Failed ALV Results listed when we reviewed the action items in CCADB on the summary screen. Subsequent manual review of the SecureTrust 2018 standard audit report indicated that all 22 intermediate thumbprints were correctly listed in the standard audit report. However, the audit report document does not have selectable text, and the thumbprints contain embedded newlines and per-octet colon delimiters, which is potentially the reason why ALV could not find the thumbprints. Comments were added to each intermediate certificate’s CCADB entry to indicate that the corresponding intermediate is accurately listed in the audit report.
The same manual review was performed for the 2018 BR audit report. Like the 2018 standard report, the thumbprints contain newlines and colon delimiters in the thumbprint and the document does not have selectable text. Seventeen of the 18 intermediate certificates were correctly listed in the audit report, but one intermediate certificate was not under the scope of the BR audit despite being technically capable of TLS certificate issuance, because it does not contain an EKU extension (details listed below):
https://crt.sh/?id=8987754
Serial Number: 1100000187 (0x4190abbb)
Issuer: C = US, OU = www.xrampsecurity.com, O = XRamp Security Services Inc, CN = XRamp Global Certification Authority
Validity
Not Before: Dec 22 23:47:40 2008 GMT
Not After: Dec 22 23:47:40 2028 GMT
Subject: C = US, ST = Illinois, L = Chicago, O = "Trustwave Holdings, Inc.", CN = "Trustwave Code Signing CA, Level 2", emailAddress = ca@trustwave.com
X509v3 CRL Distribution Points:
Full Name:
URI:http://crl.securetrust.com/XGCA.crl
SHA256 Certificate Fingerprint 95:46:9A:45:EE:12:F3:A7:D1:CA:88:0A:2F:52:44:2E:28:63:59:39:23:FE:C0:8A:F6:BC:AB:23:75:45:9E:9B
We plan to revoke this intermediate by the close of business tomorrow (Tuesday, December 3) in compliance with the required timeframe mandated in BR 4.9.1.2.
We will follow up when the intermediate certificate in question has been revoked. As expected by Mozilla Incident Reporting guidelines, we will also provide an incident report.
Updated•6 years ago
|
Assignee | ||
Comment 1•6 years ago
|
||
Yesterday, we have revoked the non-compliant intermediate certificate within the timeframe mandated by the BRs. We are now developing a long-term remediation plan to ensure that similar violations of Mozilla Root Store Policy do not occur again.
We will follow up with a comprehensive incident report no later than December 11th.
Assignee | ||
Comment 2•6 years ago
|
||
1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.
On November 26 at approximately 9 AM EST, we observed 22 Intermediate Certs with Failed ALV Results listed when we reviewed the action items in CCADB on the summary screen. Subsequent manual review of the SecureTrust 2018 standard audit report indicated that all 22 intermediate thumbprints were correctly listed in the standard audit report. However, the audit report document does not have selectable text, and the thumbprints contain embedded newlines and per-octet colon delimiters, which is potentially the reason why ALV could not find the thumbprints. Comments were added to each intermediate certificate’s CCADB entry to indicate that the corresponding intermediate is accurately listed in the audit report.
The same manual review was performed for the 2018 BR audit report. Like the 2018 standard report, the thumbprints contain newlines and colon delimiters in the thumbprint and the document does not have selectable text. Seventeen of the 18 intermediate certificates were correctly listed in the audit report, but one intermediate certificate was not under the scope of the BR audit despite being technically capable of serverAuth certificate issuance, because it does not contain an EKU extension.
2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.
2016-12-08: Gerv posts on MDSP (https://groups.google.com/d/msg/mozilla.dev.security.policy/zdI2p_BGLpQ/HZ8BX6p6BgAJ) about a proposal to modify the Root Policy language to reference technical capability of issuing serverAuth certificates as opposed to the CA intention to issue serverAuth certificates from a given intermediate.
2017-01-12: Discussion on the MDSP list concludes, and the modification to include “language of capability” is incorporated into the version 2.4 policy draft.
2017-02-28: Policy version 2.4 is published.
2017-03-21: Clarification of required audits was integrated into draft of version 2.4.1 (https://github.com/mozilla/pkipolicy/commit/0824a8630f6968d430c33f79276161a26e688858)
2017-03-31: Policy version 2.4.1 is published. There is a note that this version has no changes in normative requirements over version 2.4.
2019-11-26 9 AM EST (approximate): Logged in to CCADB and noted 22 Failed Intermediate Cert Results. Manual review of ALV failures commenced.
2019-11-26 9:40 AM EST: Discovered that the “Trustwave Code Signing CA, Level 2”, which has no EKU (and is therefore technically unconstrained), was flagged for its thumbprint not being found in the 2018 BR audit report.
2019-11-26 10 AM EST: Discovered that this intermediate has not historically been listed in BR audit reports. Internal discussion and investigation began surrounding customer impact of revocation of the intermediate.
2019-11-26 3 PM EST: Investigation revealed that there are no unexpired, non-revoked certificates chaining to this intermediate, so there is no customer impact to revocation. Began implementing revocation procedure and scheduling for key personnel to be available.
2019-12-02 12 PM EST: Finalized revocation plan.
2019-12-02 8 PM EST: Disclosed the non-compliance incident (this bug) and timely revocation plan in Bugzilla.
2019-12-03 2:15 PM EST: Started revocation procedure and other offline CA maintenance operations.
2019-12-03 4:01 PM EST: Executed revocation script for offending intermediate.
2019-12-03 5:30 PM EST: Published updated root CRL to indicate revoked status of the offending intermediate certificate.
3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.
All non-expired, non-revoked SecureTrust intermediate certificates which chain to a root that is included in the Mozilla Root Store contain an EKU extension which makes the alignment of the intended use and technical capability explicit. Given this, there will be no future confusion of the correct audit scope regarding the technical issuance capability of a given intermediate.
4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.
The single offending intermediate certificate was issued in 2008, before industry standards had developed regarding encoding EKU extensions in certificates to denote technical capability. This certificate has been under the WTCA audit scope and was also included under the WTCS (WebTrust for Code Signing) audit in recent years, but was not listed under WTBR audits.
Despite not being listed in WTBR audits, there were several compensating controls in place to prevent the intermediate from being used for non-Code Signing certificate issuance. This intermediate has never had a CA software issuance profile such that non-Code Signing certificates could be issued. Additionally, the private key of the intermediate is hosted in the same infrastructure as our webPKI intermediates (which are audited under WTBR), so it is secured by the same logical and physical access controls in place for the webPKI intermediates.
Analysis of the issued certificate corpus from the intermediate shows that this intermediate had always issued end-entity Code Signing certificates (as denoted in the end-entity EKU extension value) and had never issued any other type of certificate.
5. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.
6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
Before version 2.4 of the Mozilla Root Store Policy, it was unclear whether intermediate certificates not intended for serverAuth certificate issuance did not need to be audited under WTBR scope. In versions up to (but not including) 2.4, the language used to define audit scope was “CA operations and issuance of certificates to be used for SSL-enabled servers”. CA intent to use a given intermediate for serverAuth certificate issuance as expressed through their operations – not technical capability of the intermediate certificate itself via lack of EKU – appeared to be the determining factor on whether the BRs were applicable (and thus yearly WTBR audits were required). Naturally, such CA operations included technical controls to prevent serverAuth certificate issuance from these intermediates, so although there was a technical capability for serverAuth certificate issuance as encoded in the intermediate certificate by lack of EKU extension, controls in place on CA systems (e.g., lack of serverAuth certificate profile configurations for the intermediate) rendered serverAuth certificate issuance from such intermediates impossible.
When version 2.4. of the Policy was modified to incorporate “language of capability”, the term “CA operations” was retained (https://github.com/mozilla/pkipolicy/commit/4b3460ea2e7b9542bdc92f106477350255c4dce1). It appeared that technical controls in place on CA systems to prevent unintended certificate issuance were still sufficient to exclude a given intermediate certificate from BR scope. Version 2.4.1 more explicitly defined policy and audit scope (https://github.com/mozilla/pkipolicy/commit/0824a8630f6968d430c33f79276161a26e688858). Through the version 2.4.1 clarifications, it appears that a normative change was introduced even though version 2.4.1 was advertised as not introducing any requirements changes from version 2.4.
A review of changes between versions 2.3, 2.4, and 2.4.1 of the Policy documents shows how such a misunderstanding could occur and be perpetuated. However, reviewing the background provided for this Mozilla Policy change to incorporate “language of capability” will turn over Gerv’s policy proposal for version 2.4, the MDSP thread referenced above (https://groups.google.com/d/msg/mozilla.dev.security.policy/zdI2p_BGLpQ/HZ8BX6p6BgAJ). This thread makes clear the intent of the version 2.4 policy change.
Although the Digital Certificates team at SecureTrust regularly monitors and contributes to MDSP, the team has not to date implemented a formal step of reviewing MDSP “Policy Proposal” threads as part of reviewing Policy changes. We believe that including such a step as part of our Mozilla Policy change review would have led to the inclusion of this requirement, thereby including the intermediate certificate in WTBR audit reports.
7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.
- The offending intermediate certificate in question has been revoked within the required timeframe as mandated by the Baseline Requirements.
- All non-expired, non-revoked SecureTrust intermediate certificates which chain to roots that are included in the Mozilla Root Store have EKUs to explicitly denote technical capability. Thus, similar confusion regarding technical capability and audit scope will not occur again.
- Beginning immediately, we will formally review all MDSP “Policy Proposal” threads for a given Policy version when new Mozilla Policy documents are released to better ascertain the intent of requirements changes. Such review will include the “Policy Proposal” threads themselves and links to corresponding policy change commits on Github (and associated discussion on Github). Mapping each policy change to the underlying background and motivation will mitigate the possibility of similar future misunderstandings.
Comment 3•6 years ago
|
||
Corey: Thanks for the detailed incident report!
In the discussion of "language of capability", what sort of analysis with respect to the Baseline Requirements' Section 8.1?
Certificates that are capable of being used to issue new certificates MUST either be Technically Constrained in line
with section 7.1.5 and audited in line with section 8.7 only, or Unconstrained and fully audited in line with all
remaining requirements from this section. A Certificate is deemed as capable of being used to issue new certificates
if it contains an X.509v3 basicConstraints extension, with the cA boolean set to true and is therefore by definition a
Root CA Certificate or a Subordinate CA Certificate.
This language dates back to BRs 1.1.6, as shown in the redline, as adopted in Ballot 105
Understanding the analysis of the language of the BRs, to a similar level as that provided against the Mozilla policy, would be useful in understanding where there may be opportunities to strengthen and improve.
Assignee | ||
Comment 4•6 years ago
|
||
A review of the discussion around Ballot 105 indicates the language surrounding capability/technical constraints was largely inspired by Mozilla Policy version 2.1 (https://cabforum.org/pipermail/public/2013-March/001371.html). Item 11 of Version 2.1 sets out the WebTrust audit requirements for SSL certificate issuance, specifically the requirement for annual WTCA and WTBR audits (https://wiki.mozilla.org/CA:CertInclusionPolicyV2.1). But as noted in our incident report above, the somewhat unclear policy and audit scopes were clarified in Policy version 2.4.1. The Baseline Requirements themselves did not specify a requirement for annual WTBR audits until 2018 with the passage of Ballot 223 (https://cabforum.org/2018/05/16/ballot-223-update-br-section-8-4-for-caaudit-criteria/). Additionally, the that ballot’s intent was not to explicitly require WTBR audits, but rather to specify equivalence for the WebTrust and ETSI audit schemes. Prior to Ballot 223, the requirement for annual WTBR audits was derived from the Root Programs’ requirements themselves.
Our analysis primarily focused on determining the root cause and remediation for why this intermediate certificate was not included in WTBR audit reports after Mozilla Policy was clarified in version 2.4.1, which was released over a year before the passage of Ballot 223. The passage of Ballot 223 was a missed opportunity to identify that the offending intermediate certificate was not correctly under WTBR audit scope.
To mitigate against future errors, we will create an “Audit Applicability Matrix” for our roots and intermediate certificates to denote which audits each certificate needs to be included under. The Matrix will cite each Root Program’s policy section that specifies the Program’s audit requirements, and will be updated as policy changes that affect audit scope occur. This “living document” will help to ensure that the correct certificates are included in each audit report. Although the issue described in the incident report will not recur due to other changes reported (including universal use of EKUs to explicitly denote technical capability), the Matrix may mitigate against failure to include certificates in audits whose inclusion criteria is not solely the EKU value (e.g., EVSSL), and allow us to track policy changes and their effect on audit scope for our root and intermediate certificates.
Comment 5•6 years ago
|
||
Wayne: Corey's insights and analysis are super useful here, and really represent the kind of thorough analysis we'd like to see more of from CAs. Comment #2 going back in the timeline, and Comment #4 going in the discussion archives, really helps with that.
I have no further questions
Comment 6•6 years ago
|
||
It appears that all questions have been answered and remediation is complete.
Updated•3 years ago
|
Updated•2 years ago
|
Updated•1 year ago
|
Description
•