User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36
Steps to reproduce:
How your CA first became aware of the problem:
On December 11th 2019 the Mozilla 2.7 Policy was published and January 1st it became effective [https://blog.mozilla.org/security/2019/12/11/announcing-version-2-7-of-the-mozilla-root-store-policy/].
In support of compliance tracking as this policy developed, we monitored the development of this policy and the impact the changes would have on our current and future issuance and once the final version was published, we conducted another review. The latest review made it clear we had previously made a mistake.
During the latest review we identified that two of our subordinate CAs (GTSY3 and GTSY4) were generated with a curve-hash combination that was not permitted under Mozilla Policy 2.6.1, the policy that they were created under and this was made clear under the revised 2.7 policy.
These subordinate CAs were created to support the demo sites for GTS Root R3 and GTS Root R4. These roots used a used a P-384 signing key to sign the GTSY3 and GTSY4 certificates with the SHA256withECDSA algorithm.
The language on this topic in 2.6 [https://github.com/mozilla/pkipolicy/blob/2.6/rootstore/policy.md#51-algorithms] stated:
ECDSA keys using one of the following curve-hash pairs:
- P‐256 with SHA-256
- P‐384 with SHA-384
While the new language in 2.7 [https://github.com/mozilla/pkipolicy/blob/2.7/rootstore/policy.md#512-ecdsa] states:
If the signing key is P-256, the signature MUST use ECDSA with SHA-256. The encoded AlgorithmIdentifier MUST match the following hex-encoded bytes: 300a06082a8648ce3d040302.
If the signing key is P-384, the signature MUST use ECDSA with SHA-384. The encoded AlgorithmIdentifier MUST match the following hex-encoded bytes: 300a06082a8648ce3d040303.
In both cases, the combination of P-384 and SHA256 is not permitted and unfortunately when reviewing the certificate profile of the associated CAs prior to creation, none of the reviewers or the quality control tooling identified the issue. After investigating similar compliance issues by other CAs, we believe the more ambiguous 2.6.1 wording was a contributing factor.
While observing the evolving 2.7 policy our focus was on changes, and since the text in 2.6 was clear about the requirement and the improved text in 2.7 simply was a more explicit statement of the same we did not notice the issue until we did the final review of the 2.7 policy.
A timeline of the actions your CA took in response:
2018-10-29 GTSY3 and GTSY4 Subordinate CAs are issued under Mozilla Root Store Policy 2.6.1.
2019-03-01 An issue opens at the Mozilla Root Store Policy repository to update the language of the requirements and clarify the exact allowed curve-hash pairs as a result of some CAs misinterpreting the requirements.
2019-12-11 A review of the not yet in effect Mozilla 2.7 policy was conducted by the CA policy, operations and development teams. No changes were identified as requiring changes to existing systems or configurations
2020-01-01 Mozilla Root Store Policy 2.7 enters into effect clarifying the allowed curve-hash pairs further.
2020-01-27 A regular internal review flags 2 historical key-algorithm mismatches as worthy of deeper review in light of actions other CAs took
2020-01-27 Public discussions concerning the policy change are revisited to confirm the consensus / most conservative interpretation of applicable rules
2020-01-28 The decision is made to revoke and replace both GTSY3 and GTSY4
2020-01-30 This bug is posted.
Whether GTS has stopped, or has not yet stopped, issuing certificates with the problem:
GTS has stopped issuance of subCA certificates with this problem. The only leaf certificates issued from the two affected subCAs were the 6 required 'demo' sites.
A summary of the problematic certificates:
Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now:
When these CAs were created we relied exclusively on multiple human reviews of certificate profiles and testing to determine appropriateness of the certificate profiles. Though these reviewers were from across our organization and experienced in the area, each reviewer approved these certificate profiles and missed the curve-hash mismatch.
We have included Zlint in our review process for some time but it did not until very recently get support for testing for Mozilla policies [https://github.com/zmap/zlint/issues/277], this policy is covered in this new version.
A human review was conducted following the 2019 incidents relating to other CAs with effectively the same problem:
At the time, we still felt our erroneous interpretation of 2.6.1 was valid and that the CAs in question were being extra cautious by revoking. Historically, we have taken a consensus driven approach to evaluating reported incidents and the consensus was that our initial reviews and interpretation were still valid.
To reduce the chances of erroneous consensus reviews, we will be instituting a change to have a formal review committee consisting of both policy officers and technical staff who have experience in cryptographic algorithms review each Mozilla CA related bug and provide a report detailing similarities and differences to our setup and practices to ensure a detailed review. By incorporating this process into our standard operating procedures we believe we can reduce the likelihood of future process failures.
Steps GTS is taking to resolve the situation and ensure such issuance will not be repeated in the future (including timeline):
We are revoking the subordinateCAs on 2020-01-31.
We are also currently updating to the most recent version of ZLint, which we use as part of our policy compliance checks. This version has support for checking this issue [https://github.com/zmap/zlint/issues/277] amongst other Mozilla Policy compliance checks. Augmenting our manual review processes with this updated version of Zlint will further help ensure that we do not encounter this issue again.
Google Trust Services