Closed Bug 1612389 Opened 4 years ago Closed 4 years ago

Google Trust Services: invalid curve-hash combination

Categories

(CA Program :: CA Certificate Compliance, task)

task
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: awarner, Assigned: awarner)

Details

(Whiteboard: [ca-compliance] [ca-misissuance])

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36

Steps to reproduce:

How your CA first became aware of the problem:

On December 11th 2019 the Mozilla 2.7 Policy was published and January 1st it became effective [https://blog.mozilla.org/security/2019/12/11/announcing-version-2-7-of-the-mozilla-root-store-policy/].

In support of compliance tracking as this policy developed, we monitored the development of this policy and the impact the changes would have on our current and future issuance and once the final version was published, we conducted another review. The latest review made it clear we had previously made a mistake.

During the latest review we identified that two of our subordinate CAs (GTSY3 and GTSY4) were generated with a curve-hash combination that was not permitted under Mozilla Policy 2.6.1, the policy that they were created under and this was made clear under the revised 2.7 policy.

These subordinate CAs were created to support the demo sites for GTS Root R3 and GTS Root R4. These roots used a used a P-384 signing key to sign the GTSY3 and GTSY4 certificates with the SHA256withECDSA algorithm.

The language on this topic in 2.6 [https://github.com/mozilla/pkipolicy/blob/2.6/rootstore/policy.md#51-algorithms] stated:

ECDSA keys using one of the following curve-hash pairs:

  • P‐256 with SHA-256
  • P‐384 with SHA-384

While the new language in 2.7 [https://github.com/mozilla/pkipolicy/blob/2.7/rootstore/policy.md#512-ecdsa] states:

If the signing key is P-256, the signature MUST use ECDSA with SHA-256. The encoded AlgorithmIdentifier MUST match the following hex-encoded bytes: 300a06082a8648ce3d040302.

If the signing key is P-384, the signature MUST use ECDSA with SHA-384. The encoded AlgorithmIdentifier MUST match the following hex-encoded bytes: 300a06082a8648ce3d040303.

In both cases, the combination of P-384 and SHA256 is not permitted and unfortunately when reviewing the certificate profile of the associated CAs prior to creation, none of the reviewers or the quality control tooling identified the issue. After investigating similar compliance issues by other CAs, we believe the more ambiguous 2.6.1 wording was a contributing factor.

While observing the evolving 2.7 policy our focus was on changes, and since the text in 2.6 was clear about the requirement and the improved text in 2.7 simply was a more explicit statement of the same we did not notice the issue until we did the final review of the 2.7 policy.

A timeline of the actions your CA took in response:

2018-10-29 GTSY3 and GTSY4 Subordinate CAs are issued under Mozilla Root Store Policy 2.6.1.
2019-03-01 An issue opens at the Mozilla Root Store Policy repository to update the language of the requirements and clarify the exact allowed curve-hash pairs as a result of some CAs misinterpreting the requirements.
2019-12-11 A review of the not yet in effect Mozilla 2.7 policy was conducted by the CA policy, operations and development teams. No changes were identified as requiring changes to existing systems or configurations
2020-01-01 Mozilla Root Store Policy 2.7 enters into effect clarifying the allowed curve-hash pairs further.
2020-01-27 A regular internal review flags 2 historical key-algorithm mismatches as worthy of deeper review in light of actions other CAs took
2020-01-27 Public discussions concerning the policy change are revisited to confirm the consensus / most conservative interpretation of applicable rules
2020-01-28 The decision is made to revoke and replace both GTSY3 and GTSY4
2020-01-30 This bug is posted.

Whether GTS has stopped, or has not yet stopped, issuing certificates with the problem:

GTS has stopped issuance of subCA certificates with this problem. The only leaf certificates issued from the two affected subCAs were the 6 required 'demo' sites.

A summary of the problematic certificates:

https://crt.sh/?id=912361731
https://crt.sh/?id=912361742

Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now:

When these CAs were created we relied exclusively on multiple human reviews of certificate profiles and testing to determine appropriateness of the certificate profiles. Though these reviewers were from across our organization and experienced in the area, each reviewer approved these certificate profiles and missed the curve-hash mismatch.

We have included Zlint in our review process for some time but it did not until very recently get support for testing for Mozilla policies [https://github.com/zmap/zlint/issues/277], this policy is covered in this new version.

A human review was conducted following the 2019 incidents relating to other CAs with effectively the same problem:

  1. https://bugzilla.mozilla.org/show_bug.cgi?id=1534145
  2. https://bugzilla.mozilla.org/show_bug.cgi?id=1530971

At the time, we still felt our erroneous interpretation of 2.6.1 was valid and that the CAs in question were being extra cautious by revoking. Historically, we have taken a consensus driven approach to evaluating reported incidents and the consensus was that our initial reviews and interpretation were still valid.

To reduce the chances of erroneous consensus reviews, we will be instituting a change to have a formal review committee consisting of both policy officers and technical staff who have experience in cryptographic algorithms review each Mozilla CA related bug and provide a report detailing similarities and differences to our setup and practices to ensure a detailed review. By incorporating this process into our standard operating procedures we believe we can reduce the likelihood of future process failures.

Steps GTS is taking to resolve the situation and ensure such issuance will not be repeated in the future (including timeline):

We are revoking the subordinateCAs on 2020-01-31.

We are also currently updating to the most recent version of ZLint, which we use as part of our policy compliance checks. This version has support for checking this issue [https://github.com/zmap/zlint/issues/277] amongst other Mozilla Policy compliance checks. Augmenting our manual review processes with this updated version of Zlint will further help ensure that we do not encounter this issue again.

Kind regards,
Google Trust Services

Assignee: wthayer → awarner
Status: UNCONFIRMED → ASSIGNED
Type: enhancement → task
Ever confirmed: true
Whiteboard: [ca-compliance]

Andy: Thanks for filing this report.

From the timeline, it's unclear when the human review was conducted of Bug 1534145 and Bug 1530971. I also didn't see discussion of the timing and review of what lead to those incidents being filed, namely https://groups.google.com/d/msg/mozilla.dev.security.policy/mCKvUmYUMb0/sqZVnFvKBwAJ , nor the related Bug 1527423 that was the precursor to those bugs.

I mention this, because it's useful to understand what went wrong with the consensus review process and how the proposed steps would address it. While it's certainly valuable to ensure a variety of perspectives are represented, it also seems useful to ensure that the internal consensus review considers external clarifications and guidance as part of that.

Could you provide more timeline and background for the process of monitoring mozilla.dev.security.policy , which was also introduced in version 2.5, and how that is being changed/updated in light of this?

Flags: needinfo?(awarner)

Per historical meeting notes and email archives, Bug 1534145 and Bug 1530971 were discussed the week they were posted (via email) and the following week (in the next team meeting). I did not find any specific records of discussion of Corey's post and Bug 1527423, but it is enough different from the issue we encountered, a disallowed hash algorithm was used and not an invalid curve-hash pair with allowed curves and hashes that I believe it was reviewed quickly and deemed non-applicable.

I believe the primary problem with our consensus driven approach is that we were largely discussing the matters in regular weekly meetings where other matters were also covered. This meant that there was sometimes pressure to fit the discussion into an agenda that may have not allowed sufficient research and debate. The right participants were involved, but time pressure appears to have contributed to a bad outcome. By moving each issue to a standalone discussion we expect to avoid similar problems. If you have alternative suggestions, we'll happily consider any feedback.

Similarly, our reviews of routine m.d.s.p posts are covered in a standing weekly meeting. For large updates, we have specific checks as part of our monthly and quarterly audit procedures which trigger reviews and summaries being written up and provided to the entire team for any changes to BRs, root programs and other applicable requirements. As part of these meetings we will be adding a specific recap and review of postings from the last week to ensure each thread has been sufficiently reviewed.

Flags: needinfo?(awarner)

While I appreciate the clarifications, I'm still primarily concerned with this part of the report:

At the time, we still felt our erroneous interpretation of 2.6.1 was valid and that the CAs in question were being extra cautious by revoking. Historically, we have taken a consensus driven approach to evaluating reported incidents and the consensus was that our initial reviews and interpretation were still valid.

While the independent meetings focused on compliance sounds like a valuable and useful change, it does seem that there still exists a significant possibility that the result of evaluating other incidents and issues may lead GTS to determine the CA is being overly cautious, or to disagree with an interpretation in a way that is favorable to GTS' current operations. This creates the risk of future incidents that may arise due to GTS being mistaken in its understanding and application of the requirements.

It seems like this is an opportunity for GTS to work how to better receive clarification for interpretative differences, or to incorporate adversarial reviews (similar to the "Default Deny" discussion in the CA/B Forum), in which GTS evaluates both their policies/practices and those requirements of Root Stores in order to examine if there are other interpretations, and to seek clarification or revisions in order to address that confusion.

I'm not sure how GTS reached a conclusion that it was valid, and I'm glad that the clarifications in 2.7 reduced that ambiguity, but I'm fundamentally concerned about the other sections which don't lend themselves to 2.7's level of prescriptiveness, and how to address those, both retrospectively and going forward. Certainly, if GTS has ideas on how to better prevent misunderstanding, it would be great to have these addressed as part of the prevention aspects.

Flags: needinfo?(awarner)

We have successfully revoked and re-issued. We continually work on improvements and are factoring community feedback into discussions.

Flags: needinfo?(awarner)

Thanks for the update.

Resetting Needs-Info because I understand Comment #4 was a follow-on to Comment #0 and wasn't related to Comment #3.

Flags: needinfo?(awarner)

We understand and share your concern.

We made extensive investments in 2019 on automation of compliance verification in an effort we have been calling “continuous compliance” with the goal of automating verification of as many of the operational requirements and practices as possible. This shift to relying on code for these checks more and more has the side effect of ensuring individual decisions like this are also part of the code and design review process we leverage for other parts of our work.

This change, along with other operational and procedural improvements we have already made, we believe, would have caught this issue today.

Specifically to the recommendation of adversarial reviews with a focus on looking at other potential interpretations, this is something we added as part of the aforementioned changes and is one of the reasons why we believe that this would not have happened for a CA we would have created today and is why the issue surfaced under the new process.

Flags: needinfo?(awarner)

I'm not really convinced by Comment #6, and I'm hoping you can be a bit more transparent with the details, so we can better understand.

My takeaway from Comment #6 is:

  • We've invested in tools, rather than people
  • We're looking at requirements more carefully

Which is good, but it's neither transparent nor necessarily helps the ecosystem improve. For example, on the basis of Comment #6, using only the information in this bug, I don't think it'd be reasonable to conclude the issue would be caught today. GTS could still make a determination that it doesn't apply and similarly not write a tool. The goal of these incident reports is to try to be transparent in a way that the community can actually understand why

This change, along with other operational and procedural improvements we have already made, ... would have caught this issue today.

Such a statement is devoid of content, and thus it doesn't allow other CAs to make the same "operational and procedural improvements", or understand what good practices are there. It doesn't provide details about why you believe this issue would have been caught today. The shift to relying on code doesn't seem to address the issues discussed in Comment #3.

I'm being vague here because I don't want to suggest things, only to have the CA mentioned "Oh, yes, those things, we implemented them", because by the time we get there, the trust/faith in the CA has been eroded. That said, I'm happy to make suggestions for how to address some of these issues, and to understand if there are challenges with these suggestions / if they'd been evaluated and rejected / etc. But I do want to provide one more opportunity here for GTS to provide a similar level of transparency as other CAs have.

Flags: needinfo?(awarner)

During the internal postmortem, it was identified that while the certificate profile was reviewed by multiple parties from the Policy Authority, the review of the signature algorithm was conducted by the technical members of our team who believed that this curve-hash combination was allowed by the Mozilla policy. This happened due to the fact that the signature algorithm was considered a technical issue, and not a policy one.

When the issue was brought up by HARICA and SSL.com, the Policy Authority who handles compliance monitoring, reviewed their findings and compared them against the certificate profile. However, as it was displayed, the faulty process did not include the signature algorithm, and thus the reviewers did not identify that GTS is also affected by the same issue.

This led us to believe that the root cause of the incident was a fault in our procedures which excluded the Policy Authority from reviewing all details of a newly issued SubCA Certificate.

In order to mitigate this problem and ensure that it will not happen again in the future, we are restructuring our review process to include adversarial reviews in our procedures, and we are engaging the Policy Authority through all steps of the process. Since the investigation identified a human error as the root cause, we are also introducing additional tooling and automation processes to identify further similar problems using technical means.

Flags: needinfo?(awarner)

Thanks for the update. I think my concerns originally raised in Comment #3 still somewhat apply.

My biggest concern going forward is, regardless of whether it's the Policy Authority or technical members of the team, anything that is seen as ambiguous, or any incident reports that are deemed CAs are being "extra cautious" and thus not applicable, has the opportunity to actually be resolved in a public and transparent way.

For example, if GTS committed to raising on mozilla.dev.security.policy any requirements that members of either the technical or policy team found confusing, or engaged in incidents involving other CAs in order to seek clarity from Peers/Owners/Other UAs, it seems like this might substantially mitigate the risk.

That, combined with adversarial reviews (that is, imagining what's the worst possible interpretation, no matter how nonsense it might be, that might be argued), and sharing the results of those reviews for discussions and improvements, seems like it would cut these issues down.

I'm assigning to Wayne, as there aren't further questions, just missed opportunities here that will hopefully be considered next time.

Wayne: Any further questions or suggestions?

Flags: needinfo?(wthayer)

It appears that all questions have been answered and remediation is complete.

Status: ASSIGNED → RESOLVED
Closed: 4 years ago
Flags: needinfo?(wthayer)
Resolution: --- → FIXED
Summary: GTS - invalid curve-hash combination → google Trust Services: invalid curve-hash combination
Product: NSS → CA Program
Summary: google Trust Services: invalid curve-hash combination → Google Trust Services: invalid curve-hash combination
Whiteboard: [ca-compliance] → [ca-compliance] [ca-misissuance]
You need to log in before you can comment on or make changes to this bug.