1586792 - QuoVadis: Issuance of intermediates after 2019-01-01 that do not comply with Mozilla Policy or the BRs

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Ryan Sleevi]

During a spot-check of Mozilla Policy Compliance, I discovered that QuoVadis issued an intermediate that does not conform with Mozilla Policy 2.6.1

Mozilla Policy 2.6.1 requires (formatted for readability), in Section 5.3:

Intermediate certificates created after January 1, 2019, with the exception of cross-certificates that share a private key with a corresponding root certificate:

• MUST contain an EKU extension; and,
• MUST NOT include the anyExtendedKeyUsage KeyPurposeId; and,
• MUST NOT include both the id-kp-serverAuth and id-kp-emailProtection KeyPurposeIds in the same certificate.

The following intermediate, for "QuoVadis EU Issuing Certification Authority G4", was issued on 2019-05-14: https://crt.sh/?q=7f93e8e4bee47624ced4e8384dba96c4828d461d787d0ae3eb316de985d51c6c

This intermediate lacks an extendedKeyUsage extension.

It also asserts an organizationIdentifier within the Subject, which would conflict with the Baseline Requirements' 7.1.2.4 and 7.1.4.3.1, that latter of which does not allow for "Other Subject Attributes" (different from 7.1.4.2, which is specific to Subscriber certificates)

Please provide an Incident Report, as per https://wiki.mozilla.org/CA/Responding_To_An_Incident

Comment 1

[Stephen Davidson]

How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

In our investigation regarding bug 1581597 on 3 Oct 2019 we identified this issue with this CA. We confirmed that no TLS certificates have been been issued from the CA. In that prior bug 1581597 we requested the CA be added to OneCRL. A meeting was schedule for 7 Oct to address the issue.

A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

3 Oct 2019 We identified the issue with this CA, meeting scheduled for 7 Oct.
4 Oct 2019 We researched the possible approaches, which are complicated by the fact that this CA is listed on the EUTL to issue Qualified certificates (Non-TLS).
7 Oct 2019 QuoVadis PKI operations meet to evaluate options.

Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

All new QuoVadis CAs include EKU.

A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

https://crt.sh/?q=7f93e8e4bee47624ced4e8384dba96c4828d461d787d0ae3eb316de985d51c6c

The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

https://crt.sh/?q=7f93e8e4bee47624ced4e8384dba96c4828d461d787d0ae3eb316de985d51c6c

Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

The original “QuoVadis EU Issuing Certification Authority G4” was created in 2016 before the requirement for EKU. The CA was renewed in 2019 to extend level constraints. As this was a renewal, it was believed the lack of EKU was not an issue. QuoVadis accepts that this was a mistake.
QuoVadis typically does not renew issuing CAs; our manual checklists for new CAs require EKU.

As the CA was not intended to issue TLS, section 7.1.4.3.1 of the Baseline Requirements was not taken into account. Is it your reading of that section, that TLS-issuing CAs may only include Subject DN commonName, organizationName, and countryName?

List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

QuoVadis typically does not renew issuing CAs; our manual checklists for new CAs require EKU. The renew checklists have been changed.

We wanted to respond rapidly to this bug. This CA is included on a European Trust List for non-TLS and issues certificates mainly on QCSD. Thus steps to replace it require some additional caution. We will follow up within the week with timeframes resulting from the work started today to replace this CA with an EKU constrained version.

Comment 2

[Ryan Sleevi]

(In reply to Stephen Davidson from comment #1)

Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

The original “QuoVadis EU Issuing Certification Authority G4” was created in 2016 before the requirement for EKU. The CA was renewed in 2019 to extend level constraints. As this was a renewal, it was believed the lack of EKU was not an issue. QuoVadis accepts that this was a mistake.
QuoVadis typically does not renew issuing CAs; our manual checklists for new CAs require EKU.

Is my understanding correct that QuoVadis had trouble understanding what "intermediate certificates issued after 1-January 2019" meant? That this was not a certificate issued after then?

I mean, it's good that QuoVadis accepts it's a mistake, it'd be far worse if they didn't, but it somewhat strains incredulity that one could reach this conclusion, given the communications, and given the past discussions in m.d.s.p. regarding new certificates complying with the rules.

If QuoVadis believes there was reasonable grounds for this confusion, then it'd be useful to understand how QuoVadis proposes to address this confusion in the future.

If QuoVadis believes it was a mistake in judgement, then it'd be useful to understand how QuoVadis plans to address this, and more generally, how QuoVadis plans to ensure it has not made other mistakes in understanding or interpretation.

If we want these incident reports to be a way to drive the industry forward, and we believe the mistakes are reasonable (which I'm not agreeing this particular issue/explanation is), then provide insight on how to improve things. Alternatively, if our faith is shaken to the core about the ability of the CA to understand the requirements set forth, and that they agreed to, then how do we move forward reducing the risk from that CA operator?

As the CA was not intended to issue TLS, section 7.1.4.3.1 of the Baseline Requirements was not taken into account. Is it your reading of that section, that TLS-issuing CAs may only include Subject DN commonName, organizationName, and countryName?

I'd rather pose the opposite: What interpretation of the BRs, and of Section 7.1.4.3.1, did QuoVadis use when evaluating that this issuance would be permitted? What's the process QuoVadis uses? Given that we've identified there was misunderstanding about Mozilla Policy above, understanding the current interpretation, process, and controls is useful here.

List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

QuoVadis typically does not renew issuing CAs; our manual checklists for new CAs require EKU. The renew checklists have been changed.

We wanted to respond rapidly to this bug. This CA is included on a European Trust List for non-TLS and issues certificates mainly on QCSD. Thus steps to replace it require some additional caution. We will follow up within the week with timeframes resulting from the work started today to replace this CA with an EKU constrained version.

As I see it, the renew checklists being changed is a corrective action, but it doesn't really move to provide any long-term assurances. If we assume the process itself broke down, which seems to be the case, understanding what steps are being done to overhaul that compliance process is useful.

Comment 3

[Stephen Davidson]

Hi Ryan: As noted elsewhere, several CAs misinterpreted the applicability to renewals of the rule regarding EKU in new intermediate certificates issued after 1 January 2019. It is not an issue of competence, but rather a clear understanding of the requirement. Now it’s clear; thank you.

Since the time of that renewal, the QuoVadis PKI team has been integrated into the DigiCert PKI Operations team and adopted many of their practices, including checklists and review signoffs. That process continues and, as you know, DigiCert has a serious commitment to standards development, compliance review, and continuous improvement in operations.

Noting the separate discussion at https://groups.google.com/forum/#!topic/mozilla.dev.security.policy/3iuG8KGryC4, we do not see BR Section 7.1.4.3.1 as restricting CA certificates to include only commonName, organizationName, and countryName fields. Moreover, a review of CT finds many hundreds of trusted TLS-issuing CAs that include other fields in their Subject DN. We agree that CAs should be free to include information such as addresses or corporate identifiers in their Subject DN.

Comment 4

[Ryan Sleevi]

(In reply to Stephen Davidson from comment #3)

As noted elsewhere, several CAs misinterpreted the applicability to renewals of the rule regarding EKU in new intermediate certificates issued after 1 January 2019.

It would be useful if you could provide links to support this.

Of the three related bugs - Bug 1586795, Bug 1586847, and Bug 1586787 - no incident report has identified the confusion that QuoVadis is asserting. The closest I could see supporting QuoVadis' argument is Bug 1586795, but that bug clearly indicates that the issue was a lack of communication from staff about the Mozilla changes, and thus a failure to implement the change - not, as QuoVadis is asserting here, a misinterpretation about the applicability.

It is not an issue of competence, but rather a clear understanding of the requirement.

Just to highlight, this doesn't address the question raised in Comment #2, where I stated

If QuoVadis believes there was reasonable grounds for this confusion, then it'd be useful to understand how QuoVadis proposes to address this confusion in the future.

I don't see how QuoVadis is addressing this confusion in the future. It seems to be saying "We're not confused anymore on this specific issue", but this does not propose solutions for how to resolve future confusion, and thus it seems equally as likely that Policy 2.7 may be instituted, QuoVadis/DigiCert will confirm it understands, then cause a violation, state they were confused, and state that they're no longer confused. I want to understand if there are opportunities to break that cycle before it happens, because that would benefit not just QuoVadis, but all CAs.

Since the time of that renewal, the QuoVadis PKI team has been integrated into the DigiCert PKI Operations team and adopted many of their practices, including checklists and review signoffs. That process continues and, as you know, DigiCert has a serious commitment to standards development, compliance review, and continuous improvement in operations.

I am deferring this to Wayne if he'd like to close it out.

I'm deeply concerned with an approach to CA management that says "Under new management; all old issues are in the past". DigiCert has, for example, the largest number of compliance issues of any currently-trusted CAs, so I do not feel particularly reassured on this front. As it applies to the issuance of Sub-CAs, DigiCert has had a repeat number of issues regarding the issuance, auditing, and disclosure of Sub-CAs, so similarly, I don't feel particularly reassured.

I can understand and appreciate if QuoVadis does not have any further actionable details to share. I think such a non-response would be useful to consider, when considering the holistic set of issues a CA (such as QuoVadis + DigiCert) has, and how they respond to incidents. I know this seems extreme, but I want to reiterate: The point of these incident reports is

1. To build confidence that the CA holistically understands the issue and has addressed it
2. To develop better approaches, as an industry, to prevent such issues.

I don't believe the response really meets either of those requirements. Which is fine, I'm not going to try to pluck a diamond out of coal, but it would certainly become part of the considerations when holistically considering the CA.

If the DigiCert folks want to share any added insight, that would likely be very useful. Otherwise, I defer to Wayne to see if he's happy with the explanation provided and the steps provided to prevent future confusion by QuoVadis/DigiCert.

Comment 5

[Jeremy Rowley]

I agree this cert requires revocation, not just under the Mozilla policy but also under the Microsoft policy. I think the fact that the renewal was treated differently than new issuance is an indication of the need to integrate the two entities faster.

QuoVadis still operates, and is audited, as an independent CA. While DigiCert has taken some steps to integrate our processes and knowledge, our primary focus remains on shutting down legacy Symantec operations, which is still targeted for Q2 2020, as previously shared. Currently, we have implemented some policy controls in place with QuoVadis, including a requirement that our PKI ops team sign off on key ceremonies (although not in time for the reissue of this intermediate). We will add technical controls as we integrate QuoVadis over the next year, including linting. At that point, the'll be able to share the knowledge we've gained from the previous incident reports and operations. I'll coordinate with Stephen and get this ICA revoked right away. I'll also see if I can come up with a plan and timeline on an integration with Quovadis so we can give a better timeframe than a vague "2020".

Just to clarify, we don’t believe old issues are purely an issue of the past. They can definitely come back to haunt people. There's also a need to cleanup areas that we weren't in control of, including a non-mutual understanding of industry requirements. For example, DigiCert treats renewals as new issuance. Quovadis obviously didn't at the time this issuing CA was done. You're right that there is always an opportunity for an organization to misinterpret their industry requirement. That's why we think a transparent view is best. With a holistic view of what we're doing, we can ensure that there is a meeting of the minds on what's going on in the industry. One reason we file frequent bug reports and like it when bugs getting split into multiple parts is so we can get a wider view of browser expectations and give us a better view into what community expectations look like. Anyway, I guess what I'm saying is that we're on the same page on what we want these incident reports to be - words of wisdom that benefit the community as a whole, not just the CA re-mediating the problem.

Comment 6

[Stephen Davidson]

The CA certificate has been revoked/superceded as of CRL Number=05b3 effective ‎October ‎15, ‎2019 7:15:35 PM GMT.

https://crt.sh/?id=1473552421&opt=ocsp

Thank you for your patience. As noted previously, this transition required care as the CA was included on the European Trusted List and issued certificates on QCSD/tokens to individuals.

Comment 7

[Wayne Thayer]

It appears that all questions have been answered and remediation is complete.