Closed Bug 1667844 Opened 4 years ago Closed 3 years ago

Google Trust Services: Certificates not disclosed in CCADB

Categories

(CA Program :: CA Certificate Compliance, task)

Product:
CA Program
Component:
CA Certificate Compliance
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: kathleen.a.wilson, Assigned: rmh)

Details

(Whiteboard: [ca-compliance] [disclosure-failure])

https://crt.sh/mozilla-disclosures#undisclosed is reporting that the following two certificates are not disclosed in the CCADB.

https://crt.sh/?sha256=349dfa4058c5e263123b398ae795573c4e1313c83fe68f93556cd5e8031b3c7d&opt=mozilladisclosure
Serial Number:
02:03:e5:c0:68:ef:63:1a:9c:72:90:50:52
Signature Algorithm: ecdsa-with-SHA384
Issuer: (CA ID: 48274)
commonName = GTS Root R4
organizationName = Google Trust Services LLC
countryName = US
Validity
Not Before: Jun 22 00:00:00 2016 GMT
Not After : Jun 22 00:00:00 2036 GMT
Subject: (CA ID: 48274)
commonName = GTS Root R4
organizationName = Google Trust Services LLC
countryName = US

https://crt.sh/?sha256=d947432abde7b7fa90fc2e6b59101b1280e0e1c7e4e40fa3c6887fff57a7f4cf&opt=mozilladisclosure
Serial Number:
02:03:e5:93:6f:31:b0:13:49:88:6b:a2:17
Signature Algorithm: sha384WithRSAEncryption
Issuer: (CA ID: 48269)
commonName = GTS Root R1
organizationName = Google Trust Services LLC
countryName = US
Validity
Not Before: Jun 22 00:00:00 2016 GMT
Not After : Jun 22 00:00:00 2036 GMT
Subject: (CA ID: 48269)
commonName = GTS Root R1
organizationName = Google Trust Services LLC
countryName = US

It looks like they are self-signed certificates, but they also chain up to root certificates included in root stores. So, you probably don't intend them to be directly included in root stores. Therefore, they should be reported in the CCADB as intermediate certs.

See bug #1652581, specifically the "gtsr1.pem" and "gtsr4.pem" attachments.

Assignee: kluge → rmh

I've added the six PEMs into the CCADB as subordinate CAs, even though these are roots. (Section 5.3 of Mozilla Policy says: "All certificates that are capable of being used to issue new certificates, and which directly or transitively chain to a certificate included in Mozilla’s CA Certificate Program, MUST be operated in accordance with this policy and MUST either be technically constrained or be publicly disclosed and audited.") They have been marked "Same as Parent" in the CCADB. For audit purposes, this is the case "If the CA has a currently valid audit report at the time of creation of the certificate, then the new certificate MUST appear on the CA's next periodic audit reports." We will need GTS to file a root inclusion request to include the new roots and then remove the old roots. Section 5.3.2 requires that new CAs be publicly disclosed in the CCADB by the CA that has their certificate included in Mozilla’s root program. We would also like GTS to file an incident report in this bug so that we can learn how to ensure that new CAs are reported in the CCADB and to prevent similar miscommunication in the future. Thanks.

I believe there may be an opportunity to revise the current Mozilla policy to more clearly accomodate this particular scenario.

Some background; prior to the creation of the revised certificates for these Root CAs we of course read the policy, during that review we did not identify specific language about this use case so we contacted the Mozilla and stated what our intention was, namely to re-create the roots with the narrow changes needed to address this issue and then to, update CCAADB along with those updates. We understood from those conversations that this was the correct path and so that is what we set out to do.

Once the revised certificates were created we went to update CCADB and found that we did not have sufficient permissions to update the certificates associated with these roots (apparently the issues was because they were root certificates and not sub ca certificates) so we again reached out to Mozilla and requested that Mozilla update CCADB with the updated certificates that were attached to the bug and were told that they would.

From this bug I understand that we are now being requested to:

  1. Add these certificates as subordinate CA certificates in CCADB,
  2. And file a root inclusion request as if these are new root CA certificates.

We are of course happy to do so, and will do so this week, but should this scenario ever happen again it would be ideal if the policy on how this is to be handled were better documented so that future root program members don't run afoul of this particular workflow.

Here is a policy clarification that we're going to push in the next version (2.7.1) of the Mozilla Root Store Policy.
https://github.com/mozilla/pkipolicy/issues/186

  1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

As detailed by Ryan Hurst, we were aware of the need to update these roots in CCADB and contacted Mozilla when CCADB tooling prevented us from doing so. Initial guidance was to add the certificates to https://bugzilla.mozilla.org/show_bug.cgi?id=1652581 and that a CCADB admin would update the records. This is effectively another variation of https://github.com/mozilla/pkipolicy/issues/186 which is pending resolution and in this case caused both Mozilla and GTS representatives confusion about the best path forward.

  1. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

2020-07-07 GTS determines that the best mitigation to incident 1652581 is the reissuance of the Root Certificates

2020-07 (multiple dates) GTS communicates with the various root programs and our plan to re-issue and request guidance. All contacted Root programs agreed that reissuance was the right way to address the issue.

2020-08-13 - Re-issuance ceremony conducted.

2020-08-14 - Initial attempt to update CCADB was made and not successful.

2020-08-17 - Other GTS members with CCADB access attempted upload to confirm the issue.

2020-08-21 - GTS reached out to Mozilla root program representatives for guidance on how to proceed.

2020-08-28 - The re-issued certs were added to https://bugzilla.mozilla.org/show_bug.cgi?id=1652581.

2020-09-28 - An incident report was requested and this report is being provided.

  1. Whether your CA has stopped, or has not yet stopped, certificate issuance or the process giving rise to the problem or incident.

Original versions of the certificates that were 'not reported' to CCADB are already trust anchors. Issuance from those CAs is already disclosed. No need was identified to stop issuance of certificates.

  1. In a case involving certificates, a summary of the problematic certificates. For each problem: the number of certificates, and the date the first and last certificates with that problem were issued. In other incidents that do not involve enumerating the affected certificates (e.g. OCSP failures, audit findings, delayed responses, etc.), please provide other similar statistics, aggregates, and a summary for each type of problem identified. This will help us measure the severity of each problem.

See point 5 for a list of affected certificates.

  1. In a case involving certificates, the complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem. In other cases not involving a review of affected certificates, please provide other similar, relevant specifics, if any.

The relevant crt.sh links are:

https://crt.sh/?sha256=349dfa4058c5e263123b398ae795573c4e1313c83fe68f93556cd5e8031b3c7d&opt=mozilladisclosure
https://crt.sh/?sha256=d947432abde7b7fa90fc2e6b59101b1280e0e1c7e4e40fa3c6887fff57a7f4cf&opt=mozilladisclosure

  1. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

CCADB does not have support for re-issued roots. This manifested as a permission error when GTS tried to update CCADB. We relied on direct communications with CCADB administrators to identify a workaround to this issue. This is an area where improved community guidance would help.

  1. List of steps your CA is taking to resolve the situation and ensure that such situation or incident will not be repeated in the future, accompanied with a binding timeline of when your CA expects to accomplish each of these remediation steps.

We are working with Mozilla to resolve the issue to their satisfaction. Update policy and associated guidance should ensure that in the future others do not encounter this problem. The 2.7.1 Mozilla Root Policy Store update with resolution of https://github.com/mozilla/pkipolicy/issues/186 including coverage of re-issued certs would provide good guidance.

Here is a link to the root inclusion case information listed in the CCADB - https://ccadb-public.secure.force.com/mozilla/PrintViewForCase?CaseNumber=00000666

Also, Bugzilla bug #1675821 has been opened for adding these six root CA certificates.

We appreciate everyone’s input on this issue. In the interest of full disclosure we would like to share further information in the Mozilla Incident Report format to provide additional background on the R2 modification:

  1. How your CA first became aware of the problem.

Comments on this bug have pointed out that the certificate modification may not have met the requirements in BR 7.1.3.2.1 and/or the Mozilla Root Store Policy 5.1.3.

  1. A timeline of the actions your CA took in response.

See timeline in initial posting.

  1. Whether your CA has stopped, or has not yet stopped, certificate issuance or the process giving rise to the problem or incident.

We have stopped issuing subordinate CAs from R2.
Subordinate CAs under the modified R2 continue issuing.

  1. In a case involving certificates, a summary of the problematic certificates.

GlobalSign Root R2

  1. In a case involving certificates, the complete certificate data for the problematic certificates.

https://crt.sh/?id=3448659678

  1. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

The decision to perform certificate modifications of roots was made after a careful consideration of available options and after consultation with auditors and root program representatives.

The modification aimed to add the missing digitalSignature bit while keeping the remainder of the certificate profile as it was to prevent compatibility issues in user agents.

We believe there is a need for such modification to be possible e.g. by providing for a key usage exception similar to the EKU exception in BR 7.1.3.2.1 and Mozilla Root Store Policy 5.1.3. We would welcome the community’s input on this question.

  1. List of steps your CA is taking to resolve the situation and ensure that such situation or incident will not be repeated in the future, accompanied with a binding timeline of when your CA expects to accomplish each of these remediation steps.

We do not plan any additional modifications of the R2 certificate. Since R2 is going to expire on December 15, 2021 we are migrating issuance to a different chain over the coming months.

It appears that the serial number length in the reissued R2 certificate is different from the original certificate. Additionally, the ordering of the CRLDP and AKID extensions was reversed in the reissued certificate compared to the original certificate. Were these changes to the tbsCertificate structure intentional?

Corey, R2 was initially created by GlobalSign. We use different tooling that has current checks for all applicable requirements. As we use our own updated tooling, the minor differences you noted were expected. We reviewed all of the changes and deemed them non-material. Altering tooling to try to preserve ordering and historical serial length would have added risk without clear benefit.

Assuming there are no further comments, I will close this bug on or about 11-Dec.

Flags: needinfo?(bwilson)
Status: ASSIGNED → RESOLVED
Closed: 3 years ago
Flags: needinfo?(bwilson)
Resolution: --- → FIXED
Product: NSS → CA Program
Whiteboard: [ca-compliance] → [ca-compliance] [disclosure-failure]
You need to log in before you can comment on or make changes to this bug.