1743935 - Amazon Trust Services: Misissuance of Subordinate Per CPS

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Trevoli (Amazon Trust Services)]

Steps to reproduce:

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in the MDSP mailing list (https://groups.google.com/a/mozilla.org/g/dev-security-policy), a Bugzilla bug, or internal self-audit), and the time and date.

Amazon Trust Services received a report that subordinate certificate https://crt.sh/?id=10739079 may violate the certificate profile in our CPS, which states that for Subordinates “validity:notAfter - Not later than the notAfter date of the signing certificate”. Page 47 - https://www.amazontrust.com/repository/cps-1.0.3.pdf (version at the time of issuance).

DigiCert operates this subordinate on our behalf. However, this issue is in scope of the Amazon Trust Services CPS and area of responsibility since we issued this certificate.

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

Previously reported timeline from: https://bugzilla.mozilla.org/show_bug.cgi?id=1719920
Oct 21, 2015 - Amazon Trust Services issues intermediates that will be used to create the test certificates for its repository.
NEW Nov 27, 2015 - Amazon Trust Services adds error checking into the CA system to prevent setting the notAfter date of a certificate beyond the notAfter date of the issuer.
Nov 27-29, 2015 - Amazon Trust Services corresponds with Google regarding a question related to path building libraries.
Nov 30, 2015 - Amazon Trust Services determines that the dates on the intermediates created on Oct 21, 2015 may create undesirable behavior in certain browsers and decides to correct the dates associated with the key pairs to eliminate the identified path building issues. At this time it is also determined that this doesn’t meet the miss-issuance criteria and that revocation is not necessary.
Dec 3, 2015 - Amazon Trust Services corrects the dates associated with the previously generated key pair. The old certificates are deleted as previously described in https://bugzilla.mozilla.org/show_bug.cgi?id=1713668#c7. (https://bugzilla.mozilla.org/show_bug.cgi?id=1713668#c7)

We only deleted the certificates in our CA system associated with the key pairs that we controlled. We provided DigiCert a new certificate at this time with a 2025 expiration. We didn’t follow up with DigiCert and ask them to destroy https://crt.sh/?id=10739079. We included that certificate in our repository. Finally, we failed to revoke the certificate.

New events:
Nov 25, 2021 - 5:46am PST - Amazon Trust Services received the report.
Nov 25, 2021 - 6:59am PST - Amazon Trust Services initiates an investigation of the report.
Nov 25, 2021 - 10:36am PST - Amazon Trust Services determines that the certificate was issued in violation of the CPS and that revocation is required per 4.9.1.2.

3. Whether your CA has stopped, or has not yet stopped, certificate issuance or the process giving rise to the problem or incident. A statement that you have stopped will be considered a pledge to the community; a statement that you have not stopped requires an explanation.

Yes we have stopped.

4. In a case involving certificates, a summary of the problematic certificates. For each problem: the number of certificates, and the date the first and last certificates with that problem were issued. In other incidents that do not involve enumerating the affected certificates (e.g. OCSP failures, audit findings, delayed responses, etc.), please provide other similar statistics, aggregates, and a summary for each type of problem identified. This will help us measure the severity of each problem.

https://crt.sh/?id=10739079

5. In a case involving TLS server certificates, the complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem. When the incident being reported involves an SMIME certificate, if disclosure of personally identifiable information in the certificate may be contrary to applicable law, please provide at least the certificate serial number and SHA256 hash of the certificate. In other cases not involving a review of affected certificates, please provide other similar, relevant specifics, if any.

https://crt.sh/?id=10739079

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

This bug occurred because our software didn’t check for this condition (notAfter date of a certificate beyond the notAfter date of the issuer) and a review of the certificate against the profile in the CPS was not done. A fix for the software was added on Nov 27, 2015. This avoided detection until now because even though we introduced mechanisms to detect and prevent this as described in a previous bug we didn’t retroactively review certificates created in the past.

Process fix for new certificate issuance from previous bug in 2019:
https://bugzilla.mozilla.org/show_bug.cgi?id=1525710
“Updated our off-line issuance process to require review of artifacts both prior to and following issuance against our CPS. We have also added a requirement to lint artifacts both prior to and following issuance. ”

7. List of steps your CA is taking to resolve the situation and ensure that such situation or incident will not be repeated in the future, accompanied with a binding timeline of when your CA expects to accomplish each of these remediation steps.

This would have been prevented by the code fix that was applied on Nov 27, 2015 and the process change of reviewing our CPS introduced following: https://bugzilla.mozilla.org/show_bug.cgi?id=1525710. We are also performing an audit of all existing subordinates to ensure they meet the CPS. This will be completed before the next update.

Comment 1

[Corey Bonnell]

Note that this is very similar to https://crt.sh/?id=3958242236, which I think the community decided was not a compliance issue even though it does not align with the Identrust CPS. The Identrust CPS in force at the time of ICA signing (https://www.identrust.com/sites/default/files/resources/TrustID_Certification_Practice_Statement_v4.7.6_12282020.pdf) states in section 6.3.2 that root key pairs can only be used for 20 years. The root key in question was created no later than 2000 but the ICA was signed in 2021, which is beyond the allowed operational key lifetime. The two issues should be treated identically.

Comment 2

[Ryan Sleevi]

re: Comment #1 : Note, there are a several substantive differences here that don’t seem to support the conclusion to treat them identically, but agreed, there are similarities. In particular, the question of “directly or transitively” included is very different for the (all paths expired) of Identrust vs this. Separately from the differences, I also haven’t double checked the CPS version and in force, but I seem to recall there was an update before signing.

Can you point to discussions of the CPS you feel are relevant, and where you believe the “community decided”? Just wanting to make sure we’re on the same page (especially for folks wanting to read this bug and understand the references)

Comment 3

[Corey Bonnell]

(In reply to Ryan Sleevi from comment #2)

Hi Ryan,
Comments inline.

In particular, the question of “directly or transitively” included is very different for the (all paths expired) of Identrust vs this.

At the time of ICA signing, the DST root was unexpired and audited under the BRs. The root key pair may now no longer be in scope of the BRs (and the associated obligations to follow the CPS), but that's not relevant to whether the original signing was compliant.

Separately from the differences, I also haven’t double checked the CPS version and in force, but I seem to recall there was an update before signing.

The changelog in https://www.identrust.com/sites/default/files/resources/TrustID_Certification_Practice_Statement_v4.7.7_04262021_0.pdf makes it clear that v4.7.6 (referenced in my original comment) was the policy document in force at the time of signing.

Can you point to discussions of the CPS you feel are relevant, and where you believe the “community decided”?

My comment that the community deemed this signing to be compliant wasn't a statement specifically concerning the CPS, but rather a general statement. Namely, https://letsencrypt.org/2020/12/21/extending-android-compatibility.html states that "ISRG and IdenTrust reached out to our auditors and root programs to review this plan and ensure there weren’t any compliance concerns."

Comment 4

[Trevoli (Amazon Trust Services)]

Thank you for the feedback on the bug. We have completed an audit of our intermediates against the CPS when they were issued. We identified three additional intermediate certificates created in the same Oct 21, 2015 ceremony that have the same issue. The notAfter date is beyond the notAfter date of the issuer. These certificates were not in use, so there is no impact to relying parties for revoking them. We revoked them on Dec 8, 2021.

Timeline:
Dec 2, 2021 - Additional certificates identified.
Dec 3, 2021 - Audit findings review meeting. Date for revocation is set to Dec 8, 2021.
Dec 8, 2021 - Certificates revoked.

Comment 5

[Trevoli (Amazon Trust Services)]

SHA256 for the certificates:
2847B37EF0FF545E744A45B90119CD6C7938F6F709EA3B93499AA6E57552AB3B
40CD66A295294FD0FBDC869B10E8B98F1A454A98420C84DC26885D5565B7DEB1
B3FEEE99D4D595FA837828E14DEC2C4D91E8669F92413D007A94DB0059FD0DAC

Comment 6

[Ben Wilson]

Thanks, Trev
I assume the serial numbers associated with the SHA256 hashes for these are on a CRL somewhere?
Ben

Comment 7

[Trevoli (Amazon Trust Services)]

Yes, Ben. Here are the crls that have these listed. There is only one entry in each CRL. It corresponds to those certificates.

http://crl.rootca2.amazontrust.com/rootca2.crl
http://crl.rootca3.amazontrust.com/rootca3.crl
http://crl.rootca4.amazontrust.com/rootca4.crl

Let us know if there is other information you need.

Comment 8

[Ben Wilson]

Are there any other action items, or can this incident be closed? If the latter, then I will schedule this for closure on 22-Dec-2021.

Comment 9

[Trevoli (Amazon Trust Services)]

Thank you Ben. We have no other action items. This can be closed.