1746945 - Amazon Trust Services: Missing CAA Check For Test Website Certificates

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Trevoli (Amazon Trust Services)]

Steps to reproduce:

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in the MDSP mailing list (https://groups.google.com/a/mozilla.org/g/dev-security-policy), a Bugzilla bug, or internal self-audit), and the time and date.

During an internal self audit one of our engineers noted that we missed performing a CAA check for certificates we issued for our test websites on Dec 8, 2021.

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

Dec 8, 2021 - Amazon Trust Services performs a ceremony to revoke Intermediates (more info: https://bugzilla.mozilla.org/show_bug.cgi?id=1743935). During this ceremony 10 certificates are also created for the test websites. Five of the certificates are revoked immediately after being issued. These certificates would have been used as test revoke certificates. The other five certificates would have been used as test valid certificates.
Dec 14, 2021 - Amazon Trust Services engineer assists with a CAA check for a ceremony where CRLs and OCSP Signing Certificates were created. Notes that a CAA check is probably not relevant for this operation.
Dec 16, 2021 - Amazon Trust Services team meets to discuss CAA checking and reviews the validation done on Dec 8, 2021. It’s determined that a CAA check was required and had been missed for 10 certificates made for our test websites.
Dec 17, 2021 - Amazon Trust Services revokes the five certificates that would have been used as test valid certificates.

3. Whether your CA has stopped, or has not yet stopped, certificate issuance or the process giving rise to the problem or incident. A statement that you have stopped will be considered a pledge to the community; a statement that you have not stopped requires an explanation.

Yes we have stopped.

4. In a case involving certificates, a summary of the problematic certificates. For each problem: the number of certificates, and the date the first and last certificates with that problem were issued. In other incidents that do not involve enumerating the affected certificates (e.g. OCSP failures, audit findings, delayed responses, etc.), please provide other similar statistics, aggregates, and a summary for each type of problem identified. This will help us measure the severity of each problem.

Serial - 07:59:71:50:8A:7D:81:37:19:69:00:51:41:7E:C6:45:B3:95:81
SHA 256 Fingerprint - 8B:35:EE:4C:7A:9D:9E:06:12:88:C4:17:D3:EB:D7:1B:4C:00:4D:9C:81:53:6F:AD:59:11:EB:C5:DB:6C:51:F8

Serial - 07:59:71:71:E8:FB:1B:A9:4D:FD:8F:21:43:EF:CC:67:71:86:01
SHA 256 Fingerprint - D4:B0:79:5C:43:EA:C3:09:AD:E5:99:83:C9:B4:62:9E:48:01:35:36:8E:1F:FF:C4:84:29:21:04:CE:28:24:3F

Serial - 07:59:71:74:CB:D7:78:DB:D5:F7:98:E7:BB:E0:00:8B:F5:AE:18
SHA 256 Fingerprint - 7D:03:A9:44:88:9B:91:EB:D9:73:1F:B7:F9:3E:91:35:ED:6E:64:ED:B3:5A:04:45:A4:19:3D:76:73:14:42:DD

Serial - 07:59:71:77:17:7D:0F:B8:58:4B:B5:EE:4E:36:25:52:EF:5D:27
SHA 256 Fingerprint - 51:88:CB:96:BA:38:61:1A:1F:34:0C:5F:33:BD:C6:81:75:52:BC:FA:83:3E:DD:2E:A1:74:3D:E1:0B:F7:27:3F

Serial - 07:59:71:78:FC:88:03:1A:73:E2:FA:DF:B9:34:D4:82:FE:8C:0F
SHA 256 Fingerprint - CF:F7:BE:B5:1B:FD:9C:46:82:10:CC:4D:74:52:90:1A:60:BA:0B:4B:67:67:36:5C:3F:B9:F4:D2:36:BE:CD:96

Serial - 07:59:71:7B:78:CC:7D:73:21:73:CA:13:CF:97:E6:0D:46:46:C1
SHA 256 Fingerprint - CD:85:28:58:9F:5D:BB:BD:78:EA:E5:11:50:DB:E5:1D:01:20:AE:30:43:19:DF:26:C0:1F:E3:E7:FD:A1:29:2D

Serial - 07:59:71:82:14:97:E8:A4:23:82:E0:29:20:7D:9C:98:8A:F0:9C
SHA 256 Fingerprint - 8E:69:1D:D6:D7:9C:86:55:8B:E2:C4:4A:46:34:F9:6C:CA:FD:82:9A:44:96:75:CA:60:03:5A:8C:2D:17:8B:82

Serial - 07:59:71:86:4E:54:B1:92:B7:A2:4D:FC:09:98:DD:E7:39:AE:57
SHA 256 Fingerprint - A5:82:76:8D:FF:98:00:9B:1E:5D:5F:1B:2C:59:86:96:E9:8D:F5:A7:5D:E1:68:9D:A6:EF:04:81:80:34:3D:52

Serial - 07:59:71:8C:42:80:C4:04:2D:94:55:93:AB:FF:85:84:66:C7:42
SHA 256 Fingerprint - DA:65:C1:00:0E:BF:12:99:9F:A1:E7:F9:15:26:E1:B5:AB:16:84:F1:17:86:30:55:F9:A9:6A:2B:0A:7B:50:E5

Serial - 07:59:71:91:B2:0A:F1:27:60:8E:DA:16:BE:8E:F0:91:D6:FD:D3
SHA 256 Fingerprint - 32:A4:86:FD:D2:C1:B8:21:AA:2B:E3:37:94:4F:9C:FB:32:7F:77:9A:54:D8:73:99:82:B0:42:4A:45:24:B8:00

5. In a case involving TLS server certificates, the complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem. When the incident being reported involves an SMIME certificate, if disclosure of personally identifiable information in the certificate may be contrary to applicable law, please provide at least the certificate serial number and SHA256 hash of the certificate. In other cases not involving a review of affected certificates, please provide other similar, relevant specifics, if any.

See question 4.

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

To explain how this occurred I’m going to provide some background for how we create these certificates. The intermediates used to issue our test website certificates are offline on HSMs stored in a secure location. Every operation with these intermediates is therefore entirely manual. We employ several mechanisms to ensure that ceremonies performed on these devices are secure, consistent, and correct. There are opportunities to improve on these mechanisms to eliminate the recurrence of this issue. First, we use a Deployment Tracking template which creates a checklist for all the steps preceding and following the ceremony. This tracks actions such as scope approval for the ceremony, domain validation, pre-ceremony sign off on testing, etc. The Deployment Tracker is reviewed and approved prior to beginning the ceremony. Second, we have a Ceremony Script template that is printed and used by all parties participating in the ceremony. It specifies the actions such as operations with the HSM and who will perform them. The Ceremony Script is reviewed as part of the pre-ceremony testing sign off step. Third, we have a source controlled Commands Template that is used for the actual commands described in the ceremony script. The Commands Template is reviewed prior to testing a ceremony and during the pre-ceremony testing sign off step. Fourth, we have a Pre-Ceremony pack list that specifies which physical items need to be brought to perform a ceremony. This is used shortly before we go to access the HSMs. This list contains items such as printed copies of the Ceremony Script, pens, CDs, etc. Two people review the items being brought to the ceremony to make sure we have the necessary items.

We were aware of this requirement to check CAA records. During our review of Ballot SC46 we were aware that we could no longer use the existing exception for CAA checks. We verified that our Deployment Tracking template had a step to track validations but we failed to ensure the template contained specific details on the activities that needed to occur such as the CAA check.

For our Dec 8, 2021 ceremony, the technician in charge of the Deployment Tracker used the domain validation from our June 23, 2021 ceremony and didn’t have a second reviewer, such as a Validation Specialist, review the existing domain control check to ensure it met the required standard.

7. List of steps your CA is taking to resolve the situation and ensure that such situation or incident will not be repeated in the future, accompanied with a binding timeline of when your CA expects to accomplish each of these remediation steps.

We’ve made the following changes to reduce the chance of reoccurrence:

1. CAA checking is now explicitly listed as it’s own step on the Deployment Tracking template.
2. In the Deployment Tracking template, EV/OV has been split out from DV and CAA records checking. DV and CAA records checking now happen as the step right before the ceremony.
3. The Ceremony Script has been modified to record the CAA record check timestamp. A step has been added to verify the timestamp is within 8 hours in two places, right before the commands are run to create valid test certificates and again before the commands are run to create revoke test certificates.
4. Pre-Ceremony pack list now includes bringing a printed copy of the CAA records check so the time stamp can be verified prior to issuance during the offline ceremony.

Comment 1

[Matthias]

The intermediates used to issue our test website certificates are offline on HSMs stored in a secure location. Every operation with these intermediates is therefore entirely manual.

Assuming that you also have online intermediates that are used for automated day-to-day issuance of valid WebPKI certificates; why don't you use those CAs for the automated issuing of test webpage certificates?

Using automated systems would prevent the requirement of manual certificate data validation and input, and prevent the mistakes related to manual certificate issuance.

Comment 2

[Trevoli (Amazon Trust Services)]

Thank you for the feedback Matthias. We agree that online intermediates would allow us to automate this and other checks. Amazon Trust Services doesn’t operate any online intermediates for day to day issuance. DigiCert operates the intermediates used to issue day to day Amazon Trust Services certificates.

Comment 3

[Trevoli (Amazon Trust Services)]

Amazon Trust Services is monitoring this bug for feedback. We have no further action items planned.

Comment 4

[Ben Wilson]

I will close this on 14-Jan-2022 unless there are any items that need to be addressed further.

Comment 5

[Trevoli (Amazon Trust Services)]

Thank you Ben, we have no further items planned.