Closed Bug 1588213 Opened 5 years ago Closed 4 years ago

IdenTrust: Missing Thumbprints In Some Annual Audit Reports

Categories

(CA Program :: CA Certificate Compliance, task)

Product:
CA Program
Component:
CA Certificate Compliance
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: roots, Assigned: roots)

Details

(Whiteboard: [ca-compliance] [audit-failure])

Attachments

(2 files, 1 obsolete file)

IdenTrust - 2019 WebTrust for CAs - Report.pdf
100.82 KB, application/pdf
Details
IdenTrust - 2019 WebTrust for CAs (SSL Baseline) - Report.pdf
196.35 KB, application/pdf
Details

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:69.0) Gecko/20100101 Firefox/69.0

Steps to reproduce:

Based on this this Mozilla forum post on 10/8/2019 regarding audit details for intermediary issuing CA certificates
https://groups.google.com/forum/#!topic/mozilla.dev.security.policy/M7NGwCh14DI
We have accessed Mozilla’s CCADB and found 14 Intermediate Certs with Failed ALV Results.
We can confirm that there are neither miss-issued certificates nor security issues with any of the flagged subordinates. As requested in the Mozilla post, we will provide a formal incident report no later than October 18, 2019.

Further review of the of the failed ALV details, we confirmed that all relevant thumbprints were included in at least one or both of the WebTrust reports. For those subordinates where the thumbprints were missing in both audit reports, we confirmed that they are not in the scope of the supplied audit reports and therefore no incident report is required. Relevant notes for each flagged subordinate were added in the CCADB.

Status: UNCONFIRMED → RESOLVED
Closed: 5 years ago
Resolution: --- → INVALID

Marking this re-opened, to make sure we have sufficient detail. In general, we want to make sure someone from the Mozilla CA Certificate Policy module closes it out (since they're the ones who do enforcement).

A potential concern here is that if these certificates are flagged, it may mean that they are capable of TLS issuance. If they're capable of TLS issuance, then they need to be audited and disclosed within both audits. Appearing in one, but not the other, would only be acceptable if these certificates are not technically capable of issuance.

Could you highlight which 14 certificates you believe are in compliance?

Status: RESOLVED → REOPENED
Ever confirmed: true
Flags: needinfo?(roots)
Resolution: INVALID → ---
Assignee: wthayer → roots
Status: REOPENED → ASSIGNED
Type: defect → task

Here is a detail of the 14 subordinate CA certificates flagged as missing SHA-256 thumbprints:
13 of the 14-flagged subordinate CAs are capable of issuing TLS certificates.
(2) Have a separate AUP (Agreed Upon Procedure) agreed with Mozilla since 2016 where auditors attest each year that no TLS certificates were issued.
CA A7
B191EDEB4CB772C7983150C047517F50BD9221C8CBC9349A11D0CC3E7EF72BFD
CA A8
2FA315DC154C213DC0C16FD12FA021BA9974F0C9AE142E24BF3AED150A7CE6FC

(4) Are in the WebTrust audit report with SHA-256 thumbprints in both reports but not in the expected format. Auditors notified to correct in next report.
Let's Encrypt Authority X1
7FDCE3BF4103C2684B3ADBB5792884BD45C75094C217788863950346F79C90A3
Let's Encrypt Authority X2
EC0C6CA496A67A13342FEC5221F68D4B3E53B1BC22F6E4BCCC9C68F0415CDEA4
Let's Encrypt Authority X3
25847D668EB4F04FDD40B12B6B0740C567DA7D024308EB6C2C96FE41D9DE218D
Let's Encrypt Authority X4
A74B0C32B65B95FE2C4F8F098947A68B695033BED0B51DD8B984ECAE89571BB6

(2) Are in the WebTrust audit report but missing the thumbprints in the B.R. Audit report, not in the standard audit report. Auditors notified to correct in next report.
TrustID Server CA A52
B39C4A4596D3191AFA3B3D254D28E5C482FCD0D500E0A9337F99277CB8A2EEF8
Booz Allen Hamilton BA CA 01 – Not capable to issue TLS certificates
DCCA716167F029AA9A309EE8CA3FF1F4017D1A1F3D1981BDFF9E5AF3F503682A

(1) Is in the WebTrust audit report but missing the thumbprints in the standard audit report, but not in the B.R. report. Auditors notified to correct in next report.
IdenTrust ACES CA 2 - SHA-256 Fingerprint
C5480D7BFF952D1BE86178FF713F11F51CF74232EE5676FC5A170D4A6A6FE50A

(3) Are not included in the WebTrust Audit Report; although they have different serial number and different SHA-256 fingerprints, cryptographically they are identical to this resigned subordinate disclosed in the supplied WebTrust audit reports:
IdenTrust Commercial CA Root CA 1 5D56499BE4D2E08BCFCAD08A3E38723D50503BDE706948E42F55603019E528AE

IdenTrust Commercial Root CA 1
91B18588225035BB2F231FEF7695E497B289934B65CB87CFC2212271EBECB58C
IdenTrust Commercial Root CA 1
F49793F8DF83CE64A8C8D50DF366B64E98C2538A2AAAB2019CA0367A1FCC03CB
IdenTrust Commercial Root CA 1
AAE38F67F0A626805928507B078D89D5598D760D17335927B46606ECDFA1B946

(2) Are not included in the WebTrust Audit Report; although they have different serial number and different SHA-256 fingerprints, cryptographically they are identical to this resigned subordinate disclosed in the supplied WebTrust audit reports:
IdenTrust Public Sector Root CA 1: C5480D7BFF952D1BE86178FF713F11F51CF74232EE5676FC5A170D4A6A6FE50A
IdenTrust ACES CA 2
9D1585E63B4D03D9ABBA0C67D46730BADF0FCEBC2081611CF7B9AA572D2D64A4
IdenTrust ACES CA 2
A59740F91153C0FB1C1E37081CD7198E0BC28B58C1D561DB785CB82B4AD9DF47

Flags: needinfo?(roots)

(In reply to roots from comment #3)

Here is a detail of the 14 subordinate CA certificates flagged as missing SHA-256 thumbprints:
13 of the 14-flagged subordinate CAs are capable of issuing TLS certificates.
(2) Have a separate AUP (Agreed Upon Procedure) agreed with Mozilla since 2016 where auditors attest each year that no TLS certificates were issued.
CA A7
B191EDEB4CB772C7983150C047517F50BD9221C8CBC9349A11D0CC3E7EF72BFD
CA A8
2FA315DC154C213DC0C16FD12FA021BA9974F0C9AE142E24BF3AED150A7CE6FC

Okay

(4) Are in the WebTrust audit report with SHA-256 thumbprints in both reports but not in the expected format. Auditors notified to correct in next report.
Let's Encrypt Authority X1
7FDCE3BF4103C2684B3ADBB5792884BD45C75094C217788863950346F79C90A3
Let's Encrypt Authority X2
EC0C6CA496A67A13342FEC5221F68D4B3E53B1BC22F6E4BCCC9C68F0415CDEA4
Let's Encrypt Authority X3
25847D668EB4F04FDD40B12B6B0740C567DA7D024308EB6C2C96FE41D9DE218D
Let's Encrypt Authority X4
A74B0C32B65B95FE2C4F8F098947A68B695033BED0B51DD8B984ECAE89571BB6

The thumbprints listed above are for the cross-certs, while the audit report referenced in CCADB only lists the roots (because it's LE's audit). Are you sure the CCADB record is correct? What do you mean by "not in the expected format"?

For all the rest below, it's not acceptable to fix the problem going forward. Identrust should either (1) revoke the CA certificate, (2) ask for it to be added to OneCRL (only if not intended for serverAuth and included in the standard WebTrust audit), or (3) ask the auditor to issue a corrected report, and provide a date in this bug by when we can expect to receive the updated report.

(2) Are in the WebTrust audit report but missing the thumbprints in the B.R. Audit report, not in the standard audit report. Auditors notified to correct in next report.
TrustID Server CA A52
B39C4A4596D3191AFA3B3D254D28E5C482FCD0D500E0A9337F99277CB8A2EEF8
Booz Allen Hamilton BA CA 01 – Not capable to issue TLS certificates
DCCA716167F029AA9A309EE8CA3FF1F4017D1A1F3D1981BDFF9E5AF3F503682A

(1) Is in the WebTrust audit report but missing the thumbprints in the standard audit report, but not in the B.R. report. Auditors notified to correct in next report.
IdenTrust ACES CA 2 - SHA-256 Fingerprint
C5480D7BFF952D1BE86178FF713F11F51CF74232EE5676FC5A170D4A6A6FE50A

(3) Are not included in the WebTrust Audit Report; although they have different serial number and different SHA-256 fingerprints, cryptographically they are identical to this resigned subordinate disclosed in the supplied WebTrust audit reports:
IdenTrust Commercial CA Root CA 1 5D56499BE4D2E08BCFCAD08A3E38723D50503BDE706948E42F55603019E528AE

IdenTrust Commercial Root CA 1
91B18588225035BB2F231FEF7695E497B289934B65CB87CFC2212271EBECB58C
IdenTrust Commercial Root CA 1
F49793F8DF83CE64A8C8D50DF366B64E98C2538A2AAAB2019CA0367A1FCC03CB
IdenTrust Commercial Root CA 1
AAE38F67F0A626805928507B078D89D5598D760D17335927B46606ECDFA1B946

(2) Are not included in the WebTrust Audit Report; although they have different serial number and different SHA-256 fingerprints, cryptographically they are identical to this resigned subordinate disclosed in the supplied WebTrust audit reports:
IdenTrust Public Sector Root CA 1: C5480D7BFF952D1BE86178FF713F11F51CF74232EE5676FC5A170D4A6A6FE50A
IdenTrust ACES CA 2
9D1585E63B4D03D9ABBA0C67D46730BADF0FCEBC2081611CF7B9AA572D2D64A4
IdenTrust ACES CA 2
A59740F91153C0FB1C1E37081CD7198E0BC28B58C1D561DB785CB82B4AD9DF47

Flags: needinfo?(roots)
Summary: IdenTrust Missing Thumbprints In Some Annual Audit Reports → IdenTrust: Missing Thumbprints In Some Annual Audit Reports
Whiteboard: [ca-compliance]

Thanks.

I have to share, I'm quite concerned with understanding how we can independently verify and assess these claims. That's part of the importance for addressing in the thumbprints. The current proposal would not have us have any independent attestation for another year, and it would only cover that year - not this year. So we have no insight into what sort of activities are in-scope or out-of-scope, and from understanding how the audits are to be used, it's not the auditor who misrepresented things if things are totally non-compliant.

This is why other CAs have been requested to revoke such certificates by other root programs, and why Mozilla has moved to OneCRL a number of them. This case is interesting, because unlike those other cases, it does appear Identrust appears to intend to use these for web server authentication, and so the OneCRL/Revocation mechanism has greater impact and side-effects. However, it's important to emphasize: the relying parties/public have no way to be sure of the information presented here, and waiting until the next audit cycle won't address that.

As it relates to https://crt.sh/?q=DCCA716167F029AA9A309EE8CA3FF1F4017D1A1F3D1981BDFF9E5AF3F503682A , I'm especially concerned about the statement "Not capable to issue TLS certificates" - unless I'm missing something, which I totally could be, this certificate is quite capable of issuance. I'm hoping you can expand more here?

For the first two, regarding the AUP, note that WebTrust guidance was updated in 2017 to provide illustrative guidance on how to report to that effect. Your WebTrust auditor should be able to use that form to provide the reporting in a consistent manner.

I think, if I'm reading correctly, that the following certificates remain unresolved:

The lack of disclosure in the BR report is a significant oversight, and it means we cannot have assurance that they've been evaluated and are operating consistent with policy. The standard response has been to insist on revocation, whether OneCRL (which has been Mozilla's standard repsonse) and/or by direct revocation (what other root programs have required). I encourage you to notify other browsers/root programs for their input, and I encourage you to re-evaluate how the necessary assurance can be provided, both for this year and prior years.

(In reply to Wayne Thayer [:wayne] from comment #4)

(In reply to roots from comment #3)

Here is a detail of the 14 subordinate CA certificates flagged as missing SHA-256 thumbprints:
13 of the 14-flagged subordinate CAs are capable of issuing TLS certificates.
(2) Have a separate AUP (Agreed Upon Procedure) agreed with Mozilla since 2016 where auditors attest each year that no TLS certificates were issued.
CA A7
B191EDEB4CB772C7983150C047517F50BD9221C8CBC9349A11D0CC3E7EF72BFD
CA A8
2FA315DC154C213DC0C16FD12FA021BA9974F0C9AE142E24BF3AED150A7CE6FC

Okay

(4) Are in the WebTrust audit report with SHA-256 thumbprints in both reports but not in the expected format. Auditors notified to correct in next report.
Let's Encrypt Authority X1
7FDCE3BF4103C2684B3ADBB5792884BD45C75094C217788863950346F79C90A3
Let's Encrypt Authority X2
EC0C6CA496A67A13342FEC5221F68D4B3E53B1BC22F6E4BCCC9C68F0415CDEA4
Let's Encrypt Authority X3
25847D668EB4F04FDD40B12B6B0740C567DA7D024308EB6C2C96FE41D9DE218D
Let's Encrypt Authority X4
A74B0C32B65B95FE2C4F8F098947A68B695033BED0B51DD8B984ECAE89571BB6

The thumbprints listed above are for the cross-certs, while the audit report referenced in CCADB only lists the roots (because it's LE's audit). Are you sure the CCADB record is correct? What do you mean by "not in the expected format"?
IdenTrust (Marco) You are correct Wayne, I erroneously was referencing to the SHA256 thumbprints supplied directly by Let's Encrypt. Those were properly included in the Standard WebTrust Audit Report but not in the B.R. We are working with the auditors to correct this.

For all the rest below, it's not acceptable to fix the problem going forward. Identrust should either (1) revoke the CA certificate, (2) ask for it to be added to OneCRL (only if not intended for serverAuth and included in the standard WebTrust audit), or (3) ask the auditor to issue a corrected report, and provide a date in this bug by when we can expect to receive the updated report.
IdenTrust (Marco) Option 3. Working with auditors to correct; as soon as we have a date will update here.

(2) Are in the WebTrust audit report but missing the thumbprints in the B.R. Audit report, not in the standard audit report. Auditors notified to correct in next report.
TrustID Server CA A52
B39C4A4596D3191AFA3B3D254D28E5C482FCD0D500E0A9337F99277CB8A2EEF8
Booz Allen Hamilton BA CA 01 – Not capable to issue TLS certificates
DCCA716167F029AA9A309EE8CA3FF1F4017D1A1F3D1981BDFF9E5AF3F503682A

(1) Is in the WebTrust audit report but missing the thumbprints in the standard audit report, but not in the B.R. report. Auditors notified to correct in next report.
IdenTrust ACES CA 2 - SHA-256 Fingerprint
C5480D7BFF952D1BE86178FF713F11F51CF74232EE5676FC5A170D4A6A6FE50A

(3) Are not included in the WebTrust Audit Report; although they have different serial number and different SHA-256 fingerprints, cryptographically they are identical to this resigned subordinate disclosed in the supplied WebTrust audit reports:
IdenTrust Commercial CA Root CA 1 5D56499BE4D2E08BCFCAD08A3E38723D50503BDE706948E42F55603019E528AE

IdenTrust Commercial Root CA 1
91B18588225035BB2F231FEF7695E497B289934B65CB87CFC2212271EBECB58C
IdenTrust Commercial Root CA 1
F49793F8DF83CE64A8C8D50DF366B64E98C2538A2AAAB2019CA0367A1FCC03CB
IdenTrust Commercial Root CA 1
AAE38F67F0A626805928507B078D89D5598D760D17335927B46606ECDFA1B946

(2) Are not included in the WebTrust Audit Report; although they have different serial number and different SHA-256 fingerprints, cryptographically they are identical to this resigned subordinate disclosed in the supplied WebTrust audit reports:
IdenTrust Public Sector Root CA 1: C5480D7BFF952D1BE86178FF713F11F51CF74232EE5676FC5A170D4A6A6FE50A
IdenTrust ACES CA 2
9D1585E63B4D03D9ABBA0C67D46730BADF0FCEBC2081611CF7B9AA572D2D64A4
IdenTrust ACES CA 2
A59740F91153C0FB1C1E37081CD7198E0BC28B58C1D561DB785CB82B4AD9DF47

Flags: needinfo?(roots)

(In reply to Ryan Sleevi from comment #5)

Thanks.

I have to share, I'm quite concerned with understanding how we can independently verify and assess these claims. That's part of the importance for addressing in the thumbprints. The current proposal would not have us have any independent attestation for another year, and it would only cover that year - not this year. So we have no insight into what sort of activities are in-scope or out-of-scope, and from understanding how the audits are to be used, it's not the auditor who misrepresented things if things are totally non-compliant.

This is why other CAs have been requested to revoke such certificates by other root programs, and why Mozilla has moved to OneCRL a number of them. This case is interesting, because unlike those other cases, it does appear Identrust appears to intend to use these for web server authentication, and so the OneCRL/Revocation mechanism has greater impact and side-effects. However, it's important to emphasize: the relying parties/public have no way to be sure of the information presented here, and waiting until the next audit cycle won't address that.

As it relates to https://crt.sh/?q=DCCA716167F029AA9A309EE8CA3FF1F4017D1A1F3D1981BDFF9E5AF3F503682A , I'm especially concerned about the statement "Not capable to issue TLS certificates" - unless I'm missing something, which I totally could be, this certificate is quite capable of issuance. I'm hoping you can expand more here?

For the first two, regarding the AUP, note that WebTrust guidance was updated in 2017 to provide illustrative guidance on how to report to that effect. Your WebTrust auditor should be able to use that form to provide the reporting in a consistent manner.

I think, if I'm reading correctly, that the following certificates remain unresolved:

IdenTrust(Marco) Working with the auditors to supply a corrected report.

(In reply to Ryan Sleevi from comment #5)

Thanks.

I have to share, I'm quite concerned with understanding how we can independently verify and assess these claims. That's part of the importance for addressing in the thumbprints. The current proposal would not have us have any independent attestation for another year, and it would only cover that year - not this year. So we have no insight into what sort of activities are in-scope or out-of-scope, and from understanding how the audits are to be used, it's not the auditor who misrepresented things if things are totally non-compliant.

This is why other CAs have been requested to revoke such certificates by other root programs, and why Mozilla has moved to OneCRL a number of them. This case is interesting, because unlike those other cases, it does appear Identrust appears to intend to use these for web server authentication, and so the OneCRL/Revocation mechanism has greater impact and side-effects. However, it's important to emphasize: the relying parties/public have no way to be sure of the information presented here, and waiting until the next audit cycle won't address that.

As it relates to https://crt.sh/?q=DCCA716167F029AA9A309EE8CA3FF1F4017D1A1F3D1981BDFF9E5AF3F503682A , I'm especially concerned about the statement "Not capable to issue TLS certificates" - unless I'm missing something, which I totally could be, this certificate is quite capable of issuance. I'm hoping you can expand more here?
IdenTrust (Marco). You are correct, this SubCA can issue TLS certificates as the EKU is missing the server issuance constraint. my bad... It is disclosed in the standard audit report but missing in the B.R. Working with the auditors to correct this.

For the first two, regarding the AUP, note that WebTrust guidance was updated in 2017 to provide illustrative guidance on how to report to that effect. Your WebTrust auditor should be able to use that form to provide the reporting in a consistent manner.

I think, if I'm reading correctly, that the following certificates remain unresolved:

The lack of disclosure in the BR report is a significant oversight, and it means we cannot have assurance that they've been evaluated and are operating consistent with policy. The standard response has been to insist on revocation, whether OneCRL (which has been Mozilla's standard repsonse) and/or by direct revocation (what other root programs have required). I encourage you to notify other browsers/root programs for their input, and I encourage you to re-evaluate how the necessary assurance can be provided, both for this year and prior years.

With respect to Option 3, I do want to highlight issues like Apple and DigiCert, which involved unconstrained third-party operated sub-CAs, and issues like QuoVadis, which reportedly involve first-party operated non-TLS sub-CAs. In both of those cases, root programs requested that those certificates be revoked.

I think it's important to communicate to all Root Programs that recognize Identrust certificates regarding the situation and disclosures. Mozilla is not the only root program that requires complete disclosure of the hierarchy, whether through CCADB or directly to the Root Program operator. The clear precedent and expectations is prompt revocation.

This is especially important when proposing amended reports, in order to ensure the correct scope and activities were performed as stated.

Note, when you use the Reply feature (or Quote Reply), you don't need to include all the text. You can delete as necessary. You also don't need to include the > character, and that will properly "unquote"

Flags: needinfo?(roots)

We are still working with the auditors and expect to supply updated audit reports by November 5, 2019.

Flags: needinfo?(roots)
Whiteboard: [ca-compliance] → [ca-compliance] - Next Update - 05-November 2019

Here is IdenTrust formal incident report:

  1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.
    IdenTrust: On 10/08/2019 we learned via a Mozilla forum post from Kathleen Wilson that a new Audit Letter Validation button on intermediate records was available in the CCADB. On the same date, we logged into the CCADB and noticed that the “Intermediate Certs with Failed ALV Results” row in the home page was flagging 14 records.
  2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.
    IdenTrust: 10/08/2019: Noticed the 14 Intermediate Subordinate CAs (SubCA(s) or ICA(s)) with missing thumbprints in both of the supplied Standard WebTrust Report and BR WebTrust Report.

10/11/2019: logged forum entry https://groups.google.com/forum/#!topic/mozilla.dev.security.policy/Krr8DV4xdyg
and filed Bugzilla bug https://bugzilla.mozilla.org/show_bug.cgi?id=1588213
We can confirm that there are neither miss-issued certificates nor security issues with any of the flagged ICA(s).

10/16/2019: Closed the bug as in our view, missing ICA thumbprints in the BR WebTrust Report was a reporting oversight and all of these ICA(s) were examined for compliance with Baseline Requirements with successful testing results. We confirmed with our auditors that ICA(s) not listed in the BR WebTrust Report were tested and part of the scope of the audit engagement. It should be noted that IdenTrust has a single audit engagement every year that produces both reports. For the ICA(s) missing the thumbprints in both reports, we determined that those were not in the scope of the audit report, as they are resigned instances of disclosed and reported intermediate certificates and that no certificates were issued from those ICAs.

10/16/2019: The bug was reopened by Mr. Sleevi requesting additional details on the 14 flagged ICAs.

10/17/2019:
Logged details for the 14 ICAs as shown on comment 3.

10/21/2019:
Addressed clarifications requested by Mr. Thayer on comment 4
Addressed clarifications requested by Mr. Sleevi on comment 5
Based on the comments, we approached the auditors in order to obtain corrected audit reports reflecting the missing relevant thumbprints.

11/05/2019:
Received updated audit reports and proceeded to create this formal incident report.
We expect to get the WebTrust links updated within the next 24 hours after which we will notify relevant Root Program CA’s.

  1. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.
    IdenTrust: As this issue is relevant to missing thumbprints of ICAs in the annual webTrust audit report, we have not stopped issuance on 9 of the flagged ICAs as they were disclosed in the CCADB and in at least one of the WenTrust audit reports. On the other 5-doppelganger (same Subject+SPKI) ICAs, no issuance has taken place. As a pledge to the community, we will revoke these 5 ICA’s by November 13, 2019:
    https://crt.sh/?id=5137250; https://crt.sh/?id=12968472; https://crt.sh/?id=20231798; https://crt.sh/?id=8579786; https://crt.sh/?id=12728868

  2. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.
    IdenTrust: 2 certificates are missing in both audit reports but disclosed in a separate AUP audit letter
    6 certificates are missing in the Baseline Requirements audit report
    1 certificate is missing in the Standard WebTrust report
    5 certificates are missing in both reports but those are identical to resigned instances, except for the serial numbers and SHA256 thumbprints.

  3. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.
    IdenTrust: https://crt.sh/?id=12716314; https://crt.sh/?id=12716094
    https://crt.sh/?id=10235198; https://crt.sh/?id=10970235; https://crt.sh/?id=15706126; https://crt.sh/?id=15710291; https://crt.sh/?id=5137251; https://crt.sh/?id=20231799
    https://crt.sh/?id=11539223
    https://crt.sh/?id=5137250; https://crt.sh/?id=12968472; https://crt.sh/?id=20231798; https://crt.sh/?id=8579786; https://crt.sh/?id=12728868

  4. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
    IdenTrust:
    We have confirmed that there were inconsistencies that exist between thumbprints included in the Baseline Requirements report appendix and the WebTrust Standard audit report appendix that are the result of an oversight in our effort to get the reports out within the timeline for compliance. We had planned to ensure that the auditors correct these issues in reports from our next audit cycle, but to ensure transparency with the community, we have confirmed with the auditors that all ICAs were in scope of the audit and they have provided updated reports for this year to alleviate any concerns for compliance with regard to both Baseline Requirements and WebTrust Audit criteria.

The missing ICA’s thumbprints were not detected on 9/17/2019 when we loaded our annual WebTrust Audit report in the CCADB via case 495. As mentioned on numeral #1 above, only until 10/8/2019 we were made aware of the missing thumbprints for those ICA(s).

  1. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.
    IdenTrust: As the auditors were requested to resubmit corrected reports, they are fully aware that going forward, any Intermediate CA capable of issuing SSL certificates not technically constrained must have the SHA256 thumbprint in the Baseline Requirements report. In addition, they also understand all certificates issued to an Intermediate CA, including with the same Subject+SPKI, must be included in the Standard WebTrust audit report. We have also updated our internal audit process to include multiple levels of validation to accurately reflect all relevant ICA(s) and associated certificates listed in future audit reports.

As the WebTrust links are taking longer to update, attached are the Updated 2019 Audit Reports PDF versions

To have the new reports verified, please submit a new audit case in CCADB once the new seals are available.

Whiteboard: [ca-compliance] - Next Update - 05-November 2019 → [ca-compliance]

Replacing the SSL Baseline Report with an updated version

Attachment #9107229 - Attachment is obsolete: true

(In reply to Wayne Thayer [:wayne] from comment #15)

To have the new reports verified, please submit a new audit case in CCADB once the new seals are available.

Yes Sir. as soon as the seals are updated we will submit a new audit case.

The new reports were verified in the CCADB via case 510 and relevant root stores were notified of the updated WebTrust Reports via email.

CA A7: B191EDEB4CB772C7983150C047517F50BD9221C8CBC9349A11D0CC3E7EF72BFD
CA A8: 2FA315DC154C213DC0C16FD12FA021BA9974F0C9AE142E24BF3AED150A7CE6FC

Appear to remain undisclosed in the relevant audits, despite the disclosure within CCADB stating "Audit same as parent". These still fail ALV.

As stated in comment 3:
CA A7 & CA A8 have a separate AUP (Agreed Upon Procedure Letter) agreed with Mozilla since 2016 where auditors attest each year that no TLS certificates were issued by these ICAs.

While I understand that, Identrust has provided no public disclosure within CCADB to that effect, as far as I can tell, and no public disclosure on their web page or to other browsers.

This is especially concerning given that WebTrust provides Illustrative Guidance on how to include such certificates in scope, while positively affirming no TLS issuance.

  1. Where's the disclosure for the purposes of Mozilla?
  2. How does Identrust, and its auditors, assert compliance with the Baseline Requirements, given the potential separate reporting?
Flags: needinfo?(roots)

(In reply to Ryan Sleevi from comment #21)

While I understand that, Identrust has provided no public disclosure within CCADB to that effect, as far as I can tell, and no public disclosure on their web page or to other browsers.
IdenTrust:
Within CCADB, CA A7 & CA A8 have this as ALV comment: “ Supplied AUP Letter. Agreed to place in OneCRL

This is especially concerning given that WebTrust provides Illustrative Guidance on how to include such certificates in scope, while positively affirming no TLS issuance.

  1. Where's the disclosure for the purposes of Mozilla?
    Emailed to Kathleen/Wayne at Mozilla
  2. How does Identrust, and its auditors, assert compliance with the Baseline Requirements, given the potential separate reporting?
    IdenTrust:
    The AUP Letter for CA’s A7 & A8 exists because those 2 ICA’s fall outside of the B.R. guidelines not being technically constrained to prevent issuance of SSL certificates. The process in place validated by the auditors attest in the AUP letter each year that no SSL certificates have ever been issued from those 2 ICA’s.

In reference to comment 12, Item 3, effective today 11/13/2019 we have revoked these 5 doppelganger (same Subject+SPKI) ICAs:
https://crt.sh/?id=5137250; https://crt.sh/?id=12968472;
https://crt.sh/?id=20231798; https://crt.sh/?id=8579786; https://crt.sh/?id=12728868

(In reply to roots from comment #22)

(In reply to Ryan Sleevi from comment #21)

While I understand that, Identrust has provided no public disclosure within CCADB to that effect, as far as I can tell, and no public disclosure on their web page or to other browsers.
IdenTrust:
Within CCADB, CA A7 & CA A8 have this as ALV comment: “ Supplied AUP Letter. Agreed to place in OneCRL

Again, and I want to stress this so it’s clear: that is not disclosure. Identrust has made it clear it is unwilling or unable to provide the BR-required reporting for these intermediates, and this is indistinguishable from a CA that has maliciously issued a sub-CA certificate. In the past, CAs that have issued unconstrained, unaudited sub-CAs have been promptly and permanently distrusted. I want to make sure that it’s clear that the failure to provide appropriate reporting is extremely serious.

  1. How does Identrust, and its auditors, assert compliance with the Baseline Requirements, given the potential separate reporting?
    IdenTrust:
    The AUP Letter for CA’s A7 & A8 exists because those 2 ICA’s fall outside of the B.R. guidelines not being technically constrained to prevent issuance of SSL certificates. The process in place validated by the auditors attest in the AUP letter each year that no SSL certificates have ever been issued from those 2 ICA’s.

This does not answer the core question. These certificates appear to clearly be in scope of the Baseline Requirements, as they chain to a root trustee for TLS, and are not technically constrained. The BRs do not recognize AUP audits - and for good reason, they provide extremely limited assurance, and only to the party developing the procedures - which is Identrust, not browsers, not relying parties. Again, if these were audited as you claim, within the full and appropriate scope, WebTrust has appropriate guidance on how to provide a public attestation in line with the BRs. However, absent this, it’s clear that these certificates are unaudited and that remains deeply concerning.

We acknowledge your position and started coordination with impacted parties. An update will be supplied on Monday November 18, 2019.

Flags: needinfo?(roots)

We have contacted all impacted customers and are progressing in assessing operational impact to those customers. We expect to complete the assessment and provide an by tomorrow 11/19/2019.

We have completed operational impact assessment and are in the process of formulating a remediation plan. We will provide an update tomorrow 11/20/2019

A remediation plan has been formulated including a revocation timeline for ICA’s CA A7 and CA A8.
A new incident report is being prepared and will be shared with the CA/B Forum community no later than Friday November 22, 2019.

We have added Bug 1598807 (https://bugzilla.mozilla.org/show_bug.cgi?id=1598807) providing a new incident report and formulating a revocation timeline.

Can this be closed or resolved now as a duplicate of Bug 1598807 (https://bugzilla.mozilla.org/show_bug.cgi?id=1598807)?

Could you clarify why you believe the two issues are the same?

It might also be helpful to emphasize that issues with audits, and issues with delayed revocations, are tracked as separate issues for quite a while now, as they represent two distinct types of incidents: a failure to audit and a failure to revoke.

I'm not sure they are. There hasn't been any activity for this bug, and it seemed like they were similar. Thus the question.

Ben: Sorry, the Comment #30 was about resolving it as a duplicate.

As it stands, I think care an attention should be paid if Identrust has ALV failures going forward.

From Comment #12, we don't really have much details on the /how/ things are being done, just that:

IdenTrust: As the auditors were requested to resubmit corrected reports, they are fully aware that going forward, any Intermediate CA capable of issuing SSL certificates not technically constrained must have the SHA256 thumbprint in the Baseline Requirements report. In addition, they also understand all certificates issued to an Intermediate CA, including with the same Subject+SPKI, must be included in the Standard WebTrust audit report. We have also updated our internal audit process to include multiple levels of validation to accurately reflect all relevant ICA(s) and associated certificates listed in future audit reports.

I think there's utility in understanding Comment #12 (bolded) in detail, but if you find the answer satisfactory, I suppose closing it is acceptable with the level of information provided, and with any future failures treated even more critically.

Flags: needinfo?(bwilson)

One of the underlying causes for ALV failure might be because CAs tend to click on "Audit Same as Parent" in the CCADB rather than specify a different online location of their audit letter covering the CAs in question. Maybe a preventative step could be added to Identrust's internal process - i.e. make sure that if the CA is in a different audit letter, that the CCADB record for that CA specifies the right audit letter location containing the SHA2 hash of the certificate? Does the Identrust internal audit reporting process have steps that the Identrust CCADB administrator should follow for updating Root CA and subordinate CA information in the CCADB after an audit letter is received?

Flags: needinfo?(bwilson) → needinfo?(roots)

(In reply to Ben Wilson from comment #33)

One of the underlying causes for ALV failure might be because CAs tend to click on "Audit Same as Parent" in the CCADB rather than specify a different online location of their audit letter covering the CAs in question. Maybe a preventative step could be added to Identrust's internal process - i.e. make sure that if the CA is in a different audit letter, that the CCADB record for that CA specifies the right audit letter location containing the SHA2 hash of the certificate? Does the Identrust internal audit reporting process have steps that the Identrust CCADB administrator should follow for updating Root CA and subordinate CA information in the CCADB after an audit letter is received?

Yes, we have a written process in place to update the annual WebTrust audit reports into Mozilla’s CCADB and those instructions cross-reference https://www.ccadb.org/cas/updates as needed.

I have reviewed both of the attached audit reports (https://bugzilla.mozilla.org/attachment.cgi?id=9107228 and https://bugzilla.mozilla.org/attachment.cgi?id=9107311) and have compared them with the Identrust CA certificates listed in the CCADB.
All 14 that have been discussed in this bug appear to be appropriately covered by WT and WT BR audit reports, except CAs A7 and A8, which the CCADB indicates 8 are covered by AUP audit reports and requests to be added to OneCRL.
The following CAs are in both WT and WTBR audit reports (Let's Encrypt X1-X4, TrustID Server CA A52, IdenTrust ACES CA 2, and Booz Allen Hamilton BA CA 01). The CAs TrustID CA A12, TrustID HID Enterprise CA 1, and TrustID SAIC Public Email Issuing CA are only in the WT Audit report, but they both have EKUs of clientAuth and emailProtection (and not serverAuth). The audit reports also cover doppelganger certificates for Identrust Commercial CA Root CA1, the Identrust ACES CA2, and the Identrust Commercial CA Root CA 1.
The CCADB indicates that four CAs have been created since the last audit period ended (June 30, 2019). They are TrustID Server CA E1, HydrantID Server CA O1, TrustID Server CA O1, and TrustID CA A13 (this latter one with EKUs of only clientAuth and emailProtection). They will need to be included in the next audit reports, as appropriate according to their EKUs.
Next I intend to verify the status of CAs A7 and A8, look into the status of https://bugzilla.mozilla.org/show_bug.cgi?id=1598807, and see if there are any other issues that arise during this review.

It appears that Bug #1598807 adequately covers CA A7 and CA A8. (Those CAs are listed on OneCRL - see https://crt.sh/mozilla-onecrl, and it is also noted that the AUP report has been provided to several root store programs and can be obtained by other root store programs upon email request to problemreport@IdenTrust.com.) I believe that there are no other issues and that this bug can be closed as fixed.

Status: ASSIGNED → RESOLVED
Closed: 5 years ago → 4 years ago
Flags: needinfo?(roots)
Resolution: --- → FIXED

(In reply to Ben Wilson from comment #35)

The CCADB indicates that four CAs have been created since the last audit period ended (June 30, 2019). They are TrustID Server CA E1, HydrantID Server CA O1, TrustID Server CA O1, and TrustID CA A13 (this latter one with EKUs of only clientAuth and emailProtection). They will need to be included in the next audit reports, as appropriate according to their EKUs.

We appreciate the feedback and confirm awareness to include the new subordinates's SHA256 fingerprints in the next annual WebTrust Audit Report

Product: NSS → CA Program
Whiteboard: [ca-compliance] → [ca-compliance] [audit-failure]
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: