DigiCert: Revoked intermediate certificates not in CRL

RESOLVED FIXED

Status

task
RESOLVED FIXED
4 months ago
14 days ago

People

(Reporter: kwilson, Assigned: s.davidson)

Tracking

trunk

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [ca-compliance])

Attachments

(4 attachments)

616 bytes, application/pkix-crl
Details
2.24 KB, application/octet-stream
Details
1.09 KB, application/octet-stream
Details
1.14 KB, application/octet-stream
Details

While processing revoked intermediate certificate records in the CCADB, I ran into the following.

== Not Found in CRL ==

Subject: CN=FMH CA G2; O=FMH Verbindung der Schweizer Ärztinnen und Ärzte; C=CH
Issuer: CN=QuoVadis Root CA 3; O=QuoVadis Limited; C=BM
Certificate Serial Number: 7D2C2572D198339C0E67A4BA49BA5F690A0B094C
SHA-256 Fingerprint: B968FBFC3ECA285AADC38D2367727241FF7D3733F1278F8A4F50ECC8C0EF1E80
CRL URL(s): http://crl.quovadisglobal.com/qvrca3.crl
Date of Revocation: 3/25/2019

Subject: CN=MyTrust ID Public Basic CA; OU=Symantec Trust Network, Class 2 Managed PKI Individual Subscriber CA; O=MSC Trustgate.com Sdn. Bhd.; C=MY
Issuer: CN=MSC Trustgate.com Class 2 CA-G3; OU=Symantec Trust Network; O=MSC Trustgate.com Sdn. Bhd.; C=MY
Certificate Serial Number: 470022B06E2D08CED43458F01CF63E63
SHA-256 Fingerprint: 4EE6DA31D131BDA7FE71327E2639F9739C2A243BFB6FFBD3F2111AE7873C2B4A
CRL URL(s): http://crl.msctrustgate.com/MSCTrustgateClass2CAG3/LatestCRL.crl
Date of Revocation: 3/18/2019

Subject: CN=MyTrust ID Public CA; OU=Symantec Trust Network, Class 2 Managed PKI Individual Subscriber CA; O=MSC Trustgate.com Sdn. Bhd.; C=MY
Issuer: CN=MSC Trustgate.com Class 2 CA-G3; OU=Symantec Trust Network; O=MSC Trustgate.com Sdn. Bhd.; C=MY
Certificate Serial Number: 05E29FC4B68A977E12F32148CA7A27E2
SHA-256 Fingerprint: D50F19C9AE59A64F52AE73B037220C663020ACF7345393D317898EB948F4675D
CRL URL(s): http://crl.msctrustgate.com/MSCTrustgateClass2CAG3/LatestCRL.crl
Date of Revocation: 3/18/2019

Here's what I see:
Certificate Revocation List (CRL):
Version 2 (0x1)
Signature Algorithm: sha256WithRSAEncryption
Issuer: /C=MY/O=MSC Trustgate.com Sdn. Bhd./OU=Symantec Trust Network/CN=MSC Trustgate.com Class 2 CA-G3
Last Update: Mar 28 00:00:00 2019 GMT
Next Update: Apr 11 23:59:59 2020 GMT
CRL extensions:
X509v3 CRL Number:
4
X509v3 Authority Key Identifier:
keyid:8A:F4:B4:8C:F2:33:08:37:95:B3:67:9E:A5:45:82:BE:3D:24:14:97

Revoked Certificates:
Serial Number: 43474AC173E211E73CDD764F82C421FA
Revocation Date: Apr 12 17:14:19 2018 GMT
Signature Algorithm: sha256WithRSAEncryption
6a:a2:ed:ef:ba:a8:15:9d:9d:76:a4:04:29:93:1e:c4:b5:78:
be:63:35:71:fd:9d:14:99:a2:8f:4e:da:a4:ae:e9:b4:5b:63:
11:9d:99:f3:72:fb:dd:bb:f5:b5:e8:40:2a:25:3a:d3:6a:e7:
d7:74:c8:53:92:1d:1e:3d:cd:d0:28:73:13:bf:1c:2a:17:2d:
9b:34:85:85:ea:18:94:1d:3f:68:90:53:0d:44:87:f0:65:2c:
6d:ab:38:45:e9:71:2b:13:2c:75:bc:ca:2d:9e:28:b8:49:d5:
90:1c:d0:6a:8e:07:04:2c:a5:ff:7e:d4:53:86:db:ef:bc:6d:
ad:0e:00:d8:22:07:c2:85:bb:63:83:c6:4f:ee:c6:e4:3b:e3:
6d:f6:00:6b:d6:ed:41:ef:38:7d:13:c4:2f:94:f4:a5:8d:82:
5a:b7:b0:db:83:a8:d8:84:ee:31:82:46:0d:0b:10:67:10:e1:
bc:d2:01:f6:0f:ea:05:ec:1f:03:f1:1e:e8:87:18:d7:6f:82:
b2:0e:81:61:56:9a:81:69:8f:e7:1d:02:2b:fc:bd:56:1e:64:
a9:33:bb:18:b1:93:fe:28:6f:91:bb:5d:61:c5:c9:a8:f9:59:
f1:6d:e7:20:71:18:51:11:60:18:67:11:97:b9:98:88:a9:e6:
3a:81:87:6e

==

== CRL URL times out ==

Subject: CN=DarkMatter Assured CA; O=DarkMatter LLC; C=AE
Issuer: CN=QuoVadis Root CA 3 G3; O=QuoVadis Limited; C=BM
Certificate Serial Number: 05A6227D599C95FEB55A333A9B6B54134512DB63
SHA-256 Fingerprint: 60F066DC78A4E2E929A1C8ED102EDB707DF03181F82FDF50D53A52DAC355C65B
CRL URL(s): http://crl1.darkmatter.ae/qvrca3g3.crl, http://crl2.darkmatter.ae/qvrca3g3.crl
Date of Revocation: 3/25/2019

Subject: CN=DarkMatter High Assurance CA; O=DarkMatter LLC; C=AE
Issuer: CN=QuoVadis Root CA 2 G3; O=QuoVadis Limited; C=BM
Certificate Serial Number: 62065C489EA237C7C40BB7A3389B1D0EB9B9A358
SHA-256 Fingerprint: E0A670F4F1057E9179E9DB45E333CE37E3EE31C3499F1C584A587BD9A5F53640
CRL URL(s): http://crl1.darkmatter.ae/qvrca2g3.crl, http://crl2.darkmatter.ae/qvrca2g3.crl
Date of Revocation: 3/25/2019

Subject: CN=DarkMatter Secure CA; O=DarkMatter LLC; C=AE
Issuer: CN=QuoVadis Root CA 2 G3; O=QuoVadis Limited; C=BM
Certificate Serial Number: 623F50D83A11192F1116CC4B12785E12B0396B24
SHA-256 Fingerprint: A019811E4369CA4C62AAA80A1549613E60F6C5CED383AF9D79DF8F8F193F1DFE
CRL URL(s): http://crl1.darkmatter.ae/qvrca2g3.crl, http://crl2.darkmatter.ae/qvrca2g3.crl
Date of Revocation: 3/25/2019

Assignee: wthayer → s.davidson
Flags: needinfo?(steve.medin)
Flags: needinfo?(s.davidson)
Flags: needinfo?(steve.medin)

Kathleen, thanks for notifying us. I've attached MSC Class 2 G3 CRL number 5. MSC are currently hosting CRL number 4. We have contacted them to resolve.

Posted file qvrca3.crl

I attach CRL 70 for RCA3 which includes:

Subject: CN=FMH CA G2; O=FMH Verbindung der Schweizer Ärztinnen und Ärzte; C=CH
Certificate Serial Number: 7D2C2572D198339C0E67A4BA49BA5F690A0B094C

This CA was not ever used in production; only internal setup OCSP and test certs were issued.

More detail to follow on Monday.

Flags: needinfo?(s.davidson)
Posted file qvrca2g3.crl

I attach CRL 42 for qvrca2g3 which includes:

Subject: CN=DarkMatter High Assurance CA; O=DarkMatter LLC; C=AE
Certificate Serial Number: 62065C489EA237C7C40BB7A3389B1D0EB9B9A358

Subject: CN=DarkMatter Secure CA; O=DarkMatter LLC; C=AE
Certificate Serial Number: 623F50D83A11192F1116CC4B12785E12B0396B24

The CRL URL worked for me. Will follow up.

Posted file qvrca3g3.crl

I attach CRL 63 for qvrca3g3 which includes:

Subject: CN=DarkMatter Assured CA; O=DarkMatter LLC; C=AE
Certificate Serial Number: 05A6227D599C95FEB55A333A9B6B54134512DB63

The CRL URL worked for me. Will follow up.

  1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

Became aware of the Bug filed by Kathleen Wilson, on the morning (in EST) of May 3 2019.

  1. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

There were two issues reported.
a. It appeared the URLs for the DarkMatter crls were timing out. This was not the case for me (and users I asked to test for me in Europe). The crls were accessible and properly reflected the DM revocations. We consider this resolved.
b. The crl for the qvrca3 did not reflect the revocation of the following CA reported in CCADB. This was verified to be correct.

Subject: CN=FMH CA G2; O=FMH Verbindung der Schweizer Ärztinnen und Ärzte; C=CH
Issuer: CN=QuoVadis Root CA 3; O=QuoVadis Limited; C=BM
Certificate Serial Number: 7D2C2572D198339C0E67A4BA49BA5F690A0B094C
SHA-256 Fingerprint: B968FBFC3ECA285AADC38D2367727241FF7D3733F1278F8A4F50ECC8C0EF1E80
CRL URL(s): http://crl.quovadisglobal.com/qvrca3.crl
Date of Revocation: 3/25/2019

The crl for qvrca3 was republished at 8:45AM EST on May 3 2019, and an investigation launched into the cause.

  1. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

NA.

  1. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

NA.

  1. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

“FMH CA G2” had not been used in production by the client, and had only issued a small number of certificates for QV internal use (such as testing and setting up associated infrastructure).

On March 25, QuoVadis routinely revoked a number of issuing CAs, including the above, under several QV roots. All were reported in CCADB at the time. Our procedures include the manual push of the updated crl for each root following such revocations using a script that resides on each respective root server. In the case of qvrca3.crl, the process failed and the previous crl was republished. Although the failure was logged, it was not noticed. The other crls published that day were successfully updated.

  1. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

In both routine crl updates and following the revocation of issuing CAs, our processes have been updated to confirm the expected publication and public access of the updated crl. Improvements are being considered to the script.

(In reply to Stephen Davidson from comment #6)

There were two issues reported.
a. It appeared the URLs for the DarkMatter crls were timing out. This was not the case for me (and users I asked to test for me in Europe). The crls were accessible and properly reflected the DM revocations. We consider this resolved.

Those CRLs are still timing out for me, so I entered Alternate CRLs for these in the CCADB -- pointing to the CRLs attached to this bug.

b. The crl for the qvrca3 did not reflect the revocation of the following CA reported in CCADB. This was verified to be correct.

Subject: CN=FMH CA G2; O=FMH Verbindung der Schweizer Ärztinnen und Ärzte; C=CH
Issuer: CN=QuoVadis Root CA 3; O=QuoVadis Limited; C=BM
Certificate Serial Number: 7D2C2572D198339C0E67A4BA49BA5F690A0B094C
SHA-256 Fingerprint: B968FBFC3ECA285AADC38D2367727241FF7D3733F1278F8A4F50ECC8C0EF1E80
CRL URL(s): http://crl.quovadisglobal.com/qvrca3.crl
Date of Revocation: 3/25/2019

The crl for qvrca3 was republished at 8:45AM EST on May 3 2019, and an investigation launched into the cause.

I have verified that the CRL now has this entry.

(In reply to Steve Medin from comment #2)

Kathleen, thanks for notifying us. I've attached MSC Class 2 G3 CRL number 5. MSC are currently hosting CRL number 4. We have contacted them to resolve.

I entered an Alternate CRL for these MSC certs in the CCADB -- pointing to the CRL attached to this bug.

There is one more CRL that needs to be updated:
Subject: CN=IT Security G3; OU=Symantec Trust Network, Class 2 Managed PKI Individual Subscriber CA; O=VocaLink; C=GB
Issuer: CN=BT Class 2 CA - G3; OU=Symantec Trust Network; O=British Telecommunications plc; C=GB
Certificate Serial Number: 4803A931DA493CD982A1F5F9B30679A1
SHA-256 Fingerprint: 4F029E7543DD8A8CA0B94D692231CB5A0AE6B87CC7BAA156DE507FEB1542FDCE
CRL URL(s): http://onsitecrl.trustwise.com/BTClass2CA-G3.crl
Date of Revocation: 4/30/2019
CRL Last Update: Feb 21 00:00:00 2019 GMT

Flags: needinfo?(steve.medin)

Here is the incident report for the MSC CRL issue:

1.How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

On 3 May 2019, we became aware of the bug filed by Kathleen Wilson.

2.A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

3 May 2019 (14:20 GMT) - After confirming the CRL entries for 2 revoked CA's were missing, investigation started. It was confirmed that the number #4 CRL was currently hosted rather than the #5 most recent CRL.

3 May 2019 (14:33 GMT) -- DigiCert’s contacted MSC Trustgate and asked to rectify.

4 May 2019 (23:53 GMT) -- MSC Trustgate came back confirming the issue and stating they had rectified and replaced with the #5 CRL.

3.Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

These CAs have been decommissioned and are not issuing at this time.

4.A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

See section 5 below)

5.The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

Subject: CN=MyTrust ID Public Basic CA; OU=Symantec Trust Network, Class 2 Managed PKI Individual Subscriber CA; O=MSC Trustgate.com Sdn. Bhd.; C=MY
Issuer: CN=MSC Trustgate.com Class 2 CA-G3; OU=Symantec Trust Network; O=MSC Trustgate.com Sdn. Bhd.; C=MY
Certificate Serial Number: 470022B06E2D08CED43458F01CF63E63
SHA-256 Fingerprint: 4EE6DA31D131BDA7FE71327E2639F9739C2A243BFB6FFBD3F2111AE7873C2B4A
CRL URL(s): http://crl.msctrustgate.com/MSCTrustgateClass2CAG3/LatestCRL.crl
Date of Revocation: 3/18/2019

Subject: CN=MyTrust ID Public CA; OU=Symantec Trust Network, Class 2 Managed PKI Individual Subscriber CA; O=MSC Trustgate.com Sdn. Bhd.; C=MY
Issuer: CN=MSC Trustgate.com Class 2 CA-G3; OU=Symantec Trust Network; O=MSC Trustgate.com Sdn. Bhd.; C=MY
Certificate Serial Number: 05E29FC4B68A977E12F32148CA7A27E2
SHA-256 Fingerprint: D50F19C9AE59A64F52AE73B037220C663020ACF7345393D317898EB948F4675D
CRL URL(s): http://crl.msctrustgate.com/MSCTrustgateClass2CAG3/LatestCRL.crl
Date of Revocation: 3/18/2019

6.Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

MSC Trustgate has a script that replaces the CRL when the previous one is scheduled to expire. A new CRL (# 4) was created and was added to this script to update. However, after this update a new CRL was generated (#5). When MSC received the # 5, the CRLs were manually updated, but the script was missed for the update. This was a result of a human error.

On 28 March 2019, the script ran copying the # 4 CRL over-writing the #5 CRL. This was not noticed as the new CRL was valid and newer than the previous one date-wise.

7.List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

MSC Trustgate is putting a monitoring script into place to avoid this from happening again, which would shore up the quality control on CRL updates. The monitoring script will do a compare between current CRL and prior CRL checked. Both date and version must be equal or greater than the previous. An error message will be generated to identify any inconsistencies, and the message will be sent to their Ops team.

DigiCert will continuously monitor the CRL updates.

Here is the incident report for the BT CRL issue:

  1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

On 6 May 2019 (20:57 GMT)- Digicert became aware of the CRL bug filed by Kathleen Wilson.

  1. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

All times GMT

6 May 2019 (20:57) - Problem report awareness via this bug
7 May 2019 (01:05) - Confirmed CRL entry for revoked CA was missing, investigation started
7 May 2019 (01:07) - BT was contacted and asked to rectify the issue
7 May 2019 (07:50) - BT came back stating it was to be loaded for a maintenance window between 18:00 and 22:00 on the 8 May 2019
8 May 2019 (18:40) - BT confirmed CRL was now loaded

  1. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

This ICA had been replaced by a fully constrained some time ago and was not issuing Certificates.

  1. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.
    Only one certificate was missing from the CRL, and this one was not issuing certificates as it had been replaced.

Subject: CN=IT Security G3; OU=Symantec Trust Network, Class 2 Managed PKI Individual Subscriber CA; O=VocaLink; C=GB
Issuer: CN=BT Class 2 CA - G3; OU=Symantec Trust Network; O=British Telecommunications plc; C=GB
Certificate Serial Number: 4803A931DA493CD982A1F5F9B30679A1
SHA-256 Fingerprint: 4F029E7543DD8A8CA0B94D692231CB5A0AE6B87CC7BAA156DE507FEB1542FDCE
CRL URL(s): http://onsitecrl.trustwise.com/BTClass2CA-G3.crl
Date of Revocation: 4/30/2019
CRL Last Update: Feb 21 00:00:00 2019 GMT

5.The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

(See 4 above)

  1. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

DigiCert’s customer, British Telecom, was provided with a new CRL on the 20:06 31 April 2019. BT placed this through their change management process to get the CRL loaded and was completed on 8 May 2019.

7.List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

BT have committed to streamlining their change management for CRLs with a plan for a 24 hour turn around moving forward.

DigiCert has committed to educating their customers about expectations on timelines for loading CRLs with newly revoked certificate.

(In reply to Kathleen Wilson from comment #7)

There is one more CRL that needs to be updated:
Subject: CN=IT Security G3; OU=Symantec Trust Network, Class 2 Managed PKI Individual Subscriber CA; O=VocaLink; C=GB
Issuer: CN=BT Class 2 CA - G3; OU=Symantec Trust Network; O=British Telecommunications plc; C=GB
Certificate Serial Number: 4803A931DA493CD982A1F5F9B30679A1
SHA-256 Fingerprint: 4F029E7543DD8A8CA0B94D692231CB5A0AE6B87CC7BAA156DE507FEB1542FDCE
CRL URL(s): http://onsitecrl.trustwise.com/BTClass2CA-G3.crl
Date of Revocation: 4/30/2019
CRL Last Update: Feb 21 00:00:00 2019 GMT

I confirm this has been resolved, and the entry is in the CRL.
Incident report provided in Comment #9.

--

There's a new problem:
Subject: CN=POSCO Online CA - G3; OU=Symantec Trust Network, Class 2 Managed PKI Individual Subscriber CA; O=POSCO; C=KR
Issuer: CN=CrossCert Class 2 CA - G3; OU=Symantec Trust Network; O=KECA, Inc.; C=KR
Certificate Serial Number: 1F82F9FC59D65E75DD5ECFFCC7E043A7
SHA-256 Fingerprint: 6BA501267D83C0D78452F8F7AD9F80E6BE88572FC72BB165DF8C671E8AD9939D
Date of Revocation: 5/7/2019
CRL URL(s): http://crl.crosscert.com/offlineca/CrossCertClass2CA-G3.crl
CRL Last Update: Apr 12 00:00:00 2018 GMT
CRL Next Update: Apr 11 23:59:59 2019 GMT

Flags: needinfo?(steve.medin) → needinfo?(brenda.bernal)

Here is the incident report for the Crosscert CRL issue:

  1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

On 13 May 2019 (23:25 GMT)- DigiCert became aware of the CRL bug filed by Kathleen Wilson.

  1. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

All times GMT

13 May 2019 (23:25) - Problem report awareness via this bug
13 May 2019 (23:35) - Confirmed CRL entry for revoked CA was missing, investigation started
13 May 2019 (23:37) - Crosscert was contacted and asked to rectify the issue
14 May 2019 (00:30) - Crosscert confirmed CRL was now available

  1. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

This ICA had been replaced by a fully constrained some time ago and was not issuing Certificates.

  1. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

Only one certificate was missing from the CRL, and this one was not issuing certificates as it had been replaced.
https://crt.sh/?q=A38872203A3FD6C62C674AD8EF2DB3BE48A74235354A66FB7F5FE3C615C38E7D
Subject: CN=POSCO Online CA - G3; OU=Symantec Trust Network, Class 2 Managed PKI Individual Subscriber CA; O=POSCO; C=KR
Issuer: CN=CrossCert Class 2 CA - G3; OU=Symantec Trust Network; O=KECA, Inc.; C=KR
Certificate Serial Number: 1F82F9FC59D65E75DD5ECFFCC7E043A7
SHA-256 Fingerprint: 6BA501267D83C0D78452F8F7AD9F80E6BE88572FC72BB165DF8C671E8AD9939D
CA Validity
Not Before: Jun 4 00:00:00 2015 GMT
Not After : Jun 3 23:59:59 2020 GMT
Date of Revocation: 5/7/2019
CRL URL(s): http://crl.crosscert.com/offlineca/CrossCertClass2CA-G3.crl

5.The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

(See 4 above)

  1. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

DigiCert’s customer, Crosscert were provided a new CA on 7 May 2019.
Crosscert Posted the CRL on their site on the 9th of May. However, due to human error, a typo in the name of the CRL was included. After being notified on 13 May, CrossCert immediately investigated and corrected the typo. As the old CRL was still valid, this did not trigger a warning.

  1. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

Crosscert have already resolved the issue with the CRL. Crosscert is also adding an additional step in their CRL load process where an external test is completed by a member of a separate department from the Data Centre team to confirm correctness.

Flags: needinfo?(brenda.bernal)

Brenda: unless I am missing something, the proposed remediation for these issues is largely procedural. I do see a mention of "continuously monitor"ing, but it's not clear if DigiCert, after the repeated failures documented in this bug, is committed to oversight of these subordinate CAs using technical controls. Are there plans for DigiCert to monitor the CRL URL for all of its delegated CAs, coupled with an incident response plan when a CRL is found to be expired or otherwise unavailable? If so, when will that be implemented?

Flags: needinfo?(brenda.bernal)

Hi Wayne: You are correct that the "get well plan" in these cases are largely procedural. With that said, we are working on the programmatic controls which we will describe in an update shortly, along with timelines for implementation.

Flags: needinfo?(brenda.bernal)
Status: NEW → ASSIGNED

Hi Wayne, Here is our update:

The technical solution is an automated check that will be run at a set interval (daily) that will check for the following CRL conditions:
o Accessible - This is to check the CRL file is available to download
o Current - This is to check the CRL is valid
o If there are any issues with the 2 condition checks above, the Digicert Internal Ops team will get alerted and will notify the Compliance team on any potential issue. Compliance will work with the sub-CA to address the issue immediately. This solution is expected to be implemented by 3rd week of June, 2019.

Can you provide greater detail about how both "available" and "valid" are defined. What tests will be performed?

There's plenty of ways to implement these controls in a way that would fail to detect errors within this class, so having greater understanding about the tests that will perform and their edge conditions not only provides valuable documentation as to (potential) best practices, but helps DigiCert demonstrate a fullness of understanding of the ways in which such situations can fail, and the necessity of robust controls to prevent such failure.

Flags: needinfo?(brenda.bernal)

Phase one has now been deployed, which encompasses the following:

For every CRL, we will perform an hourly download of that CRL using the URL provided. If the CRL is available for download, this should pass the check for availability. Once the CRL is downloaded, we will perform the parsing on the CRL to check for nextUpdate date and assert that the nextUpdate date should be in the future. We will also confirm that lastUpdate is in the past so as to confirm that a CRL has not been loaded early. This monitor will be implemented in multiple locations globally to check for regional consistency of the CRL.

If the CRL fails any of those checks, an alert will be generated and will notify our Compliance team to escalate for handling with our partners.

During the second phase targeted for the 3rd week of June, we will implement additional checks which will be performed on the CRL including CRL number, lastUpdate timestamp, nextUpdate timestamp and signature.

If you have any feedback on the checks described or additional ones we need to consider, welcome your input. Thank you.

Flags: needinfo?(brenda.bernal)

Suggestions:

  • I would suggest evaluating Section 4.9.7 of the BRGs as part of your checks
  • I would suggest evaluating Section 4.10.1 of the BRGs as part of your checks
  • I would suggest evaluating Section 4.10.2 of the BRGs as part of your checks
  • I would suggest validating your CRLs conform to the ASN.1 module expressed in RFC 5912 . For example, you can use asn1c, which is what AWS's certlint module does for certificates.
  • I would suggest adding validation logic for issuingDistributionPoints, particularly if multiple CRLs are produced for the same publisher

Are things on track for the 3rd week of June?

Flags: needinfo?(brenda.bernal)
Whiteboard: [ca-compliance] CRL → [ca-compliance] CRL - Next Update 24-June 2019

The new monitoring in now in effect, the ICA CRL’s our subCAs host are being monitored on an hourly run from three geographic regions (US, EU, APAC). If an error is triggered, the SubCA will be contacted to resolve the issue and an incident report is generated.

The checks that are being done now are:
-CRL is available to download and can be parsed correctly
-CRL nextUpdate is in the future
-CRL thisUpdate is in the Past
-CRL thisUpdate is equal to or newer than previous timestamps
-CRL was signed by correct Issuer CA.

Regarding your suggestions Ryan, here are our responses:
I would suggest evaluating Section 4.9.7 of the BRGs as part of your checks.
We will evaluate what we need to implement this properly. We do check the validity period of the CRLs through the timestamp check.

I would suggest evaluating Section 4.10.1 of the BRGs as part of your checks.
Operational Characteristics - Entries are not removed from prior CRLs when generated and all CRLs are verified by our Compliance team prior to issuing to SubCAs.

I would suggest evaluating Section 4.10.2 of the BRGs as part of your checks.
Service availability is being covered by checks every hour via the internet checking availability of CRLs.

I would suggest validating your CRLs conform to the ASN.1 module expressed in RFC 5912 . For example, you can use asn1c, which is what AWS's certlint module does for certificates.
This is noted and will be added to future roadmap. Good suggestion.

I would suggest adding validation logic for issuingDistributionPoints, particularly if multiple CRLs are produced for the same publisher
We are not using issuingDistributionPoints for CRLs currently so there is nothing to validate.

Flags: needinfo?(brenda.bernal)
Flags: needinfo?(wthayer)
Whiteboard: [ca-compliance] CRL - Next Update 24-June 2019 → [ca-compliance]

Per the suggestion above, evaluating Section 4.9.7 of the BRGs as part of your checks, we looked at the benefits of automating vs. procedural controls implemented by our trusted role holders. Our PKI operations team maintains calendar to manage the periodic CRL updates, and we believe these are sufficient, and we have adequate compensating controls for these periodic occurrences.

We have nothing further to update. Please let us know if there are additional questions or if we can close this case.

It appears that all questions have been answered and remediation is complete.

Status: ASSIGNED → RESOLVED
Closed: 14 days ago
Flags: needinfo?(wthayer)
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.