1483639 - DigiCert / ADACOM: published expired CRLs

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Wayne Thayer]

In bug 1483370, it was determined that four intermediate certificates that should have been revoked were not in ADACOM's CRLs. After DigiCert contacted ADACOM, the CRLs were replaced with versions that do contain the missing entries. It appears that ADACOM published new CRLs with the revocations, then reverted back to still-valid earlier version that essentially caused these intermediates to become "unrevoked".

Please provide an incident report, as described here:
https://wiki.mozilla.org/CA/Responding_To_A_Misissuance#Incident_Report
The incident report should be posted to the mozilla.dev.security.policy forum and added to this bug.

Comment 1

[Wayne Thayer]

Attachment: Outdated c2ca-g3.crl downloaded on 14-August 2018

Comment 2

[Wayne Thayer]

Attachment: Outdated c2ca-g4.crl downloaded on 14-August 2018

Comment 3

[Ben Wilson]

Issue notice

On Wed 15-Aug-18 2:28 AM, ADACOM was made aware by DigiCert that the following four intermediate certificates, which had been revoked, were not in ADACOM's CRLs.
•	http://crl.adacom.com/c2ca-g4.crl
o	216E6BC44090E4B1324463DE644F75BA, CYTA CA
o	61F742656C91575B12C795F1CA85CA49, CYTA CA
o	5122125855911C6A79DF45507364B5FD, Universal Bank Ukraine CA
•	http://crl.adacom.com/c2ca-g3.crl	
o	7BD7E9F29A09E230EAA47C5573EED2A0, Alpha Bank CA
‎
Issue resolution

On Wed 15-Aug-18 10:57 AM, ADACOM replaced the CRLs with the current versions that do contain the missing entries

Issue actions timeline

1.	On Thu 24-May-18 2:03 AM, ADACOM received new CRLs from DigiCert, containing the four (4) above revoked certificates.
•	c2ca-g3.crl, CRL #0b with validity period 23/05/2018 - 04/12/2019
•	c2ca-g4.crl, CRL #07 with validity period 23/05/2018 - 23/05/2019
2.	On Thu 24-May-18 12:50 PM, ADACOM did the following:
•	for c2ca-g3.crl, replaced CRL #09 with new CRL #0b
•	for c2ca-g4.crl, replaced CRL #04 with new CRL #07
3.	On Mon 30-Jul-18 10:49 PM, as explained below, the CRL files were automatically reverted back to the following files:
•	CRL #09 with validity period 19/11/2017 - 04/12/2018 for c2ca-g3.crl
•	CRL #04 with validity period 24/11/2017 - 09/12/2018 for c2ca-g4.crl
4.	On Wed 15-Aug-18 2:28 AM, ADACOM was informed that the above CRLs were reverted back to previous versions
5.	On Wed 15-Aug-18 10:57 AM, ADACOM restored the CRL files to their most current versions 

Issue explanation 

CRL publication is an automated process executed on a time schedule of when and which CRL version must be published. The process replaces CRL files on all public web servers. An automatic monitoring mechanism is in place to ensure the published CRLs operate properly and are not expired.

When ADACOM received the new CRLs from DigiCert, the current published CRL files were not yet expired. ADACOM proceeded, outside the predefined timetable, with manually replacing the CRLs with the newly generated CRLs.

On Mon 30-Jul-18, there was a system halt on one of the public web servers causing the server cluster to change to another public web server. Systems that should have synchronized CRLs to the failover web server had  not copied the new CRL files from the main public web server, so when the cluster changed to the failover web server, the old files began to be re-published. The automatic monitoring mechanism did not raise any incident, since the CRL files appeared to operate correctly and not expired.

Steps taken to resolve and avoid happening again

ADACOM will modify its process for non-automated publication of CRL files to implement:
•	Automatic checks that synchronization of the CRL directory is working correctly
•	Manual checking that new CRLs are replaced in all public web servers
•	Remove outdated CRL files from the list and work with DigiCert to update the list with the next CRL files

I would like to thank the community members that discovered this issue, as well as ADACOM and DigiCert teams that worked together to resolve it quickly.


Kostas Nousias
Senior Manager PKI & Authentication
ADACOM S.A.

Comment 4

[Wayne Thayer]

Ben: can you confirm that ADACOM has implemented all the proposed remediations?

Comment 5

[Ben Wilson]

Yes. ADACOM has implemented a monitoring alert on the status of CA synchronizations to alert personnel if something goes wrong with synchronization between servers. Additionally, ADACOM now reviews CRL postings to ensure that the correct CRL is in place.  It will also continue to work on additional system improvements. Also ADACOM wanted to note that the incident had no impact to third parties or services since all issued certificates from these CA(s) had expired before the revocation date.