1925239 - Microsec: Expired Certificates on test Pages for Revocation

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[dr. Sándor SZŐKE]

Incident Report

Summary

Microsec operates web pages to test valid, revoked and expired certificates in accordance with the requirements of CABF BR section 2.2.
Microsec received an email reporting an issue with the revoked certificate test pages.
The problem is that these test pages are protected by expired certificates.

Impact

Because the certificates are expired, these test pages are not applicable for their intended purpose, i.e. testing the correct operation of the CA revocation service.

Timeline

2023-09-06

• Microsec currently has 2 recognized active root certificates in the CCADB system, and accordingly maintains 2*3 test websites.
• According to our previous practice, we used certificates with a validity period of 365 days on the test pages and manually renewed the certificates at least once a year.
• The last manual renewal of the 6 certificates took place on 2023-09-06.

2024-01-09

• In 2023, Microsec set up a dedicated hierarchy issuing exclusively TLS certificates, and in January 2024 requested the inclusion of the new root in the Root Programs.

2024-03-14

• In accordance with Google's expectations, we introduced automatic certificate renewal on our VALID test pages in live operation.
  • the certificates are valid for 10 days
  • certificates are automatically renewed on a weekly basis
  • the change does not affect the certificates of the REVOKED and EXPIRED test pages.
  • Since we do not issue a new certificate for the key of a revoked or expired certificate, the issuance of new certificates always requires the production of a new key. Due to the more complicated process that also requires human intervention, we plan to renew the certificates of revoked and expired test pages annually.

2024-03-21

• In connection with the new dedicated root, Microsec has created 3 additional websites for test purposes.
• In the new system, due to a human mistake, the revoked certificate was also issued with a validity period of only 10 days.

2024-03-31

• The certificate on the REVOKED test page which belongs to the new dedicated hierarchy, expired.

2024-09-06

• The 2 certificates on the REVOKED test pages which belong to the active hierarchies, expired.

2024-10-14

• 13:02 UTC (15:02 CETS)
  • Microsec received an email from Chris Clements (Google) to the central email address info@... reporting a potential problem with 2 of our Revoked test pages
  • Microsec's OTRS system received the email messages and sent an automatic notification email with a registration number: Ticket#2024101484005614
    The message was put to the standard "Customer service INBOX" waiting list with normal priority
• 13:25 UTC (15:25 CETS)
  • OTRS ticket was moved from "standard INBOX" to "High Priority" waiting list and assigned to the head of Customer Service
• 14:06 UTC (16:06 CETS)
  • MSTeams notification was sent to Sándor Szőke, dep. director
  • The technical director and the IT Administrator on call was notified abouth the problem
  • The investigation of the issue started immediately
• 14:21 UTC (16:21 CETS)
  • OTRS ticket was formally assigned to Sándor Szőke
• 14:32 UTC (16:32 CETS)
  • The IT Administrator sent us the full list of the test certificates by email
• 14:40 UTC (16:40 CETS)
  • The IT Administrator sent us the full list of the expired test certificates by email
  • we recognized that there are 3 test web pages published in CCADB with the same problem
• 15:45 UTC (17:45 CETS)
  • The IT Administrator informed us by email, that all the 3 problematic test web pages have been corrected
    • 3 new keys were generated
    • 3 new certificates were issued with longer lifetime (3xx days)
    • 3 new certificates Revoked soon after issuance
    • certificates were replaced on the test web pages
• 16:10 UTC (18:10 CETS)
  • Sándor Szőke sent a manual notification email to Chris Clements with the findings and promising the quick replacement of the expired certificates

2024-10-15

• 16:51 UTC (18:51 CETS)
  • JIRA ticket was opened to manage the issue

2024-10-17

• 10:00 UTC (12:00 CETS)
  • This incident report was opened in Bugzilla.

Root Cause Analysis

• Due to the introduction of the automatic renewal of VALID certificates, the annual renewal of other certificates was out of focus, so in the case of 2 active roots, the expired certificates were not replaced on 2024-09-05.
• In the case of the new dedicated root, which is not yet operational, an additional problem was caused by the fact that the REVOKED certificate was issued for 10 days instead of the usual 365-day validity, so its validity already expired on 2024-03-31.
• Summarizing the above, we see the main cause of the problem in the fact that, when introducing automatic renewal, we did not properly assess which processes were affected by the change and which processes still had to be carried out manually. By introducing the new process, we eliminated old processes without taking care of their adequate replacement.

Lessons Learned

• In case of any change in our processes we need to make more careful investigation analyzing all the potential affects of the change
• increasing the level of automation may help preventing these type of problems.

What went well

• we responded to the notification in a timely manner
• we have collected the affected certificates
• we issued the new certificates
• we found the root of the problem

What didn't go well

• we made a mistake when introducing automatic certificate renewal

Where we got lucky

• We were lucky that the fault did not affect production user certificates, and it alerted us to carefully review our designed processes before going live with automatic certificate renewal.

Action Items

Action Item	Kind	Due Date	Status
Issuing new certificates with long lifetime	Repair	2024-10-14	Done
Supervision of our processes	Prevent	2024-11-15	In Progress

Appendix

Details of affected certificates

There are 3 affected test web sites, the certificate details can be seen in the attached excel file.

ROOT CA	test page link
Microsec e-Szigno Root CA 2009	https://qtlsca2018-revoked.e-szigno.hu/
e-Szigno Root CA 2017	https://eqtlsca2018-revoked.e-szigno.hu/
e-Szigno TLS Root CA 2023	https://edvtlsca2023-revoked.e-szigno.hu/

Based on Incident Reporting Template v. 2.0

Comment 1

[dr. Sándor SZŐKE]

Attachment: Microsec_Incident_Expired_Revoked_Certificates.xlsx

Comment 2

[Jaime Hablutzel]

Are you considering introducing automated monitoring (e.g. every day) to check for the expected status (valid, expired or revoked) of all your currently deployed test Web pages certificates?

This can be easily accomplished from a monitoring system that allows to execute custom commands/scripts like Nagios.

Comment 3

[dr. Sándor SZŐKE]

Answer to Jaime Hablutzel

Thank you for your question.

Definitely, it will be part of our solution to prevent this problem from happening again.

Microsec already uses Nagios to monitor its IT system.
We have a monitoring system to check the validity (revoked or expired) of our certificates used in our services every day, which sends a notification to our system administrators if a given certificate will expire within less than 25 days.
Normal cases the revoked certificate shall be replaced, but it in this special case the revoked status is OK, and our administratos turned off the warning for these certificates, this way we were not informed about the expiry of these test purpose certificates.

We need to reconfigure this system to properly handle these special revoked certificates for "REVOKED certificate testing website".

Comment 4

[dr. Sándor SZŐKE]

Status Report 2024-10-25

Microsec has begun a review of external requirements and internal processes to ensure the availability and proper functioning of its test websites.
Without goig into details, I summarise the main topics of our work.

• An in-depth review of external requirements.
  • CABF BR and EVG
  • Major Root Programs (Apple, Google, Microsoft, Mozilla)
• Clarifying the requirements for test websites
  • Certificate types
  • detailed specifications
    • Certificate validity periods
    • Key reuse for expired and revoked certificates
• Full automation for certificate exchange
  • with key generation if necessary/possible
  • for all types of test pages
  • automatic exchange of certificates (and keys) on test websites
• introduction of automatic daily testing of test websites
  • availability, functioning as expected
  • certificate revocation status
  • certificate expiry coming soon
  • alert IT administrators in the event of an error
  • detailed audit logs
• The extension of independent systems audits to this area
  • monitoring the proper functioning of automatic test programs
• Immediate implementation of regular manual checks of test web pages and their operation until the introduction of automated testing.

Action Items

Action Item	Kind	Due Date	Status
Issuing new certificates with long lifetime for REVOKED test sites	Repair	2024-10-14	Done
Supervision of our processes	Prevent	2024-11-15	In Progress

Based on Incident Reporting Template v. 2.0

Comment 5

[dr. Sándor SZŐKE]

Status Report 2024-11-08

Microsec is working on problems simultaneously and making progress in several areas.
I will give a short summary without going into details.

An in-depth review of external requirements.

• We have completed the review and found the following relevant requirements:

  • CABF BR

    The CA SHALL host test Web pages that allow Application Software Suppliers to test their software
    with Subscriber Certificates that chain up to each publicly trusted Root Certificate. At a minimum,
    the CA SHALL host separate Web pages using Subscriber Certificates that are

    i. valid,

    ii. revoked, and

    iii. expired.

  • CABF EVG

    The CA MUST host test Web pages that allow Application Software Suppliers to test their software
    with EV Certificates that chain up to each EV Root Certificate. At a minimum, the CA MUST host
    separate Web pages using certificates that are:

    i. valid,

    ii. revoked, and

    iii. expired.

  • Google Chrome Root Program

    ... the Chrome Root Program will only accept CCADB “Root Inclusion Requests” from Applicant PKI hierarchies that support at least one automated solution for certificate issuance and renewal for each Baseline Requirements certificate policy OID (i.e., IV, DV, OV,EV) the corresponding hierarchy issues.

    For each Baseline Requirements certificate policy OID the corresponding hierarchy issues, the Applicant MUST use its automated solution to issue test TLS server authentication certificates (i.e., "automation test certificates") intended to demonstrate its automation capabilities to the Chrome RootProgram. Automation test website certificates MUST be renewed at least once every 30 days...

Clarifying the requirements for test websites

• After analysing the requirements, Microsec decided to reorganize its test web pages
• We plan to operate 6 test web pages under each root

Chrome	CABF BR	CABF EVG	Certificate Type	Validity Status	Renewal Method	Validity Period	Renew Frequency	Key Reuse
+			DV	VALID	ACME	10 days	7 days	Reuse
+			OV	VALID	ACME	10 days	7 days	Reuse
+			IV	VALID	ACME	10 days	7 days	Reuse
+	+	+	EV	VALID	ACME	10 days	7 days	Reuse
	+	+	EV	REVOKED	manual/ACME	max. 398 days	at least yearly	Generate new key
	+	+	EV	EXPIRED	manual/ACME	5 minutes	at least yearly	Generate new key

• Microsec currently has 2 roots in operation and 2 new roots waiting for inclusion

Root	Status	Certificate Validity End
Microsec e-Szigno Root CA 2009	in operation	2029-12-30 11:30:18 UTC
e-Szigno Root CA 2017	in operation	2042-08-22 12:07:06 UTC
e-Szigno TLS Root CA 2023	inclusion requested	2038-07-17 14:00:00 UTC
e-Szigno TLS Root CA 2024	waiting for first AAL	2039-07-14 08:42:19 UTC

• as a result, Microsec plans to set up and operate 24 test web sites

Full automation for certificate exchange

• currently, all VALID test site certificates are automatically renewed weekly using ACME
• further planned developments
  • manage automatic key generation for each certificate issuance
  • manage different validity periods
  • manage different renewal periods
  • automatic revocation of test certificates

Introduction of automatic daily testing of test websites

• Currently, we test the availability and expiration of test certificates internally using Nagios, but this test was developed for one-year certificate validity
• we plan to improve automatic testing
  • external test from the public internet will check
    • availability of test pages
    • in the event of an error, an e-mail and SMS will be sent to the system administrators on duty
  • internal test will be mainly based on Nagios and will test
    • availability of test certificates
    • revocation status
    • validity end date, incoming expiry
    • system alarms will be configured for different faults
    • in the event of an error, an e-mail and SMS will be sent to the system administrators on duty
    • detailed logs will be included in audit logs
    • audit logs will be routed into the independent system audits

Extension of independent systems audits to this area

• waiting for the previously described audit logs to be available
• will supervise the proper operation of automatic test programs

Immediate implementation of regular manual checks of test web pages and their operation until the introduction of automated testing.

• Regular checks are already running and will continue until fully automated tests are available

Action Items

Action Item	Kind	Due Date	Status
Issuing new certificates with long lifetime for REVOKED test sites	Repair	2024-10-14	Done
Supervision of our processes	Prevent	2024-11-15	Done
Immediate implementation of regular manual checks of test web pages and their operation until the introduction of automated testing	Prevent	2024-10-30	Done
An in-depth review of external requirements	Prevent	2024-10-30	Done
Clarifying the requirements for test websites	Prevent	2024-11-08	Done
Full automation for certificate exchange	Prevent	2024-11-30	In Progress
Introduction of automatic daily testing of test websites	Prevent	2024-11-15	In Progress
Extension of independent systems audits to this area	Prevent	2024-12-31	Planned

Based on Incident Reporting Template v. 2.0

Comment 6

[dr. Sándor SZŐKE]

Status Report 2024-11-15

Microsec is working on problems simultaneously and making progress in several areas.
I will give a short summary without going into details.

Clarifying the requirements for test websites

• Microsec reviewed the original plan and made some changes. The two new dedicated roots certifiy the same suburdinate CA units, there is no reason to duplicate each test website. Instead of duplicating, Microsec sets up each test websites so that the certificates can be validated by tracing back to both root certificates.
• as a result, Microsec plans to set up and operate 18 test web sites
  • 6+6 test websites for the mutipurpose roots
  • 6 special websites for the two dedicated roots
• all the test websites have been set up according to the new specification

Full automation for certificate exchange

• currently, all VALID test site certificates are automatically renewed weekly
• further developments for REVOKED and EXPIRED test certificates to be decided
  • manage automatic key generation for each certificate issuance
  • manage different validity periods
  • manage different renewal periods
  • automatic revocation of test certificates

Introduction of automatic daily testing of test websites

• we improved our automatic testing
  • we use the same script for external tests from the public internet as well as internal tests. We perform the following tests
    • availability of the test sites
    • certificate expiration, incoming expiration warning
    • the revocation status is checked using OCSP, the actual status is compared with the expected validity status
    • in the event of an error, an e-mail and SMS will be sent to the system administrators on duty
    • detailed logs are included in audit logs (syslog)
    • audit logs will be processed by the independent system audits

Extension of independent system audits to this area

• new task opened in jira to add new test case(s)
• currently we are working on the processing of the received log entries

Weekly manual checks of test web pages

• checks are already running and will continue until fully automated tests are available

Action Items

Action Item	Kind	Due Date	Status
Issuing new certificates with long lifetime for REVOKED test sites	Repair	2024-10-14	Done
Supervision of our processes	Prevent	2024-11-15	Done
Immediate implementation of regular manual checks of test web pages and their operation until the introduction of automated testing	Prevent	2024-10-30	Done
An in-depth review of external requirements	Prevent	2024-10-30	Done
Clarifying the requirements for test websites	Prevent	2024-11-08	Done
Full automation for certificate exchange	Prevent	2024-11-30	In Progress
Introduction of automatic daily testing of test websites	Prevent	2024-11-15	Done
Extension of independent system audits to this area	Prevent	2024-12-31	In progress
Weekly manual checks of test web pages	Prevent/Detect	until fully automated tests are available	In progress

Based on Incident Reporting Template v. 2.0

Comment 7

[dr. Sándor SZŐKE]

Status Report 2024-11-22

Full automation for certificate exchange

• we have made a decision regarding REVOKED and EXPIRED test sites.
  • the renewal of these test certificates will initiated manually, when
    • a new CPS comes into affect
    • there will be any change to the certificate profile
    • before expiry of the REVOKED test certificate
    • the issueance will be done using ACME to automatically support as many functions as possible
    • the revocation will be done manually

Extension of independent system audits to this area

• we are working on processing the received log entries

Weekly manual checks of test web pages

• checks are already running and will continue until fully automated tests are available

Action items status changed or still open

Action Item	Kind	Due Date	Status
Full automation for certificate exchange	Prevent	2024-11-30	Done
Extension of independent system audits to this area	Prevent	2024-12-31	In progress
Weekly manual checks of test web pages	Prevent/Detect	until fully automated tests are available	In progress

Based on Incident Reporting Template v. 2.0

Comment 8

[dr. Sándor SZŐKE]

Closing Status Report 2025-01-10

Extension of independent system audits to this area

• log entries have been processed since the end of November 2024
  • monitors for the absence of required log entries
  • monitors ERROR messages in log entries

Weekly manual checks of test web pages

• manual checks were terminated after one month successful running of the automated tests on 2025-01-01

Action items status changed or still open

Action Item	Kind	Due Date	Status
Extension of independent system audits to this area	Prevent	2024-12-31	Done
Weekly manual checks of test web pages	Prevent/Detect	until fully automated tests are available	Terminated

Comment 9

[Ben Wilson]

I'll close this on or about Monday, 13-Jan-2025.