2047337 - Government of Korea: Delayed revocation related to Bugzilla 2032478

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Ji Eun Seong]

Steps to reproduce:

Preliminary Incident Report

Summary

• Incident description:

  Certain certificates with the identified non-compliant profile values described in Bugzilla Bug 2032478 were not revoked within the 120-hour revocation timeline required by BR Section 4.9.1.1.

  Government of Korea became technically aware on 2026-04-17 that the reported certificates shared the same underlying certificate profile issue. Based on that awareness date, the applicable 120-hour revocation deadline was 2026-04-22.

  Some certificates remained valid beyond that deadline. As of 2026-06-02, all certificates with the identified non-compliant profile values had been revoked or had expired. Government of Korea confirmed on 2026-06-05 that no such certificates remained valid.

• Relevant policies:

  CA/Browser Forum TLS Baseline Requirements, Section 4.9.1.1

• Source of incident disclosure:

  Self-reported by Government of Korea, following the discussion in Bugzilla Bug 2032478

Government of Korea will provide a full incident report for this delayed revocation issue in this Bugzilla ticket.

Comment 1

[Ji Eun Seong]

Attachment: Certificate Details (Revoked and Expired) under GPKIRootCA1.csv

Full Incident Report

Summary

• CA Owner CCADB unique ID: Government of Korea, KLID (A000749)

• Incident description:

  This report addresses only the delayed revocation aspect related to the certificate profile issue reported in Bugzilla Bug 2032478.

  Government of Korea became technically aware on 2026-04-17 00:10 UTC that certain TLS Subscriber Certificates issued under the legacy GPKIRootCA1 hierarchy, specifically through CA131100001 and CA131100002, shared the same underlying certificate profile issue described in Bugzilla Bug 2032478.

  Under BR Section 4.9.1.1, the applicable 120-hour revocation deadline was 2026-04-22 00:10 UTC.

  Some affected certificates remained valid beyond that 120-hour deadline. This report is limited to that delayed revocation issue.

• Timeline summary:

  • Non-compliance start date: 2026-04-22 00:10 UTC

  • Non-compliance identified date: 2026-04-22 00:10 UTC

  • Non-compliance end date: 2026-06-02

    This is the date when the last remaining valid certificate with the identified non-compliant profile values was revoked. Government of Korea completed additional verification on 2026-06-05 and confirmed that no such certificates remained valid.

• Relevant policies:

  • CA/Browser Forum TLS Baseline Requirements, Section 4.9.1.1

• Source of incident disclosure:

  Self-reported by Government of Korea following the investigation and discussion in Bugzilla Bug 2032478.

---

Impact

• Total number of certificates

  82 certificates remained valid beyond the 120-hour revocation deadline and are included in the delayed revocation population for this report.

  From the total profile-issue population of 98 certificates, 16 certificates were resolved within the 120-hour revocation deadline, consisting of:

  • 9 revoked certificates
  • 7 expired certificates

  The remaining 82 certificates are the population for this delayed revocation report.

• Total number of "remaining valid" certificates

  0

  Government of Korea confirmed on 2026-06-05 that no certificates with the identified non-compliant profile values remained valid.

• Affected certificate types

  TLS Subscriber Certificates issued under the legacy GPKIRootCA1 hierarchy through CA131100001 and CA131100002.

• Incident heuristic

  The delayed revocation population consists of certificates from the final 98-certificate profile-issue population identified in Bugzilla Bug 2032478 that remained valid after 2026-04-22 00:10 UTC, the 120-hour revocation deadline calculated from Government of Korea’s technical awareness on 2026-04-17 00:10 UTC.

• Was issuance stopped in response to this incident, and why or why not?

  No issuance stoppage was required as a direct response to this delayed revocation incident.

  This incident concerns revocation execution rather than new issuance.

  In addition, no new publicly trusted TLS server certificates were issued under the affected legacy GPKIRootCA1 hierarchy after the Microsoft NotBefore restriction date of 2025-09-15.

• Analysis

  Government of Korea treated the underlying certificate profile issue as a population-level issue after technical awareness on 2026-04-17.

  However, not all certificates in the affected population that remained valid were revoked within the 120-hour period required by BR Section 4.9.1.1.

  The delayed revocation was caused by a combination of:

  • insufficient deadline-based revocation tracking for the full affected certificate population; and
  • manual reconciliation errors during certificate population documentation.

  These issues resulted in some certificates remaining valid beyond the required revocation deadline.

• Additional considerations

  This report does not re-open the underlying certificate profile issue.

  That issue is addressed in Bugzilla Bug 2032478.

  This report is limited to the separate delayed revocation incident.

---

Timeline

Date / Time	Event
2026-04-16	Bugzilla Bug 2032478 was opened, reporting certificate profile issues in certificates issued under the legacy GPKIRootCA1 hierarchy.
2026-04-17 00:10 UTC	Government of Korea’s engineering team became technically aware that the reported certificates shared the same underlying certificate profile issue. This timestamp was used as the start of the 120-hour revocation timeline under BR Section 4.9.1.1.
2026-04-22 00:10 UTC	The 120-hour revocation deadline elapsed. Certificates from the affected population that remained valid after this timestamp became part of this delayed revocation incident.
2026-04-22	Government of Korea performed the initial revocation batch for affected certificates. Nine certificates were revoked, and seven certificates in the affected population had expired.
2026-04-29	Government of Korea revoked 80 additional certificates with the identified non-compliant profile values. These certificates were revoked after the 120-hour deadline.
2026-05-04	Government of Korea identified and revoked one additional certificate with the identified non-compliant profile values that had been omitted from the previously documented certificate population.
2026-06-02	Government of Korea identified eight additional records that had been omitted due to manual reconciliation error when reorganizing the internal DB extract into the uploaded Appendix and Full Incident Report. Seven had already been revoked, and one remained valid at that time.
2026-06-02	Government of Korea revoked the last remaining valid certificate with the identified non-compliant profile values. Government of Korea also incorporated PKIMetal-based verification into the improved verification process for GPKIRootCA1 and performed a PKIMetal linting scan.
2026-06-05	Government of Korea confirmed that no certificates with the identified non-compliant profile values remained valid.
2026-06-13	Government of Korea opened Bugzilla Bug 2047337 to track this delayed revocation issue as a separate incident.

---

Related Incidents

Bug	Date	Description
Bugzilla Bug 2032478	2026-04-16	Government of Korea: Misissuance detected by PKIMetal. This is the underlying certificate profile issue that triggered the revocation obligation. This report addresses only the delayed revocation aspect of that incident.

---

Root Cause Analysis

Contributing Factor #1: Insufficient 120-hour revocation deadline tracking for the full affected certificate population

Description

After Government of Korea became technically aware that the reported certificates shared the same underlying certificate profile issue, all affected certificates that remained valid should have been tracked against a single 120-hour revocation deadline.

Government of Korea did not complete revocation of the full affected population within that deadline. Although revocation actions were performed, the controls were insufficient to confirm that all valid affected certificates had been revoked or had expired before 2026-04-22 00:10 UTC.

Timeline

Government of Korea became technically aware of the shared profile issue on 2026-04-17 00:10 UTC.

The 120-hour deadline elapsed on 2026-04-22 00:10 UTC.

Additional affected certificates were revoked on:

• 2026-04-29
• 2026-05-04
• 2026-06-02

Detection

The delayed revocation issue was identified through review of the revocation timeline, certificate status, and subsequent reconciliation and verification of the affected certificate population.

Interaction with other factors

This factor interacted with the manual reconciliation error described below.

Because the full affected population was not managed through a complete deadline-based register, omitted records were not detected early enough to ensure revocation within the 120-hour deadline.

Root Cause Analysis methodology used

Government of Korea reviewed:

• Bugzilla incident timeline
• affected certificate population
• revocation timestamps
• DB query results
• uploaded Appendix

Contributing Factor #2: Manual reconciliation error in certificate population documentation

Description

Government of Korea determined that some inconsistencies in the documented affected certificate population resulted from manual reconciliation error when reorganizing the internal DB extract into reporting materials.

The final affected certificate population for the underlying profile issue was 98 certificates valid as of 2026-04-16. During subsequent verification, eight records were identified as omitted from the uploaded Appendix and Full Incident Report. Seven had already been revoked, and one remained valid until it was revoked on 2026-06-02.

Timeline

The certificate population review began after technical awareness on 2026-04-17.

• 2026-05-04: One omitted certificate was additionally identified and revoked.
• 2026-06-02: Eight omitted records were additionally identified.
• 2026-06-02: The last remaining valid certificate was revoked.

Detection

This issue was detected during subsequent verification and reconciliation among:

• the internal certificate DB,
• DB SQL query results,
• the uploaded Appendix, and
• the Full Incident Report.

Interaction with other factors

Manual reconciliation increased the risk that both the operational revocation population and the publicly reported population would be incomplete. This contributed to the delayed identification and delayed revocation of the last remaining valid certificate.

Root Cause Analysis methodology used

Government of Korea performed:

• DB record review,
• SQL query result comparison,
• certificate status review,
• revocation timestamp review, and
• reconciliation against the uploaded Appendix and Full Incident Report.

---

Lessons Learned

What went well

Government of Korea investigated the externally reported certificate profile issue, identified the affected legacy GPKIRootCA1 issuance paths, and revoked all affected certificates that remained valid.

Government of Korea also incorporated PKIMetal-based verification into the improved verification process for GPKIRootCA1 and confirmed that no affected certificates remained valid as of 2026-06-05.

What didn't go well

Government of Korea did not revoke all affected certificates that remained valid within the 120-hour deadline.

In addition, the initial incident response relied heavily on manual reconciliation when converting the DB extract into the documented affected certificate population.

Effective 2026-06-26, Government of Korea established a step-by-step procedure to:

• prohibit manual reconciliation,
• require system-based reconciliation,
• require reviewer cross-checking, and
• require the responsible owner to confirm the accuracy of the final result.

Where we got lucky

This issue was externally reported and reviewed by the community.

Subsequent reconciliation also identified omitted records, and no affected certificate remained valid after 2026-06-02.

Additional

For future certificate profile-level non-compliance issues, Government of Korea will treat the issue as a full-population revocation matter from the earliest point of technical awareness and will track each affected certificate against the applicable BR revocation deadline.

---

Action Items

Action Item	Kind	Corresponding Root Cause(s)	Evaluation Criteria	Due Date	Status
Revoke all affected certificates that remained valid beyond the 120-hour revocation deadline.	Mitigate	Contributing Factor #1, Contributing Factor #2	Remaining valid certificate count for the identified non-compliant profile issue is 0.	2026-06-02	Complete
Complete final verification that no certificates with the identified non-compliant profile values remain valid.	Detect	Contributing Factor #1, Contributing Factor #2	Verification result confirms 0 remaining valid certificates.	2026-06-05	Complete
Incorporate PKIMetal-based verification into the improved verification process for GPKIRootCA1.	Detect	Contributing Factor #1	PKIMetal scan results are retained and reviewed.	2026-06-02	Complete
Update the Appendix to reflect the final verified affected certificate population and identify the delayed revocation subset.	Correct	Contributing Factor #2	The Appendix count and delayed revocation subset match the internal DB query results and revocation timestamps.	2026-06-13	Complete
Update and implement a BR 4.9 deadline-based revocation tracking procedure for future revocation incidents.	Prevent	Contributing Factor #1	The revised procedure became effective on 2026-06-26 and requires recording the certificate identifier, applicable BR revocation deadline, certificate status, revocation timestamp, owner, and reviewer.	2026-06-26	Complete
Update the incident response procedure to prohibit manual-only reconciliation of affected certificate populations before public reporting.	Prevent	Contributing Factor #2	The revised procedure became effective on 2026-06-26 and requires generation of an affected-population report based on DB query results, independent reviewer review followed by reviewer cross-checking, and final responsible-owner verification of reconciliation evidence and internal CA system integrity before submission of an Appendix or Full Incident Report.	2026-06-26	Complete

---

Appendix

The attached CSV Appendix identifies the delayed revocation certificate population for this report.

The Appendix includes, at minimum:

• Certificate Identifier
• Issuing CA
• Certificate Status
• NotBefore
• NotAfter
• Revocation Date
• Whether the certificate remained valid beyond the deadline

The final profile-issue population consists of 98 certificates valid as of 2026-04-16.

This delayed revocation report covers the subset of that population that remained valid beyond the 2026-04-22 00:10 UTC revocation deadline.

Comment 2

[James]

The current Root Cause Analysis (RCA) is too abstract to understand what actually went wrong.
Specifically, the following statement:

"insufficient deadline-based revocation tracking for the full affected certificate population; and manual reconciliation errors during certificate population documentation."

is merely a high-level summary of the symptoms, not a root cause. It completely fails to explain the exact technical or operational mechanics of why these failures occurred.

Because the root cause remains unknown, it is impossible for the community to understand or trust your prevention plan.

Before discussing any remediation, please perform a proper "5 Whys" analysis and clarify:

1. What was the exact technical or operational bottleneck that caused 82 certificates to miss the 120-hour deadline?
2. What specific breakdown in your data extraction process caused the verification team to miss the 8 remaining records three separate times (April 22, May 4, and June 2)?

Comment 3

[Ji Eun Seong]

(In reply to James from comment #2)

Before discussing any remediation, please perform a proper "5 Whys" analysis and clarify:

1. What was the exact technical or operational bottleneck that caused 82 certificates to miss the 120-hour deadline?
2. What specific breakdown in your data extraction process caused the verification team to miss the 8 remaining records three separate times (April 22, May 4, and June 2)?

Government of Korea provides the following clarification regarding the specific operational causes requested in Comment 2.

1. Failure to complete revocation within 120 hours

Following a previous certificate issuance incident involving domain validation, GPKI added a procedure to re-check domain-validation records and subscriber application information before performing forced revocation.

As previously explained in Comment 14 of Bug 2032478, Government of Korea’s incident response process prioritized the review of internal certificate databases, certificate profiles, and domain-validation-related records based on its previous incident response experience concerning domain validation.

As also described in Comment 12 of Bug 2032478, the process included verification of subscriber requests, confirmation that the domain names in the certificates matched the original requests, individual validation of revocation candidates, and written and telephone notifications to subscribers.

During this incident, the re-checking of domain-validation records and communications with subscribers to confirm the relevant facts were performed before revocation. These verification and communication activities delayed completion of the revocations, resulting in certain certificates not being revoked within the 120-hour period required by the CA/Browser Forum Baseline Requirements.

As acknowledged in Comment 16 of Bug 2032478, this delayed revocation did not comply with BR Section 4.9.1.1, and subscriber administrative coordination or operational impact cannot supersede or delay a mandatory revocation deadline.

Government of Korea has therefore revised the procedure so that re-checking domain-validation records and confirming facts with subscribers are no longer treated as prerequisites to mandatory revocation.

For future incidents, once a mandatory revocation condition under the CA/Browser Forum Baseline Requirements has been established, revocation will be prioritized and tracked against the applicable deadline calculated from the technical-awareness timestamp.

Subscriber notifications and any additional factual confirmation will be performed in parallel with, or after, revocation and will not be permitted to delay the applicable mandatory revocation deadline.

2. Omission from the documented certificate population

Government of Korea reviewed the source certificate database again and confirmed that all 98 records valid as of 2026-04-16 remained present in the database and had not been modified or deleted.

The omission did not occur in the certificate database or during the SQL query process. The database query results were exported to Excel for preparation of the internal reporting materials. During this reporting step, an Excel function was manually applied, which caused certain rows to be excluded from the internal working and reporting list.

Therefore, the direct cause of the omission was not a structural defect or data loss in the certificate database. It was a manual processing error that occurred after the database results had been exported and reorganized in Excel for reporting purposes.

This further clarifies the explanation provided in Comment 14 of Bug 2032478: no structural flaw was identified in the issuance database itself, and the omission occurred when data extracted from the internal certificate database was reorganized into the documented affected population.

To prevent recurrence, Government of Korea discontinued the process of manually reorganizing the affected certificate population in Excel. The affected population is now determined using a report generated directly from the CA system.

The revised process also requires reviewer cross-checking of the system-generated results and final confirmation by the responsible owner. This improved procedure became effective on 2026-06-26 and corresponds to the system-based reconciliation, reviewer cross-checking, and responsible-owner verification controls described in Comment 1 of Bug 2047337.

Comment 4

[Ji Eun Seong]

Report Closure Summary

Incident Description

Government of Korea confirmed that certain publicly trusted TLS server certificates affected by the non-compliant profile values identified in Bug 2032478 were not revoked within the 120-hour revocation timeline required by Section 4.9.1.1 of the CA/Browser Forum TLS Baseline Requirements.

Government of Korea became technically aware on April 17, 2026, that the certificates shared the same underlying certificate profile issue. The corresponding revocation deadline was April 22, 2026. However, certain certificates remained valid beyond that deadline.

Incident Root Cause(s)

The incident response process initially relied on reviews of internal certificate databases, certificate profiles, domain validation-related records, and subscriber notification procedures instead of immediately performing an automated linting review of the complete valid certificate population.

Manual reconciliation errors also occurred when database extracts were reorganized into the documented affected certificate population. Consequently, the complete affected certificate population was not identified rapidly and comprehensively.

In addition, the point of technical awareness was not clearly distinguished from the timing of individual subscriber notification and revocation execution, resulting in certain certificates remaining valid beyond the applicable 120-hour revocation deadline.

Remediation Description

Government of Korea reverified the affected certificate population and completed the revocation of all affected certificates that remained valid.

By June 2, 2026, all affected certificates had either been revoked or had expired. On June 5, 2026, Government of Korea confirmed that no valid certificates affected by the identified non-compliant profile values remained.

Government of Korea also incorporated automated full-population linting using PKIMetal (or an equivalent tool) into its incident response process.

The reconciliation process was revised so that database extracts are preserved and reviewed without alteration.

The incident reporting process was also updated to clearly distinguish the point of technical awareness from subscriber notification and revocation execution.

Commitment Summary

Issuance of publicly trusted TLS server certificates under the GPKIRootCA1 hierarchy remains discontinued, and Government of Korea has no plans to resume such issuance.

For future certificate-related incidents, Government of Korea will:

• Perform automated verification of the complete valid certificate population
• Calculate the applicable revocation timeline from the point of technical awareness
• Track and manage revocation activities against the applicable Baseline Requirements deadlines

All Action Items disclosed in this report have been completed as described, and Government of Korea respectfully requests closure of this incident report.

Comment 5

[incident-reporting]

This is a final call for comments or questions on this Incident Report.

Otherwise, it will be closed on approximately 2026-08-04.