Closed
Bug 2016672
Opened 1 month ago
Closed 3 days ago
certSIGN: certificates with delayed SCT signature
Categories
(CA Program :: CA Certificate Compliance, task)
CA Program
CA Certificate Compliance
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: gabriel.petcu, Assigned: gabriel.petcu)
Details
(Whiteboard: [ca-compliance] [ov-misissuance])
Steps to reproduce:
Preliminary Incident Report
Summary
Incident description: During investigation of a certificate reported by the Chrome Root Program, we identified a total of seven (7) leaf certificates where the notBefore value appears to exceed the 48-hour window defined in BR 7.1.2.7 when compared against the final certificate signing operation.
Relevant requirements: Baseline Requirements for the Issuance and Management of Publicly-Trusted TLS Server Certificates - 7.1.2.7 Subscriber (Server) Certificate Profile – notBefore: a value within 48 hours of the certificate signing operation.
**Source of incident disclosure: ** Certificate Problem Reporting by Chrome Root Program<chrome-root-program@google.com> through Revokecsgn@certsign.ro for 1 certificate, followed by internal investigation.
The certificates will be revoked.
Our investigation is ongoing, and a full incident report will follow.
|
||
Updated•1 month ago
|
Assignee: nobody → gabriel.petcu
Status: UNCONFIRMED → ASSIGNED
Type: defect → task
Ever confirmed: true
Whiteboard: [ca-compliance] [__-misissuance]
|
||
Comment 1•1 month ago
|
||
Thanks for posting the preliminary incident report, Gabriel.
Just FYI, whilst our team was reviewing it this afternoon, Martijn Katerbarg pointed out that this is something that ctlint could check for. So I've added a lint to ctlint, which you can see in action at (for example):
|
|
||
Comment 2•1 month ago
|
||
Over at Zlint, I see Adriano had the same idea:
|
Assignee | |
Comment 3•1 month ago
|
||
Full Incident Report
Summary
- CA Owner CCADB unique ID: A000013
- Incident description: certSIGN issued 14 subscriber certificates where the certificate notBefore value was more that 48 hours earlier than the effective signing operation as interpreted form the last embedded SCT timestamp. Affected certificates failed to meet the Baseline Requirements for the Issuance and Management of Publicly-Trusted TLS Server Certificates - 7.1.2.7 Subscriber (Server) Certificate Profile – notBefore: a value within 48 hours of the certificate signing operation.
- Timeline summary:
- Non-compliance start date: 2023-09-15 00:00:00 GMT
- Non-compliance identified date: 2026-02-13 13:52:00 GMT
- Non-compliance end date: 2026-02-16 10:47:00 GMT
- Relevant policies: Baseline Requirements for the Issuance and Management of Publicly-Trusted TLS Server Certificates - https://cabforum.org/working-groups/server/baseline-requirements/documents/CA-Browser-Forum-TLS-BR-2.2.2.pdf
- Source of incident disclosure: Certificate Problem Reporting by Chrome Root Program<chrome-root-program@google.com> through Revokecsgn@certsign.ro, followed by internal investigation.
Impact
- Total number of certificates: 14
- Total number of "remaining valid" certificates: 7
- Affected certificate types: OV: 14
- Incident heuristic: List of affected certificates, see appendix
- Was issuance stopped in response to this incident, and why or why not?: No operational impact was reported by subscribers. Certificate issuing was not stopped because the issue was determined to be related to CT logging timing controls and did not affect validation procedures, certificate contents or key material..
- Analysis: on any exception or delay on the responses from CT Logs we had a manual retry through an operator
Timeline
| Time (UTC) | Event |
|---|---|
| 2023-09-15 00:00:00 | Effective date for applying the Certificate Profiles Update – in Section 7 of Baseline Requirements for the Issuance and Management of Publicly-Trusted TLS Server Certificates v2.0.0 |
| 2026-02-12 13:52:00 | Chrome Root Program chrome-root-program@google.com reported to Revokecsgn@certsign.ro a potential non-compliance for one certificate with valid SCTs outside the temporal interval of 48 hours requested by TLS BR. |
| 2026-02-12 14:13:00 | certSIGN acknowledged to Chrome Root Program the notification by email and begins investigation. |
| 2026-02-12 14:14:00 | certSIGN opened an internal ticket (SVCCA-13) Potential CA/Browser Forum TLS BR Non-compliance – for analysis of the issue. |
| 2026-02-12 14:33:00 | certSIGN notified the appropriate DRA about the problem under investigation. |
| 2026-02-12 16:14:00 | certSIGN notified the appropriate DRA to work with Subscriber to reissue/revoke the affected certificate. |
| 2026-02-12 16:48:00 | certSIGN analysis pending. No impact in Production. The list of affected certificates had been updated. |
| 2026-02-12 16:55:00 | certSIGN opened an internal ticket (CSFCA-1273) for developing and testing the alternative solutions on the problem. |
| 2026-02-13 09:00:00 | The solution to fix the issue is developed and started to be tested. |
| 2026-02-13 11:01:00 | certSIGN opens Bugzilla ticket #2016672 with the Preliminary Incident Report. |
| 2026-02-13 16:19:00 | certSIGN deploys a patch to fix the issue and prevent future delays (CSFCA-1273). |
| 2026-02-16 10:47:00 | certSIGN revoked the valid non-conformant certificates. |
Related Incidents
| Bug | Date | Description |
|---|---|---|
| N/A | N/A | N/A |
| --- | --- | --- |
Root Cause Analysis
Contributing Factor #1: Lack of a limit for logging to CT logs
- Description: In certSIGN logic for logging precertificates to CT logs, on any failure, the CT logging retry mechanism lacked an upper time boundary aligned with the 48 hour requirement defined by BR 7.1.2.7; as a result, in rare cases, SCT occurs outside the permitted interval.
- Timeline: 2023-09-15
- Detection: Manual investigation on how certificates could be attempting to use CT logs over the 48 hours allowed by TLS BR since Precertificate creation.
- Interaction with other factors: lack of controls on the delay between the Precertificate issuance and the timestamp of the CT logs, through a linter check, made this harder to detect.
- Root Cause Analysis methodology used: Barrier/Comparison Analysis
Contributing Factor #2: Lack of delay check in Linter
- Description: certSIGN linter, that is calling a set of other external linters also, like zlint, did not included the controls of the delay between the Precertificate issuance and the timestamp of the CT logs.
- Timeline: 2023-09-15
- Detection: Manual investigation on how certificates could be attempting to use CT logs over the 48 hours allowed by TLS BR since Precertificate creation.
- Interaction with other factors: the certSIGN logic on logging that lacks a limit for logging to CT logs.
Lessons Learned
- What went well: once we were aware of the issue, it wasn’t difficult to quicly replace the impacted certificates and to fix the issue of CT logging limits.
- What didn’t go well: the lack of control on the specific limit allowed the existence of a small numer of non-conformant certificates.
- Where we got lucky: the clients were not impacted and were cooperant on the renewal of the affected certificates.
Action Items
| Action Item | Kind | Corresponding Root Cause(s) | Evaluation Criteria | Due Date | Status |
|---|---|---|---|---|---|
| Patch system ensure CT Log delay intervals are respected | Prevent | Lack of a limit for logging to CT logs | Unit tests pass, logs timestamp within the accepted delay | 2026-02-13 | Complete |
| Add linter check for the interval to SCT | Detect | Lack of delay check in Linter | Testing for SCT delay validations correctly block SCT embedding if delay interval is not correct | 2026-02-28 | Ongoing |
| Reach out to affected customers for reissuance and revocation | Mitigate | What didn’t go well: non-conformant certificates | No. of non-conformant certificates | 2026-02-16 | Complete |
Appendix
| No | Serial | crt.sh | Status | notBefore | CTLog Time |
| 1 | 2305930542140877E3B16D | https://crt.sh/?id=12848345724 | Expired | Apr 25 13:43:18 2024 GMT | 2024-05-02 08:59:53 UTC |
| 2 | 230E2ABFB78EAA3FE418D9 | https://crt.sh/?id=14930081717 | Expired | Oct 11 16:36:36 2024 GMT | 2024-10-14 15:06:44 UTC |
| 3 | 230E2907CBC3F06D125671 | https://crt.sh/?id=14942771657 | Expired | Oct 11 16:36:34 2024 GMT | 2024-10-15 11:44:26 UTC |
| 4 | 230E285E0865137EC58BAC | https://crt.sh/?id=14942784495 | Expired | Oct 11 16:36:31 2024 GMT | 2024-10-15 11:45:20 UTC |
| 5 | 230E272ACB14B3437CD250 | https://crt.sh/?id=14942788098 | Expired | Oct 11 16:36:29 2024 GMT | 2024-10-15 11:45:55 UTC |
| 6 | 230E263E9887E927E4156A | https://crt.sh/?id=14942787996 | Expired | Oct 11 16:36:27 2024 GMT | 2024-10-15 11:46:43 UTC |
| 7 | 230E2B7626186A33A542CA | https://crt.sh/?id=14942801784 | Expired | Oct 11 16:36:38 2024 GMT | 2024-10-15 11:47:25 UTC |
| 8 | 231728A4FE6EF5774A309D | https://crt.sh/?id=18823008375 | Revoked | Jun 3 08:05:31 2025 GMT | 2025-06-05 11:16:57 UTC |
| 9 | 231855188B1963F023FA93 | https://crt.sh/?id=19367848301 | Revoked | Jun 27 07:03:41 2025 GMT | 2025-07-01 07:25:08 UTC |
| 10 | 231BD77D6ECBF508262817 | https://crt.sh/?id=20947013362 | Revoked | Sep 9 08:49:38 2025 GMT | 2025-09-11 15:27:43 UTC |
| 11 | 231BD95C569CBA87EA1CFC | https://crt.sh/?id=20965162440 | Revoked | Sep 9 08:49:54 2025 GMT | 2025-09-12 09:42:45 UTC |
| 12 | 231D8B79D2A2961B1B1CAF | https://crt.sh/?id=21443305897 | Revoked | Oct 3 12:14:26 2025 GMT | 2025-10-06 06:09:45 UTC |
| 13 | 231D8C444B359E71878847 | https://crt.sh/?id=21446809901 | Revoked | Oct 3 15:24:22 2025 GMT | 2025-10-06 06:10:26 UTC |
| 14 | 23209430142FEBF90DE536 | https://crt.sh/?id=23131354166 | Revoked | Dec 10 09:53:26 2025 GMT | 2025-12-15 10:12:30 UTC |
|
|
Assignee | |
Comment 4•1 month ago
|
||
Full Incident Report
Summary
- CA Owner CCADB unique ID: A000013
- Incident description: certSIGN issued 14 subscriber certificates where the certificate notBefore value was more that 48 hours earlier than the effective signing operation as interpreted form the last embedded SCT timestamp. Affected certificates failed to meet the Baseline Requirements for the Issuance and Management of Publicly-Trusted TLS Server Certificates - 7.1.2.7 Subscriber (Server) Certificate Profile – notBefore: a value within 48 hours of the certificate signing operation.
- Timeline summary:
- Non-compliance start date: 2023-09-15 00:00:00 GMT
- Non-compliance identified date: 2026-02-13 13:52:00 GMT
- Non-compliance end date: 2026-02-16 10:47:00 GMT
- Relevant policies: Baseline Requirements for the Issuance and Management of Publicly-Trusted TLS Server Certificates - https://cabforum.org/working-groups/server/baseline-requirements/documents/CA-Browser-Forum-TLS-BR-2.2.2.pdf
- Source of incident disclosure: Certificate Problem Reporting by Chrome Root Program<chrome-root-program@google.com> through Revokecsgn@certsign.ro, followed by internal investigation.
Impact
- Total number of certificates: 14
- Total number of "remaining valid" certificates: 7
- Affected certificate types: OV: 14
- Incident heuristic: List of affected certificates, see appendix
- Was issuance stopped in response to this incident, and why or why not?: No operational impact was reported by subscribers. Certificate issuing was not stopped because the issue was determined to be related to CT logging timing controls and did not affect validation procedures, certificate contents or key material..
- Analysis: on any exception or delay on the responses from CT Logs we had a manual retry through an operator
Timeline
| Time (UTC) | Event |
|---|---|
| 2023-09-15 00:00:00 | Effective date for applying the Certificate Profiles Update – in Section 7 of Baseline Requirements for the Issuance and Management of Publicly-Trusted TLS Server Certificates v2.0.0 |
| 2026-02-12 13:52:00 | Chrome Root Program chrome-root-program@google.com reported to Revokecsgn@certsign.ro a potential non-compliance for one certificate with valid SCTs outside the temporal interval of 48 hours requested by TLS BR. |
| 2026-02-12 14:13:00 | certSIGN acknowledged to Chrome Root Program the notification by email and begins investigation. |
| 2026-02-12 14:14:00 | certSIGN opened an internal ticket (SVCCA-13) Potential CA/Browser Forum TLS BR Non-compliance – for analysis of the issue. |
| 2026-02-12 14:33:00 | certSIGN notified the appropriate DRA about the problem under investigation. |
| 2026-02-12 16:14:00 | certSIGN notified the appropriate DRA to work with Subscriber to reissue/revoke the affected certificate. |
| 2026-02-12 16:48:00 | certSIGN analysis pending. No impact in Production. The list of affected certificates had been updated. |
| 2026-02-12 16:55:00 | certSIGN opened an internal ticket (CSFCA-1273) for developing and testing the alternative solutions on the problem. |
| 2026-02-13 09:00:00 | The solution to fix the issue is developed and started to be tested. |
| 2026-02-13 11:01:00 | certSIGN opens Bugzilla ticket #2016672 with the Preliminary Incident Report. |
| 2026-02-13 16:19:00 | certSIGN deploys a patch to fix the issue and prevent future delays (CSFCA-1273). |
| 2026-02-16 10:47:00 | certSIGN revoked the valid non-conformant certificates. |
Related Incidents
| Bug | Date | Description |
|---|---|---|
| N/A | N/A | N/A |
| --- | --- | --- |
Root Cause Analysis
Contributing Factor #1: Lack of a limit for logging to CT logs
- Description: In certSIGN logic for logging precertificates to CT logs, on any failure, the CT logging retry mechanism lacked an upper time boundary aligned with the 48 hour requirement defined by BR 7.1.2.7; as a result, in rare cases, SCT occurs outside the permitted interval.
- Timeline: 2023-09-15
- Detection: Manual investigation on how certificates could be attempting to use CT logs over the 48 hours allowed by TLS BR since Precertificate creation.
- Interaction with other factors: lack of controls on the delay between the Precertificate issuance and the timestamp of the CT logs, through a linter check, made this harder to detect.
- Root Cause Analysis methodology used: Barrier/Comparison Analysis
Contributing Factor #2: Lack of delay check in Linter
- Description: certSIGN linter, that is calling a set of other external linters also, like zlint, did not included the controls of the delay between the Precertificate issuance and the timestamp of the CT logs.
- Timeline: 2023-09-15
- Detection: Manual investigation on how certificates could be attempting to use CT logs over the 48 hours allowed by TLS BR since Precertificate creation.
- Interaction with other factors: the certSIGN logic on logging that lacks a limit for logging to CT logs.
Lessons Learned
- What went well: once we were aware of the issue, it wasn’t difficult to quicly replace the impacted certificates and to fix the issue of CT logging limits.
- What didn’t go well: the lack of control on the specific limit allowed the existence of a small numer of non-conformant certificates.
- Where we got lucky: the clients were not impacted and were cooperant on the renewal of the affected certificates.
Action Items
| Action Item | Kind | Corresponding Root Cause(s) | Evaluation Criteria | Due Date | Status |
|---|---|---|---|---|---|
| Patch system ensure CT Log delay intervals are respected | Prevent | Lack of a limit for logging to CT logs | Unit tests pass, logs timestamp within the accepted delay | 2026-02-13 | Complete |
| Add linter check for the interval to SCT | Detect | Lack of delay check in Linter | Testing for SCT delay validations correctly block SCT embedding if delay interval is not correct | 2026-02-28 | Ongoing |
| Reach out to affected customers for reissuance and revocation | Mitigate | What didn’t go well: non-conformant certificates | No. of non-conformant certificates | 2026-02-16 | Complete |
Appendix
| No | Serial | crt.sh | Status | notBefore | CTLog Time |
| 1 | 2305930542140877E3B16D | https://crt.sh/?id=12848345724 | Expired | Apr 25 13:43:18 2024 GMT | 2024-05-02 08:59:53 UTC |
| 2 | 230E2ABFB78EAA3FE418D9 | https://crt.sh/?id=14930081717 | Expired | Oct 11 16:36:36 2024 GMT | 2024-10-14 15:06:44 UTC |
| 3 | 230E2907CBC3F06D125671 | https://crt.sh/?id=14942771657 | Expired | Oct 11 16:36:34 2024 GMT | 2024-10-15 11:44:26 UTC |
| 4 | 230E285E0865137EC58BAC | https://crt.sh/?id=14942784495 | Expired | Oct 11 16:36:31 2024 GMT | 2024-10-15 11:45:20 UTC |
| 5 | 230E272ACB14B3437CD250 | https://crt.sh/?id=14942788098 | Expired | Oct 11 16:36:29 2024 GMT | 2024-10-15 11:45:55 UTC |
| 6 | 230E263E9887E927E4156A | https://crt.sh/?id=14942787996 | Expired | Oct 11 16:36:27 2024 GMT | 2024-10-15 11:46:43 UTC |
| 7 | 230E2B7626186A33A542CA | https://crt.sh/?id=14942801784 | Expired | Oct 11 16:36:38 2024 GMT | 2024-10-15 11:47:25 UTC |
| 8 | 231728A4FE6EF5774A309D | https://crt.sh/?id=18823008375 | Revoked | Jun 3 08:05:31 2025 GMT | 2025-06-05 11:16:57 UTC |
| 9 | 231855188B1963F023FA93 | https://crt.sh/?id=19367848301 | Revoked | Jun 27 07:03:41 2025 GMT | 2025-07-01 07:25:08 UTC |
| 10 | 231BD77D6ECBF508262817 | https://crt.sh/?id=20947013362 | Revoked | Sep 9 08:49:38 2025 GMT | 2025-09-11 15:27:43 UTC |
| 11 | 231BD95C569CBA87EA1CFC | https://crt.sh/?id=20965162440 | Revoked | Sep 9 08:49:54 2025 GMT | 2025-09-12 09:42:45 UTC |
| 12 | 231D8B79D2A2961B1B1CAF | https://crt.sh/?id=21443305897 | Revoked | Oct 3 12:14:26 2025 GMT | 2025-10-06 06:09:45 UTC |
| 13 | 231D8C444B359E71878847 | https://crt.sh/?id=21446809901 | Revoked | Oct 3 15:24:22 2025 GMT | 2025-10-06 06:10:26 UTC |
| 14 | 23209430142FEBF90DE536 | https://crt.sh/?id=23131354166 | Revoked | Dec 10 09:53:26 2025 GMT | 2025-12-15 10:12:30 UTC |
|
|
||
Updated•1 month ago
|
Whiteboard: [ca-compliance] [__-misissuance] → [ca-compliance] [ov-misissuance]
|
|
Assignee | |
Comment 5•24 days ago
|
||
certSIGN completed in February 2026 the linter tests for SCT delay validations, as planned.
Please note that following the CCADB Incident Reporting Guidelines updates should be provided weekly, unless a 'next update' date has been set in advance. That is a separate incident to be raised.
When are reports updated?
CA Owners SHOULD respond promptly to comments and questions, and MUST respond within 7 days, even if only to acknowledge the request and provide a timeline for a full response.
If you believe this incident is resolved please submit a closure report.
|
|
Assignee | |
Comment 7•11 days ago
|
||
Report Closure Summary
- Incident description:
certSIGN issued 14 subscriber certificates where the certificate notBefore value was more that 48 hours earlier than the effective signing operation as interpreted form the last embedded SCT timestamp. Affected certificates failed to meet the Baseline Requirements for the Issuance and Management of Publicly-Trusted TLS Server Certificates - 7.1.2.7 Subscriber (Server) Certificate Profile – notBefore: a value within 48 hours of the certificate signing operation. - Incident Root Cause(s):
#1: Lack of a limit for logging to CT logs.
In certSIGN logic for logging precertificates to CT logs, on any failure, the CT logging retry mechanism lacked an upper time boundary aligned with the 48 hour requirement defined by BR 7.1.2.7; as a result, in rare cases, SCT occurs outside the permitted interval.
#2: Lack of delay check in Linter.
certSIGN linter, that is calling a set of other external linters also, like zlint, did not included the controls of the delay between the Precertificate issuance and the timestamp of the CT logs. - Remediation description:
The affected certificates had been revoked. certSIGN linter was updated for SCT delay validation, was tested and deployed. - Commitment summary:
certSIGN is continuously monitoring the requirements from all CA/Browser Form members and is committed to continue to follow the rules and recommendation of the Forum.
All Action Items disclosed in this report have been completed as described, and we request its closure.
|
|
Assignee | |
Comment 8•11 days ago
|
||
Report Closure Summary
- Incident description:
certSIGN issued 14 subscriber certificates where the certificate notBefore value was more that 48 hours earlier than the effective signing operation as interpreted form the last embedded SCT timestamp. Affected certificates failed to meet the Baseline Requirements for the Issuance and Management of Publicly-Trusted TLS Server Certificates - 7.1.2.7 Subscriber (Server) Certificate Profile – notBefore: a value within 48 hours of the certificate signing operation. - Incident Root Cause(s):
#1: Lack of a limit for logging to CT logs.
In certSIGN logic for logging precertificates to CT logs, on any failure, the CT logging retry mechanism lacked an upper time boundary aligned with the 48 hour requirement defined by BR 7.1.2.7; as a result, in rare cases, SCT occurs outside the permitted interval.
#2: Lack of delay check in Linter.
certSIGN linter, that is calling a set of other external linters also, like zlint, did not included the controls of the delay between the Precertificate issuance and the timestamp of the CT logs. - Remediation description:
The affected certificates had been revoked. certSIGN linter was updated for SCT delay validation, was tested and deployed. - Commitment summary:
certSIGN is continuously monitoring the requirements from all CA/Browser Form members and is committed to continue to follow the rules and recommendation of the Forum.
All Action Items disclosed in this report have been completed as described, and we request its closure.
|
|
Assignee | |
Updated•11 days ago
|
Flags: needinfo?(incident-reporting)
|
|
||
Comment 9•10 days ago
|
||
This is a final call for comments or questions on this Incident Report.
Otherwise, it will be closed on approximately 2026-03-30.
Whiteboard: [ca-compliance] [ov-misissuance] → [close on 2026-03-30] [ca-compliance] [ov-misissuance]
|
|
||
Updated•3 days ago
|
Status: ASSIGNED → RESOLVED
Closed: 3 days ago
Flags: needinfo?(incident-reporting)
Resolution: --- → FIXED
Whiteboard: [close on 2026-03-30] [ca-compliance] [ov-misissuance] → [ca-compliance] [ov-misissuance]
You need to log in
before you can comment on or make changes to this bug.
Description
•