Open Bug 2004654 Opened 2 months ago Updated 4 days ago

SECOM: Invalid stateOrProvinceName

Categories

(CA Program :: CA Certificate Compliance, task)

Product:
CA Program
Component:
CA Certificate Compliance
Type:
task

Tracking

(Not tracked)

Status:
ASSIGNED

People

(Reporter: cainfo, Assigned: cainfo)

Details

(Whiteboard: [close on 2026-02-12] [ca-compliance] [ov-misissuance])

Attachments

(3 files)

NII Open Domain CA - G7 RSA_140_certs.txt
12.17 KB, text/plain
Details
NII Open Domain CA - G7 ECC_27_certs.txt
2.35 KB, text/plain
Details
CERTS_20251210_FULL_Record.csv
70.11 KB, text/csv
Details
No description provided.

Preliminary Incident Report

Summary

  • Incident description:

    • TLS server certificates issued via ACME from the subordinate CA "NII Open Domain CA - G7 RSA" (https://crt.sh/?caid=195167) were found to violate the Baseline Requirements.
    • The affected scope is ACME‑issued TLS server certificates on or after 2025-10-31.
    • In affected end‑entity certificates, the Subject’s stateOrProvinceName should contain only the state or province (prefecture) name; however, values like "ST=Tokyo, C=JP" were present, where ", C=JP" is not expected within that attribute.
    • A linting tool was in use, but it did not detect this issue.
    • We confirmed that pkimetal v1.31 (pkilint v0.12.13, zlint v3.6.8) did not detect this issue.
    • ACME issuance was stopped at 2025-12-08.

  • Relevant policies:

    Baseline Requirements for the Issuance and Management of Publicly‑Trusted TLS Server Certificates

  • Source of incident disclosure:

    2025-12-08 06:00 UTC: We discovered the issue during a customer meeting.

Assignee: nobody → cainfo
Status: UNCONFIRMED → ASSIGNED
Type: defect → task
Ever confirmed: true
Whiteboard: [ca-compliance] [ov-misissuance]

Incident Report Update (Interim Progress)

Summary

In the Preliminary Incident Report, we stated that a Baseline Requirements violation occurred in TLS server certificates issued by "NII Open Domain CA - G7 RSA" (https://crt.sh/?caid=195167).
Upon further investigation, we have determined that Baseline Requirements violations occurred in certificates issued by both "NII Open Domain CA - G7 RSA" (https://crt.sh/?caid=195167) and "NII Open Domain CA - G7 ECC" (https://crt.sh/?caid=195168).
These two subordinate CAs use a common application for certificate issuance.

Impact

Actions Taken

  • As the incident was found to affect OV TLS server certificates issued via ACME from both CAs, ACME issuance was suspended for both CAs at 2025-12-08 09:00 UTC.
  • On 2025-12-09 09:00 UTC, the application was updated to ensure that stateOrProvinceName no longer included ", C=JP", and TLS server certificate issuance was resumed.
  • By 2025-12-10 09:20 UTC, all affected certificates had been revoked.

Next Steps

  • A Full Incident Report is being prepared.

Full Incident Report

Summary

  • CA Owner CCADB unique ID:
    SECOM Trust Systems: A000045

  • Incident description:
    TLS server certificates issued via ACME from the subordinate CAs “NII Open Domain CA – G7 RSA” (https://crt.sh/?caid=195167) and “NII Open Domain CA – G7 ECC” (https://crt.sh/?caid=195168) were found to violate the Baseline Requirements.
    Affected certificates included malformed stateOrProvinceName values such as "ST=Tokyo, C=JP", where ", C=JP" must not appear within that attribute.
    ACME issuance was suspended on 2025-12-08.
    The latest linting tools (pkimetal v1.32.0, pkilint v0.13.0, zlint v3.6.8) are unable to detect the issue.

  • Timeline summary:

    • Non‑compliance start date: 2025‑10‑17 05:51 UTC
    • Non‑compliance identified date: 2025‑12‑08
    • Non‑compliance end date: 2025‑12‑08 06:57 UTC

  • Relevant policies:
    Baseline Requirements §7.1.2.7.4 — Organization Validated
    Requirement: If present, stateOrProvinceName MUST contain only the subject’s state or province.

  • Source of incident disclosure:
    Discovered during a customer meeting on 2025‑12‑08 at 06:00 UTC.

Impact

  • Total number of certificates:
    167

  • Total number of “remaining valid” certificates:
    0 (all revoked)

  • Affected certificate types:
    167 OV TLS server certificates

  • Incident heuristic:
    ACME‑issued certificates from "NII Open Domain CA – G7 RSA/ECC" between 2025‑10‑17 05:51 UTC and 2025‑12‑08 06:57 UTC.

  • Was issuance stopped in response to this incident, and why or why not?:
    Yes. ACME issuance was suspended after discovery to prevent further mis‑issuance.

  • Analysis:

    • In the none-automated issuance process, DN values excluding C=JP were sent to the common management system, which appended C=JP as part of its issuance configuration to construct the full DN.
    • The newly designed ACME system was implemented to include C=JP in the DN values before sending the request to the common management system, which resulted in C=JP being included twice and ultimately caused the mis-issuance.
    • DN construction logic differed between none-automated and ACME issuance, and no clear ACME environment build procedure existed.
    • Required production settings were missing.
    • Post‑issuance Subject checks were manually performed without a detailed profile checklist, allowing the anomaly to be missed.
    • Linting tools could not detect this semantic BR violation.
    • In total, 167 malformed certificates were issued and subsequently revoked.

  • Additional considerations:
    None. No additional impact types or different behaviors were observed beyond the scope described.

Timeline

  • Policy, process, or software changes leading to the root cause:
    2025‑10‑16 09:00 UTC — ACME issuance functionality deployed for "NII Open Domain CA – G7 RSA/ECC".

  • Incident began:
    2025‑10‑17 05:51:15 UTC — First non‑compliant certificate issued.

  • CA Owner became aware:
    2025‑12‑08 06:00 UTC — Issue discovered during customer meeting.

  • Scope and impact determined and disclosure initiated:
    2025‑12‑08 07:55 UTC — Subscriber notification began.

  • Issuance ceased and resumed:

    • 2025‑12‑08 09:00 UTC — ACME issuance suspended.
    • 2025‑12‑09 09:00 UTC — Application fixed to prevent malformed stateOrProvinceName; issuance resumed.

  • Expected revocation completion:
    2025‑12‑12 09:00 UTC

  • Actual revocation completion:
    2025‑12‑10 09:12 UTC — All affected certificates revoked.

Related Incidents

Bug ID Date Description
1978186 2025‑07‑19 SHECA: Incorrect stateOrProvinceName and streetAddress values in certificate DN
1718552 2021‑06‑28 IdenTrust: Certificates with invalid stateOrProvinceName values
1710243 2021‑05‑08 Sectigo: Invalid stateOrProvinceName
1668007 2020‑09‑29 GlobalSign: Invalid stateOrProvinceName value
1670894 2020‑10‑13 SwissSign: Invalid stateOrProvinceName field
1763173 2022‑04‑05 certSIGN: Incorrect stateOrProvinceName data

Root Cause Analysis

Contributing Factor #1 — Application defect

  • Description:

    • None-automated issuance workflows sent DN data without C=JP, leaving the common management system to append it.
    • ACME issuance sent DN data including C=JP; the common management system appended it again, creating malformed attributes.
    • The ACME release on 2025‑10‑16 was believed to be correct, but misconfiguration was present from initial deployment.

  • Timeline:
    Misconfiguration introduced at ACME release on 2025‑10‑16.

  • Detection:
    Detected during customer meeting.

  • Interaction with other factors:

    • Separate development teams for ACME and customer systems caused misunderstanding of production behavior.
    • Specification did not anticipate the conflicting DN construction logic.
    • Differences existed between test and production environments.

  • Root Cause Analysis methodology used:
    5 Whys.

Contributing Factor #2 — Failure to detect abnormal Subject fields

  • Description:

    • The common management system failed to block malformed Subject data.
    • None-automated checks lacked detailed profile validation, leading to oversight.
    • The issue was discovered during a customer meeting. Neither pkilint nor zlint reported an error because these tools validate the syntactic correctness of Subject DN attributes but do not evaluate the semantic correctness of attribute values. In this case, the value "ST=Tokyo, C=JP" is syntactically valid as a DirectoryString and therefore passed all lint checks. If linting tools had included rules to detect invalid patterns inside attribute values—such as unexpected '=' or substrings resembling an additional RDN—this issue could potentially have been detected automatically.

  • Timeline:
    First malformed certificate issued on 2025‑10‑17.

  • Detection:
    Detected during customer meeting.

  • Interaction with other factors:

    • Integration of newly connected systems introduced unexpected behaviors.
    • Additional robustness is needed to prevent malformed Subject attributes.

  • Root Cause Analysis methodology used:
    5 Whys.

Lessons Learned

  • What went well:

    • Subscriber notification began promptly on 2025‑12‑08.
    • All revocations completed by 2025‑12‑10.
    • Quick identification and fix allowed reissuance before revocation for most subscribers.

  • What didn’t go well:

    • Some subscribers experienced revocation before completing reissuance.

  • Where we got lucky:

    • Only 0.7% of the 24,000 valid certificates under G7 RSA/ECC had transitioned to ACME, limiting impact.
    • A scheduled transition to a new subordinate CA on 2025‑12‑17 allowed remediation before migration.

  • Additional:
    None.

Action Items

Action Item Kind Corresponding Root Cause(s) Evaluation Criteria Due Date Status
#1 Production environment configuration fix Mitigate #1 ACME build checklist created and used to prevent misconfiguration 2025‑12‑26 Ongoing
#2 Automatic Subject field validation Prevent #2 Issuance blocked when duplicate/invalid Subject fields exist 2026‑01‑16 Ongoing
#3 Enhanced post‑issuance profile checks Detect #2 Joint review by operations and compliance teams 2025‑12‑26 Ongoing
#4 Notify linting tool authors Detect #2 Incident shared with lint developers 2025‑12‑26 Ongoing

Appendix

A CSV file listing all 167 affected certificates—including precertificate hash, certificate hash, Subject, Issuer, validity period, serial number, dNSNames, and revocation details—is included as required.

Weekly Update – 2025‑12‑24

This is a scheduled weekly update regarding the status of the Action Items disclosed in our Full Incident Report.
No changes to the incident scope, root causes, or impact have been identified since the previous update. Issuance remains stable, and no recurrence of the issue has been observed.

Action Items

Action Item Kind Corresponding Root Cause(s) Evaluation Criteria Due Date Status
#1 Production environment configuration fix Mitigate #1 ACME build checklist created and used to prevent misconfiguration 2025‑12‑26 Ongoing
#2 Automatic Subject field validation Prevent #2 Issuance blocked when duplicate/invalid Subject fields exist 2026‑01‑16 Ongoing
#3 Enhanced post‑issuance profile checks Detect #2 Joint review by operations and compliance teams 2025‑12‑26 Ongoing
#4 Notify linting tool authors Detect #2 Incident shared with lint developers 2025‑12‑26 Complete

Additional Explanation for Action Item #4

Action Item #4 (Notify linting tool authors) was completed on 2025‑12‑17, ahead of the planned due date.

The following actions were taken:

Information sharing with relevant linting tool authors has therefore been fully completed.

Next Steps

  • Action Items #1 and #3 remain on track for completion by 2025‑12‑26.
  • The next update will be provided according to the Whiteboard schedule or earlier if any material changes occur.

Weekly Update – 2025‑12‑31

This is a scheduled weekly update regarding the status of the Action Items disclosed in our Full Incident Report.
No changes to the incident scope, root causes, or impact have been identified since the previous update. Issuance remains stable, and no recurrence of the issue has been observed.

Action Items

Action Item Kind Corresponding Root Cause(s) Evaluation Criteria Due Date Status
#1 Production environment configuration fix Mitigate #1 ACME build checklist created and used to prevent misconfiguration 2025‑12‑26 Complete
#2 Automatic Subject field validation Prevent #2 Issuance blocked when duplicate/invalid Subject fields exist 2026‑01‑16 Ongoing
#3 Enhanced post‑issuance profile checks Detect #2 Joint review by operations and compliance teams 2025‑12‑26 Complete
#4 Notify linting tool authors Detect #2 Incident shared with lint developers 2025‑12‑26 Complete

Additional Explanation for Action Item #1 and #3

Action Items #1 (Production environment configuration fix) and #3 (Enhanced post‑issuance profile checks) were completed on 2025‑12‑26.

  • #1 Production environment configuration fix
    An ACME build checklist was created and is now being used to prevent misconfiguration.

  • #3 Enhanced post‑issuance profile checks
    The post‑issuance profile review process was updated so that operations and compliance teams jointly review certificate profiles, strengthening the overall verification process.

Next Steps

  • Action Item #2 remains on track for completion by 2026‑01‑16.
  • The next update will be provided according to the Whiteboard schedule or earlier if any material changes occur.

Weekly Update – 2026-01-07

Due to temporary resource availability constraints during this period, the completion of Action Item #2 (Automatic Subject field validation) has been rescheduled to 2026‑01‑30.

Action Items

Action Item Kind Corresponding Root Cause(s) Evaluation Criteria Due Date Status
#1 Production environment configuration fix Mitigate #1 ACME build checklist created and used to prevent misconfiguration 2025‑12‑26 Complete
#2 Automatic Subject field validation Prevent #2 Issuance blocked when duplicate/invalid Subject fields exist 2026‑01‑30 Ongoing
#3 Enhanced post‑issuance profile checks Detect #2 Joint review by operations and compliance teams 2025‑12‑26 Complete
#4 Notify linting tool authors Detect #2 Incident shared with lint developers 2025‑12‑26 Complete

Next Steps

We will complete Action Item #2 (Automatic Subject field validation) by 2026‑01‑30.

Weekly Update – 2026-01-14

During this week, we continued working on Action Item #2 (Automatic Subject field validation).
The completion date remains scheduled for 2026‑01‑30, and we are currently preparing test scenarios and validation steps in the staging environment.

Action Items

Action Item Kind Corresponding Root Cause(s) Evaluation Criteria Due Date Status
#1 Production environment configuration fix Mitigate #1 ACME build checklist created and used to prevent misconfiguration 2025‑12‑26 Complete
#2 Automatic Subject field validation Prevent #2 Issuance blocked when duplicate/invalid Subject fields exist 2026‑01‑30 Ongoing
#3 Enhanced post‑issuance profile checks Detect #2 Joint review by operations and compliance teams 2025‑12‑26 Complete
#4 Notify linting tool authors Detect #2 Incident shared with lint developers 2025‑12‑26 Complete

Next Steps

  • We will complete Action Item #2 (Automatic Subject field validation) by 2026‑01‑30.

Weekly Update – 2026-01-21

During this week, we continued progressing on Action Item #2 (Automatic Subject field validation).
Test scenario preparation and validation planning in the staging environment have been proceeding as expected, and no blockers have been identified to date.
Overall, the work remains on schedule toward the planned completion date of 2026‑01‑30.

Action Items

Action Item Kind Corresponding Root Cause(s) Evaluation Criteria Due Date Status
#1 Production environment configuration fix Mitigate #1 ACME build checklist created and used to prevent misconfiguration 2025‑12‑26 Complete
#2 Automatic Subject field validation Prevent #2 Issuance blocked when duplicate/invalid Subject fields exist 2026‑01‑30 Ongoing
#3 Enhanced post‑issuance profile checks Detect #2 Joint review by operations and compliance teams 2025‑12‑26 Complete
#4 Notify linting tool authors Detect #2 Incident shared with lint developers 2025‑12‑26 Complete

Next Steps

  • Complete Action Item #2 (Automatic Subject field validation) by 2026‑01‑30, as scheduled.

Weekly Update – 2026-01-28

During this week, we completed Action Item #2 (Automatic Subject field validation) on 2026‑01‑27T10:00Z (UTC).
All planned validation tasks in the staging environment were finished without any blockers, and the implementation has been verified as functioning as intended.

Action Items

Action Item Kind Corresponding Root Cause(s) Evaluation Criteria Due Date Status
#1 Production environment configuration fix Mitigate #1 ACME build checklist created and used to prevent misconfiguration 2025‑12‑26 Complete
#2 Automatic Subject field validation Prevent #2 Issuance blocked when duplicate/invalid Subject fields exist 2026‑01‑30 Complete
#3 Enhanced post‑issuance profile checks Detect #2 Joint review by operations and compliance teams 2025‑12‑26 Complete
#4 Notify linting tool authors Detect #2 Incident shared with lint developers 2025‑12‑26 Complete

Next Steps

  • Prepare the Closure Report for this incident.

Report Closure Summary

  • Incident description:
    TLS server certificates issued via ACME from the subordinate CAs “NII Open Domain CA – G7 RSA” and “NII Open Domain CA – G7 ECC” violated the Baseline Requirements due to malformed stateOrProvinceName values (e.g., values containing , C=JP within the attribute). ACME issuance was suspended upon discovery.

  • Incident Root Cause(s):

    1. Application defect and configuration gap in DN construction: the ACME path supplied C=JP while the common management system also appended C=JP, leading to malformed Subject attributes; differences between test and production and the lack of a clear ACME environment build procedure contributed.
    2. Insufficient Subject validation and post-issuance profile checks; available lint tools at the time did not detect the semantic error inside the attribute value.

  • Remediation description:

    • ACME issuance was suspended on 2025-12-08 (UTC), an application fix was deployed, and issuance resumed on 2025-12-09 (UTC).
    • All 167 affected certificates were revoked by 2025-12-10T09:12Z (UTC).
    • Action Items are now complete:
      #1 Production environment configuration fix (ACME build checklist) — Complete.
      #2 Automatic Subject field validation — Completed on 2026-01-27T10:00Z (UTC).
      #3 Enhanced post-issuance profile checks — Complete.
      #4 Notify linting tool authors — Complete.

  • Commitment summary:
    We will (a) maintain and enforce the ACME build checklist as the controlled procedure for all ACME deployments, (b) keep the automatic Subject field validator permanently enabled in staging and production and extend its rules as needed, (c) continue engagement with lint tool projects to contribute rules that detect malformed Subject attribute values, and (d) run periodic audits comparing ACME and manual issuance DN construction to ensure parity and prevent regressions.

All Action Items disclosed in this report have been completed as described, and we request its closure.

This is a final call for comments or questions on this Incident Report.

Otherwise, it will be closed on approximately 2026-02-12.

Whiteboard: [ca-compliance] [ov-misissuance] → [close on 2026-02-12] [ca-compliance] [ov-misissuance]
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: