Closed Bug 2057433 Opened 3 months ago Closed 26 days ago

SHECA: Disclosure alerts indicating audit information missing for 2 Sub-CAs

Categories

(CA Program :: CA Certificate Compliance, task)

Product:
CA Program
Component:
CA Certificate Compliance
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: wangjiatai, Assigned: wangjiatai)

Details

(Whiteboard: [ca-compliance] [disclosure-failure])

Preliminary Incident Report

Summary

  • Incident description:

SHECA received a third-party email regarding the disclosure alerts on the Mozilla disclosures page on crt.sh. The alert indicates audit information was missing for 2 Sub-CAs. The alert was triggered since SHECA failed to tick the checkbox "audit same as parent" for 2 newly‑added Sub-CAs on CCADB. SHECA has fixed this problem. 

  • Relevant policies:

CCADB Policy 5. Audit Disclosures

The entry for each subordinate CA certificate has checkboxes to indicate "Audits Same as Parent". When the checkbox is checked, the details do not need to be duplicatedfrom the parent certificate.

  • Source of incident disclosure:

Third Party Reported: SHECA received a third-party email regarding the disclosure alerts on 2026-07-14 at 08:02 UTC+8.

Assignee: nobody → wangjiatai
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true
Whiteboard: [ca-compliance] [disclosure-failure]

SHECA will provide the full incident report soon.

Full Incident Report

Summary

  • CA Owner CCADB unique ID:
    A000261

  • Incident description:

    SHECA received a third-party email regarding the disclosure alerts on the Mozilla disclosures page on crt.sh. The alert indicates audit information was missing for 2 Sub-CAs.The alert was triggered since SHECA failed to tick the checkbox "audit same as parent" for 2 newly‑added Sub-CAs on CCADB. After ticking the checkboxes, the alert was cleared.

  • Timeline summary:

    • Non-compliance start date:
      2026-02-08 10:00 UTC+8 (Sub-CAs uploaded to CCADB)
    • Non-compliance identified date:
      2026-07-14 08:02 UTC+8 (email received)
    • Non-compliance end date:
      2026-07-14 09:00 UTC+8 (alert cleared)

  • Relevant policies:

    • CCADB Policy 5. Audit Disclosures

      The entry for each subordinate CA certificate has checkboxes to indicate "Audits Same as Parent". When the checkbox is checked, the details do not need to be duplicatedfrom the parent certificate.

  • Source of incident disclosure:

    Third Party Reported: SHECA received a third-party email regarding the disclosure alerts on 2026-07-14.

Impact

  • Total number of certificates:

    N/A. This incident only involves non-compliance with Sub-CA certificate disclosure; it does not cover violations related to certificate issuance or revocation. No certificates were affected.

  • Total number of "remaining valid" certificates:

    N/A. No certificates were impacted.

  • Affected certificate types:

    N/A. No certificates were impacted.

  • Incident heuristic:

    N/A. No certificates were impacted.

  • Was issuance stopped in response to this incident, and why or why not?:
    No. This non-compliance relates to Sub-CA certificate disclosure. It does not affect certificate issuance workflows, validation criteria, or certificate security.

  • Analysis:
    N/A

  • Additional considerations:
    SHECA determined that the delayed publication of an incident report constituted a separate non-compliance with the applicable incident-reporting requirements. That issue is being tracked separately in Bug 2057439.

Timeline

  • 2026-02-08 10:00 UTC+8: SHECA added two newly issued subordinate CA certificates to CCADB but did not select the applicable “Audits Same as Parent” checkboxes.
  • 2026-07-14 08:02 UTC+8: SHECA received an email from a third party notifying SHECA of the alerts shown on the Mozilla Disclosures page on crt.sh.
  • 2026-07-14 09:00 UTC+8: SHECA completed its initial investigation and identified that the applicable “Audits Same as Parent” checkboxes had not been selected in CCADB. SHECA selected the checkboxes, and the alerts were cleared.
  • 2026-07-14 09:23 UTC+8: SHECA replied to the third party and confirmed that the disclosure alerts had been resolved.
  • 2026-07-24 07:52 UTC+8: SHECA received a follow-up email from the third party, suggesting the issue deserves a Bugzilla ticket to ensure sustained improvements are made.
  • 2026-07-24 14:00 UTC+8: SHECA determined that the incident should be formally reported to support transparency and sustained process improvement. SHECA also recognized that more than 72 hours had elapsed between initial discovery and Bugzilla reporting, which constitutes non-compliance with CCADB Incident Reporting Guidelines. A separate ticket should be opened for this violation.
  • 2026-07-24 16:27 UTC+8: Preliminary Incident Report published for the disclosure alert issue.
  • 2026-07-24 16:50 UTC+8: Preliminary Incident Report published for the delayed reporting (Bug 2057439).

Related Incidents

Bug Description
2009542 Microsoft PKI Services: Improper Disclosure of CRLs – IDP – New CAs
2052377 Apple: Audit Documents missing 2 Sub-CAs

Root Cause Analysis

Contributing Factor #1: Lack of automated monitoring for CCADB disclosure information

  • Description:

    SHECA's CCADB information review process adopted a manual dual-operation mode, with one staff responsible for uploading certificate information and another for manual verification. However, reliance on manual inspection poses inherent human error risks.In this incident, the omission of the “Audits Same as Parent” checkboxes was not identified during the manual review.

    As an immediate improvement, SHECA added crt.sh | mozilla-disclosures to its daily compliance-monitoring checklist on 2026-07-14. SHECA is also developing an automated monitoring function to detect new alerts on this page.

  • Timeline:

    This deficiency has existed historically.

  • Detection:

    During the investigation on 2026-07-14, SHECA found that the alerts were triggered by the applicable “Audits Same as Parent” checkboxes not being selected. SHECA then detected that its existing review process for CCADB disclosure relied on manual checks.

  • Interaction with other factors:

    No other factors contributed to this incident. The sole contributing factor is the long-term reliance on manual verification and the lack of automated monitoring.

  • Root Cause Analysis methodology used:
    5-Whys

    Why 1: Why was the alert triggered? The checkbox "audit same as parent" was not ticked.

    Why 2: Why was the checkbox not ticked? The mistake was made during upload and was not detected.

    Why 3: Why was the mistake not detected? The verification mechanism failed.

    Why 4: Why did the verification mechanism failed? It depended on manual work.

    Why 5: Why did it depended on manual work? There was no automated monitoring.

Lessons Learned

  • What went well:

    SHECA identified the cause and corrected the disclosure records within one hour of receiving the third-party notification.

    The remediation was effective: after the applicable checkboxes were selected, the alerts were cleared.

  • What didn’t go well:

    SHECA did not publish an incident report within the required reporting timeframe. This delayed-reporting issue is being tracked separately in Bug 2057439. The related action items are documented in that ticket and are not duplicated in this report.

  • Where we got lucky:

    The issue was limited to the completeness of the public CCADB disclosure records. The applicable audit coverage existed through the parent CA, and the incident did not affect certificate issuance, validation, revocation, CA key security, or the validity of issued certificates.

  • Additional:
    A dual-review process based on manual inspection is not sufficient by itself to ensure that all CCADB disclosure fields are complete. Public disclosure status should also be independently verified after each CCADB submission or update, supported by automated alert monitoring where possible.

Action Items

Action Item Kind Corresponding Root Cause(s) Evaluation Criteria Due Date Status
add crt.sh | mozilla-disclosures to daily monitoring list prevent Factor #1 the page was added to the daily monitoring list 2026-07-14 Completed
implement automated monitoring for alerts on crt.sh | mozilla-disclosures prevent Factor #1 the function is implemented 2026-08-31 Ongoing

Appendix

N/A

The action items are in progress. We request to set the next update date to August 31.

Flags: needinfo?(incident-reporting)
Flags: needinfo?(incident-reporting)
Whiteboard: [ca-compliance] [disclosure-failure] → [ca-compliance] [disclosure-failure] Next update 2026-08-31

(In reply to SHECA from comment #3)

The action items are in progress. We request to set the next update date to August 31.

Update Action Items

Action Item Kind Corresponding Root Cause(s) Evaluation Criteria Due Date Status
add crt.sh | mozilla-disclosures to daily monitoring list prevent Factor #1 the page was added to the daily monitoring list 2026-07-14 Completed
implement automated monitoring for alerts on crt.sh | mozilla-disclosures prevent Factor #1 the function is implemented 2026-08-31 postponed to 2026-09-15 Ongoing

Hi,
SHECA is merging and refactoring internal code for Bugzilla, CCADB and other monitoring systems to unify scattered modules and enhance native compliance alert features. To mitigate risks caused by rushed release, the overall schedule is postponed to 15 September. Manual monitoring of crt.sh disclosures will remain in place until automation implementation.

Whiteboard: [ca-compliance] [disclosure-failure] Next update 2026-08-31 → [ca-compliance] [disclosure-failure]

No content to update, monitoring in progress！

No content to update, monitoring in progress！

Update Action Items

Action Item Kind Corresponding Root Cause(s) Evaluation Criteria Due Date Status
add crt.sh | mozilla-disclosures to daily monitoring list prevent Factor #1 the page was added to the daily monitoring list 2026-07-14 Completed
implement automated monitoring for alerts on crt.sh | mozilla-disclosures prevent Factor #1 the function is implemented 2026-08-31 postponed to 2026-09-15 Completed

More details related to Action Item 2 (automated monitoring for crt.sh Mozilla Disclosures)

crt.sh Mozilla Disclosures Monitoring
Monitors the crt.sh Mozilla Disclosures page for certificate disclosures matching configured Root Owner keywords.

Data Source: https://crt.sh/mozilla-disclosures (parsed via Jsoup)
Table Parsing: Only matches summary tables where "Root Owner" is the first column header (skips detail tables)
Max Tables: Parses the first 3 disclosure tables on the page
Keyword Matching: Case-insensitive contains match against Root Owner names only
Schedule: Every 3 hours
Retry: Exponential backoff (30s → 60s → 120s) on HTTP 5xx / 429 errors, up to 3 attempts
Alert Content: All matched items grouped by disclosure category, each with owner name and matched keyword
Manual Trigger: GET /api/bugzilla/disclosure

This feature is integrated into CA Sentinel. CA Sentinel is a Spring Boot-based CA certificate monitoring and alerting service. It monitors Mozilla CA Program Bugzilla cases, crt.sh Mozilla Disclosures, and incoming incident emails — sending bilingual (Chinese/English) alerts through multiple channels with CCADB-compliant timer reminders.
Key Features of CA Sentinel:

  1. Bugzilla Case Monitoring
  2. crt.sh Mozilla Disclosures Monitoring
  3. Incident Event Mail Monitor
  4. Incident Event Timer Reminder

For the open-source repository link and detailed parameter documentation, please refer to: https://github.com/SHECA-Alvin/ca-sentinel

All action items are completed. SHECA will send the closure report next week.

Report Closure Summary

  • Incident description:

    A crt.sh alert was triggered because SHECA failed to tick the "audit same as parent" checkbox for two newly added Sub-CAs in CCADB. After checking the boxes, the alert was cleared.

  • Incident Root Cause(s):

    The incident stemmed from the absence of automated monitoring for CCADB disclosure information. SHECA’s CCADB information review process used a manual dual-control workflow: one team member uploaded certificate data while another performed manual verification. Nevertheless, manual reviews inherently carry risks of human error. In this case, the missing selection of the “Audits Same as Parent” checkboxes was not caught during the manual review.

  • Remediation description:

    SHECA added https://crt.sh/mozilla-disclosures to its daily compliance monitoring checklist and enabled automated monitoring for alerts from https://crt.sh/mozilla-disclosures. This capability is integrated within CA Sentinel (a Spring Boot-based CA certificate monitoring and alerting service).

  • Commitment summary:

    SHECA commits to continuously improving the timeliness and accuracy of disclosure information, refining compliance-related automated monitoring capabilities, and replacing manual checks with automated tools to the greatest extent possible.

All Action Items disclosed in this report have been completed as described, and we request its closure.

Flags: needinfo?(incident-reporting)

This is a final call for comments or questions on this Incident Report.

Otherwise, it will be closed on approximately 2026-10-01.

Flags: needinfo?(incident-reporting)
Whiteboard: [ca-compliance] [disclosure-failure] → [close on 2026-10-01] [ca-compliance] [disclosure-failure]

No update this week.

Status: ASSIGNED → RESOLVED
Closed: 26 days ago
Resolution: --- → FIXED
Whiteboard: [close on 2026-10-01] [ca-compliance] [disclosure-failure] → [ca-compliance] [disclosure-failure]
You need to log in before you can comment on or make changes to this bug.