Open Bug 1925293 Opened 15 days ago Updated 2 days ago

Firmaprofesional: Incorrect publication of information for "Test Website - Revoked" URL in the CCADB.

Categories

(CA Program :: CA Certificate Compliance, task)

Tracking

(Not tracked)

ASSIGNED

People

(Reporter: ext-antoni.camon, Assigned: ext-antoni.camon)

Details

(Whiteboard: [ca-compliance] [policy-failure])

Incident Report

Summary

The 'Test Website - Revoked' URL [https://testrevokedsslev.firmaprofesional.com] appeared as expired. The cause was that the certificate configured for this URL was the same certificate that had been issued and already configured for the 'Test Website - Expired' [https://testexpiredsslev.firmaprofesional.com] value in the CCADB.

Impact

The misconfigured URL https://testrevokedsslev.firmaprofesional.com due to the wrong certificate being applied.

No certificate issuance had to be stopped.

Timeline

All times are UTC.

2024-03-25

  • 15:35 - 15:39: Firmaprofesional's test certificates are issued.

2024-03-26

  • 14:35: 1Password's certificate synchronization script is deployed on the server.
  • 14:36: Firmaprofesional's test certificates are configured in Apache Virtual Hosts.

2024-10-14

  • 13:31: Firmaprofesional compliance account receives an email from Google's Chris Clements reporting the incident.
  • 14:00: The PKI team is made aware of the issue.
  • 14:05: The misconfiguration is found and manually fixed.

2024-10-15

  • 15:15: Simplified Apache configuration deployed to production.

Root Cause Analysis

Firmaprofesional uses test URLs for each of the four types of SSL certificates issued under their hierarchies. Each type has three states (valid, expired, and revoked), resulting in a total of 12 URLs that need to be configured.

The error occurred due to a misconfiguration in the deployment script, which mistakenly assigned the certificate for the URL testexpiredsslev.firmaprofesional.com to the URL testrevokedsslev.firmaprofesional.com. This happened because the script had hardcoded file names, and it did not properly distinguish between the expired and revoked certificates during deployment.

Proactive monitoring of the test URLs in the CCADB was not in place, allowing this error to go unnoticed until it was externally reported.

Contributing Factors:

  1. Misconfiguration in the deployment script: The script used hardcoded references to certificate files, which led to the wrong certificate being deployed for the testrevokedsslev.firmaprofesional.com URL.
  2. Complexity of the overall configuration: The configuration involved multiple files and certificates, making the deployment process prone to errors.
  3. Lack of proactive monitoring: Automated verification of the test URL outcomes was missing, which could have detected the error earlier.

Lessons Learned

What went well

  • Rapid response to the issue: Once the misconfiguration was reported, the PKI team responded quickly, identifying and fixing the issue within a short timeframe.
  • Efficient communication: The Firmaprofesional compliance account received the external report promptly, and the issue was quickly escalated to the PKI team for resolution.

What didn't go well

  • Overly complex configuration: The Apache configuration and the deployment process were too complex, which led to the error in certificate deployment. We have now simplified the configuration in order reduce the chances of human error.
  • Hardcoded certificate references in deployment scripts: The deployment scripts contained hardcoded file names, which increased the likelihood of deploying the wrong certificate, as was the case in this incident. The scripts should be dynamic and verify which certificate is being deployed.
  • Lack of proactive monitoring: There was no automated system in place to verify that the correct test URLs and certificates were being deployed, which delayed detection of the misconfiguration. Implementing proactive monitoring could have caught the issue earlier.

Where we got lucky

  • Early detection by external security researcher: The issue was reported by an external source. However, relying on external detection is not a sustainable solution, and internal monitoring should be prioritized.

Action Items

Action Item Kind Due Date
Update deployment scripts to dynamically verify and deploy the correct certificate based on the domain, removing hardcoded references. Prevent 2024-10-31
Implement proactive monitoring to automatically verify the correct certificates and expected outcomes for all test URLs. Prevent 2024-10-31
Deploy ACME for all test URLs. Prevent 2025-01-31

Appendix

Details of affected certificates

https://crt.sh/?sha256=4654AF7E46CCB5B713FA1DC11740903114C22970840BB32BD1028CEB1D355042 testexpiredsslev.firmaprofesional.com
https://crt.sh/?sha256=A7508F80016C5EB1DEE358D1B815033426D155E6C37F3FFFAAB48950223C2000 testrevokedsslev.firmaprofesional.com

Assignee: nobody → ext-antoni.camon
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true
Whiteboard: [ca-compliance] [policy-failure]

We confirm that the first two action items are in place:

  • Our deployment scripts now dynamically verify and deploy the correct certificate based on the domain, with hardcoded references removed.
  • We have set up proactive monitoring to automatically verify the correct certificates and outcomes for all test URLs.
You need to log in before you can comment on or make changes to this bug.