Firmaprofesional: Incorrect publication of information for "Test Website - Revoked" URL in the CCADB.
Categories
(CA Program :: CA Certificate Compliance, task)
Tracking
(Not tracked)
People
(Reporter: ext-antoni.camon, Assigned: ext-antoni.camon)
Details
(Whiteboard: [ca-compliance] [policy-failure])
Incident Report
Summary
The 'Test Website - Revoked' URL [https://testrevokedsslev.firmaprofesional.com] appeared as expired. The cause was that the certificate configured for this URL was the same certificate that had been issued and already configured for the 'Test Website - Expired' [https://testexpiredsslev.firmaprofesional.com] value in the CCADB.
Impact
The misconfigured URL https://testrevokedsslev.firmaprofesional.com due to the wrong certificate being applied.
- Incorrect certificate: https://crt.sh/?sha256=4654AF7E46CCB5B713FA1DC11740903114C22970840BB32BD1028CEB1D355042
- Correct certificate: https://crt.sh/?sha256=A7508F80016C5EB1DEE358D1B815033426D155E6C37F3FFFAAB48950223C2000
No certificate issuance had to be stopped.
Timeline
All times are UTC.
2024-03-25
- 15:35 - 15:39: Firmaprofesional's test certificates are issued.
2024-03-26
- 14:35: 1Password's certificate synchronization script is deployed on the server.
- 14:36: Firmaprofesional's test certificates are configured in Apache Virtual Hosts.
2024-10-14
- 13:31: Firmaprofesional compliance account receives an email from Google's Chris Clements reporting the incident.
- 14:00: The PKI team is made aware of the issue.
- 14:05: The misconfiguration is found and manually fixed.
2024-10-15
- 15:15: Simplified Apache configuration deployed to production.
Root Cause Analysis
Firmaprofesional uses test URLs for each of the four types of SSL certificates issued under their hierarchies. Each type has three states (valid, expired, and revoked), resulting in a total of 12 URLs that need to be configured.
The error occurred due to a misconfiguration in the deployment script, which mistakenly assigned the certificate for the URL testexpiredsslev.firmaprofesional.com
to the URL testrevokedsslev.firmaprofesional.com
. This happened because the script had hardcoded file names, and it did not properly distinguish between the expired and revoked certificates during deployment.
Proactive monitoring of the test URLs in the CCADB was not in place, allowing this error to go unnoticed until it was externally reported.
Contributing Factors:
- Misconfiguration in the deployment script: The script used hardcoded references to certificate files, which led to the wrong certificate being deployed for the
testrevokedsslev.firmaprofesional.com
URL. - Complexity of the overall configuration: The configuration involved multiple files and certificates, making the deployment process prone to errors.
- Lack of proactive monitoring: Automated verification of the test URL outcomes was missing, which could have detected the error earlier.
Lessons Learned
What went well
- Rapid response to the issue: Once the misconfiguration was reported, the PKI team responded quickly, identifying and fixing the issue within a short timeframe.
- Efficient communication: The Firmaprofesional compliance account received the external report promptly, and the issue was quickly escalated to the PKI team for resolution.
What didn't go well
- Overly complex configuration: The Apache configuration and the deployment process were too complex, which led to the error in certificate deployment. We have now simplified the configuration in order reduce the chances of human error.
- Hardcoded certificate references in deployment scripts: The deployment scripts contained hardcoded file names, which increased the likelihood of deploying the wrong certificate, as was the case in this incident. The scripts should be dynamic and verify which certificate is being deployed.
- Lack of proactive monitoring: There was no automated system in place to verify that the correct test URLs and certificates were being deployed, which delayed detection of the misconfiguration. Implementing proactive monitoring could have caught the issue earlier.
Where we got lucky
- Early detection by external security researcher: The issue was reported by an external source. However, relying on external detection is not a sustainable solution, and internal monitoring should be prioritized.
Action Items
Action Item | Kind | Due Date |
---|---|---|
Update deployment scripts to dynamically verify and deploy the correct certificate based on the domain, removing hardcoded references. | Prevent | 2024-10-31 |
Implement proactive monitoring to automatically verify the correct certificates and expected outcomes for all test URLs. | Prevent | 2024-10-31 |
Deploy ACME for all test URLs. | Prevent | 2025-01-31 |
Appendix
Details of affected certificates
https://crt.sh/?sha256=4654AF7E46CCB5B713FA1DC11740903114C22970840BB32BD1028CEB1D355042 testexpiredsslev.firmaprofesional.com
https://crt.sh/?sha256=A7508F80016C5EB1DEE358D1B815033426D155E6C37F3FFFAAB48950223C2000 testrevokedsslev.firmaprofesional.com
Updated•15 days ago
|
Assignee | ||
Comment 1•2 days ago
|
||
We confirm that the first two action items are in place:
- Our deployment scripts now dynamically verify and deploy the correct certificate based on the domain, with hardcoded references removed.
- We have set up proactive monitoring to automatically verify the correct certificates and outcomes for all test URLs.
Description
•