SSL.com: Insufficient serial number entropy
Categories
(CA Program :: CA Certificate Compliance, task)
Tracking
(Not tracked)
People
(Reporter: wthayer, Assigned: me)
Details
(Whiteboard: [ca-compliance] [ca-misissuance] [ev-misissuance] [ov-misissuance] [dv-misissuance])
Attachments
(3 files)
Fotis Loukos posted the following incident report to the mozilla.dev.security.policy forum:
SSL.com has been following the recent discussions at
mozilla.dev.security.policy regarding the behavior of EJBCA based CAs in
the matter of serial number generation.
SSL.com is using EJBCA internally and is affected by the same issue.
After consulting with our auditors, we would like to post a preliminary
report of our findings.
How your CA first became aware of the problem (e.g. via a problem
report submitted to your Problem Reporting Mechanism, a discussion in
mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit),
and the time and date.
SSL.com has been following discussion of this issue in
mozilla.dev.security.policy and initiated a review on 05/03/2019.
A timeline of the actions your CA took in response. A timeline is a
date-and-time-stamped sequence of all relevant events. This may include
events before the incident was reported, such as when a particular
requirement became applicable, or a document changed, or a bug was
introduced, or an audit was done.
- 10/07/2016 - Ballot 164 on Certificate Serial Number Entropy is voted.
- 30/09/2016 - Ballot 164 enters into effect.
- 05/03/2019 - Initial review initiated.
- 05/03/2019 - Confirmed issue exists in SSL.com certificates.
- 05/03/2019 - Tested and deployed correction to production systems.
- 06/03/2019 - A full plan for the revocation of all certificates has
been initiated. We plan on revoking all certificates within 30 days.
Whether your CA has stopped, or has not yet stopped, issuing
certificates with the problem. A statement that you have will be
considered a pledge to the community; a statement that you have not
requires an explanation.
A patch was deployed as soon as we confirmed that the issue exists in
SSL.com certificates. Certificate issuance has been resumed with serials
meeting all requirements.
A summary of the problematic certificates. For each problem: number
of certs, and the date the first and last certs with that problem were
issued.
- 6931 End Entity TLS Certificates
We will post an update that will include all S/MIME and CA certificate
information.
The complete certificate data for the problematic certificates. The
recommended way to provide this is to ensure each certificate is logged
to CT and then list the fingerprints or crt.sh IDs, either in the report
or as an attached spreadsheet, with one list per distinct problem.
Please find attached the file tlseelist.txt containing the fingerprints
of all affected end entity TLS certificates. S/MIME and CA certificate
information will be posted during an update.
Explanation about how and why the mistakes were made or bugs
introduced, and how they avoided detection until now.
EJBCA's method of generating serial numbers has led to a discrepancy
between expected and actual behavior and output, such that any CA using
EJBCA with the default settings will encounter this issue (and be
therefore in violation of BR 7.1).
List of steps your CA is taking to resolve the situation and ensure
such issuance will not be repeated in the future, accompanied with a
timeline of when your CA expects to accomplish these things.
The number of bits of entropy used for the generation of serial numbers
has been increased to 127.
In addition to remediation related to this issue, we will initiate a
review of other technical requirements of CA/B Forum and how they are
implemented by EJBCA in order to ensure no more problematic practices
are followed.
SSL.com intends to exceed minimum technical requirements where ever
possible to guard against similar issues in the future and ensure the
highest possible level of security and compliance.
|
Assignee | |
Comment 1•6 years ago
|
||
SSL.com has been following the recent discussions at mozilla.dev.security.policy regarding the behavior of EJBCA based CAs in the matter of serial number generation.
SSL.com is using EJBCA internally and is affected by the same issue. After consulting with our auditors, we would like to post a preliminary
report of our findings.
How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.
SSL.com has been following discussion of this issue in mozilla.dev.security.policy and initiated a review on 2019/03/05.
A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.
- 2016/07/10 - Ballot 164 on Certificate Serial Number Entropy is voted.
- 2016/09/30 - Ballot 164 enters into effect.
- 2019/03/05 - Initial review initiated.
- 2019/03/05 - Confirmed issue exists in SSL.com certificates.
- 2019/03/05 - Tested and deployed correction to production systems.
- 2019/03/06 - A full plan for the revocation of all certificates has been initiated. We plan on revoking all certificates within 30 days.
- 2019/03/14 – A first database scan was conducted to estimate the number of affected certificates.
- 2019/03/14-18 – Simulation and testing of the mass renewal and revocation process for affected TLS Certificates took place.
- 2019/03/22 – A Full scan with an updated custom linting tool that scans for the serial number issue was conducted in order to find the exact number of valid, expired and revoked certificates.
- 2019/03/22 – Based on the scan results, Management approved the issuance of new CAs to replace older ones affected by the issue
- 2019/03/26 – A key ceremony was performed to issue replacement CA Certificates
- 2019/03/28 – Prepare notification messages to Subscribers affected by the incident explaining the steps of the plan.
- 2019/03/28 – Issuance of new end-entity certificates is resumed using the replacement CA Certificates that were generated in March 26th.
- 2019/03/28 – Bugzilla id 1534147 is updated with the timeline of the remediation plan.
The following steps have not been performed yet but are part of the approved remediation plan:
- 2019/03/29 – TLS Certificates affected by the incident that expire after April 6th 2019 will be renewed and a notification will be sent to Subscribers that the new certificates must be installed by April 6th, 2019 because their affected certificate(s) will be revoked on that date.
- 2019/03/30 – 06/04/2019: Monitoring the replacement of certificates to affected web sites and send periodic reminders to the Subscribers that didn’t replace their web site certificates
- 2019/04/05 – The affected CA Certificates technically capable of issuing a TLS Certificate will be revoked.
- 2019/04/06 – The affected TLS end-entity Certificates will be revoked.
Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.
A patch was deployed as soon as we confirmed that the issue exists in SSL.com certificates. Certificate issuance has been resumed with serials meeting all requirements.
A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.
- 5485 End Entity TLS Certificates
- 20 End Entity S/MIME Certificates
- 12 CA Certificates capable of issuing TLS Certificates
- 5 CA Certificates capable of issuing S/MIME Certificates
The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.
Please find attached a file containing the fingerprints of all affected end entity TLS certificates.
Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
EJBCA's method of generating serial numbers has led to a discrepancy between expected and actual behavior and output, such that any CA using EJBCA with the default settings will encounter this issue (and be therefore in violation of BR 7.1).
List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.
The number of bits of entropy used for the generation of serial numbers has been increased to 127.
In addition to remediation related to this issue, we will initiate a review of other technical requirements of CA/B Forum and how they are implemented by EJBCA in order to ensure no more problematic practices are followed.
SSL.com intends to exceed minimum technical requirements where ever possible to guard against similar issues in the future and ensure the highest possible level of security and compliance.
|
|
Assignee | |
Comment 2•6 years ago
|
||
|
Reporter | |
Comment 3•6 years ago
|
||
Fotis: Thank you for the incident report. Please update this bug with the results of your "review of other technical requirements of CA/B Forum and how they are implemented by EJBCA" Also, please state if you plan to revoke the problematic certificates, and if so, by when.
|
|
Assignee | |
Comment 4•6 years ago
|
||
The review of the technical requirements of CA/B Forum and the EJBCA implementation has been initiated, but is not complete yet. The results will be published as soon as we finish it.
We plan on revoking all CA and end-entity TLS certificates that are affected by 2019/04/06 based on the timeline mentioned at the last update. We agree with Kathleen's interpretation of the requirements on S/MIME certificates as it is displayed at https://github.com/mozilla/pkipolicy/issues/176, and we will not revoke any certificates that are not used in the WebPKI and are not under the CA/B Forum Baseline Requirements. However, we have reissued all S/MIME issuing CAs that were affected and we have already started deprecating the old ones.
|
|
Assignee | |
Comment 5•6 years ago
|
||
SSL.com has been following the recent discussions at mozilla.dev.security.policy regarding the behavior of EJBCA based CAs in the matter of serial number generation.
SSL.com is using EJBCA internally and is affected by the same issue. After consulting with our auditors, we would like to post a preliminary report of our findings.
How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.
SSL.com has been following discussion of this issue in mozilla.dev.security.policy and initiated a review on 2019/03/05.
A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.
- 2016/07/10 - Ballot 164 on Certificate Serial Number Entropy is voted.
- 2016/09/30 - Ballot 164 enters into effect.
- 2019/03/05 - Initial review initiated.
- 2019/03/05 - Confirmed issue exists in SSL.com certificates.
- 2019/03/05 - Tested and deployed correction to production systems.
- 2019/03/06 - A full plan for the revocation of all certificates has been initiated. We plan on revoking all certificates within 30 days.
- 2019/03/14 – A first database scan was conducted to estimate the number of affected certificates.
- 2019/03/14-18 – Simulation and testing of the mass renewal and revocation process for affected TLS Certificates took place.
- 2019/03/22 – A Full scan with an updated custom linting tool that scans for the serial number issue was conducted in order to find the exact number of valid, expired and revoked certificates.
- 2019/03/22 – Based on the scan results, Management approved the issuance of new CAs to replace older ones affected by the issue
- 2019/03/26 – A key ceremony was performed to issue replacement CA Certificates
- 2019/03/28 – Prepare notification messages to Subscribers affected by the incident explaining the steps of the plan.
- 2019/03/28 – Issuance of new end-entity certificates is resumed using the replacement CA Certificates that were generated in March 26th.
- 2019/03/28 – Bugzilla id 1534147 is updated with the timeline of the remediation plan.
- 2019/03/29 – TLS Certificates affected by the incident that expire after April 6th 2019 were renewed and a notification was sent to Subscribers that the new certificates must be installed by April 6th, 2019 because their affected certificate(s) will be revoked on that date.
- 2019/03/30 – 06/04/2019: Monitoring the replacement of certificates to affected web sites and periodic reminders sent to the Subscribers that didn't replace their web site certificates
- 2019/04/05 – Affected CA Certificates technically capable of issuing a TLS Certificate revoked.
- 2019/04/06 – Affected TLS end-entity Certificates revoked
- 2019/04/10 – The final incident report was approved by Management
Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.
A patch was deployed as soon as we confirmed that the issue exists in SSL.com certificates. Certificate issuance has been resumed with serials meeting all requirements.
A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.
- 5485 End Entity TLS Certificates
- 20 End Entity S/MIME Certificates
- 12 CA Certificates capable of issuing TLS Certificates
- 5 CA Certificates capable of issuing S/MIME Certificates
Here is the list of CA Certificates that were affected by this incident:
- https://crt.sh/?id=164243746
- https://crt.sh/?id=1044441049
- https://crt.sh/?id=1044438805
- https://crt.sh/?id=1044438815
- https://crt.sh/?id=1220120852
- https://crt.sh/?id=1220120782
- https://crt.sh/?id=1220120858
- https://crt.sh/?id=1243471012
- https://crt.sh/?id=1243471036
- https://crt.sh/?id=1243470987
- https://crt.sh/?id=1143349807
- https://crt.sh/?id=1138282622
- https://crt.sh/?id=1143349721
- https://crt.sh/?id=1143455464
- https://crt.sh/?id=1143456852
- https://crt.sh/?id=1143349701
- https://crt.sh/?id=1138282621
All of the afforementioned certificates were revoked within 30 days.
The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.
Please find attached the files containing the fingerprints of all affected end entity certificates per EKU.
Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
EJBCA's method of generating serial numbers has led to a discrepancy between expected and actual behavior and output, such that any CA using EJBCA with the default settings will encounter this issue (and be therefore in violation of BR 7.1).
The large number of affected TLS certificates (~5500) issued to a great number of different Subscribers with heterogenous systems, made SSL.com extend the 5-day revocation requirement for TLS Certificates to 30 days, acknowledging that this was yet another violation of the Baseline Requirements. SSL.com did not extend the revocation requirement lightly but considered the incident circumstances, especially the ones related to the security implications to Subscribers and Relying Parties. This incident had a significant impact mainly on Subscribers with TLS Certificates, because they had to replace these Certificates in a short timeframe. No security threats were detected and no certificate was compromised or suspected to be compromised. Here are some supporting arguments from the WebPKI community related to this incident:
- https://groups.google.com/d/msg/mozilla.dev.security.policy/7WuWS_20758/Tz5EGv1UCAAJ
- https://www.feistyduck.com/bulletproof-tls-newsletter/issue_51_trouble_with_a_missing_random_bit_in_serial_numbers
List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.
The number of bits of entropy used for the generation of serial numbers has been increased to 127.
In addition to remediation related to this issue, we have made arrangements with Primekey and we are moving forward with checking all BR technical requirements and their implementation in EJBCA. Our plan is to create an initial set of technical requirements as they are verified by linters, and add any missing requirements. The implementation will be verified with the help of Primekey. Our findings will be made available to all parties who express their interest.
Furthermore, in order to be reassured that we will keep up with the CA/B Forum's updated requirements and to contribute to the CA community, we have pushed upstream our EJBCA validator patch which allows linting of certificates before any issuance (a function not supported until now), and its correct operation has been verified by Primekey's technical team. This SSL.com provided validator will be included in the EJBCA 7.1 release.
SSL.com intends to exceed minimum technical requirements where ever possible to guard against similar issues in the future and ensure the highest possible level of security and compliance.
We also plan on deploying and encouraging use of automation installation tools in 2019.
We already offer a REST API, and we are going to create training material for all of our users.
In addition, we have plans of implementing an ACME server in order to move to a more standardized model of automation.
|
|
Assignee | |
Comment 6•6 years ago
|
||
|
|
Reporter | |
Comment 7•6 years ago
|
||
Fotis: thank you for confirming that all TLS certificates have been revoked, and for the thorough remediation plan that goes beyond addressing the immediate issue, including your efforts to work with Primekey to improve EJBCA and plans to support automated provisioning.
|
||
Updated•2 years ago
|
|
||
Updated•2 years ago
|
Description
•