GDCA: Issuance of SSL/TLS certificates with Non-critical Basic Constraints
Categories
(CA Program :: CA Certificate Compliance, task)
Tracking
(Not tracked)
People
(Reporter: capoc, Assigned: capoc)
Details
(Whiteboard: [ca-compliance] [ov-misissuance])
Attachments
(2 files)
1. Current Bugzilla Incidents Monitoring and Response Steps.jpg
User Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.5735.289 Safari/537.36
Actual results:
This is a preliminary report acknowledging that GDCA has issued a number of SSL/TLS certificates between 15 September and 8 October 2023, with the Basic Constraints extension included by not setting as Critical.
We will provide more details within the next few days.
|
||
Updated•1 year ago
|
|
|
||
Updated•1 year ago
|
Incident Report
Issuance of SSL/TLS certificates with Non-critical Basic Constraints.
Summary
GDCA has issued 20 SSL/TLS certificates from 15 September to 8 October 2023, with the Basic Constraints extension included but not set as Critical.
Impact
A total of 20 SSL/TLS certificates were affected by this incident, including 12 OV certificates, 7 DV certificates, and 1 EV certificate. As of 18:00 1 April 2024 (UTC+8), 6 certificates have expired, 5 revoked, and the rest are within validity.
Timeline
All times are UTC+8.
2023-3-14:
-Ballot SC 62 V2 passed.
2023-4-22:
-Ballot SC 62 V2 was adopted as part of Baseline Requirements v2.0.0, effective as of 15 September 2023.
2023-5-8:
-We completed the review and analysis of Ballot SC 62 V2 and Baseline Requirements v2.0.0, an updated version of our CP/CPS was published, with revisions made to section 7 regarding certificate profiles.
2023-9-15:
-First problematic certificate issued.
2023-10-8:
-Last problematic certificate issued.
2024-3-26:
-23:54: A certificate problem report was received by our CPR mechanism from a concerned third party, indicating that we have mis-issued a number of SSL/TLS certificates with Basic Constraints not set as critical and should be treated as an incident .
2024-3-27:
-14:15: We confirmed this issue and published a preliminary incident report on Bugzilla.
2024-3-27:
-We began to contact the impacted customers to communicate about certificate revocation and replacement.
Root Cause Analysis
When determining whether or not the issuance of a certificate complies with the Baseline Requirements, such as requirements regarding certificate profiles, we primarily consider two aspects. Firstly we assess whether the issuance of a certificate constitutes a mis-issuance incident based on the requirement level keywords of a particular requirement in the Baseline Requirements, for example, requirements containing keywords such as “MUST”, “MUST NOT” “REQUIRED”, “SHALL”, and “SHALL NOT” are mandatory, and issuing certificates that do not comply with such requirements constitutes a violation or mis-issuance incident, these keywords in the Baseline Requirements are interpreted in accordance with RFC 2119. Secondly, we rely on zlint for pre and post issuance compliance checks that cover certificate profiles, a certificate issuance is considered as mis-issuance if any aspect related to the certificate profiles is flagged as “Fatal” or “Error” by zlint. And from our understanding, certificate linters including zlint also adhere to the Baseline Requirements and these keywords defined in RFC 2119 as the basis for flagging issues of different severity levels.
The root cause of the mis-issuance, in our view, lies primarily in our failure to find a clear statement during our review of Ballot SC 62 V2 regarding Basic Constraints must be a critical extension if present in the subscriber certificates, section 7.1.2.7.6 did specify the presence and criticality of the Basic Constraints extension in the subscriber certificates through the use of a table, however, it does not use keywords to indicate the requirement level, such as an explicit statement like “this extension may be present, and MUST be marked critical if present”, and section 7.1.2.7.8 also lacks such a statement. Additionally, the stipulation on Basic Constraints extension in the subscriber certificate in RFC 5280 (section 4.2.1.9) says “this extension MAY appear as a critical or non-critical extension in end entity certificates”. Based on the reasons mentioned above, during the review and analysis of Ballot SC 62 V2, we believed that there is a lack of clarity regarding the criticality of Basic Constraints extension if it is present in a subscriber certificate, although Ballot SC 62 V2 may be intended to mandate Basic Constraints as a critical extension if present in the subscriber certificate, and intended to be more restrictive than RFC 5280 in this regard, the failure to find an explicit statement in the Baseline Requirements led us to mistakenly believe that this requirement was not mandatory and could be treated as an "Optional" level requirement.
Additionally, as far as we know, it was not until the release of zlint v3.6.0-rc1 in December 2023 that such certificates began to be marked as invalid, and the certificates affected by this issue were issued from 15 September to 8 October 2023, and the zlint version deployed in our production environment at the time was not this version. Therefore, we were not able to block the issuance of these certificates through pre-issuance linting or identify this issue using post-issuance linting feature during a period of time after such certificates were issued.
However, we did adjust our issuance system on 8 October 2023 to require Basic Constraints as a critical extension if present in the subscriber certificate, although there was an uncertainty on our side regarding the understanding of the criticality requirement of the Basic Constraints extension as proposed by Ballot SC 62 V2.
Lessons Learned
When determining whether or not a requirement is mandatory, it is important to not only focus on the keywords indicating the requirement levels but also to carefully consider the intended purposes behind the relevant requirements or ballot proposals.
What went well
N/A
What didn't go well
We have been overly reliant on zlint for certificate linting, and if the zlint version updates do not cover relevant requirements within its linting scope in a timely manner before they become effective, it may lead to the failure of compliance checks with requirements that are already effective.
Where we got lucky
The number of affected certificates is small, and the scope of impact on subscribers is limited.
Action Items
| Action Item | Kind | Due Date |
|---|---|---|
| Revoke and replace the problematic certificates. | Mitigate | In progress |
| Deploy the latest version of zlint in our production environment. | Prevent | 7 April 2024 |
| Revise our CP/CPS to clearly state the criticality of the Basic Constraints extension in SSL/TLS certificates. | Mitigate | 15 April 2024 |
Appendix
N/A
Details of affected certificates
https://crt.sh/?id=10719936783
https://crt.sh/?id=10470545541
https://crt.sh/?id=10445635011
https://crt.sh/?id=10493475350
https://crt.sh/?id=10493412807
https://crt.sh/?id=10477997256
https://crt.sh/?id=10478083391
https://crt.sh/?id=10478023477
https://crt.sh/?id=10444099057
https://crt.sh/?id=10469199584
https://crt.sh/?id=10478050726
https://crt.sh/?id=10494148757
https://crt.sh/?id=10469206693
https://crt.sh/?id=10421315957
https://crt.sh/?id=10493688664
https://crt.sh/?id=10386032449
https://crt.sh/?id=10385760399
https://crt.sh/?id=10410654215
https://crt.sh/?id=10419351537
https://crt.sh/?id=10409984698
|
||
Comment 2•1 year ago
|
||
Bug 1875820 was filed on January 22, 2024, and demonstrated the same mis-issuance we observe disclosed in this report (i.e., basicConstraints not marked critical). We later observed other reports opened for the same issue, for example https://bugzilla.mozilla.org/show_bug.cgi?id=1885132.
Question #1: How is GDCA monitoring Bugzilla such that it is able to identify potential issues of its own and remediate them in a timely manner?
Question #2: Can you describe how GDCA evaluates linting tools to fully comprehend each one's scope, capabilities, and limitations, including as updates are made available - including tools that might not yet be in use today (e.g., pkilint)?
Question #3: What is GDCA doing to remedy the self-described over-reliance on zlint? What corresponding actions will be added to the Actions list to address this? How should these actions be tracked and measured to ensure they are effective?
Question #4: Can you describe how GDCA validates that linting tools are working as expected?
Question #5: Is there consideration for incorporating pkilint? It appears that pkilint could have also been helpful in identifying, and possibly preventing, the mis-issuance reported in this incident report.
Request for update #1: The “Due Date" for “revoke and replace problematic certificates" is listed as “In progress.” Please provide a date for when this activity is expected to be completed. If this timeline is more than 5 days since receiving the Certificate Problem Report, a separate bug should be opened focused on the failure to revoke in a timely manner.
Hello Ryan,
Thanks for following up.
Regarding the request for update #1, as of 17:10, 2 April 2024 (UTC+8), all these certificates are either revoked or expired, there are 13 certificates that have been revoked not within 5 days since receiving the Certificate Problem Report, we just filed a separate bug to track this issue, please see Bug 1889062.
We will provide responses to your other questions later.
Thanks.
Xiu Lei
(In reply to Ryan Dickson from comment #2)
Bug 1875820 was filed on January 22, 2024, and demonstrated the same mis-issuance we observe disclosed in this report (i.e., basicConstraints not marked critical). We later observed other reports opened for the same issue, for example https://bugzilla.mozilla.org/show_bug.cgi?id=1885132.
Question #1: How is GDCA monitoring Bugzilla such that it is able to identify potential issues of its own and remediate them in a timely manner?
We regularly review and monitor the CA incidents dashboard on Bugzilla, and summarize important issues that have emerged or existed weekly, with the attempt to identify similar issues that could affect ourselves. We have discovered our own issues by monitoring Bugzilla in the past, for example the insufficient serial number entropy issue, CRL validity exceeding the maximum allowed value etc., and relevant incident reports with remediation had been submitted.
Question #2: Can you describe how GDCA evaluates linting tools to fully comprehend each one's scope, capabilities, and limitations, including as updates are made available - including tools that might not yet be in use today (e.g., pkilint)?
We take a multidimensional approach to evaluate linting tools to support certificates compliance validation, including analyzing the official documentation in relation to the tool, understanding the standards it supports, and its use limitations. We also assess the linting capabilities, accuracy, and stability of the linting tools through conducting testing in our test environment, and furthermore, we examine the tool regarding its community activities, user feedback, and the quality of support services provided by the development team.
Question #3: What is GDCA doing to remedy the self-described over-reliance on zlint? What corresponding actions will be added to the Actions list to address this? How should these actions be tracked and measured to ensure they are effective?
We are currently working on enhancing the diversity of pre-certificates linting to reduce the over reliance on a single tool like zlint. To this end, we plan to incorporate pkilint to work alongside zlint, and relevant system development, testing, and deployment is expected to be completed by 29 July 2024.
Question #4: Can you describe how GDCA validates that linting tools are working as expected?
To validate linting tools are working as expected, we conduct functional testing, anomaly testing in the test environment and acceptance testing in the production environment. Anomaly testing includes linting of problematic test certificates to evaluate whether the tool accurately identifies anomalies and returns corresponding anomaly values, and assess the effectiveness of the integrated system in identifying problematic pre-certificates and blocking the issuance of such certificates. And with regard to the acceptance testing in the production environment, we ensure that the linting tools work as expected primarily through post issuance linting which includes reviewing the logs of linting tools, human review of certificate profiles, and using the linting tools to check the issued certificates.
Question #5: Is there consideration for incorporating pkilint? It appears that pkilint could have also been helpful in identifying, and possibly preventing, the mis-issuance reported in this incident report.
Yes we plan to incorporate pkilint to work alongside zlint, and relevant system development, testing, and deployment is expected to be completed by 29 July 2024.
Updated action Items
| Action Item | Kind | Due Date |
|---|---|---|
| Revoke and replace the problematic certificates. | Mitigate | Completed |
| Revise our CP/CPS to clearly state the criticality of the Basic Constraints extension in SSL/TLS certificates. | Mitigate | 15 April 2024 |
| Deploy the latest version of zlint in our production environment. | Prevent | 8 May 2024 |
| Deploy pkilint in production environment. | Prevent | 29 July 2024 |
Thank you.
|
|
||
Comment 5•1 year ago
|
||
We regularly review and monitor the CA incidents dashboard on Bugzilla, and summarize important issues that have emerged or existed weekly, with the attempt to identify similar issues that could affect ourselves. We have discovered our own issues by monitoring Bugzilla in the past, for example the insufficient serial number entropy issue, CRL validity exceeding the maximum allowed value etc., and relevant incident reports with remediation had been submitted.
Bug 1875820 was opened on January 22, 2024. GDCA filed this incident report on March 26, 2024. In between these dates, several other incident reports were filed demonstrating the same failure mode (i.e., basicConstraints NOT marked critical).
- Bug 1883416 (opened March 4, 2024)
- Bug 1885132 (opened March 13, 2024)
- Bug 1886135 (opened March 19, 2024)
- Bug 1887008 (opened March 22, 2024)
Questions:
-
Can you help us understand why the incident monitoring process described in Comment 4 did not detect GDCA's issue, given we observe numerous incident reports being filed to Bugzilla focusing on this same issue?
-
What changes does GDCA intend to make to improve future incident detection and response?
-
How will these changes be disclosed and tracked in the Action Items Table?
(In reply to Ryan Dickson from comment #5)
We regularly review and monitor the CA incidents dashboard on Bugzilla, and summarize important issues that have emerged or existed weekly, with the attempt to identify similar issues that could affect ourselves. We have discovered our own issues by monitoring Bugzilla in the past, for example the insufficient serial number entropy issue, CRL validity exceeding the maximum allowed value etc., and relevant incident reports with remediation had been submitted.
Bug 1875820 was opened on January 22, 2024. GDCA filed this incident report on March 26, 2024. In between these dates, several other incident reports were filed demonstrating the same failure mode (i.e., basicConstraints NOT marked critical).
- Bug 1883416 (opened March 4, 2024)
- Bug 1885132 (opened March 13, 2024)
- Bug 1886135 (opened March 19, 2024)
- Bug 1887008 (opened March 22, 2024)
Questions:
Hello Ryan,
These are fair questions to ask.
- Can you help us understand why the incident monitoring process described in Comment 4 did not detect GDCA's issue, given we observe numerous incident reports being filed to Bugzilla focusing on this same issue?
We did notice Bug 1875820 in early February, as well as other related incident reports in March, and we revisited the cause of the same issue with ourselves at the time. During the follow-up discussions on these incident reports, we did not find discussions regarding the clarity of the criticality requirement of the basicConstraints extension if present in SSL/TLS subscriber certificates, and we believed that there was a lack of clarity regarding the criticality requirement of basicConstraints extension if it is present in a subscriber certificate, which led us fail to file an incident report regarding this issue in a timely manner.
- What changes does GDCA intend to make to improve future incident detection and response?
According to our current Bugzilla incidents monitoring and response steps, when noticing new incident reports on Bugzilla, we classify them into different categories and check whether we are affected by the same problem, and if similar issues identified, we would confirm the corresponding compliance requirements, after which an incident report will be filed. In case the compliance requirements are considered unclear or ambiguous to us, we may file the incident report later after further confirmation. We plan to adjust our current incident monitoring and response steps to require an incident report to be filed shortly after we identify similar issues with us, to discuss any potential issues publicly, regardless of whether compliance requirements can be confirmed by us timely.
- How will these changes be disclosed and tracked in the Action Items Table?
Updated Action Items
| Action Item | Kind | Due Date |
|---|---|---|
| Revoke and replace the problematic certificates. | Mitigate | Completed |
| Update the Bugzilla incidents monitoring and response steps. | Mitigate | 15 April 2024 |
| Revise our CP/CPS to clearly state the criticality of the Basic Constraints extension in SSL/TLS certificates. | Mitigate | 15 April 2024 |
| Deploy the latest version of zlint in our production environment. | Prevent | 8 May 2024 |
| Deploy pkilint in production environment. | Prevent | 29 July 2024 |
Appendix
- Current Bugzilla Incidents Monitoring and Response Steps
- Updated Bugzilla Incidents Monitoring and Response Steps (Planned)
Thank you!
|
||
Comment 9•1 year ago
|
||
(In reply to capoc from comment #1)
However, we did adjust our issuance system on 8 October 2023 to require Basic Constraints as a critical extension if present in the subscriber certificate, although there was an uncertainty on our side regarding the understanding of the criticality requirement of the Basic Constraints extension as proposed by Ballot SC 62 V2.
What prompted you to make this change on October 8? What steps did you take to clarify the requirements if you were uncertain? Basically, this is when the incident report should have been created, and there should be an analysis of what decisions led to the miss and what actions can be taken to prevent similar mistakes in the future.
|
Assignee | |
Comment 10•1 year ago
|
||
(In reply to Mathew Hodson from comment #9)
(In reply to capoc from comment #1)
However, we did adjust our issuance system on 8 October 2023 to require Basic Constraints as a critical extension if present in the subscriber certificate, although there was an uncertainty on our side regarding the understanding of the criticality requirement of the Basic Constraints extension as proposed by Ballot SC 62 V2.
What prompted you to make this change on October 8? What steps did you take to clarify the requirements if you were uncertain? Basically, this is when the incident report should have been created, and there should be an analysis of what decisions led to the miss and what actions can be taken to prevent similar mistakes in the future.
Hello Mathew,
Regarding the change we made on October 8, we revisited the criticality requirement of Basic Constraints extension after Ballot SC 62 V2 became effective and did a research on the criticality of this extension in the certificates issued by other CAs at the time, and we found that the certificates issued by certain CAs contain this extension and were marked as non-critical, a few CAs issued certificates without this extension, while the majority of CAs, especially some leading CAs, issued certificates with this extension and flagged it as critical. Based on this, we believed it was necessary to make the change.
For the clarification of this requirement, we did a further analysis on Ballot SC 62 V2 at the time by searching for the basis of this requirement change in the revision and discussion records related to Ballot SC 62 V2 on GitHub and in the Server Certificate Working Group mailing list, we did not find a clear discussion thread indicating that the requirement level of this extension had been changed to the MUST level. Additionally, considering that the zlint at the time for both pre and post issuance linting did not flag this issue as invalid, we mistakenly believed that this requirement remained as an "Optional" level requirement, therefore we did not decide to file an incident report and revoke the certificates around 8 October.
To prevent similar mistakes in the future, we expect to use pkilint and zlint to effectively identify any potential issues, and additionally, during discussions and analyses of Ballots, if we have questions about changes to the requirements themselves or changes in the requirement levels, we should raise them promptly within the relevant CABF Working Groups to seek the opinions from a wider community, this should help us to have a more accurate and definitive understanding of the requirement changes.
Thank you.
|
|
Assignee | |
Comment 11•1 year ago
|
||
Following is the latest status of the action items:
Action Items
| Action Item | Kind | Due Date | Status |
|---|---|---|---|
| Revoke and replace the problematic certificates. | Mitigate | 2 April 2024 | Completed |
| Update the Bugzilla incidents monitoring and response steps. | Mitigate | 15 April 2024 | Completed |
| Revise our CP/CPS to clearly state the criticality of the Basic Constraints extension in SSL/TLS certificates. | Mitigate | 15 April 2024 | Completed |
| Deploy the latest version of zlint in our production environment. | Prevent | 8 May 2024 | In progress |
| Deploy pkilint in production environment. | Prevent | 29 July 2024 | In progress |
|
|
Assignee | |
Comment 12•1 year ago
|
||
We are monitoring this bug for further comments or questions.
|
|
Assignee | |
Comment 13•1 year ago
|
||
Following is the latest status of the action items:
Action Items
| Action Item | Kind | Due Date | Status |
|---|---|---|---|
| Revoke and replace the problematic certificates. | Mitigate | 2 April 2024 | Completed |
| Update the Bugzilla incidents monitoring and response steps. | Mitigate | 15 April 2024 | Completed |
| Revise our CP/CPS to clearly state the criticality of the Basic Constraints extension in SSL/TLS certificates. | Mitigate | 15 April 2024 | Completed |
| Deploy the latest version of zlint in our production environment. | Prevent | 8 May 2024 | Completed |
| Deploy pkilint in production environment. | Prevent | 29 July 2024 | In Progress |
|
|
Assignee | |
Comment 14•1 year ago
|
||
We are monitoring this bug for further comments or questions.
|
|
Assignee | |
Comment 15•1 year ago
|
||
We are monitoring this bug for further comments or questions.
|
|
Assignee | |
Comment 16•1 year ago
|
||
We are monitoring this bug for further comments or questions.
|
|
Assignee | |
Comment 17•1 year ago
|
||
We are monitoring this bug for further comments or questions.
|
|
Assignee | |
Comment 18•1 year ago
|
||
Following is a status update of the proposed action items:
Action Items
| Action Item | Kind | Due Date | Status |
|---|---|---|---|
| Revoke and replace the problematic certificates. | Mitigate | 2 April 2024 | Completed |
| Update the Bugzilla incidents monitoring and response steps. | Mitigate | 15 April 2024 | Completed |
| Revise our CP/CPS to clearly state the criticality of the Basic Constraints extension in SSL/TLS certificates. | Mitigate | 15 April 2024 | Completed |
| Deploy the latest version of zlint in our production environment. | Prevent | 8 May 2024 | Completed |
| Deploy pkilint in production environment. | Prevent | 29 July 2024 | Completed |
|
|
Assignee | |
Comment 19•1 year ago
|
||
We are monitoring this bug for further comments or questions.
|
|
Assignee | |
Comment 20•1 year ago
|
||
We are monitoring this bug for further comments or questions.
|
|
Assignee | |
Comment 21•1 year ago
|
||
We are monitoring this bug for further comments or questions.
|
|
Assignee | |
Comment 22•1 year ago
|
||
We are monitoring this bug for further comments or questions.
|
|
Assignee | |
Comment 23•1 year ago
|
||
We are monitoring this bug for further comments or questions.
|
|
Assignee | |
Comment 24•1 year ago
|
||
We are monitoring this bug for further comments or questions.
|
|
Assignee | |
Comment 25•1 year ago
|
||
We are monitoring this bug for further comments or questions.
|
|
Assignee | |
Comment 26•1 year ago
|
||
We are monitoring this bug for further comments or questions.
|
|
Assignee | |
Comment 27•1 year ago
|
||
We are monitoring this bug for further comments or questions.
|
|
Assignee | |
Comment 28•1 year ago
|
||
We are monitoring this bug for further comments or questions.
|
|
||
Comment 29•11 months ago
|
||
You will need to provide a Closure Summary. See also https://www.ccadb.org/cas/incident-report#incident-closure-summary.
|
|
Assignee | |
Comment 30•11 months ago
|
||
(In reply to Ben Wilson from comment #29)
You will need to provide a Closure Summary. See also https://www.ccadb.org/cas/incident-report#incident-closure-summary.
Report Closure Summary
Incident description:
GDCA issued 20 SSL/TLS certificates from 15 September to 8 October 2023, with the Basic Constraints extension included but not set as Critical.
Incident Root Cause(s):
We believe that the root cause of the incident lies in our confusion regarding the criticality requirement of Basic Constraints extension defined in SC 62v2, coupled with the fact that the zlint version deployed in our production environment at the time failed to detect this issue in a timely manner.
Remediation description:
GDCA adjusted our Bugzilla incidents monitoring and response steps, updated the zlint version deployed in the production environment, and also deployed pkilint.
Commitment summary:
We will continue to actively monitor compliance incident reports on Bugzilla, conduct internal reviews of potential issues that may arise within our own systems, and promptly update the deployed versions of zlint and pkilint in our production environment to prevent the recurrence of similar issues.
All Action Items disclosed in this report have been completed as described, and we request its closure.
|
|
||
Comment 31•11 months ago
|
||
Unless there are questions or concerns, I will close this on Wed. 5-Mar-2025.
|
|
||
Updated•11 months ago
|
Description
•