TWCA: TLS EV certificates with invalid subject attribute order
Categories
(CA Program :: CA Certificate Compliance, task)
Tracking
(Not tracked)
People
(Reporter: hcli, Assigned: hcli)
Details
(Whiteboard: [ca-compliance] [ev-misissuance] Next update 2024-06-30)
Attachments
(1 file)
|
2.81 KB,
text/plain
|
Details |
Steps to reproduce:
Incident Report
This is a preliminary report.
Summary
TWCA received an email on 2024-03-04 reporting a potentially mis-issued certificate.
After initial analysis we confirmed the issue and found it is the EV TLS system that has been issuing certificates with subject attribute order nonconforming to BR Section 7.1.2.4.
We have stopped issuing EV TLS certificates.
We are still investigating the issue and expecting to provide further details this week.
|
||
Updated•4 months ago
|
|
Assignee | |
Comment 1•4 months ago
|
||
|
|
Assignee | |
Comment 2•4 months ago
|
||
Incident Report
This is an updated preliminary report.
Summary
TWCA received an email on 2024-03-04 reporting a potentially mis-issued certificate.
We found there are 90 EV TLS certificates issued with subject attribute order nonconforming to BR Section 7.1.4.2.
The issuing system has been patched and we are working with affected customers to replace their certificates.
Impact
90 EV TLS certificates were issued since 2023-09-15 with incorrect subject attribute order.
All affected certificates are revoked/going to be revoked, except for one that was already expired.
Timeline
All times are UTC+8.
2023-09-15:
- 08:00 BR for TLS 2.0.0 has become effective.
2024-03-04:
- 21:00 Email reporting the issue received.
2024-03-05:
- 07:28 Compliance team confirmed the issue and started the investigation. TWCA stopped issuance of EV TLS certificates.
- 17:30 Patch for issuing system is completed.
- 20:28 First preliminary report is posted.
2024-03-06:
- 07:46 All affected certificates identified.
- 08:55 TWCA confirmed the patch is working as intended and resumed EV TLS certificates issuance.
- 13:24 TWCA started working on replacing certificate.
2024-03-08:
- 20:45 Post this update.
Appendix
Details of affected certificates
Incident Report
Summary
TWCA received an email on 2024-03-04 reporting a potentially mis-issued certificate.
We found that 90 EV TLS certificates had been issued with the subject attribute order not conforming to BR Section 7.1.4.2.
Impact
Since 2023-09-15, 90 EV TLS certificates were issued with incorrect subject attribute order. The issuance of EV TLS certificates was stopped until the issuing system had been fixed.
77 of the affected certificates have been revoked or expired within 5 days. We have created another bug (https://bugzilla.mozilla.org/show_bug.cgi?id=1884568) for the remaining 13 certificates and expect to revoke all certificates no later than 2024-03-23.
Timeline
All times are UTC+8.
2023-04-11:
- TLS BR 2.0.0 was published.
2023-05~08:
- We have reviewed the certificate profile changes in TLS BR 2.0.0 and scheduled relevent development tasks.
2023-09-15:
- 08:00 TLS BR 2.0.0 became effective.
2024-03-04:
- 21:00 Email reporting the issue received.
2024-03-05:
- 07:28 Compliance team confirmed the issue and started the investigation. TWCA stopped the issuance of EV TLS certificates.
- 17:30 Patch for thie issuing system is completed.
- 20:28 A preliminary report was posted.
2024-03-06:
- 07:46 All affected certificates identified.
- 08:55 TWCA confirmed the patch was working as intended and resumed EV TLS certificate issuance.
- 13:24 TWCA started contacting customers for certificate replacement.
2024-03-09:
- 15:52 After risk assessment, we concluded there are 13 certificates that cannot be revoked in time.
- 19:30 77 of the affected certificates have been revoked or expired.
2024-03-10：
- 20:44 A preliminary report for delayed revocation was posted. (https://bugzilla.mozilla.org/show_bug.cgi?id=1884568)
Root Cause Analysis
TLS BR version 2.0.0 introduced a completely rewritten section 7, focusing mostly on clarifications and encoding canonicalizations. While we reviewed the changes in TLS BR version 2.0.0, the requirement for subject attribute order in Section 7.1.4.2 was overlooked.
We have been using linting to detect these problems but did not identify this issue until now because we primarily rely on zlint, and there were no errors reported since zlint only implemented this change on 2024-01-08. We had not completed testing and integrating the new version at the time this issue was reported.
Additionally, we might have noticed this requirement earlier if we had been closely monitoring compliance issues on Bugzilla.
Lessons Learned
What went well
N/A
What didn't go well
- We overlooked the subject attribute order requirement.
- The textual changes in TLS BR 2.0.0 were substantial, making it difficult to distinguish requirements from format changes. A change list that only contains requirement changes would be helpful.
- Delays occurred before the linter could be updated to check for the new requirement.
- We did not update the linter frequently enough to detect the issue earlier.
- We were not monitoring Bugzilla closely enough to notice this requirement.
Where we got lucky
- Only attributes used in EV TLS certificates were affected.
Action Items
| Action Item | Kind | Due Date |
|---|---|---|
| Assign more personnel to review the requirements and establish a verification mechanism for double-checking. | Prevent | 2024-03-31 |
| Add pkilint to the certificate issuance process. | Prevent/Detect | 2024-06-30 |
| Establish a procedure to review Bugzilla issues at least once a week and confirm whether we are affected by similar problems. | Prevent/Detect | 2024-03-31 |
Appendix
Details of affected certificates
Incident Report
Final report on this incident.
Summary
TWCA received an email on 2024-03-04 reporting a potentially mis-issued certificate.
We found that 90 EV TLS certificates had been issued with the subject attribute order not conforming to BR Section 7.1.4.2.
Impact
Since 2023-09-15, 90 EV TLS certificates were issued with incorrect subject attribute order. The issuance of EV TLS certificates was stopped until the issuing system had been fixed.
77 of the affected certificates have been revoked or expired within 5 days. We have created another bug (Bug 1884568) for the remaining 13 certificates and expect to revoke all certificates no later than 2024-03-23.
Timeline
All times are UTC+8.
2023-04-11:
- TLS BR 2.0.0 was published.
2023-05~08:
- We have reviewed the certificate profile changes in TLS BR 2.0.0 and scheduled relevent development tasks.
2023-09-15:
- 08:00 TLS BR 2.0.0 became effective.
2024-03-04:
- 21:00 Email reporting the issue received. We define this as the point of incident occurrence.
2024-03-05:
- 07:28 Compliance team confirmed the issue and started the investigation. TWCA stopped the issuance of EV TLS certificates.
- 17:30 Patch for thie issuing system is completed.
- 20:28 A preliminary report was posted.
2024-03-06:
- 07:46 All affected certificates identified.
- 08:55 TWCA confirmed the patch was working as intended and resumed EV TLS certificate issuance.
- 13:24 TWCA started contacting customers for certificate replacement.
2024-03-09:
- 15:52 After risk assessment, we concluded there are 13 certificates that cannot be revoked in time.
- 19:30 77 of the affected certificates have been revoked or expired.
- 21:00 According to BR requirements, all affected certificates should be revoked before this point in time.
2024-03-10：
- 20:44 A preliminary report for delayed revocation was posted. (Bug 1884568)
Root Cause Analysis
TLS BR version 2.0.0 introduced a completely rewritten section 7, focusing mostly on clarifications and encoding canonicalizations. While we reviewed the changes in TLS BR version 2.0.0, the requirement for subject attribute order in Section 7.1.4.2 was overlooked.
We have been using linting to detect these problems but did not identify this issue until now because we primarily rely on zlint, and there were no errors reported since zlint only implemented this change on 2024-01-08. We had not completed testing and integrating the new version at the time this issue was reported.
Additionally, we might have noticed this requirement earlier if we had been closely monitoring compliance issues on Bugzilla.
Lessons Learned
What went well
N/A
What didn't go well
- We overlooked the subject attribute order requirement.
- The textual changes in TLS BR 2.0.0 were substantial, making it difficult to distinguish requirements from format changes. A change list that only contains requirement changes would be helpful.
- Delays occurred before the linter could be updated to check for the new requirement.
- We did not update the linter frequently enough to detect the issue earlier.
- We were not monitoring Bugzilla closely enough to notice this requirement.
Where we got lucky
- Only attributes used in EV TLS certificates were affected.
Action Items
| Action Item | Kind | Due Date |
|---|---|---|
| Assign more personnel to review the requirements and establish a verification mechanism for double-checking. | Prevent | 2024-03-31 |
| Add pkilint to the certificate issuance process. | Prevent/Detect | 2024-06-30 |
| Establish a procedure to review Bugzilla issues at least once a week and confirm whether we are affected by similar problems. This procedure will be documented in the TWCA internal ISMS documents. | Prevent/Detect | 2024-04-30 |
| Establish a procedure to check at least once a week whether the certificate linter tools has been updated, and upon discovering an update, initiate a series of self-checks and tool update operations. This procedure will be documented in the TWCA internal ISMS documents. | Prevent/Detect | 2024-04-30 |
Appendix
Details of affected certificates
Due to the fact that this incident is related to [Bug 1885132], with both occurring around the same time and their root causes being the failure to thoroughly review the BR specifications leading to the issuance of incorrect credentials, the corrective and preventative measures for both incidents will be merged. To avoid inconsistencies in the follow-up tasks for both incidents, our subsequent actions will be consolidated into [Bug 1885132] for explanation and tracking. Unless there are specific issues related to this incident, we hope to close it. Thank you.
Incident Report
- Update the Timeline.
- Correct the Root Cause Analysis content.
Summary
TWCA received an email on 2024-03-04 reporting a potentially mis-issued certificate.
We found that 90 EV TLS certificates had been issued with the subject attribute order not conforming to BR Section 7.1.4.2.
Impact
Since 2023-09-15, 90 EV TLS certificates were issued with incorrect subject attribute order. The issuance of EV TLS certificates was stopped until the issuing system had been fixed.
77 of the affected certificates have been revoked or expired within 5 days. We have created another bug (Bug 1884568) for the remaining 13 certificates and expect to revoke all certificates no later than 2024-03-23.
Timeline
All times are UTC+8.
2023-04-11:
- TLS BR 2.0.0 was published.
2023-05~08:
- We have reviewed the certificate profile changes in TLS BR 2.0.0 and scheduled relevent development tasks.
2023-09-15:
- 08:00 TLS BR 2.0.0 became effective.
2024-03-04:
- 21:00 Email reporting the issue received. We define this as the point of incident occurrence.
2024-03-05:
- 07:28 Compliance team confirmed the issue and started the investigation. TWCA stopped the issuance of EV TLS certificates.
- 17:30 Patch for thie issuing system is completed.
- 20:28 A preliminary report was posted.
2024-03-06:
- 07:46 All affected certificates identified.
- 08:55 TWCA confirmed the patch was working as intended and resumed EV TLS certificate issuance.
- 13:24 TWCA started contacting customers for certificate replacement.
2024-03-09:
- 15:52 After risk assessment, we concluded there are 13 certificates that cannot be revoked in time.
- 19:30 77 of the affected certificates have been revoked or expired.
- 21:00 According to BR requirements, all affected certificates should be revoked before this point in time.
2024-03-10:
- 20:44 A preliminary report for delayed revocation was posted. (Bug 1884568)
2024-03-22:
- 18:40 All affected certificates in this incident have been revoked.
Root Cause Analysis
TLS BR version 2.0.0 introduced a completely rewritten section 7, focusing mostly on clarifications and encoding canonicalizations. While we reviewed the changes in TLS BR version 2.0.0, the requirement for subject attribute order in Section 7.1.4.2 was overlooked.
Since July 2020, we have relied on ZLint to detect compliance issues with certificates, but until the incident was reported, the tool was still unable to detect the problem of incorrect order of the subject attributes. We did not pay attention to and utilize other detection tools (such as pkilint) to mitigate the problem of insufficient detection by a single tool.
Additionally, we might have noticed this requirement earlier if we had been closely monitoring compliance issues on Bugzilla.
Lessons Learned
What went well
N/A
What didn't go well
- We overlooked the subject attribute order requirement.
- The textual changes in TLS BR 2.0.0 were substantial, making it difficult to distinguish requirements from format changes. A change list that only contains requirement changes would be helpful.
- Delays occurred before the linter could be updated to check for the new requirement.
- We did not update the linter frequently enough to detect the issue earlier.
- We were not monitoring Bugzilla closely enough to notice this requirement.
Where we got lucky
- Only attributes used in EV TLS certificates were affected.
Action Items
| Action Item | Kind | Due Date |
|---|---|---|
| Assign more personnel to review the requirements and establish a verification mechanism for double-checking. | Prevent | 2024-03-31 |
| Add pkilint to the certificate issuance process. | Prevent/Detect | 2024-06-30 |
| Establish a procedure to review Bugzilla issues at least once a week and confirm whether we are affected by similar problems. This procedure will be documented in the TWCA internal ISMS documents. | Prevent/Detect | 2024-04-30 |
| Establish a procedure to check at least once a week whether the certificate linter tools has been updated, and upon discovering an update, initiate a series of self-checks and tool update operations. This procedure will be documented in the TWCA internal ISMS documents. | Prevent/Detect | 2024-04-30 |
Appendix
Details of affected certificates
After a detailed investigation, we believe that the cause of this incident and the action items still differ from those related to [Bug 1885132]. Therefore, we will continue to update the incident report until all action items are completed.
We continue to monitor the incident and complete action items as planned.
|
||
Comment 9•3 months ago
|
||
Thank you for this report, the timely updates, and for noting what changed in the revision of the final version.
Related to your Action Items:
- When your first action item is completed, it might be helpful to know what verification mechanism was deployed and why TWCA believes this will help avoid overlooking future requirements. It might also be helpful to highlight how TWCA plans to measure the effectiveness of this change.
- It's encouraging to see the adoption of pkilint. Can you elaborate on which part of the certificate issuance process you intend to introduce this lint? Will this be for pre-issuance, similar to what was stated in 1885132?
- It might be helpful to the community if you were to describe some of the detail behind the procedures you are creating for your last two action items and how you know those implemented changes are effective. Understandably, you describe detailing them in internal documents, but maybe there are non-sensitive details that can be shared to help others improve.
|
||
Comment 10•3 months ago
|
||
(In reply to Chris Clements from comment #9)
Thank you for this report, the timely updates, and for noting what changed in the revision of the final version.
Related to your Action Items:
- When your first action item is completed, it might be helpful to know what verification mechanism was deployed and why TWCA believes this will help avoid overlooking future requirements. It might also be helpful to highlight how TWCA plans to measure the effectiveness of this change.
In the past, we overly relied on the opinion of a single individual, lacking a mechanism for discussion among multiple parties, which could lead to misjudgment by an individual. Going forward, we will assign different individuals to produce comparative documents after reviewing the standards, and these documents will be reviewed by a supervisor. At the same time, these documents will also become part of the scope for external audits.
- It's encouraging to see the adoption of pkilint. Can you elaborate on which part of the certificate issuance process you intend to introduce this lint? Will this be for pre-issuance, similar to what was stated in 1885132?
Yes, as described in 1885132, we will perform a pkilint check before uploading to the CT log server. If there is an anomaly, the issuance will fail. This tool is expected to be introduced by 6/30.
- It might be helpful to the community if you were to describe some of the detail behind the procedures you are creating for your last two action items and how you know those implemented changes are effective. Understandably, you describe detailing them in internal documents, but maybe there are non-sensitive details that can be shared to help others improve.
We plan to have two automated schedules.
One of the schedules is set to regularly fetch incidents created in Bugzilla from the previous week on a weekly basis and send a summary to the relevant personnel. These incidents are then discussed in the weekly meeting, with the content of the meeting including self-assessment and review.
The other schedule runs daily, conducting checks on the targets we monitor (including the GitHub site of lint tools) and sends the results to the relevant personnel via email. If there is an update in the version of the monitored targets, related assessments will be performed, including checks on issued certificates and evaluations of tool updates.
These related measures and the detailed execution methods will be documented in the internal ISMS documents.
|
|
||
Comment 11•3 months ago
|
||
We've assigned more staff to study the standards and produce discrepancy reports, and we've established a review mechanism.
Action Items
| Action Item | Kind | Due Date |
|---|---|---|
| Assign more personnel to review the requirements and establish a verification mechanism for double-checking. | Prevent | Done |
| Establish a procedure to review Bugzilla issues at least once a week and confirm whether we are affected by similar problems. This procedure will be documented in the TWCA internal ISMS documents. | Prevent/Detect | 2024-04-30 |
| Establish a procedure to check at least once a week whether the certificate linter tools has been updated, and upon discovering an update, initiate a series of self-checks and tool update operations. This procedure will be documented in the TWCA internal ISMS documents. | Prevent/Detect | 2024-04-30 |
| Add pkilint to the certificate issuance process. | Prevent/Detect | 2024-06-30 |
|
|
||
Comment 12•3 months ago
|
||
The current draft of our internal ISMS documents has been completed and is now undergoing the review and issuance process. We request to set the next update for April 30, 2024. Thank you.
|
|
||
Updated•3 months ago
|
|
|
||
Comment 13•2 months ago
|
||
Progress Update (synchronized with Bug 1885132):
- The internal procedures have been established, relevant ISMS documents have been issued, and the company is currently complying with the standards in its operations.
- We use automated scheduling to monitor Bugzilla and lint tools. This schedule sends information via email to internal colleagues at daily and weekly intervals:
- Daily monitoring: An email is sent out every day to monitor standards (such as BRs) or new release information of lint tools, and version change information is marked in the email.
- Weekly monitoring: An email is sent every Monday summarizing incidents created on Bugzilla from the previous week, which are then reviewed in weekly meetings to analyze the causes of incidents and whether similar situations have occurred within the company.
- The implementation of pkilint is progressing smoothly; software development has been completed and is currently in the testing phase, expected to be completed on schedule.
Action Items
| Action Item | Kind | Due Date |
|---|---|---|
| Assign more personnel to review the requirements and establish a verification mechanism for double-checking. | Prevent | Done |
| Establish a procedure to review Bugzilla issues at least once a week and confirm whether we are affected by similar problems. This procedure will be documented in the TWCA internal ISMS documents. | Prevent/Detect | Done |
| Establish a procedure to check at least once a week whether the certificate linter tools has been updated, and upon discovering an update, initiate a series of self-checks and tool update operations. This procedure will be documented in the TWCA internal ISMS documents. | Prevent/Detect | Done |
| Add pkilint to the certificate issuance process. | Prevent/Detect | 2024-06-30 |
|
|
||
Comment 14•2 months ago
|
||
We request to set the next update for May 30, 2024. Thank you.
|
|
||
Updated•2 months ago
|
|
|
||
Comment 15•1 month ago
|
||
The CA code has currently been handed over to the operations team for final validation tests before going live, and the administrative procedures for the launch are underway. Everything is progressing smoothly, and the plan to implement pkilint by June 30th remains unchanged.
We request to set the next update for June 30, 2024. Thank you.
|
|
||
Updated•1 month ago
|
|
|
||
Comment 16•10 days ago
|
||
Our CA system has successfully integrated pkilint (v0.11.0). Currently, before a precertificate is uploaded to the CT log server, it undergoes a pkilint check. If the linting fails, it will not be uploaded to the log server, and a final certificate will not be issued, reducing the chance of incorrect issuance. Additionally, during the linting process, the lint results and the current pkilint version are recorded in the database for future auditing purposes.
Action Items
| Action Item | Kind | Due Date |
|---|---|---|
| Assign more personnel to review the requirements and establish a verification mechanism for double-checking. | Prevent | Done |
| Establish a procedure to review Bugzilla issues at least once a week and confirm whether we are affected by similar problems. This procedure will be documented in the TWCA internal ISMS documents. | Prevent/Detect | Done |
| Establish a procedure to check at least once a week whether the certificate linter tools has been updated, and upon discovering an update, initiate a series of self-checks and tool update operations. This procedure will be documented in the TWCA internal ISMS documents. | Prevent/Detect | Done |
| Add pkilint to the certificate issuance process. | Prevent/Detect | Done |
|
|
||
Comment 17•7 days ago
|
||
For this bug, all item actions have been completed. We kindly request that it be closed.
|
|
||
Updated•2 days ago
|
Description
•