1883731 - Actalis: Certificates issued with invalid RDN order

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Marco Menonna]

Incident Report

This is a preliminary report.

Summary

Actalis received a report concerning some potential issues related to order requirements for Subject Attributes in TLS EV certificates.
We have started investigations and will publish an incident report soon.

Comment 1

[Marco Menonna]

Attachment: Affected certificates

Comment 2

[Marco Menonna]

Incident Report

This is an updated preliminary report

Summary

Actalis received a report concerning a potential issue related to order requirements for Subject Attributes in TLS EV certificates as defined by section 7.1.4.2 in BR.

Impact

After internal investigation, a total of 263 certificates have been confirmed to be affected by the issue.

Timeline

All times are GMT + 1

2024-03-04:
• 14:00 We received an email reporting a potential issue affecting an EV certificate
• 14:30 We started investigating on the potential Issue related on the EV certificate
• 15:00 Internal checks to identifed that the potential issue could affect the EV certificates
• 15:12 We have stopped issuing new TLS EV certificates as a precautionary measure.
• 16:40 Alternative CA software configuration has been explored in the test environment

2024-03-05:
• 10:20 We verified if the new configuration of CA software could resolve the potential issue
• 18:26 The preliminary report has been posted to Bugzilla

2024-03-06:
• 08:07 We verified that the new configuration of CA software resolve the potential issue

2024-03-11:
• 09:40 The issue is confirmed as a mis-issuance, and we have decided to revoke all affected certificates within 5 days
• 18:30 We have posted an updated of the preliminary report on Bugzilla

Appendix

Details of affected certificates

Affected certificates

Comment 3

[Marco Menonna]

Update: All certificates affected by this bug are revoked.

Comment 4

[Marco Menonna]

Incident Report

Summary

Actalis received a report concerning some potential issues related to order requirements for Subject Attributes in TLS EV certificates as defined by section 7.1.4.2 in BR.

After internal investigation we confirmed the mis-issuance for 263 certificates with the streetAddress attribute in a different relative encoding order and revoked all certificates impacted by the bug.

Impact

After internal investigation, a total of 263 certificates have been confirmed to be affected by the issue.

Timeline

All times are GMT + 1

2023-04-11:

• BR 2.0.0 have been published

2023-09-15:

• The certificate profile changes in BR 2.0.0 has become effective

2024-03-04:

• 14:00 We received an email reporting a potential issue affecting an EV certificate
• 14:30 We started investigating on the potential Issue related on the EV certificate
• 15:00 Internal checks to identifed that the potential issue could affect the EV certificates
• 15:12 We have stopped issuing new TLS EV certificates as a precautionary measure.
• 16:40 Alternative CA software configuration has been explored in the test environment

2024-03-05:

• 10:20 We verified if the new configuration of CA software could resolve the potential issue
• 18:26 The preliminary report has been posted to Bugzilla

2024-03-06:

• 08:07 We verified that the new configuration of CA software resolve the potential issue

2024-03-11:

• 09:40 The issue is confirmed as a mis-issuance, and we have decided to revoke all affected certificates within 5 days
• 18:30 We have posted an updated of the preliminary report on Bugzilla

2024-03-15:

• 11:59 All the affected certificates have been revoked

Root Cause Analysis

The publication of version 2.0.0 of the CABF BR, as described within the ballot, has been approached by Actalis primarily as a reorganization of the certification profiles required by Section 7.

The issue is partially being caused by the usage of EJCBA as CA Software, which applies its own default ordering, often correct but not in this case.

In addition, the ZLINT in use at the date did not detect an incorrect RDN Subject Attributes order as an Issue.

Lessons Learned

What went well

• Most of the affected certificates were issued for internal use, so it has been easier to work for their smooth replacement
• For the affected certificates issued to customers, our service manager team worked intensively together with customers to ensure their reissuance without disruption
• After the recognition of the bug, the new configuration of the EJBCA for the correct encoding order has not been difficult

What didn't go well

• We reviewed Certificate Profile Updates after the publication of BR 2.0.0 but we didn't detect the Subject Attribute Encoding Order requirements
• ZLINT, in use by Actalis, did not check the Subject Attribute Encoding Order; even the additional linter internally developed by Actalis didn't detect the issue
• The default ordering applied by EJBCA has not been double-checked

Where we got lucky

• Only a limited number of certificates have been impacted by the bug, most of them for internal use

Action Items

Action Item	Kind	Status	Due Date
Revocation of all affected certificates	Corrective	Complete	2023-03-15
Development of a new lint (PR#813) in ZLINT project for the "relative order" prescribed by CABF BR section 7.1.4.2 Subject Attribute Encoding )	Preventive	Complete	2023-03-10
Implement an internal lint for the "relative order" prescribed by CABF BR section 7.1.4.2 Subject Attribute Encoding in our internally developed linter	Preventive	Complete	2023-03-10

Appendix

Details of affected certificates

Affected Certificates

Comment 5

[Ryan Dickson]

Hi Marco,

Thanks for filing this bug.

General Comments:

• First, thank you for contributing to the ZLint project. This is really helpful. I hope contributions like this become considered “good practice" in response to future mis-issuance incidents, as it decreases the likelihood of other members of the ecosystem being affected by the same issue.

• When providing crt.sh links, it’s generally considered more helpful to reference sha256 hashes, rather than serial #s (From CCADB.org: “The recommended format is to ensure that all affected certificates are logged to CT, then to attach a text file where each line is of the form https://crt.sh/?sha256=sha256 fingerprint of the certificate.)”

• Actalis’ revocation timeline does not appear to satisfy the expectations of the Baseline Requirements.

• Specific updates and answers to questions are requested below.

Updates Requested:

A. The Timeline Section does not describe when the certificate problem report was responded to or when Actalis provided a preliminary report on its findings to both the affected Subscribers and the entity that filed the report (BRs 4.9.5). Also, you should add specific line items for when revocation of the misissued certificates was expected (as required by the BRs, more on this below).

B. The Root Cause Analysis does not clearly present a root cause. It generally describes a combination of human error, a software misconfiguration, and a misunderstanding of the scope of lints provided by ZLint. The description provided does not describe how these contributing causes avoided detection. You could try the “5 Whys” methodology observed in 1878106.

C. A separate bug should be opened focusing on the delayed revocation of these misissued certificates.

The Baseline Requirements specify:

4.9.1.1: “With the exception of Short-lived Subscriber Certificates, the CA SHOULD revoke a certificate within 24 hours and MUST revoke a Certificate within 5 days and use the corresponding CRLReason (see Section 7.2.2) if one or more of the following occurs:” … “The CA is made aware that the Certificate was not issued in accordance with these Requirements or the CA’s Certificate Policy or Certification Practice Statement (CRLReason #4, superseded).”

4.9.5: Within 24 hours after receiving a Certificate Problem Report, the CA SHALL investigate the facts and circumstances related to a Certificate Problem Report and provide a preliminary report on its findings to both the Subscriber and the entity who filed the Certificate Problem Report. After reviewing the facts and circumstances, the CA SHALL work with the Subscriber and any entity reporting the Certificate Problem Report or other revocation-related notice to establish whether or not the certificate will be revoked, and if so, a date which the CA will revoke the certificate. The period from receipt of the Certificate Problem Report or revocation-related notice to published revocation MUST NOT exceed the time frame set forth in Section 4.9.1.1.

As described by the bold language above, the 5-day revocation window was considered to begin on the date of receiving the problem report (March 4, 2024), not when Actalis determined revocation should take place (March 11, 2024).

Questions:

Q1) The “Root Cause Analysis” Section describes, “The publication of version 2.0.0 of the CABF BR, as described within the ballot, has been approached by Actalis primarily as a reorganization of the certification profiles required by Section 7.” What specific practices will Actalis be changing such that this mistake, or others like it, do not repeat? (i.e., how does Actalis plan to more closely follow the discussions and corresponding outcomes within the CA/Browser Forum?). It is possible, when considering how this condition avoided detection, that some specific preventative practices are discovered. Please add corresponding actions to the Action Items list for tracking.

Q2) Can you describe how Actalis performs linting today (i.e., is it strictly post-issuance linting? If so, can you describe whether pre-issuance linting has been considered?)?

Q3) Can you describe how Actalis evaluates linting tools to fully comprehend each one's scope, capabilities, and limitations, including as updates are made available - including tools that might not yet be in use today (e.g., pkilint)?

Q4) Can you describe how Actalis validates linting tools are working as expected?

Comment 6

[Marco Menonna]

Attachment: cancel

Comment 7

[Marco Menonna]

Hi Ryan,
we've received your comments and questions. After an internal review, here's some feedback.
First of all, we've opened a separate bug (see https://bugzilla.mozilla.org/show_bug.cgi?id=1887941) for revocation delay.
Regarding your questions, you can find answers to Q1 within the updated incident report (new root cause analysis with inclusion of the 5 why methodology)
Linting activities (Q2, Q3, Q4) are performed by Actalis as follows:

Q2) Can you describe how Actalis performs linting today (i.e., is it strictly post-issuance linting? If so, can you describe whether pre-issuance linting has been considered?)?

Since long time, Actalis performs pre-issuance linting. Specifically, linting activities are performed by Actalis both:
• before issuance, blocking issuance based on Zlint results;
• after issuance. In this case, linting is based on Actalis' proprietary tool.
The second lint, based on our proprietary linter, is configured to send an alert to several people if problems that the first phase did not detect arise.

Q3) Can you describe how Actalis evaluates linting tools to fully comprehend each one's scope, capabilities, and limitations, including as updates are made available - including tools that might not yet be in use today (e.g., pkilint)?

We have been monitoring the linting tools landscape for several years and evaluating what to do when interesting new developments arise. We also consider the use of tools that we are not currently using, we might decide to dismiss linting tools that are no longer useful for our purpose.
We are aware of the existence of "pklint" because we regularly follow the discussions on the various mailing lists of the CAB Forum as well as Mozilla. We have also already tested it and we are considering its use, taking into account various factors, including architectural, systemic and management aspects.

Q4) Can you describe how Actalis validates linting tools are working as expected?

We normally use the latest available version of ZLint (and occasionally contribute to its development); to ensure this, we monitor its releases on GitHub.
When a new release includes additional lints that apply to our services, we prioritize the update on our systems. in particular, when a new version of ZLint "addresses" new requirements introduced by the Baseline Requirements and/or Guidelines of the CAB Forum (or by other regulations or standards applicable to our context), we plan the upgrade urgently.
Otherwise, we plan the update with medium priority. In all cases, the upgraded version is first installed in the Test environment, subjected to a few "targeted" test cases, both positive and negative, and released to Production when we are confident that all is going to be OK.
The above also applies, mutatis mutandis, to Actalis' proprietary linting tool: we work to keep it up-to-date taking into account the evolving requirements of the CABF and the various Root CA programs.

Incident Report

Summary

Actalis received a report concerning some potential issues related to order requirements for Subject Attributes in TLS EV certificates as defined by section 7.1.4.2 in BR.
After internal investigation we confirmed the mis-issuance for 263 certificates with the streetAddress attribute in a different relative encoding order and revoked all certificates impacted by the bug.

Impact

After internal investigation, a total of 263 certificates have been confirmed to be affected by the issue.

Timeline

All times are GMT + 1
2023-04-11:
• BR 2.0.0 have been published

2023-09-15:
• The certificate profile changes in BR 2.0.0 has become effective

2024-03-04:
• 14:00 We received an email reporting a potential issue affecting an EV certificate
• 14:22 We've confirmed to have take in charge the issue to the reporting entity
• 14:30 We started investigating on the potential Issue related on the EV certificate
• 15:00 Internal checks to identifed that the potential issue could affect the EV certificates
• 15:12 We have stopped issuing new TLS EV certificates as a precautionary measure.
• 16:40 Alternative CA software configuration has been explored in the test environment

2024-03-05:
• 10:20 We verified if the new configuration of CA software could resolve the potential issue
• 18:26 The preliminary report has been posted to Bugzilla

2024-03-06:
• 08:07 We verified that the new configuration of CA software resolve the potential issue
• 08:27 Preliminary communication was given to customers affected

2024-03-09:
• 14:22 Misissued certificates revocation was expected (as required by BRs 4.9.1.1.)

2024-03-11:
• 09:40 The issue is confirmed as a mis-issuance, and we have decided to revoke all affected certificates within 5 days
• 18:30 We have posted an updated of the preliminary report on Bugzilla

2024-03-15:
• 11:59 All the affected certificates have been revoked

2024-03-26:
• 18:50 a separate bug (see https://bugzilla.mozilla.org/show_bug.cgi?id=1887941) for revocation delay has been opened

Root Cause Analysis

The publication of version 2.0.0 of the CABF BR, as described within the ballot, has been approached by Actalis primarily as a reorganization of the certification profiles required by Section 7.
The issue is partially being caused by the usage of EJCBA as CA Software, which applies its own default ordering, often correct but not in this case.
In addition, the ZLINT in use at the date did not detect an incorrect RDN Subject Attributes order as an Issue.

For a more accurate analysis, we have applied the “5 whys” root cause methodology as follow:

Why was there a problem?
→ Because the Subject Attribute Order was incorrect due to the introduction of BR version 2.0.0, which required that there be an order defined

Why applied Attribute order was incorrect?
→ because the RDN ordering applied by default by the EJBCA software was not entirely correct in the case of EV certificates.

Why was the problem not detected?
→ because our linters, in the versions we had in use up to March 4, 2024, were not able to detect it

Why did ZLINT and EJBCA's default order not meet the CABF requirements of BR section 7.1.4.2?
→ because these tools are intended to assist the CA, but they are not to be used as a substitute for the analysis that the CA must do on the impacts that new policy versions have on the CA's activities

Why did Actalis not adequately intercept the new requirements introduced by the new policy versions?
→ Because the monitoring activities for CABF, ballots, the Bugzilla CA Program forum, and the [e-mail] group did not highlited the issue

Lessons Learned

What went well

• Most of the affected certificates were issued for internal use, so it has been easier to work for their smooth replacement
• For the affected certificates issued to customers, our service manager team worked intensively together with customers to ensure their reissuance without disruption
• After the recognition of the bug, the new configuration of the EJBCA for the correct encoding order has not been difficult

What didn't go well

• We reviewed Certificate Profile Updates after the publication of BR 2.0.0 but we didn't detect the Subject Attribute Encoding Order requirements
• ZLINT, in use by Actalis, did not check the Subject Attribute Encoding Order; even the additional linter internally developed by Actalis didn't detect the issue
• The default ordering applied by EJBCA has not been double-checked

Where we got lucky

• Only a limited number of certificates have been impacted by the bug, most of them for internal use

Action Items

| Action Item | Kind | Status | Due Date |
| Revocation of all affected certificates | Mitigate | Complete | 2024-03-15 |
| Development of a new lint (PR#813) in ZLINT project for the "relative order" prescribed by CABF BR section 7.1.4.2 Subject Attribute Encoding | Prevent/Detect | Complete | 2024-03-10 |
| Implement an internal lint for the "relative order" prescribed by CABF BR section 7.1.4.2 Subject Attribute Encoding in our internally developed linter | Prevent/Detect | Complete | 2024-03-10 |
| Increased participation and monitoring of Bugzilla CA program (in order to dectect issues occurred) and [e-mail] group | Detect/ Prevent | In progress | 2024-04-30 |
| Strengthened internal proccess for monitoring applicable regulations and defining internal requirements for compliance | Prevent | In progress | 2024-04-30 |

Appendix

Details of affected certificates

Affected certficates

Comment 8

[Marco Menonna]

At the moment, we have no further updates on this bug.

Comment 9

[Wayne]

Your action items have a due date up to 2024-04-30, is there an update?

Comment 10

[Marco Menonna]

Here I am with an update of the implementation of the actions outlined in our previous incident report. All the actions have been successfully completed:

Increased participation and monitoring of the Bugzilla "CA Program" and [e-mail] group.

We set up an automated task has been established to ensure systematic checks are completed and all pertinent outputs are internally summarized and analyzed.

Strengthened internal processes for monitoring applicable regulations and defining internal requirements for compliance.

Our internal processes for monitoring applicable regulations and defining internal compliance requirements have been comprehensively strengthened and a dedicated team has been tasked with continuously monitoring regulatory updates and assessing their impact on our operations.

If you have any questions or require further details, please feel free to post your queries.

Comment 11

[Marco Menonna]

We don't foresee any further actions on our part and we request that this bug be closed unless there are additional questions.

Comment 12

[Ben Wilson]

I will close this on Wed. 12-June-2024 unless there are any questions or comments.

Comment 13

[amir]

Just because this can be useful for the plethora of other CAs that have been struggling with keeping up with Bugzilla, CCADB, MDSP, can you expand more on how

We set up an automated task has been established to ensure systematic checks are completed and all pertinent outputs are internally summarized and analyzed.

This works? How does the work get discovered, assigned, etc?

Comment 14

[Marco Menonna]

This is an additional component of our existing regulatory daily monitoring activity that already checks for working group ballots and the release of new BR and Browser policies. This new control enhances our ability to effectively track ongoing discussions and identify open issues reported by other Certificate Authorities, facilitating information sharing.
Our findings are formalized, and significant elements are forwarded to the relevant regulatory departments to address potential internal dissemination or further investigation needs. This activity is integrated into an already structured and existing process, enhancing and expanding its benefits and impacts.

Comment 15

[Ben Wilson]

I will close this sometime next week (June 17-21).

Comment 16

[Wayne]

(In reply to Ryan Dickson from comment #5)

• First, thank you for contributing to the ZLint project. This is really helpful. I hope contributions like this become considered “good practice" in response to future mis-issuance incidents, as it decreases the likelihood of other members of the ecosystem being affected by the same issue.

I'd like to echo this sentiment and also raise a question regarding the Root Cause of RDN mis-ordering itself and the underlying mis-issuance. The report states this is due to an error within EJCBA itself, so I'd expect more CAs impacted if they were linting.

Do we have a more definitive source of truth of element order that the linter relies on:

expectedRDNOrder := []string{"DC", "C", "ST", "L", "postalCode", "street", "O", "SN", "givenName", "OU", "CN"}

As after looking around I've already submitted another CPR to a different CA for hitting the error on this linter. Looking at the BR itself though I'm a bit puzzled at how you could take other elements such as jurisdictionCountry and apply them in practice such as the above.

I understand this comes from X.500 and X.520 processes focusing on Big->Small and that each batch are taken in from 7.1.4.2 but only sorted within their own elements (i.e. 7.1.4.2.6 here does C->ST->L->postalCode->street). Are we ordering based off of OID?

The reason I ask is that looking at 7.1.4.2.4 in censys shows that jurisdictionCountryName (A) vs jurisdictionStateOrProvince (B):
A->B: 77,297
B->A: 46

Likewise with jurisdictionCountryName (A) vs jurisdictionLocality (B):
A->B: 4228
B->A: 43

I would presume that the above would be incidents raised for TWCA, SSL.com and E-Tugra as a basic example? The lack of such a basis is why I'm not sending reports and conferring here.

Comment 17

[chtsai]

(In reply to Wayne from comment #16)

(In reply to Ryan Dickson from comment #5)

• First, thank you for contributing to the ZLint project. This is really helpful. I hope contributions like this become considered “good practice" in response to future mis-issuance incidents, as it decreases the likelihood of other members of the ecosystem being affected by the same issue.

I'd like to echo this sentiment and also raise a question regarding the Root Cause of RDN mis-ordering itself and the underlying mis-issuance. The report states this is due to an error within EJCBA itself, so I'd expect more CAs impacted if they were linting.

Do we have a more definitive source of truth of element order that the linter relies on:

expectedRDNOrder := []string{"DC", "C", "ST", "L", "postalCode", "street", "O", "SN", "givenName", "OU", "CN"}

As after looking around I've already submitted another CPR to a different CA for hitting the error on this linter. Looking at the BR itself though I'm a bit puzzled at how you could take other elements such as jurisdictionCountry and apply them in practice such as the above.

I understand this comes from X.500 and X.520 processes focusing on Big->Small and that each batch are taken in from 7.1.4.2 but only sorted within their own elements (i.e. 7.1.4.2.6 here does C->ST->L->postalCode->street). Are we ordering based off of OID?

The reason I ask is that looking at 7.1.4.2.4 in censys shows that jurisdictionCountryName (A) vs jurisdictionStateOrProvince (B):
A->B: 77,297
B->A: 46

Likewise with jurisdictionCountryName (A) vs jurisdictionLocality (B):
A->B: 4228
B->A: 43

I would presume that the above would be incidents raised for TWCA, SSL.com and E-Tugra as a basic example? The lack of such a basis is why I'm not sending reports and conferring here.

We hereby respond to indicate that we are also concerned and are researching this issue.

ChyaHung
TWCA

Comment 18

[Martijn Katerbarg]

(In reply to Wayne from comment #16)

Do we have a more definitive source of truth of element order that the linter relies on:

expectedRDNOrder := []string{"DC", "C", "ST", "L", "postalCode", "street", "O", "SN", "givenName", "OU", "CN"}

TLS BR Section 7.1.4.2 has a table for this (emphesis mine): "Table: Encoding and Order Requirements for Selected Attributes", which this linter seems to have been built against.

The same section goes on about some other fields, such as jurisdictionCountry, for which no order requirements are stated.

Comment 19

[Wayne]

Ah that'd explain that. It is interesting that a choice was made to prescribe specific ordering for only part of the Subject's DN but not cover the full list. As it stands the DN is a bit of a mess all around, but that's outside of the scope of this incident.

Comment 20

[Rebecca Kelley]

We agree that there is no issue as the EVGs do not require a specific ordering of the JOI fields. As a result SSL.com is not affected by the original argument proposed in this bug.

Comment 21

[Ben Wilson]

I am closing this bug. We can re-open it if necessary.