1860750 - SwissSign: EV code in JurisdiktionStateOrProvinceName

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[raffaela.achermann]

Attachment: SwissSign EV JoiST.csv

Incident Report

Summary

An internal employee raised a question about the content of the EV juristictionStateOrProvinceName field. This question was reviewed by our product manager and he realized that we use the ISO 3166-2 code for this field instead of the full name of the State or Province as required by EV Guidelines chapter 9.2.4. The product manager then started the internal compliance incident process.

Impact

The mis-issued certificates are clearly not compliant to the EV Guidelines. However, we assign a low risk to the ecosystem by these mis-issued certificates because to the best of our knowledge, certificate consumers generally do not consider/evaluate the EV jurisdiction fields. Additionally, we assume that the ISO 3166-2 code is potentially better defined than the ISO 3166-2 full name (some states have several full names depending on the language).

Timeline

2023-10-20:

• 12:30 UTC A SwissSign internal employee raises a question about EV juristictionStateOrProvince Name field

2023-10-23:

• 05:30 UTC Product manager reviews question and discovers non-compliance to EV Guidelines 9.2.4, compliance incident raised
• 06:15 UTC Compliance team confirms non-compliance and gives order to stop issuance of EV certificates and started root cause analysis
• 07:00 UTC List of affected certificates compiled
• 07:15 UTC Start informing affected customers, Audit body informed
• 08:30 UTC Root cause identified, developers identify quick emergency fix
• 09:45 UTC Emergency fix reviewed, tested and deployed
• 09:15 UTC Issuance of EV certificates resumed

2023-10-24:

• 08:50 UTC CPR TLS updated and published
• 09:50 UTC Bugzilla posted

Root Cause Analysis

As part of the mitigation of Bugzilla https://bugzilla.mozilla.org/show_bug.cgi?id=1613334 we introduced the use of ISO 3166-1/2 codes for country and state fields (see comment 15 https://bugzilla.mozilla.org/show_bug.cgi?id=1613334#c15).
As we aim to standardize as much as possible, we used the ISO codes in all fields and updated our public documents accordingly. See https://repository.swisssign.com/SwissSign_CPR_TLS_R06.pdf (current Version 7.0 is already corrected) chapter 3.3.1/.2 where we allowed names and codes. However, the EV guidelines explicitly demand names for the 'jurisdictionStateOrProvinceName' field and not codes. This change happened in 2020-06-05 (time of comment 15 in the referenced Bugzilla).

Lessons Learned

What went well

• Internal incident reporting, analysis of the root cause and remediation worked well

What didn't go well

• It is unclear how such an oversight could go undetected for such a long time

Where we got lucky

• The fix was easy (change table reference), so we could re-enable EV issuance quickly

Action Items

Action Item	Kind	Due Date
1. Revocation of all affected EV certificates: Customers are currently being notified to replace the affected certificates by 27 October 2023 23:59 (CEST) latest. SwissSign will revoke any still valid affected EV certificates latest on 2023-10-28 06:00 UTC	Reactive	2023-10-28 06:00 UTC
2. The emergency fix consists of referencing the name of the state/province instead of its code for all new customers and replacing the existing codes with their name for existing customers.	Reactive	Done
3. Adjustment CPR EV chapter 3.3.2 / 3.3.1 https://repository.swisssign.com/SwissSign_CPR_TLS.pdf	Reactive	Done
4. Additional consideration: We are considering to put the EV requirement that jurisdictionStateOrProvinceName must be the full name to discussion in the CA/B server certificate working group. It may actually be desirable to allow ISO 3166-2 codes in the same way that jurisdictionCountryName is the ISO code of the country.	Preventive	n/a

Appendix

Details of affected certificates

In the attachement are only the valid certificates listed. If necessary, we provide lists with revoked and expired certificates.

Comment 1

[Dimitris Zacharopoulos]

I'm not sure if 9.2.4 of the EV Guidelines has such a strict requirement for the JOIstateOrProvinceName value. I have not searched CENSYS but perhaps a simple search could reveal a variety of values CAs use for that field to represent US states.

Comment 2

[raffaela.achermann]

EV Guidelines 9.2.4, end of second last paragraph states:

"State or province or locality information (where applicable) for the Subject’s Jurisdiction of Incorporation or Registration MUST be specified using the full name of the applicable jurisdiction."
We believe that this leaves no room for codes or abbreviations.

Comment 3

[Dimitris Zacharopoulos]

Thank you Raffaela, indeed, this language has been in the EV Guidelines since the very early versions. It might actually make sense to consider allowing known abbreviations that are documented in an ISO document. We can discuss further within the CA/B Forum's SCWG.

Comment 4

[raffaela.achermann]

Thank you Dimitris, we are of the same opinion and support this very much. See our action item number 4.

Comment 5

[Roman Fischer]

Created additional Bugzilla due to delayed revocation of 40 certificates: https://bugzilla.mozilla.org/show_bug.cgi?id=1861682

Comment 6

[Ryan Dickson]

Hi SwissSign, thank you for the report. A few initial comments:

Request for update #1: The Impact section should be improved by including the total count of certificates impacted (e.g., “The Impact section should contain a short description of the size and nature of the incident.” as detailed on the CCADB incident report page).

Request for update #2: The Actions Items list should be updated to minimally include a Prevent, Mitigate, or Detect type of action for each thing that did not go well. Ideally, at least one of each of these three types is included in the list of action items, as stated on the CCADB incident report page. Specific to this report:

• "reactive" is not in the set of response classifications described on CCADB.org
• there is no action addressing the one item listed as “What didn’t go well”: "It is unclear how such an oversight could go undetected for such a long time."

Question: It seems SwissSign has not adopted a technical control that would have prevented certificates that included an invalid juristictionStateOrProvinceName value (i.e., using the ISO 3166-2 code instead of the “full name" requirements described in the EV Guidelines) from being issued. Even if we assume the EV Guidelines are updated to instead adopt the ISO 3166-2 codes, it still seems like a technical control would be worthwhile (i.e., to ensure intended issuance is consistent with the set of permitted values). Can you describe whether SwissSign has considered a preventative technical control, like linting, to avoid repeating this mistake in the future? If SwissSign has decided against adopting a practice such as this, can you describe why?

Thanks,
Ryan

Comment 7

[raffaela.achermann]

Dear Ryan

Thank you for your comments, so we can improve our incident reporting. Below our updates:

Re "Request for update #1":

Impact

Since 2020-06-05 we issued in total 6176 certificates with ISO 3166-2 in the EV jurisdiction fields. At the time of the posting this bugzilla (2023-10-20) there were 2505 valid certificates. These are the certificates in the attached CSV.

Re "Request for update #2":

Action Items

Action Item	Kind	Due Date
1. Revocation of all affected EV certificates: Customers are currently being notified to replace the affected certificates by 27 October 2023 23:59 (CEST) latest. SwissSign will revoke any still valid affected EV certificates latest on 2023-10-28 06:00 UTC	Mitigate	2023-10-28 06:00 UTC
2. The emergency fix consists of referencing the name of the state/province instead of its code for all new customers and replacing the existing codes with their name for existing customers.	Prevent	Done
3. Adjustment CPR EV chapter 3.3.2 / 3.3.1 https://repository.swisssign.com/SwissSign_CPR_TLS.pdf	Mitigate	Done
4. Additional consideration: We are considering to put the EV requirement that jurisdictionStateOrProvinceName must be the full name to discussion in the CA/B server certificate working group. It may actually be desirable to allow ISO 3166-2 codes in the same way that jurisdictionCountryName is the ISO code of the country.	Prevent	n/a

Lessons Learned

What didn't go well

When we implemented the change to use ISO 3316-1/2 codes in 2020 the main goal was to standardize certificate contents as much as possible. We probably were kind of "blinded" by the consistent and efficient solution with the ISO codes that we didn't realize it was in violation of the EV guidelines.

This change was also reviewed and tested by our audit body (see Bugzilla 1613334). Additionally we documented in Bugzilla, comment 19 that we were going to use this setup for all relevant processes which included the EV 'jurisdictionStateOrProvinceName' field.

Even though multiple pairs of eyes looked at it, nobody realized the violation of the EV guideline we introduced.

Since that change in 2020, there was no indication that the 'jurisdictionStateOrProvinceName' fields does not contain the correct values and there was no change in this part of the regulation since then. Therefore, we have not checked the 'jurisdictionStateOrProvinceName' fields in the certificates separately.
If we had implemented an internal technical control / linter, this linter would had checked for the ISO-codes we implemented because we thought we were doing the right thing. So such a linter wouldn't have helped.
We believe that only a public linter (not developed by us) would have detected this, but no such linter was / is available.

As briefly described in action item #2 in the report above, we use database tables to store the allowed values for JOI fields (both the codes and the full names). These tables are the "single source of truth" for what gets written in the respective certificate fields. We have strict dual-control on these tables. An additional linter would take the same table as input and compare the certificate fields against it. But that would in essence be a control that controls itself and we don't think it would make our system more robust or effectively prevent mis-issuances of such a kind.

Update

Action Item	Kind	Due Date
1. Revocation of all affected EV certificates: Customers are currently being notified to replace the affected certificates by 27 October 2023 23:59 (CEST) latest. SwissSign will revoke any still valid affected EV certificates latest on 2023-10-28 06:00 UTC	Mitigate	Done

We revoked all certifcates except the ones metioned in Bugzilla 1861682.

Comment 8

[raffaela.achermann]

For this Bugzilla there is no further action planned on our side. All remaining tasks are handled in Bugzilla 1861682 .
If there are no further questions from the community, we would ask to close this Bugzilla.

Comment 9

[Ben Wilson]

I will close this on Wed. 2023-11-08, unless further discussion is needed.