Closed Bug 1613334 Opened 1 year ago Closed 10 months ago

SwissSign: Misissuance with mispellings in Location for a number of Certificates

Categories

(NSS :: CA Certificate Compliance, task)

task
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: nathalie.weiler, Assigned: michael.guenther)

References

Details

(Whiteboard: [ca-compliance] Next Update 31-July 2020)

Attachments

(1 file)

26.11 KB, application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
Details

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.18363

Steps to reproduce:

  1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.
    In the verification of all our issued certificates, the auditor made us aware of issues with some certificates. The last certificates were delivered for analysis on November 14, 2019. The subsequent analysis revealed that we have misissued certificates with the wrong location abbreviation reps. with misspellings in the location (entry did not match the verified input data).

  2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.
    Dec 11, 2019: The misissue on the exact count of certificates was acknowledged and the revocation triggered.
    Dec 13, 2019: Process for implementation of check was agreed and planned for delivery in early 2020; manual intermediate process was agreed internally
    Dec 16, 2019: Revocation was performed by SwissSign
    Jan 27, 2020: Revocation of certificates on critical infrastructure (hospitals, police, other emergency services).
    Feb 05, 2020: Posting of this report

  3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.
    Manual process for avoiding misissuance is in place as of Dec 13, 2019.

  4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.
    47 SSL certificates
    113 S/MIME certificates

  5. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each
    certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.
    SSL certificates list:
    02:D4:07:C9:66:FC:2F:54:EC:F7:9C:97:3C:3A:F5:91:EF:48:79:39
    10:23:d9:3c:dd:00:60:92:1f:ef:8c:83:0a:ae:52:2a:08:10:88:f4
    11:b9:f7:f4:e9:56:f2:bf:21:37:e0:19:7c:5c:e9:1d:bc:1c:05:91
    12:7d:dc:ec:51:77:64:c5:d6:e9:40:d5:84:e3:18:54:6a:d2:7a:1e
    13:92:da:ee:71:c5:18:4f:9a:a4:fc:e8:e1:32:98:79:9c:27:e5:24
    15:cc:f7:81:d6:ae:f8:6d:75:d0:19:a5:64:38:bb:81:39:25:8f:f8
    16:ea:d7:58:39:f3:b2:f1:5e:69:e9:81:1a:4f:39:97:86:28:be:ad
    18:3f:80:97:81:2c:d8:b4:df:77:93:e0:ac:75:15:af:c7:45:c6:e7
    18:7d:9d:cd:8b:b7:c4:37:b9:b9:e9:67:08:1f:3b:b4:69:09:d4:93
    18:f6:3e:49:24:b8:73:b8:04:ea:db:c8:7d:45:2d:eb:7a:0c:12:80
    19:0f:34:32:3c:49:21:f4:43:bf:a6:82:41:2d:86:58:22:05:99:3c
    1b:33:f7:ab:06:a2:d6:6a:7c:d7:52:f5:36:37:88:33:fd:47:db:99
    1f:bd:e7:df:9c:70:4b:0d:22:ae:ca:01:15:7e:36:17:14:a0:5a:ad
    20:e5:56:31:66:a3:6f:04:fe:b2:cf:42:77:81:5b:4e:5b:3a:42:55
    24:bb:96:55:30:4b:3a:3c:ca:12:97:09:ed:12:69:d5:6b:37:47:e
    29:25:fc:a3:9f:51:58:78:2f:59:0b:0d:d2:d0:6e:53:03:79:6e:c2
    29:e7:b8:ce:f8:43:04:c1:cf:7d:02:b3:0c:b3:bc:a8:36:44:4b:23
    29:e9:76:72:2e:b6:f2:5a:b3:cc:c4:58:ff:b4:cc:13:e0:f1:3d:4e
    32:33:a8:41:b5:83:16:10:97:9f:6c:e2:c5:f8:1f:96:a5:a3:7b:be
    32:a6:64:2a:b3:6f:1a:e8:2a:44:30:66:0d:bd:7f:6f:b9:95:17:c6
    35:a0:34:f6:b6:6f:35:3b:99:c4:63:1c:1c:58:2a:7b:ba:0e:27:de
    35:d9:39:2b:8d:09:54:26:55:7d:92:52:58:ef:a9:6d:6b:9a:b6:0a
    38:BC:97:66:72:96:0C:96:4D:FE:A2:DE:1A:80:B1:C8:59:29:4F:FD
    39:0a:b8:f2:59:66:af:3c:70:85:b8:f9:a6:9e:8a:7f:59:2f:42:f1
    3a:3c:3d:34:56:ed:ba:df:03:ce:ae:8b:dc:d4:74:29:21:42:d9:cc
    3a:8d:30:fd:0e:06:9c:cf:a2:78:dd:66:0b:6f:bc:ca:71:30:0b:b6
    3b:6f:4c:56:6b:b2:3b:e9:c1:e9:27:5b:5c:38:7e:23:ed:4c:0a:6f
    41:d2:36:50:5e:e0:43:23:9d:9c:fc:f4:71:de:d0:1c:08:8e:c5:e4
    4a:4b:59:37:95:b5:a7:b7:ca:93:92:d1:ab:d0:a7:d4:25:14:0e:a7
    4b:51:ef:4e:95:f8:fe:f9:45:47:8c:5f:5e:18:3a:71:49:23:57:5f
    50:76:ad:6f:42:63:f9:f9:7a:a4:f3:f5:0f:6c:7d:19:94:27:15:fc
    51:31:a4:9f:f2:7d:a9:16:9a:7e:c7:c2:f7:48:ef:76:e4:1d:59:2d
    56:48:ac:f9:97:25:fc:9b:2b:40:44:98:55:55:b6:06:13:24:8b:e2
    60:8e:9e:d0:14:d7:c8:8f:f8:0c:4e:c3:fa:2b:62:4d:c7:2e:f9:3c
    65:2e:9f:7b:f0:2b:2a:ea:f1:38:11:d7:10:e3:03:f2:d3:5e:bb:ef
    68:41:65:b1:ea:43:5f:06:11:de:16:41:30:20:03:f5:f1:1a:a4:90
    68:c3:b9:8a:c3:26:5a:58:d7:17:28:7e:c7:8e:31:fd:c6:08:78:40
    76:1f:ca:5f:20:b4:ac:b0:30:2b:32:a5:aa:37:b6:14:73:16:0a:3d
    77:41:75:ab:b5:e5:41:da:d6:b6:03:e6:86:f3:34:df:3a:ef:0b:5d
    7a:89:c8:d8:ae:51:8b:1c:06:f6:db:25:2a:e5:8a:c3:c4:24:db:ad
    7d:d0:93:94:7d:69:64:66:0a:39:61:e8:73:14:3d:1d:a4:e8:bb:a9
    83:C1:7B:75:66:B5:E9:35:88:A6:A1:26:1A:4C:07:6C:AC:BB:B6:9A
    86:39:AF:25:F9:02:7B:BB:26:11:4B:31:9B:D3:BD:C0:71:79:D8:3E
    90:C2:FD:3A:90:65:56:70:25:28:37:31:C7:91:AD:2F:79:E2:F5:6D
    AA:E2:C1:E8:F3:5A:44:FF:B7:DA:D8:A3:15:38:E4:EC:4B:4D:5C:F0
    FB:B9:F1:B1:94:F8:79:2D:3C:EC:19:29:14:94:C6:D8:26:3D:49:17
    FF:3B:5A:B9:CF:A2:D1:11:FA:79:C2:9E:00:DC:90:F1:8F:E5:2C:DF

S/MIME certificates not discloses publicly to adhere to data protection law in place.

  1. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
    The mistake went undiscovered because there was no automatic validation on the corresponding fields.

  2. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

Manual check has been put in place by Dec 13, 2019. The automatic check according to an official source for standard values will be implemented by end of March 2020 (e.g. ISO 3166-1&-2). The automatic check will be part of the audit in next summer.

Type: defect → task

Why is this only filed on February 5 and not somewhere between November 14 and December 11? Per https://wiki.mozilla.org/CA/Responding_To_An_Incident#Incident_Report incident reports are expected sooner rather than later, and "certainly within two weeks of the initial issue report".

Flags: needinfo?(nathalie.weiler)

We waited with the report until we got the certificates on critical infrastructure revoked (in total 21 certificates). The bulk of the certificates was revoked as mentioned within 5 days. The preliminary report was ready by that time, but not filed. This may have been a mistake from our side which I apologize for.

Flags: needinfo?(nathalie.weiler)

As Julien mentioned, the expectation is that a CA will file a report as soon as they know about the incident. These sorts of delays are not acceptable, and confusion about the policy is not reassuring.

Further, because you failed to abide by the BRs revocation timeline, a separate and additional bug must be opened for those certificates, detailing precisely why you delayed revocation, and what systemic steps are being taken so that SwissSign never delays revocation again.

While I can understand and appreciate that there has been a non-trivial amount of turnover with staff, this represents several serious issues:

  1. The failure to timely report
  2. The failure to timely revoke
  3. The failure to timely detect this, given how much discussion has happened via CA incidents and via m.d.s.p. regarding the correct spelling and data entry of locality information.

Further, the failure to minimally disclose the S/MIME hashes and serial numbers is not acceptable. This has been an issue in the past, and if data protection laws are used as a shield to prevent the disclosure of misissued certificates, then the only way to be confident in harm remediation is to remove trust in the CA. To avoid that, please disclose both the hashes and the issuer+SerialNumber tuples to allow verification of the revocation status of these S/MIME certificates and to verify the complete set SwissSign detected and remediated.

Assignee: wthayer → nathalie.weiler
Flags: needinfo?(nathalie.weiler)
Whiteboard: [ca-compliance]
Blocks: 1613406

I went and created Bug 1613406 for you to fill in.

Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true
Attached file smime_ids.xlsx
Flags: needinfo?(nathalie.weiler)

The failure to meet timely reporting will be further documented in the separate bug that Ryan Sleevi opened ( https://bugzilla.mozilla.org/show_bug.cgi?id=1613406).

In the attachment, we added for all S/MIME certificate the tuple (serial id, key id, revocation date) for transparency on revocation status. We would like to point out that we did not have any intention on instrumentalizing the privacy laws in place to not provide transparency on revocation status.

The failure to timely revoke of the critical certificates is due to the nature of the usage of these certificates: They are used for emergency services provided to the public. Due to the Christmas/New Year freeze, the providers of these critical infrastructures were not able to conduct changes before mid of January 2020. Revocation would have meant putting the related services at risk.
SwissSign made the judgement call for delaying revocation based on the current Mozilla root store policy (https://wiki.mozilla.org/CA/Responding_To_An_Incident#Revocation) stating:
Mozilla recognizes that in some exceptional circumstances, revoking misissued certificates within the prescribed deadline may cause significant harm, such as when the certificate is used in critical infrastructure and cannot be safely replaced prior to the revocation deadline, or when the volume of revocations in a short period of time would result in a large cumulative impact to the web.

The failure to timely detect the misspellings are related to not having yet in place an automatic check against a predefined setting for the locality field. To avoid this ambiguity in the future, we opted for the usage of an official source for standard values (e.g. ISO 3166-1&-2). This is the last part of a series of additional automated validations with we started during 2019 as a consequence of the public discussions around the "SomeState" and "DefaultCity" misissuance.

Please provide weekly updates on "automatic check according to an official source for standard values will be implemented by end of March 2020 (e.g. ISO 3166-1&-2)" in this bug.

Analysis of ISO 3166-2 (State Information) revealed ambiguities (states are not uniquely defined). Therefore, we are currently analyzing other alternatives such as the UN/LOCODE Code List by Country and Territory.

Thanks for the update!

Could you describe more about the problems/ambiguities you ran into? Understanding a bit more about the challenges you’re seeing helps us figure out opportunities to improve, as well as helps us better understand the challenges CAs are facing.

Flags: needinfo?(nathalie.weiler)

With regard to ambiguities, we noticed that the codes for some states are based on numbers. For example, for a state commonly known as Schaan in Liechtenstein the code would be 07 according to the ISO 3166. Also it is uniquely defined with the country indication, we think it will potentially introduce a new source of errors as people cannot relate to the numbering.
We are progressing fine with the UN/LOCODE list analysis.

Flags: needinfo?(nathalie.weiler)

I'm not sure I understand? While LI-07 is the code for Schaan within Lichtenstein, the ISO 3166 MA publishes the names as well; that is, that the entity LI-07 is named Schaan.

I totally agree that placing the value LI-07 would be confusing; but the 3166 MA publishes both, and I think the idea here was to make sure that the locality is represented within one of the names published by the 3166 MA.

That said, I also agree that for non-jurisdictional information, the use of UN/LOCODE seems reasonable, since that's necessarily going to include the internationally-recognized addressing schemes. I'm curious to know/learn more, though, if there are situations where the UN/LOCODE ends up diverging from the ISO 3166 named subdivisions.

Also, resetting N-I because I wasn't sure if Comment #10 was responding to Comment #7, meaning the UN/LOCODE list is implemented, or whether it's still being analyzed.

Flags: needinfo?(nathalie.weiler)

In the meantime, we further analysed both lists and decided to use ISO 3166-1 and -2 codes for country and state and the UN/LOCODE for localities.
We will also implement an exception process for some ISO 3166-2 shortcomings (at the moment Liechtenstein, where the code is alpha-numeric and the "subdivision name"in the standard corresponds to the localities).

Flags: needinfo?(nathalie.weiler)
Flags: needinfo?(wthayer)

Nathalie or Mike: please provide an update on the status of the work that SwissSign is doing to remediate this issue, and if the work is not complete, provide a timeline describing the milestones for completing the remaining work.

Flags: needinfo?(wthayer)
Flags: needinfo?(nathalie.weiler)
Flags: needinfo?(michael.guenther)

@Wayne: Please assign this ticket to me.

Currently the manual check (see comment 0) is still in place.

The automated checks are currently in our staging area and are working. We have a pre-assessment presentation for our auditors this week. Their feedback will be taken into account and then we will be able to present a final timeline.

I will update this ticket at the latest at the end of next week.

Flags: needinfo?(michael.guenther)
Assignee: nathalie.weiler → michael.guenther

After gathering the auditors feedback we now have a definite time schedule. The new automation which includes the ISO 3166-1 and -2 codes for country and state as well as the UN/LOCODE for localities and exception list for contradictory/inconclusive cases will be integrated with the next release in July 2020.
The new technical capabilities allow us to additionally go through all of our existing customer configuration once more. Therefore we will be able to close the cleanup at the latest on 31 July 2020.

Hi Mike,
Could you please provide a status update?
Thanks,
Ben

Flags: needinfo?(nathalie.weiler) → needinfo?(michael.guenther)
QA Contact: wthayer → bwilson
Whiteboard: [ca-compliance] → [ca-compliance] Next Update 15-July 2020

Ben, we are still on schedule. The release is fully tested and ready.
On Tuesday, 21 July 2020 we have the final demonstration for our auditors. The release into production will happen at the latest on Wednesday, 22 July 2020. At that point we will go through the existing customer configuration. We are confident that we will deliver in the committed time (31 July 2020).

Flags: needinfo?(michael.guenther)

Just a reminder that, in the absence of an update to Next Update, weekly updates are expected

For example, this would have addressed whether the release to production happened.

Flags: needinfo?(michael.guenther)
Whiteboard: [ca-compliance] Next Update 15-July 2020 → [ca-compliance] Next Update 31-July 2020

As of last Tuesday the release is life. We are using the ISO3166-1/2 and UN/LOCODE for all TSP relevant processes (see comment 15).
The legacy check is still ongoing. I will update this ticket next Wednesday.

Flags: needinfo?(michael.guenther)

The legacy check is done and has been reviewed & verified today. All mis-configurations have been corrected.

The following issues were detected:
- For Liechtenstein we had the state 'Oberland' configured
- For Liechtenstein we had the state 'Fürstentum Liechtenstein' configured (another customer)
- For Singapore there was a triplet C=Singapore, ST=Singapore, Location=Singapore. The ST is false
- For GB we had the state 'Berkshire' instead of 'WBK'
- For Germany we had 2 misspellings 'Donauschingen' instead of 'Donaueschingen' and 'Neustadt am Rübenberg' instead of 'Neustadt am Rübenberge'

In total 39 certificates are effected. These certificates are going to be revoked within the next 5 days.
Web-certificates
- https://crt.sh/?id=1770396279
- https://crt.sh/?id=3154887838
- https://crt.sh/?id=2393991708
- https://crt.sh/?id=1442302100
- https://crt.sh/?id=2410260909
- https://crt.sh/?id=2410253873
- https://crt.sh/?id=1695786087
- https://crt.sh/?id=1705420615

S/MIME
- 10D3261912223F7A1E9E6377582E58FC2B180131
- 10F2BF24D1B10456B0A7DBC46A90A9990D4D4F27
- 110C8440FC7F1048C7EBE16C6B030773BD67481C
- 11688D9BF875E488500A489B58ABE987E8C07422
- 11E90124A8BF8B1037FD70D2573F44B206433342
- 15EAC4BD52E412140E0DC9C50405B4DFD7642AAB
- 171D9FFB71680ABDB17EEA4E9E0DC8A5303D4DE1
- 1A79FE274DC402130E987BDA45473BBF854DFC8E
- 1B3D3AA40E741D916AAC1703A806B939279E61F7
- 265327580AB6704F90F9B25F77F5DBEF6FA308D3
- 295B8CDC8A4D6C91A90BA1EFB3160241E97BDDDF
- 303F35A4C5DDD9BA832844ABA2BC02125CA3F28F
- 328AEDB6511E34921CA005D89FAD86A612EE50A0
- 37A6FB4381939C58C372A6B6985231CA6EB8DF87
- 3ADC26A42BCCCC3FB3585CB1C843177594B9640C
- 431FF6C62819F5DE2B65A1D831F4F87504F25831
- 432B65F57B7633DECDC43A93C54E5C35CFFE0AE1
- 58CAAAAD3AA689F641A24216E9D83AA676F29C26
- 61476588789F736A190E363612573440A56E22B4
- 6260B7ED55D80BC7921A3CB0D5EA7CA6B291F92E
- 64FAC2280A505A1705690235AF3B8CF1F1EF4CA2
- 67E04D4E5F0A3F7F548B10560BC28788F0297242
- 6A9AF354ABC677B31D97247A7E1B5DD30DE9B796
- 6ADF7618CB04E49922CD38879F2ED4819A54FAF0
- 6CCF150D8F9D44871E23FE05D974499F2B231AAE
- 370790EC6852741F4A702356ABA53772A4B7B2C2
- 1ACC62C7CE2C82E0959B5B758B46EEF186055FE4
- 342B043401AC17B41CD847313D711ABDB39E71BF
- 44EF88B20D1E226CAF8AC2C81F35C4970BACB8F0
- 4CF7E89E9C19048B10C16D7D8E3352D679E4666B
- 68BE244B8DCA7DE877036E0413AC388A6AC57B5C

Next update is planned for Tuesday, 11 August 2020.

All the above certificates are revoked as of yesterday. As all planned improvements and tasks are in place/have been executed are there any other open questions. If not may I ask this Bugzilla to be closed. Thanks

Status: ASSIGNED → RESOLVED
Closed: 10 months ago
Resolution: --- → FIXED
See Also: → 1670894
You need to log in before you can comment on or make changes to this bug.