1609501 - WISeKey: Typo in Organization field

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Pedro Fuentes]

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_2) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0.4 Safari/605.1.15

Steps to reproduce:

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.
   15.01.2020 - 04:25 CET: We received a mail sent to our notifications address [e-mail]
   This mail made us aware of a "misuse" of certain certificates. This was initially confusing until we figured out it could be about a "misissuance".

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.
   15.01.2020 - 04:25 CET: The notification is sent, making reference to a list of certificates found by this query: https://crt.sh/?Identity=Allscritps&iCAID=124836
   15.01.2020 - 07:30 CET: After a first analysis, no apparent problem is found
   15.01.2020 - 16:00 CET: After a second analysis, it's found that the "Organization" of these certificates had a typo, as where it said "Allscritps" it should say "Allscripts"
   15.01.2020 - 16:15 CET: The customer is notified about the problem and about the need to replace their certificates. At that point the reason of the typo is unclear
   15.01.2020 - 17:30 CET: After a detailed analysis, it's found the source of the typo (See Item #6 for details)
   15.01.2020 - 18:00 CET: New certificates have been issued and sent to the customer to be replaced ASAP and agreeing to revoke the wrong certificates within the least delay

3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.
   The problem originating the typo in the Organization field has been solved and the necessary steps to avoid similar issues have been defined

4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.
   The query used to list the problematic certificates is https://crt.sh/?Identity=Allscritps&iCAID=124836
   This list informs of 10 certificates issued for the same customer, of which there are 2 repetitions of Precertificate/Leaf-Certificate, so there are 8 certificates that show the typo. Of these 8, one is expired and doesn't require further action

5. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.
   https://crt.sh/?id=2209977337
   https://crt.sh/?id=2149916576
   https://crt.sh/?id=2137410061
   https://crt.sh/?id=2137407066
   https://crt.sh/?id=2151331473
   https://crt.sh/?id=2074890791
   https://crt.sh/?id=1864166725
   https://crt.sh/?id=1790623073 (EXPIRED)

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
   This particular customer is using a MPKI interface that allows him to create certificates for a list of authorized and validated domains. When processing a request, the platform replaces some values of the CSR to ensure consistency in certain fields, like is the case of the "Organization" field.
   Unfortunately, a mistake was done when entering the fixed string to replace the Organization field, introducing the typo that would be replicated in a number of issuances.
   Despite of several tests and validations at WISeKey and by the customer, the problem was not detected, as all the involved people just misread the value without noticing the inverted letters.

7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.
   We have corrected the fixed string in the account configuration and reviewed that there weren't similar cases for other customers.
   The setting of these fixed values to override the CSR data implies a manual typing that can't be automated, but we are introducing a second validation of a different person once an account is configured in the MPKI interface.

Comment 1

[Wayne Thayer]

Pedro: thank you for this incident report.

The setting of these fixed values to override the CSR data implies a manual typing that can't be automated

Will you please explain this statement? I'm not certain that I understand.

I presume that the organization name is verified with some online data source? If so, is there a way to determine that the string in the account doesn't match?

Also, please update this bug when these certificates have been revoked.

Comment 2

[Pedro Fuentes]

(In reply to Wayne Thayer [:wayne] from comment #1)

The setting of these fixed values to override the CSR data implies a manual typing that can't be automated

Will you please explain this statement? I'm not certain that I understand.

I presume that the organization name is verified with some online data source? If so, is there a way to determine that the string in the account doesn't match?

Also, please update this bug when these certificates have been revoked.

Hello Wayne,
We always do Organization name validations using online services like the ones provided by the chambers of commerce.
In this particular type of customers that use the Managed PKI interface, we do the previous validations on the company and its domains and then we configure its account in the MPKI platform, to restrict the domains and certain fields of the CSR, so in ulterior certificate requests our platform will only allow the prevalidated domains and will override, for example, the Organization value coming in a CSR to ensure that it matches the value that was validated.
I refer above as « manual typing «  to the configuration of these details, and it’s here where we introduced the typo.
its the first time it happens, but we modified the procedure to ensure an additional pair of eyes to review the configuration of a new account in the MPKI platform.

The customer already has the new certificates and he’s proceeding to the replacement, but they told us that for some certificates the deadline of 5 days is too tight.

Comment 3

[Wayne Thayer]

(In reply to Pedro Fuentes from comment #2)

Hello Wayne,
We always do Organization name validations using online services like the ones provided by the chambers of commerce.
In this particular type of customers that use the Managed PKI interface, we do the previous validations on the company and its domains and then we configure its account in the MPKI platform, to restrict the domains and certain fields of the CSR, so in ulterior certificate requests our platform will only allow the prevalidated domains and will override, for example, the Organization value coming in a CSR to ensure that it matches the value that was validated.
I refer above as « manual typing «  to the configuration of these details, and it’s here where we introduced the typo.

Can the MPKI platform compare the Organization value with an online service in order to prevent this type of problem?

its the first time it happens, but we modified the procedure to ensure an additional pair of eyes to review the configuration of a new account in the MPKI platform.

The customer already has the new certificates and he’s proceeding to the replacement, but they told us that for some certificates the deadline of 5 days is too tight.

Per Mozilla's incident reporting guidance, please submit a separate incident report covering the failure to meet the revocation deadline.

Comment 4

[Pedro Fuentes]

(In reply to Wayne Thayer [:wayne] from comment #3)

Can the MPKI platform compare the Organization value with an online service in order to prevent this type of problem?

The MPKI platform will only allow the issuance on the pre-validated organization value, overriding it if required. As I tried to explain, we introduced a typo when configuring this setting in the platform.

Per Mozilla's incident reporting guidance, please submit a separate incident report covering the failure to meet the revocation deadline.

We hope this is not required, as the replacement and revocation is already on going, but we’d have to do that if Monday the certificates aren’t revoked. The customer insisted to get an extension of the deadline, but we are forcing to keep it.

Comment 5

[Pedro Fuentes]

Dear all,
as per today, 6 out of the 7 certificates have been revoked.
The one that is missing is https://crt.sh/?id=1864166725

The customer needs to involve an external contractor to replace the certificate and this takes some time. Due to having a weekend and a bank holiday for the US in the middle of the 5-day deadline, they requested us some more time to do the replacement.

As this certificate is used in internal communications that are important for the customer operations, we don't consider adequate to force a revocation and create a disruption in the customer, so we will give them some more time to replace the certificate.

We assume that this implies a failure to meet the revocation deadline and we will open the necessary incident report once the last certificate is replaced.

Comment 6

[Ryan Sleevi]

Pedro: Please open the new incident report sooner than later. The procedures outlined in https://wiki.mozilla.org/CA/Responding_To_An_Incident#Revocation are meant to ensure the public has the full information as soon as possible, and particularly, to discourage CAs providing incidents once they've completed the revocation.

In particular, please refer to the following:

If your CA will not be revoking the certificates within the time period required by the BRs, our expectations are that:

• The decision and rationale for delaying revocation will be disclosed to Mozilla in the form of a preliminary incident report immediately; preferably before the BR mandated revocation deadline. The rationale must include an explanation for why the situation is exceptional. Responses similar to “we deem this misissuance not to be a security risk” are not acceptable. When revocation is delayed at the request of specific Subscribers, the rationale must be provided on a per-Subscriber basis.

Comment 7

[Pedro Fuentes]

(In reply to Ryan Sleevi from comment #6)

Pedro: Please open the new incident report sooner than later. The procedures outlined in https://wiki.mozilla.org/CA/Responding_To_An_Incident#Revocation are meant to ensure the public has the full information as soon as possible, and particularly, to discourage CAs providing incidents once they've completed the revocation.

Hello Ryan. I posted the additional incident report here: https://bugzilla.mozilla.org/show_bug.cgi?id=1610767

Comment 8

[Ryan Sleevi]

Wayne: If I'm understanding your Comment #3 correctly, you're wanting to know why the pre-validated organization value can't come directly from the information source (e.g. QGIS/QIIS) directly, rather than require human entry, is that correct? If so, I'm not sure if Comment #4 fully answers that, but I wasn't sure if you wanted to reframe.

I think it's great to ignore the organization value within the CSR, and to make sure it's tied directly back to the validated organization field. That said, it seems ideal to make sure that the validation organization field comes directly from the data source itself. For example, human transcription can involve situations like typos, homoglyphs, transliterations, or other issues that can impact accuracy. I believe Wayne is suggesting that second part, and wanting to understand the challenges involved.

Comment 9

[Wayne Thayer]

(In reply to Ryan Sleevi from comment #8)

Wayne: If I'm understanding your Comment #3 correctly, you're wanting to know why the pre-validated organization value can't come directly from the information source (e.g. QGIS/QIIS) directly, rather than require human entry, is that correct? If so, I'm not sure if Comment #4 fully answers that, but I wasn't sure if you wanted to reframe.

Correct, my question is about populating and/or validating the pre-validated Organization information programmatically. Pedro, will you please explain why that's not possible?

Comment 10

[Pedro Fuentes]

(In reply to Wayne Thayer [:wayne] from comment #9)
Pedro, will you please explain why that's not possible?

Hello Wayne. Well, I didn't say that it's not possible, I said that this is not how our platform works.

There are two reasons that made us leave it like this:

1. Lack of standardization on how this information is made available across the world. Given that we only have a very low number of customers using MPKI for SSL, in different countries, we can't find an efficient approach to have something fully automated.
2. Security architecture. The MPKI administration interface is accessed on a restricted environment where connectivity is kept to a minimum.

Frankly speaking, I haven't seen yet any MPKI platform that fully automates this, but the ones I saw had the same approach of automating some validations (i.e. domain validations) and making other validations offline (e.g. Organization name) and then having an administrator entering some settings manually.

For now, to mitigate the risk we introduce a dual person validation in the process, and I don't fully eliminate the possibility of introducing further automations in the future.

Comment 11

[Pedro Fuentes]

Hello,
I'd like to inform you that the last certificate has been revoked after being safely replaced by the customer.
Please let us know if further actions are required.
Thanks,
Pedro

Comment 12

[Wayne Thayer]

Pedro: thank you for your response in comment #10.

it appears that all questions have been answered and remediation is complete.