Closed Bug 1705647 Opened 3 years ago Closed 2 years ago

KIR S.A.: Invalid organizationName

Categories

(CA Program :: CA Certificate Compliance, task)

Product:
CA Program
Component:
CA Certificate Compliance
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: michel, Assigned: piotr.grabowski)

Details

(Whiteboard: [ca-compliance] [ov-misissuance])

Attachments

(1 file)

3881266052_ocsp.txt
6.88 KB, text/plain
Details
Attached file 3881266052_ocsp.txt —

Hello,
I found 2 certificates with the organizationName set to Bank Testowy:
https://crt.sh/?id=3667478749&opt=ocsp (Not Revoked)
https://crt.sh/?id=3881266052&opt=ocsp (CRL Revoked, OCSP Unknown)

Assignee: bwilson → piotr.grabowski
Status: NEW → ASSIGNED
Whiteboard: [ca-compliance]

https://crt.sh/?id=3881266052&opt=ocsp is CRL Revoked, OCSP Revoked
We will revoke tomorrow https://crt.sh/?id=3667478749&opt=ocsp

We were kindly requested to postpone the revocation of https://crt.sh/?id=3667478749&opt=ocsp for the next day.

It sounds like you will not be revoking in the time required.

Please read https://wiki.mozilla.org/CA/Responding_To_An_Incident#Revocation carefully for expectations.

Flags: needinfo?(piotr.grabowski)

It is already revoked (2021-04-21 08:18:00 UTC)
Although the certificate was installed in the very critical infrastructure, I think we managed to revoke it in the time required.
I was assigned to the bug 2021-04-21 08:09 PDT

Flags: needinfo?(piotr.grabowski)

so(In reply to Piotr Grabowski from comment #4)

It is already revoked (2021-04-21 08:18:00 UTC)
Although the certificate was installed in the very critical infrastructure, I think we managed to revoke it in the time required.
I was assigned to the bug 2021-04-21 08:09 PDT

typo above

It is already revoked (2021-04-21 08:18:00 UTC)
Although the certificate was installed in the very critical infrastructure, I think we managed to revoke it in the time required.
I was assigned to the bug 2021-04-16 08:09 PDT

It's been 17 days since this was filed, and still no incident report.

Have I overlooked anything?

Flags: needinfo?(piotr.grabowski)

You didn't overlook anything. I will report an incident here until Thursaday (06-05-2021)

Flags: needinfo?(piotr.grabowski)

Bug report:

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

KIR was informed by the third-party in the bug https://bugzilla.mozilla.org/show_bug.cgi?id=1705647. 2 certificates were found with the organizationName set to Bank Testowy. Proper value for this field should be:
Krajowa Izba Rozliczeniowa S.A. Both certificates are requested by KIR itself and are used in KIR S.A. systems.

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

2021-04-16 00:43 PDT https://bugzilla.mozilla.org/show_bug.cgi?id=1705647 bug was created.
2021-04-16 08:09 PDT KIR was assigned to this bug.
2021-04-21 01:35 PDT https://crt.sh/?id=3667478749&opt=ocsp certificate was revoked.
2021-04-24 10:00 CEST Full scan was performed to check if there can be more certificates affected. No more certifcates were found with similar problem.

3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

KIR has stopped issuing certificates with the problem.

4. summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

https://crt.sh/?id=3667478749&opt=ocsp
https://crt.sh/?id=3881266052&opt=ocsp

5. The complete certificate data for the problematic certificates.

https://crt.sh/?id=3667478749&opt=ocsp
https://crt.sh/?id=3881266052&opt=ocsp

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

According to the internal procedures of KIR, the operator was obliged to reject such a request and ask for a new one to be prepared with correct data in the Organization (O) field. Correct data for this field in these 2 certificates are: Krajowa Izba Rozliczeniowa S.A. The current procedures required only single operator verification/validation of orders and requests from KIR.
We felt that double checking for internal orders was unnecessary and it was the root cause of the problem. All other orders (coming not from KIR) and all other certificate requests are always verified by two operators.

7. List of steps CA is taking to resolve the situation and ensure it will not be repeated.

Full scan was performed to check if there can be more certificates affected. No more certifcates were found with similar problem. The orders and certificate requests comming from KIR itself are now subject to same rules
like requests from any external customer. No other exceptions exist in the processes of data verifications. The internal register has been created with forbidden field names (field blacklist). All requests will be validated against this register on 2 levels: operator and supervisor. If any of the new request will be rejected, the field value from the rejected request will be added to the field blacklist with specific key (organization in this case).
Cyclic scans will be executed as a post issaunce tests against said register. All KIR departments were informed about the issue and instructed how to prepare internal orders and requests.

We felt that double checking for internal orders was unnecessary and it was the root cause of the problem.

Why have you decided that it was unnecessary? Did you assume that internal orders will always be correct?

Flags: needinfo?(piotr.grabowski)

(In reply to Michel Le Bihan from comment #9)

We felt that double checking for internal orders was unnecessary and it was the root cause of the problem.

Why have you decided that it was unnecessary? Did you assume that internal orders will always be correct?

We thought that verification by one person will be enough, because we know our systems better than systems of other customers.

Flags: needinfo?(piotr.grabowski)

We thought that verification by one person will be enough, because we know our systems better than systems of other customers.

Could you please explain why knowing a system made you think that verification of data in the certificate request can be relaxed? How did check if the org listed in organizationName even exists? How did you know that the org to verify was Krajowa Izba Rozliczeniowa S.A.?

Flags: needinfo?(piotr.grabowski)

(In reply to Michel Le Bihan from comment #11)

We thought that verification by one person will be enough, because we know our systems better than systems of other customers.

Could you please explain why knowing a system made you think that verification of data in the certificate request can be relaxed? How did check if the org listed in organizationName even exists? How did you know that the org to verify was Krajowa Izba Rozliczeniowa S.A.?

Absolutely not relaxed but verified only by 1 operator.
We all know the systems operating at KIR and we know who can apply for certificates for them. The order contains data for the certificate for a given system (domain) and data of KIR and is signed - authorized by an employee of KIR responsible for the given system.

Flags: needinfo?(piotr.grabowski)

No specific update here. Blacklist register status is described in p.7 https://bugzilla.mozilla.org/show_bug.cgi?id=1705337#c4 ( "automation" project)

We all know the systems operating at KIR and we know who can apply for certificates for them. The order contains data for the certificate for a given system (domain) and data of KIR and is signed - authorized by an employee of KIR responsible for the given system.

In this case the data contained an unknown entity in the organizationName. I don't quite understand how you knew that it was related to KIR. Only because it was signed by a KIR employee? How does this confirm that the data (SAN and ON) in the certificate belongs to or is controlled by KIR?

Flags: needinfo?(piotr.grabowski)

Like Michel, I'm finding the incident report in Comment #8 difficult to understand how this issue happened, and how/why KIR S.A. is confident in the explanation.

I think a more detailed explanation about the issuance process is useful, and I think (although Michel, feel free to disagree if I'm wrong here) that Michel is also highlighting this in asking "but how do you know?"

When looking at the timeline (the answer to Question 2), we have zero understanding about how this request flowed through KIR S.A.'s systems, and that seems essential to understanding. When you talk about having one person verifying, it seems like when they verified should be part of the timeline here.

The Baseline Requirements require that the CA keep a detailed, write-once event log of all the relevant events for the issuance of a certificate, and that's what we're looking to understand with Question 2. I realize there are two certificates at play, and so that might seem like "twice as much information", but the goal here is to help us understand the entire flow of these certificates through KIR S.A.'s systems, so that we can better understand where things went wrong, as well as the opportunities to improve them.

Hello,
When can we expect an update on this?

Currently, orders for SSL certificates for KIR's infrastructure are carried out in the same way as any other order.

  1. KIR employee selects the product - SSL certificate in KIR e-shop.
  2. He enters the required data. One of the required section is ordering party data - in this case KIR's data (full name, tax number, address)
  3. Pays for the order (uses voucher code).
  4. Downloads the order available in the summary page on the website. The order contains the entered data and formal consents and declarations of the Ordering Party.
  5. Makes sure that the entered data are correct.
  6. Sends the order signed by persons entitled to represent the Ordering Party to the address provided in the summary of the order (electronically using qualified certificates or in paper form).
  7. While the order is fully verified by the operator, he gets an e-mail contact from KIR's operator to verify the domain with all instructions for placing authorization code in DNS-TXT or on the website.
  8. Confirms that he is under control of the domain indicated in the order by starting the separate process with flow controlled by change management process to validate KIR's domain (usually several departments involved).
    When the domain is validated he recieves an e-mail confirmation.
  9. Delivers the request to KIR (csr file) - personally to a selected branch of KIR or electronically signed with a qualified signature of the person authorized to deliver the request indicated in the order.
    CSR is verified and compared to the data in the order.
  10. Collect the certificate.

Based on the flow above we are always sure the order is for KIR system: KIR's data as ordering party, signed by person from KIR entitled to represent KIR. Domain name in the order can be also easily
correlated to specific KIR system. In these two cases we have: euroexecertsym.kir.pl and srpncertsym.kir.pl (kir.pl is official KIR's website). There is always internal communications
during this flow for support and awareness (dedicated mail group certyfikaty_szafir@kir.pl).

The problem was in point 9. We had the correct data in the order but wrong value in organizationName in th CSR which was not noticed by only one operator.

Flags: needinfo?(piotr.grabowski)

In what format is the data in the order? Can it be automatically compared with the data in the csr and a warning shown if it differs?

Flags: needinfo?(piotr.grabowski)

It's pdf format. It can be compared with the data in the csr and with customer data from out internal CRM. It's already the planned feature of our new 'automation' project which is in the process of implementation.

Flags: needinfo?(piotr.grabowski)

No update here.

Piotr: As with the other KIR S.A. bugs, I fail to see a binding timeline on the automation project.

It would appear you're using values from the CSR separate from the order, and that's long been flagged by CAs as a source of compliance issues. The decision to rely on manual review is by no means encouraging, and this incident report's root cause analysis seems strikingly similar to the "human error" discussion on https://wiki.mozilla.org/CA/Responding_To_An_Incident#Incident_Report

Can you also describe, precisely, what you scanned for? I'm concerned, given KIR S.A.'s repeated demonstrations of problematic understanding, that the scan that resulted in "No more certifcates were found with similar problem." may have simply scanned for "Bank Testowy" as opposed to actually looking at the certificates and data.

Given KIR S.A.'s issues like Bug 1705337, or other CAs' issues like Bug 1712188, that there's almost certainly a deeper, systemic risk in how KIR S.A. is controlling its data and scanning its certificates.

Flags: needinfo?(piotr.grabowski)

(In reply to Ryan Sleevi from comment #21)

Piotr: As with the other KIR S.A. bugs, I fail to see a binding timeline on the automation project.

  • CA software upgrade 10/2021
  • Changes in internal KIR software components - 11/2021
  • New intergation component - 12/2021
  • New hardware / printers - 12/2021
  • Testing i bug fixing - 01/2022
  • Documentation 02/2022
  • Production deployment 03/20212

It would appear you're using values from the CSR separate from the order, and that's long been flagged by CAs as a source of compliance issues. The decision to rely on manual review is by no means encouraging, and this incident report's root cause analysis seems strikingly similar to the "human error" discussion on https://wiki.mozilla.org/CA/Responding_To_An_Incident#Incident_Report

We are fully aware that manual (even double-check) review can be source of errors. That is why we will have this technical validation in place once the autmation project will go live.

Can you also describe, precisely, what you scanned for? I'm concerned, given KIR S.A.'s repeated demonstrations of problematic understanding, that the scan that resulted in "No more certifcates were found with similar problem." may have simply scanned for "Bank Testowy" as opposed to actually looking at the certificates and data.

We scanned certificates data against CRM data where fully verified customer and order data are stored. In the scan we compared orgranization value from CRM db and orgranization value from the certificate. Additionally we executed tests for specic values in certificates like for example 'Bank Testowy', 'Test' etc..

Given KIR S.A.'s issues like Bug 1705337, or other CAs' issues like Bug 1712188, that there's almost certainly a deeper, systemic risk in how KIR S.A. is controlling its data and scanning its certificates.

Flags: needinfo?(piotr.grabowski)

No update here . Are there any additional questions?

Whiteboard: [ca-compliance] → [ca-compliance] Next update 2021-09-01

Please provide an update now, and then I will set the "Next Update" so that you provide another update in a month from now (unless you can provide your own suggested "Next Update" along with supporting rationale for your recommendation).

Flags: needinfo?(piotr.grabowski)

Referring to the plan outlined in https://bugzilla.mozilla.org/show_bug.cgi?id=1705647#c22 we are in the process of upgrading CA software which should be completed by 10/2021. On the other hand, we are working on extracting this functionality from the automation project so that we can prioritize it and complete it faster than we intended. I think 15/10/2021 is the date when we will be able to provide more information on the progress.

Flags: needinfo?(piotr.grabowski)
Whiteboard: [ca-compliance] Next update 2021-09-01 → [ca-compliance] Next update 2021-10-15

We have fninshed the process of extraction functionality from the automation project to be able to fix the issue CSR being separated from the order much eariler than expeted Production deployment date 03/20212. We plan to deploy these updates by the end of 2021 together with ACME server as stated here https://bugzilla.mozilla.org/show_bug.cgi?id=1709872#c24 . The deployment date should be confirmed 15/11/2021.

We have confirmation that we will be able to complete development by the end of 2021, but production deployment is planned on 21/01/2022

Whiteboard: [ca-compliance] Next update 2021-10-15 → [ca-compliance] Next update 2022-01-05

Are you still on schedule for 21/01/2022?

Flags: needinfo?(piotr.grabowski)
Whiteboard: [ca-compliance] Next update 2022-01-05 → [ca-compliance] Next update 2022-01-22

(In reply to Ben Wilson from comment #28)

Are you still on schedule for 21/01/2022?

Yes, I confirm, we are on schedule for 21/01/2022

Flags: needinfo?(piotr.grabowski)

The changes have successfully deployed to the production.

Are there any other mitigation steps to be taken? If not, I could close this sometime during the upcoming week.

Flags: needinfo?(bwilson)

You can close it. There are no more additional actions here.

Status: ASSIGNED → RESOLVED
Closed: 2 years ago
Flags: needinfo?(bwilson)
Resolution: --- → FIXED
Product: NSS → CA Program
Whiteboard: [ca-compliance] Next update 2022-01-22 → [ca-compliance] [ov-misissuance]
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: