Closed Bug 1711208 Opened 5 years ago Closed 5 years ago

Asseco DS / Certum: Incorrect localityName

Categories

(CA Program :: CA Certificate Compliance, task)

Product:
CA Program
Component:
CA Certificate Compliance
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: aleksandra.kurosz, Assigned: aleksandra.kurosz)

Details

(Whiteboard: [ca-compliance] [ev-misissuance])

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36

Steps to reproduce:

While analyzing bug 1709392, we found 3 certificates with an invalid localityName field.
List of certificates:

https://crt.sh/?serial=426a3f6fb6081b1b17da7502ae3f5602
https://crt.sh/?serial=27bfb3534ef06c2c78aec769a0e3654b
https://crt.sh/?serial=13341deef1ad0e4c78cc52e13f96c4ee

The reasons are similar to bug 1709392, we will post a report soon.

Thanks for reporting this and scanning through your existing certificates quickly, this seems really good from Certum.

However, I don't see how the first two certificates are misissued, "Praha x" seems to be valid here as they are municipalities of Prague. The last certificate does seem to have a different root cause though so I hope that you can focus on this specifically in your incident report.

Assignee: bwilson → aleksandra.kurosz
Status: UNCONFIRMED → ASSIGNED
Type: defect → task
Ever confirmed: true
Whiteboard: [ca-compliance]
  1. How your CA first became aware of the problem. raport Incorrect localityName.docx

While analyzing the bug 1709392, we have checked the localityName field, and on the 2021-05-14 we found 3 incorrectly issued certificates.

  1. A timeline of the actions your CA took in response.

Time in UTC.

2021-02-03 Bug 1667986 was closed.

2021-02-15 Analysis of possible localityName checking with additional tools or external APIs as operator support started.

2021-03-08 Additional localityName checking was introduced as a part of the address validation. We decided to add this as a manual step at this point and add automated validation to future development roadmap.

2021-03-29 Analysis of the possibility of creating the role of an operator with lower powers to issue certificates started. Operator with lower powers would require his decisions to be confirmed by a second person with regular powers.

2021-04-01 Role of an operator with lower powers was accepted and added to current development roadmap for Q2 2021.

2021-05-04 18:00 The bug 1709392 was created.

2021-05-11 The analysis of the stateOrProvinceName field was closed.

2021-05-12 We started a new analysis of the localityName for the active certificates.

2021-05-14 3 Certificates with an incorrect localityName were found.

2021-05-19 Certificates were revoked.

  1. Confirmation that your CA has stopped issuing TLS/SSL certificates with the problem.

As we described in bug 1709392 we constantly take measures to reduce the number of errors that may result from the manual processing of applications for certificates. Manual checking of the localityName field is still in progress.

  1. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

The first certificate was issued 2019-12-05, and the last certificate was issued 2020-12-08.

  1. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

https://crt.sh/?serial=426a3f6fb6081b1b17da7502ae3f5602
https://crt.sh/?serial=27bfb3534ef06c2c78aec769a0e3654b
https://crt.sh/?serial=13341deef1ad0e4c78cc52e13f96c4ee

  1. Explanation about how and why the mistakes were made, or bugs introduced, and how they avoided detection until now.

The misissuance occured before the first bug 1667986 was submitted, after that we have made changes to the verification process.

For certificate https://crt.sh/?serial=13341deef1ad0e4c78cc52e13f96c4ee explanation of this missisuance from the person from vetting team who issued this certificate is that the organization from the source verification document was copy-pasted to organizationName and organizationUnitName and accidentally again to the localityName.

For certificates https://crt.sh/?serial=426a3f6fb6081b1b17da7502ae3f5602 and https://crt.sh/?serial=27bfb3534ef06c2c78aec769a0e3654b as pointed out in this bug, it is debatable whether this field was filled in correctly or not. However, Praha 1 to 15 were removed from list of districts in 2018-11-26 according to ISO 3166-2 (https://www.iso.org/obp/ui/#iso:code:3166:CZ), and based on this we decided to revoke these certificates and reissue them with Praha in localityName field.

  1. List of steps your CA is taking to resolve the situation.

The solution to this bug is similar to the bug 1709392, so we suggest the same steps.

We decided to add double-check for OV certificates as well, since most of the affected certificates are OV. In addition, we created the role of an operator with lower powers to issue certificates. Operator with lower powers would require his decisions to be confirmed by a second person with regular powers. We started implementing this solution after closing the previous bug, it is now in the testing phase, and we expect to bring it to the production system in June.

We are going to introduce automated address verification supporting operators to minimize the possibility of human error. This change will apply not only to localityName field but also other address fields, e.g. stateOrProvinceName. We are done with the initial research and now we are in process of checking the effectiveness of available solutions. We plan to deploy it into production by the end of this year.

In addition to the self-audit specified in the CABF BR, we decided to introduce a monthly checking of issued certificates for localityName until the automated address verification will be implemented. It should help us to detect misissuance in a timely manner.

We are conducting analysis of all active certificates for any type of errors for localityName. We will inform about the results of our scan in this bug.

There are not update in this bug. We are in the process of additional analyzes.

All certificate mentioned in comment 4 are revoked, the last one: 2021-06-16 11:28

Aleksandra: Thanks for the update.

I note Comment #2 was dated 2021-05-21, which stated Asseco DS was beginning the analysis, and that analysis completed on 2021-06-11, based on Comment #4.

Could you share more about why the analysis took so long? For example, did this require creating custom code, was it done by hand, etc?

I think the goal here to understand is "What are the factors that made this analysis take so long, and what can be done to make such analysis go quicker in the future?". While obviously, there's no one-size-fits-all solution, understanding, for example, if a CA has the ability to run a lint or check against 100% of its existing certificates, and how long that would take, would give a good idea for how quickly a CA can respond to an incident. Since incidents can end up quite bespoke (e.g. "We need to find all certificates issued by this validation agent" or "All certificates that used this registration database as a QGIS"), it's understandable that this may not be something "as simple as" a lint, but hopefully the goal of these questions is clear.

Flags: needinfo?(aleksandra.kurosz)

We did the analyse manually, additionally independently checking one of the prototype automatic methods that we are developing. After doing both, our team compared the results and analyzed all the differences and inconsistencies in the human and system assessment, so the process took a long time.

Flags: needinfo?(aleksandra.kurosz)

No updates. If there are no additional questions, can this bug be closed?

Thanks for your answer in Comment #7. I was hoping you might be able to more deeply analyze the question in #6. Namely, Comment #7 doesn't give any hope or indication that there might be steps to take to improve analysis in the future. The more detailed you can be about what processes you had, and what you're thinking to improve, the better things can be.

Of course, if the view is that this was as quick as it possibly could be, then we can go ahead and proceed closing, but I'm not sure we have a clear understanding why, which isn't the greatest.

Flags: needinfo?(aleksandra.kurosz)

I assure that with each analysis we make every effort to ensure that it is as fast and accurate as possible. We approach each analysis individually and complexy, therefore, depending on the amount of data to be analyzed manualy by the team, it may take more or less time.

Flags: needinfo?(aleksandra.kurosz)

I will schedule this for closure to be done on or about this Friday, 16-July-2021, unless there are questions that still need to be answered.

Flags: needinfo?(bwilson)
Status: ASSIGNED → RESOLVED
Closed: 5 years ago
Flags: needinfo?(bwilson)
Resolution: --- → FIXED
Product: NSS → CA Program
Whiteboard: [ca-compliance] → [ca-compliance] [ev-misissuance]
You need to log in before you can comment on or make changes to this bug.