1675314 - Telekom Security: Wrong jurisdiction entries in certificates

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Arnold Essing]

Six EV certificates for a single costumer have been issued with incorrect jurisdiction entries. The certificates have been revoked the next working day and have never been in use.
A detailed incident report will follow.

Comment 1

[Arnold Essing]

A German customer (subscriber) requested 6 EV certificates for its Spanish subsidiary company (subject). Due to a software bug, the customer was not able to fill in the jurisdiction information of the Spanish company in its customer master data (which provides a basis for requests). To complete the order, the required information was then provided via a different channel (and validated accordingly), while the technically mandatory fields in the customer master data were temporarily filled with the jurisdiction information of the German parent company until the software bug would be fixed in the future and the information could be corrected. It was assumed, that this data would only be relevant for the customer master data as well as the validation process and would not be written into the certificates themselves. As a result, the final certificates contained the wrong information from the master data.

Note: Times are given in CET.

1.How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.
Our internal auditor currently checks all EV certificates being issued and identified the 6 misissued certificates during his regular check on 2020-11-02 at 09:27.

2.A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.
2020-10-28 Our RA-Team processes the request of the customer and consults the technical expert for the customer service portal regarding the customer’s problem of not being able to enter the correct jurisdiction information. Since the problem cannot be solved in short term, the work-around as described in the summary above (as well as point 6 of this incident report) is utilized. As mentioned before, the correct data was still validated!

2020-10-30 06:25 Our RA-Team issues the six certificates.

2020-11-02 09:27 Our Internal Auditor identifies the certificates with the unusual entries.

2020-11-02 10:08 The Internal Auditor reports his findings to the responsible solution manager/product manager and to the compliance team.

2020-11-02 13:00 A conference with the Internal Auditor, compliance team, responsible solution managers is held, in which the problem is initially evaluated. As a result, it is decided to inform the management, evaluate the possibility of potential further occurrences in the past, prepare the revocation of certificates (within 24 hours after the finding) and stop the issuance of EV certificates.

2020-11-02 14:11 Issuance of new EV certificates is put on hold until further notice.

2020-11-03 09:00 A conference is held in which the management is informed about the finding in detail. The planned revocation of the certificates is confirmed and carried out (see next point). Additionally, the evaluation by the responsible solution managers has revealed how the erroneous certificates came into being (as described in the summary above or under point 6 of this incident report) and, hence, only subjects from other countries than Germany or Switzerland are potentially affected. It is therefore decided that issuance for EV certificates may be resumed for German and Swiss subjects after another safeguarding phase of at least one day. This decision was made to not rush things and also to wait for the evaluation of potential other / similar occurrences to be finished.

2020-11-03 09:17 The certificates are revoked.

2020-11-04 10:30 The evaluation of potential further affected certificates is finished without additional findings.

2020-11-04 11:30 Issuance of EV certificates is resumed for German and Swiss EV certificates in connection with a sensitization of the RA-Team and an improvement of the RA-GUI with additional information on which content will be part of the certificate.

2020-11-04 16:00 Another conference is held in which it is decided that issuance of EV certificates will from now on be limited to German and Swiss subjects for which the software as well as the processes have proven to be reliable and working. Issuance of EV certificates to subjects from other countries will only be enabled after appropriate preparations, tests and other measures that are to be thoroughly evaluated. This includes the customer’s request which will be required to be denied.

3.Whether your CA has stopped, or has not yet stopped, certificate issuance or the process giving rise to the problem or incident. A statement that you have stopped will be considered a pledge to the community; a statement that you have not stopped requires an explanation.
The issuance of EV Certificates has been stopped temporarily on 2020-11-02 at 14:11. It was partially resumed on 2020-11-04 after the analysis revealed that the software error would only occur with requests for countries other than Germany and Switzerland. Issuance of EV certificates for other countries is still on hold (even technically disabled) for the time being and will only be resumed after all bugs have been fixed and correct issuance can be ensured. It is planned to enable this function on a country by country basis after appropriate analysis.

4.In a case involving certificates, a summary of the problematic certificates. For each problem: the number of certificates, and the date the first and last certificates with that problem were issued. In other incidents that do not involve enumerating the affected certificates (e.g. OCSP failures, audit findings, delayed responses, etc.), please provide other similar statistics, aggregates, and a summary for each type of problem identified. This will help us measure the severity of each problem.
All of the six certificates were part of one order from one customer (German subscriber with a Spanish subsidiary company as subject) and were issued on 2020-10-30. They have not been put into operation by the customer.
Prior to this, there have not been any EV certificates issued to customers from Spain (or any other country besides Germany and Switzerland).

5.In a case involving certificates, the complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem. In other cases that are not involving a review of affected certificates, please provide other similar, relevant specifics, if any.
https://crt.sh/?id=3576606064
https://crt.sh/?id=3576585791
https://crt.sh/?id=3576590239
https://crt.sh/?id=3576601296
https://crt.sh/?id=3576579994
https://crt.sh/?id=3576597467

6.Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
Note:
In case of German and Swiss organizations, the jurisdiction information can be selected from given entries, while for other countries such information is required to be entered by the customer via free text. This is due to the fact, that almost all our customers are from Germany or Switzerland and errors during input can be reduced this way (e.g. typos). Regarding Extended Validation, certificates have actually only been issued to German and Swiss subjects until now.

In this case, a customer (subscriber, seated in Germany) requested 6 EV certificates for a subsidiary company (subject, seated in Spain) for the first time.
Due to a bug in the software, the option to enter the jurisdiction information as free text was not available to the customer. The information itself was then provided via a different channel and validated accordingly by our RA-Team. However, it is technically mandatory to provide the information in the customer data to request EV certificates. When the technical expert for the configuration of this software was contacted by the RA-Team, that person was not able to solve the problem in short term and suggested to fill the required master data with substitute information instead (i.e. the jurisdiction information of the German parent company which was also the subscriber) until the bug was fixed and the customer can correct the information afterwards. It was assumed by the technical expert as well as the members of the RA, that the data would only be relevant for the customer master data as well as the validation process and would not be written into the certificates themselves. This assumption is further supported by the unfortunate design of the GUI which explicitly highlights the contents of the CSR (although this highlighting is unrelated to which information is written into the certificate). As a result, the final certificates contained the wrong information from the master data.

Our internal auditor currently checks all EV certificates and identified the mistake the very next work day. This is the first occurrence of the software bug as well as the resulting mistakes.

7.List of steps your CA is taking to resolve the situation and ensure that such situation or incident will not be repeated in the future, accompanied with a binding timeline of when your CA expects to accomplish each of these remediation steps.
a) The affected certificates have been revoked (within 24 hours after the finding).
b) Besides our general quality improvement processes, an explicit analysis of the RA-processes has been initiated to find potential other/similar sources of error.

We came to the conclusion that the main root cause was that we were not properly prepared for the issuance of EV certificates to subjects outside of Germany and Switzerland and that the RA-Team is not sufficiently supported from a technical point of view. Therefore, the following actions have been taken to tackle these problems.
c) The GUI for the RA-Team received a quick update and now highlights the information that will be part of the certificate’s content (this change was possible without a software update and is only based on configuration of the software). Further improvements to this GUI are currently being discussed as part of the long term improvements and will be implemented in future releases of the software.
d) It was decided that for the time being EV certificates shall only be issued to subjects from Germany or Switzerland, for which the software and the processes have been thoroughly tested and proven to be working as well as reliable. Should other countries be considered in the future, then the support for those countries will only be added country-by-country and after appropriate preparations, testing and other measures yet to be evaluated.
e) The software has been configured to technically enforce the decision mentioned in d) and any work-arounds have been explicitly forbidden without consulting an extended body of experts from different areas first. This information has been distributed to all members of the RA-Team as well as all product managers (including the technical managers).

Comment 2

[Ryan Sleevi]

Thanks. I think one area of concern is understanding how and why this incident wasn't also caught by the controls CAs were required to introduce in Ballot SC30. It seems that the "Free-text approach" would have naturally been discouraged by such an approach, which also would have appropriately flagged that there was an unusual request, and greater attention to detail was required.

Similarly, why was it that the internal auditor detected this, and not the RA that knew of the appropriate/expected information that should have appeared in the certificate?

Comment 3

[Arnold Essing]

There might be a misunderstanding in regard to the free-text field. The information is only gathered with the help of the free-text field and then forwarded to the RA and CA. The validation process by the RA is uneffected and always requires that the information can be validated with the available sources in the first place (which obviously is guaranteed for the selectable entries but not limited to them). Should the validation not be possible with the available sources, the request will be dropped. In this case, the problem was solely that the information had to be provided via a different channel due to the software bug. The validation itself was not a problem case to be flagged (in our opinion).

In regard to your question why the internal auditor detected this instead of the RA: The RA validated all information for correctness as required. As it was not possible for the client to fill in the free-text field in this particular case, the RA contacted the technical expert. This technical expert wrongly assumed that it was not an EV but an OV-certificate where this data is not included in the certificate and suggested the workaround. The RA trusted this statement of the technical expert. The GUI for the RA also did not explicitly point out that this was a field that would also appear later in the certificate. From the RA point of view, the statement of the technical expert combined with the display in the GUI seemed conclusive and therefore the RA did not critically question the workaround and approved the issuance of the six certificates. In order to prevent such cases from occurring again, we have taken the measures described in point 7 of the incident report.

Comment 4

[Ben Wilson]

I believe this matter can be closed, so unless I hear otherwise, I'll schedule this for closure on 7-Apr-2021.