1736020 - Telia: Invalid email contact address was used for few domains

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[pekka.lahtiharju]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.81 Safari/537.36

Steps to reproduce:

invalid domain validation

Actual results:

email address was invalid in email domain validation

Expected results:

email should have been sent to right contact

Comment 1

[pekka.lahtiharju]

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in the MDSP mailing list, a Bugzilla bug, or internal self-audit), and the time and date.

In internal routines related to email validation Telia CA found exceptional domain validation information where email address was one of the five standard ones but it wasn’t using same domain. We decided to verify all domains validated by email. We found few similar cases. All issues happened in early 2020 when Telia still was using old validation software.

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.
   Thu 2021-10-14 16:00 We discovered one pre-validated domain object where information was confusing because it wasn’t either one of the 5 static ones or from DNS contacts.
   Thu 2021-10-14 16:00-18:00 We evaluated the problem. It was identified to be happened only to few domains during 03-06/2020. All except one were Telia's own domains.
   Fri 2021-10-15 8:00-15:30 We continued evaluation. We found that this was bug in our previous validation software. In multi-SAN case in Telia's SSL ordering system it could use same email target email address for all unvalidated domains in the same request. Thus the domain could be approved by the wrong hostmaster. Current software hasn't similar erroneous behavior. Also was found that there are 7 valid certificates that were still using those illegally validated domains. All illegal domains are already expired so new illegal certificates can't be created anymore. We started a process to revoke all valid certificates that were still using the illegally validated domains.
   Fri 2021-10-15 15:45 Mozilla incident was created

3. Whether your CA has stopped, or has not yet stopped, certificate issuance or the process giving rise to the problem or incident. A statement that you have stopped will be considered a pledge to the community; a statement that you have not stopped requires an explanation.

Change in domain reuse period 1 Oct 2021 expired all problematic domains from early 2020 so Telia has stopped creating illegal certificates.

4. In a case involving certificates, a summary of the problematic certificates. For each problem: the number of certificates, and the date the first and last certificates with that problem were issued. In other incidents that do not involve enumerating the affected certificates (e.g. OCSP failures, audit findings, delayed responses, etc.), please provide other similar statistics, aggregates, and a summary for each type of problem identified. This will help us measure the severity of each problem.

7 certificates using these illegally validated domains are still valid. ~30 were using them at some point of time.

5. In a case involving TLS server certificates, the complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem. When the incident being reported involves an SMIME certificate, if disclosure of personally identifiable information in the certificate may be contrary to applicable law, please provide at least the certificate serial number and SHA256 hash of the certificate. In other cases not involving a review of affected certificates, please provide other similar, relevant specifics, if any.

Certificates that used domains that were illegally validated were:
https://crt.sh/?id=3488033760
https://crt.sh/?q=4276441541
https://crt.sh/?q=2661557546
https://crt.sh/?q=2963992540
https://crt.sh/?q=4349009623
https://crt.sh/?q=3314879972
https://crt.sh/?q=2700974232

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

Telia's previous order validation software that was closed October 2020 had a bug when sending emails to host masters. Bug triggered only in special circumstances that were undetected by all tests and users. In the most common Telia process customers are using self-service system that hadn't any problems. Email method is not the most used method. In ordering system domains are typically prevalidated and only afterwards customer will set orders. Bug required that ordering system was used and email method was used and there were several previously unvalidated domains in same CSR that were using different email details. In Telia CA history there have been only 3 such orders. They caused 7 domains to be illegally validated. It is hard to detect this kind of error when it requires rare circumstances and the actual host master is practically from the same team (from the same company).

7. List of steps your CA is taking to resolve the situation and ensure that such situation or incident will not be repeated in the future, accompanied with a binding timeline of when your CA expects to accomplish each of these remediation steps.

In addition to the steps listed above (e.g. revocations) Telia will create a new monitoring tool next week to create alarm if email will be sent to different domain than the target domain. Also testing will be improved from now on to include use cases like the one that triggered this bug.

Comment 2

[Andrew Ayer]

Thanks for promptly disclosing this issue in Bugzilla.

In multi-SAN case in Telia's SSL ordering system it could use same email target email address for all unvalidated domains in the same request.

Two of the certificates (https://crt.sh/?q=2963992540 and https://crt.sh/?q=4276441541) have only a single SAN, so it's not clear how the bug was triggered for them. Could you clarify?

Comment 3

[pekka.lahtiharju]

This problem was directly related to only seven domains in three certificates. During validation of the three Telia stored that those seven domains were pre-validatated and reused the validation for some later certificates. That's why the named single-SAN certificates are also in the problem list. Telia CA used logs, certificates and CA database to find all affected certificates that used the illegally validated or pre-validated domains. This is also the reason why some of the affected certificates are from year 2021 even though the validation errors happened in 03-08/2020. Only small part of the domains in these affected certificates had the problem because most of the domains were correctly pre-validated at the time of issuance and didn't need any (failing) revalidation.

The issue was caused by these three certificates (orders):
https://crt.sh/?q=2700974232
https://crt.sh/?q=2661557546
https://crt.sh/?q=3314879972

Comment 4

[pekka.lahtiharju]

Status:
The planned new alarm system has been implemented to notify us if any similar error should happen in the future. All related systems were verified that this kind of error can't happen with them. 6/7 related domain validations are renewed with valid methods and one domain has been closed. Domain control still belongs to same parties meaning that replacement certificates will be identical to invalid/revoked ones.

Status is now that 2/7 of illegal certificates are revoked/expired. The rest (5/7) of the affected certificates are in critical heavily used systems (login and email) which are continuously used by tens of thousands of persons. It looks like it isn't possible to revoke those in BR specified 5 day limit without causing major disturbance to Nordic community. Because the replacement certificate is identical we evaluate the security threat is very low compared to bad impact revoking those swiftly. Thus, we are willing to give some extra days for these 5 certificate owners before we revoke/close their systems. According to discussions with owners, those could be replaced in decent service breaks during next week latest 2021-10-29. All replaces are scheduled. We have set this as the final revocation date.

Comment 5

[Ben Wilson]

A delayed revocation requires that you file an additional incident report in Bugzilla. We place it o the whiteboard as [ca-compliance] [delayed-revocation-leaf].

Comment 6

[pekka.lahtiharju]

We created [Bug 1737808] Telia CA: Delayed revocation of 5 EE certificates in connection to id=1736020 on last Tuesday 2021-10-26.

Comment 7

[Ali Gholami]

As of today October 20, 2021 all violating certificates are revoked.

Comment 8

[Ali Gholami]

(In reply to Ali Gholami from comment #7)

As of today October 20, 2021 all violating certificates are revoked.

I appologize for a typo in the above date. Correct date is: October 29, 2021.

Comment 9

[Ben Wilson]

I'm going to close this on next Wed. 2-16-2022, unless there are any questions or issues to discuss.