1524050 - Telia: Misissued certificate - invalid dnsName

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Jonathan Rudenberg]

Telia issued the following certificate with an invalid dnsName: "apmdc.han.telia.se " (note the trailing space)

https://crt.sh/?opt=zlint&id=282641081

I notified them via an email to their problem reporting address ([e-mail]) at 2019-01-25 15:25 UTC with the subject problem report: certificate with invalid dnsName.

As of the filing of this bug, the certificate has not been revoked.

Comment 1

[Ryan Sleevi]

Telia: Please provide an incident report, as described at https://wiki.mozilla.org/CA/Responding_To_An_Incident

Comment 2

[pekka.lahtiharju]

We'll do it during next few hours. Internal Telia CA investigation was started immediately when we were informed about the problem. We evaluated that this problem didn't pose an urgent security concern. More information later in incident report. We could't reach the customer yet and we didn't want to close down functional server because of this error but we'll replace the certificate soon.

Comment 3

[pekka.lahtiharju]

Incident Report

Case "apmdc.han.telia.se "

Telia CA was informed about the problem late Friday afternoon local time (Friday, January 25, 2019 5:25:45 PM (UTC+02:00)) using our normal problem reporting address [e-mail]. Telia CA procedure for urgent cases wasn't triggered by reporter (check https://support.trust.telia.com/palvelinvarmenneturvallisuus_en.html). Thus this problem message was opened at Telia on Monday morning. Problem certificate was created in 2016 and problem was that SAN value had extra space after the real FQDN. Incorrect certificate was in use by active service/server. Early Monday morning Telia CA Security Board decided that this problem didn't pose an urgent security concern and Telia will replace the certificate without urgently closing the related service/server. Customer representative who was originally created this certificate wasn't any more available but his boss was informed using high importance email at Monday, January 28, 2019 11:07 AM (UTC+02:00) to replace this certificate ASAP so that CA could revoke the incorrect certificate. Customer didn't inform Telia CA that they had renewed the certificate on Monday, January 28, 2019 13:07 AM (UTC+02:00). They didn't revoke the old certificate by themselves. Telia CA was waiting the Customer until status was rechecked on Thursday morning. Then CA could verify that certificate was renewed by Customer and CA was able to revoke the problem certificate Thursday, January 31, 2019.

During the week Telia CA analyzed the reason for the problem. Reason was that the CSR verification process used in 2016 didn't verify FQDN string from CSR properly. At the time CN was allowed to use space because allowed character sets were common for SSL and SMIME certificates. CN was copied to SAN field if SAN was missing. Other hidden characters (like Null) have been always been denied. Space handling was later changed/fixed. Domain validation didn't prevent this error either because in 2016 Telia used visual comparison against domain register and trailing space was almost impossible to be seen in the process. Now domain validation would prevent this because new methods are more automated without any visual phases. To be sure Telia also did database check to verify all existing SAN values and we didn't find any other space problems in our active SSL certificates. So this "apmdc.han.telia.se " was the only one this kind.

This kind of error is impossible in current Telia CA systems because of better CSR verification and better domain validation methods. We verified that Openssl still let you put prefixing or trailing spaces to FQDN but Telia CA won't accept those. Also nowadays lint check is done regularly against CT log by Telia to find any kind of syntax error early on. It would disclose the error if same would happen now. It wasn't our normal practice in 2016 why problem wasn't discovered at the time. This particular certificate was never included to internal or external audit verification set to be disclosed in those.

Comment 4

[Wayne Thayer]

Pekka: thank you for the incident report.

(In reply to pekka.lahtiharju from comment #3)

Incident Report

Case "apmdc.han.telia.se "

Telia CA was informed about the problem late Friday afternoon local time (Friday, January 25, 2019 5:25:45 PM (UTC+02:00)) using our normal problem reporting address [e-mail]. Telia CA procedure for urgent cases wasn't triggered by reporter (check https://support.trust.telia.com/palvelinvarmenneturvallisuus_en.html). Thus this problem message was opened at Telia on Monday morning.

Please be aware that the BRs do not provide any provisions for a different procedure for "urgent cases". BR section 4.9.3 requires the CA to list their contact for revocation in section 1.5.2 of the CPS, which for Telia does list the email address that was contacted and does not list the special phone number in the link above. Given this fact, how does Telia comply with the BRs? Section 4.9.5 requires an initial investigation and problem report to be completed within 24 hours.

Furthermore, BR section 4.9.5 requires that an initial problem report be provided to the reporter within 24 hours. Please explain why this was not done and how this problem will be prevented in the future.

Problem certificate was created in 2016 and problem was that SAN value had extra space after the real FQDN. Incorrect certificate was in use by active service/server. Early Monday morning Telia CA Security Board decided that this problem didn't pose an urgent security concern and Telia will replace the certificate without urgently closing the related service/server.

What, if any, consideration was give to the BR section 4.9.1.1 revocation requirements that apply to this certificate in making this decision? Your response implies that Telia believes that BR compliance is optional.

Customer representative who was originally created this certificate wasn't any more available but his boss was informed using high importance email at Monday, January 28, 2019 11:07 AM (UTC+02:00) to replace this certificate ASAP so that CA could revoke the incorrect certificate. Customer didn't inform Telia CA that they had renewed the certificate on Monday, January 28, 2019 13:07 AM (UTC+02:00). They didn't revoke the old certificate by themselves. Telia CA was waiting the Customer until status was rechecked on Thursday morning. Then CA could verify that certificate was renewed by Customer and CA was able to revoke the problem certificate Thursday, January 31, 2019.

The revocation was more than 5 days after the incident report. Please explain why Telia violated BR section 4.9.1.1 and what is being done to prevent future violations.

During the week Telia CA analyzed the reason for the problem. Reason was that the CSR verification process used in 2016 didn't verify FQDN string from CSR properly. At the time CN was allowed to use space because allowed character sets were common for SSL and SMIME certificates. CN was copied to SAN field if SAN was missing. Other hidden characters (like Null) have been always been denied. Space handling was later changed/fixed. Domain validation didn't prevent this error either because in 2016 Telia used visual comparison against domain register and trailing space was almost impossible to be seen in the process. Now domain validation would prevent this because new methods are more automated without any visual phases. To be sure Telia also did database check to verify all existing SAN values and we didn't find any other space problems in our active SSL certificates. So this "apmdc.han.telia.se " was the only one this kind.

Excellent, I am glad that you scanned the database of active certificates for this problem and found no others.

This kind of error is impossible in current Telia CA systems because of better CSR verification and better domain validation methods. We verified that Openssl still let you put prefixing or trailing spaces to FQDN but Telia CA won't accept those. Also nowadays lint check is done regularly against CT log by Telia to find any kind of syntax error early on. It would disclose the error if same would happen now. It wasn't our normal practice in 2016 why problem wasn't discovered at the time. This particular certificate was never included to internal or external audit verification set to be disclosed in those.

Comment 5

[pekka.lahtiharju]

Thank you for your comments. The fact that BR won't understand concept of normal and urgent cases is tricky. In real world problems have priority and Mozilla has formulated that nicely in your chapter on your Incident instruction page:

"However, it is not our intent to introduce additional problems by forcing the immediate revocation of certificates that are not BR-compliant when they do not pose an urgent security concern. Therefore, we request that your CA perform careful analysis of the situation. If there is justification to not revoke the problematic certificates, then your report will need to explain those reasons and provide a timeline for when the bulk of the certificates will expire or be revoked/replaced."

Telia has now divided its CPS contact instruction in chapter 1.5.2 according to this Mozilla instruction providing two options to contact Telia. Either reporter can use email address in non-urgent cases (non BR compliant) or use phone if immediate handling is required (BR compliant). We have tried but maybe failed to put the wording so that reporter would understand that only phone call is the BR compliant way. Note! You say that our contact phone number on our security instruction page is missing from CPS 1.5.2; it is there like this: "Revocation Service: [phone] (revocation requests or any urgent issues)". Our current wording in 1.5.2 instruction chapter is (note the last sentence):

"Subscribers, relying parties, application software vendors, and other third parties can send emails to [e-mail] reporting complaints or suspected private key compromise, certificate misuse, or other types of fraud, compromise, misuse, or inappropriate conduct related to certification. In urgent cases we recommend contacting Telia Company or revoking the certificate by calling and using the above contact phone numbers. "

What we can do now is to improve the wording in CPS 1.5.2 so that it is absolutely clear for everybody that only phone call is the fully BR compliant channel and email channel works only on business hours. CPS wording will be improved in Feb 2019. So this explains why the used process wasn't fully BR compliant (= reporter didn't get the response within 24 hours and initial evaluation wasn't done within 24 hours). Your constructive comments also triggered us to improve our 24/7 service functionality. We started a new plan to create 24/7 email address for Telia CA because we prefer getting this kind of problem reports by email instead of phone. Also this case disclosed that we have to modify our 24/7 phone number so that it is always functional from all countries. Current one may have issues from abroad. These improvements are ongoing and expected to be finished in February.

Your other main concern was BR 4.9.1.1 compatibility (not revoked within 1 or 5 days). In the initial evaluation on Monday Telia Security Board notified that this certificate was incorrect according to rule 7 (certificate was not issued according to BR). It was not causing any security concerns and no new similar errors are possible. There is 5-day BR revocation requirement for such. This requirement wasn't kept optional but because of human mistakes in communication it took almost 6 days (counting from initial report) to revoke this. Telia CA incorrectly thought that Customer certificate creator would revoke the incorrect one or at least inform CA (renewal was done on Monday). Telia has now re-instructed its personnel to avoid such delays in the future. In new instructions it is stated that Telia CA can't trust Customers and such tasks are verified by CA personnel in each case in time.

Comment 6

[Wayne Thayer]

Pekka: thank you for this informative response. Please update this bug as the actions described in comment #5 (update CPS, create new 24/7 email address) are completed.

Comment 7

[pekka.lahtiharju]

Telia did mass lint scan in February 8, 2019 and found one additional invalid certificate for this error category. It had space before the actual FQDN so it wasn't detected by our previous database scan. But otherwise almost everything in this incident report match to that case also. That certificate is " vpncare.dk.telia.net" with serial 044fccb2b83b6db8531491ea6bfc2e4e created May 6, 2016 for three years. It was revoked on the same day Telia found it meaning February 8, 2019.

Comment 8

[pekka.lahtiharju]

PEM of the other affected certificate is:

-----BEGIN CERTIFICATE-----
MIIGmjCCBIKgAwIBAgIQBE/Msrg7bbhTFJHqa/wuTjANBgkqhkiG9w0BAQsFADBG
MQswCQYDVQQGEwJGSTEUMBIGA1UECgwLVGVsaWFTb25lcmExITAfBgNVBAMMGFRl
bGlhU29uZXJhIFNlcnZlciBDQSB2MjAeFw0xNjA1MDYxMTUzNDNaFw0xOTA1MDYx
MTUzNDNaMG4xCzAJBgNVBAYTAlNFMRIwEAYDVQQHDAlTdG9ja2hvbG0xFzAVBgNV
BAoMDlRlbGlhU29uZXJhIEFCMRMwEQYDVQQLDApPcGVyYXRpb25zMR0wGwYDVQQD
DBR2cG5jYXJlLmRrLnRlbGlhLm5ldDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCC
AQoCggEBALndd1aYCSt1y4G5y+o8ot2mKsMksYbmIgbtO6qCAz1wGxLI26Nn2YOn
uszTK7n4epihLwz2D4Ct4m62KlPx+NOAVViDrwh+xJ3AoVpsLp/Gh0zhTWpfQdvj
mwROMjzyd6MnjoF1YY6N+HdJ1tfMPPAQ0wLYJme8hTBopOxmGgsnbTXYJQ4dg16p
7z1hNlezere+vsBgyZVzGJTl+PuKC+b/Uo1xjRfMKloHr02gEnT00LpIt7nA2ogY
qo7yMYhDF+rBMUDJdpIGhkr66PLagKc5LYC0UEzoKGPJhk0GoDTkxFyhw1xQGdS3
MeaTOGOFC4/gFiIbaQsZpdERwMlhxkECAwEAAaOCAlowggJWMIGNBggrBgEFBQcB
AQSBgDB+MC0GCCsGAQUFBzABhiFodHRwOi8vb2NzcC50cnVzdC50ZWxpYXNvbmVy
YS5jb20wTQYIKwYBBQUHMAKGQWh0dHA6Ly9yZXBvc2l0b3J5LnRydXN0LnRlbGlh
c29uZXJhLmNvbS90ZWxpYXNvbmVyYXNlcnZlcmNhdjIuY2VyMB8GA1UdIwQYMBaA
FC9JPClP1wcl+caM1WT1Zj0SgyKVMFQGA1UdIARNMEswSQYMKwYBBAGCDwIDAQEP
MDkwNwYIKwYBBQUHAgEWK2h0dHA6Ly9yZXBvc2l0b3J5LnRydXN0LnRlbGlhc29u
ZXJhLmNvbS9DUFMwgcoGA1UdHwSBwjCBvzBCoECgPoY8aHR0cDovL2NybC0zLnRy
dXN0LnRlbGlhc29uZXJhLmNvbS90ZWxpYXNvbmVyYXNlcnZlcmNhdjIuY3JsMHmg
d6B1hnNsZGFwOi8vY3JsLTEudHJ1c3QudGVsaWFzb25lcmEuY29tL2NuPVRlbGlh
U29uZXJhJTIwU2VydmVyJTIwQ0ElMjB2MixvPVRlbGlhU29uZXJhP2NlcnRpZmlj
YXRlcmV2b2NhdGlvbmxpc3Q7YmluYXJ5MB0GA1UdJQQWMBQGCCsGAQUFBwMBBggr
BgEFBQcDAjAOBgNVHQ8BAf8EBAMCBLAwMgYDVR0RBCswKYIVIHZwbmNhcmUuZGsu
dGVsaWEubmV0ghB2cG5jYXJlLnRlbGlhLmRrMB0GA1UdDgQWBBRHDZ+AX+aYii75
TNrM5oTK3JAj6jANBgkqhkiG9w0BAQsFAAOCAgEAK/FEtH5VAr2amV6oEuB4Fhda
Iq+1ftyGr4SunKtmdtUEMEQcefnWWXyAUEZhLqQKtMqHqdUgM1PdvLIRmpQm2cn/
yjSq9W+5ijmPic+ERbIKxOxgvZGlbHmsrO9WvQub24qepeHBkxnKfowLJQ4dLZgE
kyxJcsD+3IOGoQdX1KjHb844PWd1mOy8KyLSRKq0A+5pLanLua5dfyT36OVtdcYL
GC+4UGXrHABDLjBOANojf0S5XaGrnGqEG8XUXCcbY4ZP819vZcja4Ai7e1L+5yUh
rhWkuGieDZ0HmtJPxUEZRdGrC0KUQMey5piYbfxK0I9PzrLxGp/mFRZx/jQ1XeVJ
xXViiRro9D6spFqbjJMdH3+2ZxLo2LoHSwLdEc2nqj/CzDfRnM5QOhyFJ7wOQqMh
HnYikZ3lnZOv/17s3n3w4PeDdSgBJRj1mn/HfDcHEg3YGzJ5rNx8b1TAx/kha5E+
LpPhmMCc7TPkPojQvVqq09AIewb48DIQQzZyduVZlUzOWBOsw9SrH4sntXD5Wlpp
/kgdE/NoYKTXMHDwDYaGRF1m2HZPLVKRm78GblJC8pq9DZ8RPl5qYhdM7soV5EP5
imFwATBkkhhTTZIs5I3lublPibcoIbk4FuL3mouW3hMoYoKM6IviWf1RPCVD3yIf
x5vBGNRGrhlVhLsMEAo=
-----END CERTIFICATE-----

Comment 9

[Tim Smith (inactive) 👨‍🔬 [:tdsmith]]

https://crt.sh?q=03A634F4A7988470C18644107A0EB003C44C5081EF4FA596A9AF6376FCA74779

Comment 10

[pekka.lahtiharju]

Telia has now improved the wording in CPS chapter 1.5.2. The new one is clearer how to contact if BR 24h response is expected. However, we want to still maintain also our normal support email that give service only on business days. The new wording and support channel email address will be (effective on 15th March):

Certificate problem reporting:

Subscribers, relying parties, application software vendors, and other third parties can use two optional methods to contact Telia CA:

1. [e-mail] Support channel. Not necessarily handled within 24 hours.
2. [e-mail] Important reports. Always handled within 24 hours (BR compliant)

Use either of these channels to report complaints or suspected private key compromise, certificate misuse, or other types of fraud, compromise, misuse, or inappropriate conduct related to certification. In urgent cases we recommend contacting Telia Company or revoking the certificate by calling and using the above contact phone numbers also.

Hopefully this makes it clearer for reporter what channel to use in each case.

Comment 11

[Ryan Sleevi]

Wayne: I think this has what you asked for.

Comment 12

[Wayne Thayer]

It appears that all questions have been answered and remediation is complete.