Closed Bug 1524567 Opened 8 months ago Closed 3 months ago

Telia: invalid IP value in SAN DNS field

Categories

(NSS :: CA Certificate Compliance, task)

task
Not set

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: pekka.lahtiharju, Assigned: pekka.lahtiharju)

Details

(Whiteboard: [ca-compliance])

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36 Edge/15.15063

Steps to reproduce:

Telia had issued some invalid certificates in 2016:

https://crt.sh/?opt=zlint&id=41922753
https://crt.sh/?opt=zlint&id=282640412
https://crt.sh/?opt=zlint&id=282652592

All are revoked now.

Actual results:

Certificates had IP value in SAN DNS field.

Expected results:

IP values should be stored in SAN IP field.

Incident report:

Telia was informed on Tuesday 29 January 2019 about three invalid certificates that were issued in 2016. Problem was that IP value was stored incorrectly on DNS field. Problem was verified by Telia and these certificates were revoked on Thursday 31 January.

Reason was Telia's previous CA code that copied CN value to wrong field. Issue has been fixed since. Telia has added automatic testing to prevent this in the future. Also Telia is now using regular lint checking to find all irregularities quickly.

Assignee: wthayer → pekka.lahtiharju
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true
Whiteboard: [ca-compliance]

This incident report is incomplete, please use the template in https://wiki.mozilla.org/CA/Responding_To_An_Incident#Incident_Report

Flags: needinfo?(pekka.lahtiharju)

Complete incident report:

  1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

Telia was informed by jonathan@titanous.com on Tuesday, January 29, 2019 7:54:07 PM (UTC+02:00) about three invalid certificates that were issued in 2016. Problem was that IP value was stored incorrectly into DNS field. He was using Telia's problem reporting email address cainfo@telia.fi to report these problems.

  1. A timeline of the actions your CA took in response.

Tuesday, January 29, 2019 7:54:07 PM (UTC+02:00): Problem report received
Wednesday, January 30, 2019 10:00 AM (UTC+02:00): Telia Security Board had meeting that confirmed the problem to be invalid dns name (IP address in DNS field; value was in SAN DNS instead of SAN IP because of an old Telia CA bug). Board estimated this problem against BR 4.9.1.1 and estimated that this wasn't posing any immediate security threat. This issue was BR revoke reason 7 which has requirement "CA SHOULD revoke a certificate within 24 hours and MUST revoke a Certificate within 5 days". These three certificates had been used more than two years and were still in active use. Telia decided to replace those ASAP without doing emergency shutdown/revocation to the related services. Also Board stated that this particular problem had been previously fixed (fix date was January 9, 2017) so that this can't happen again.
Thursday, January 31, 2019: Server operators were able to replace the problematic reported certificates and all three were revoked.
Friday, February 1, 2019: Telia did CA database check to verify if there are any other illegal certificates like these. 9 others were found and replacement for those was immediately started. 6 were revoked on Friday, 3 remaining for later revoke. These nine were also created in 2016 because of the same bug in CA at the time. Note: These nine are in private networks that may not be visible to CT agents. Preliminary incident report was created on Friday.
Monday, February 4, 2019: a new incident report (this) was created. The remaining illegal certificates will be revoked.

  1. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

All problematic certificates were issued in 2016. At the time there was a bug that copied IP formatted CN-value to SAN-DNS field instead of SAN-IP. Since 2017 this error hasn't happened and can't happen.

  1. A summary of the problematic certificates.

3+9 certificates had incorrect value (IP Address) in SAN DNS field. All were created in 2016 either by "TeliaSonera Gateway CA v2" or by "TeliaSonera Server CA v2". Otherwise certificates were OK, in use and browsers accepted those.

  1. The complete certificate data for the problematic certificates.

Full data of reported illegal certificates is:
https://crt.sh/?opt=zlint&id=41922753
https://crt.sh/?opt=zlint&id=282640412
https://crt.sh/?opt=zlint&id=282652592

Serial numbers of the other affected certificates are:
C5BC673506CE07BC885DC32B1244A6DF
A3BC79A93D8013A2C7F7845C71115175
64B9FE86003EC4C4C6C0BCCE936D3BDA
58E4D161A728867C1AA872BA9EA3E989
AB0842CD66B6527ED2435E987D286DA5
0AD986CBA0DC598974F036AE012EF5D0
01B671D39EBE7042E9E9E39B2EAA133F
A8937999ED44E209C3181E20ED07E68B
E8C44962E84947BDB5534EFC33981632

These may include private data but Telia may give details of these for reasonable purposes if separately requested. In practice they have exactly the same error that can be seen in the three CT logged certificates.

  1. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

It is still a common practice that SSL customers won't add SAN fields to CSR but instead the server name is added to CN field. Telia accepts such requests but in that case Telia automatically copies the value from CN to appropriate SAN field. It was originally incorrectly implemented and only SAN DNS was used. The IP address field was introduced to Telia CN-to-SAN code in early 2017. Before that in this special scenario the Telia SSL IP certificates may be formatted incorrectly. Because Telia, browsers, auditors or community didn't detect these before now these 12 incorrect certificates were in live production until now. Any/all of those parties could/should have detected these before now. No security threat known by Telia was posed to anybody but this was clearly against BR. The same would be detected by Telia systems (check next chapter).

  1. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.
  • All 12 this kind of illegal certificates were/will be replaced and revoked within 5 days from discovery. Mostly they were revoked within 24h.
    Telia has prevented similar to happen in the future by utilizing this:
  • This particular issue was fixed 9 January 2017. Error was possible only before that.
  • Now there are much more comprehensive Telia tests including lots of automatic test cases. New tests would probably disclose similar problem in the new Telia systems.
  • Regular weekly lint verification to all Telia SSL certificates. All Telia certificates are now verified against lint by Telia CA to disclose any irregularity.
  • CT logging. Telia is logging all pre-certificates to CT to let CT community find out if there are any issues.
Flags: needinfo?(pekka.lahtiharju)

Thank you for the prompt incident notification and report!

I have a few questions around improvements that would help avoid this type of incident in the future:

  1. Do you have an established procedure of a) checking previously issued certificates for specific misissuance and b) filing an incident report here each time you discover a bug and fix it in your CA system?
  2. Have you in the past or do you plan to lint all unexpired certificates that you've issued in order to discover overlooked misissuance like this?
Flags: needinfo?(pekka.lahtiharju)
  1. Do you have an established procedure of a) checking previously issued certificates for specific misissuance and b) filing an incident report here each time you discover a bug and fix it in your CA system?
    ANSWERS:
    a) Related to each new issue Telia has a procedure to do a database scan to find out if there are any similar issues in other Telia SSL certificates.

b) Current Telia policy is to file an incident report of all problems to Telia's internal Bugzilla system. In addition Telia will report here in Mozilla Bugzilla those cases that were reported by Mozilla community. Internal systems are used also in development phase to report issues found in testing.

  1. Have you in the past or do you plan to lint all unexpired certificates that you've issued in order to discover overlooked misissuance like this?
    ANSWERS:
    Only Telia certificates after ~June 2018 have been lint verified by Telia. Telia could do this kind of mass lint scan for older Telia certificates. Could you recommend how and which tools should be used if we do this. We prefer Linux and Python systems.
Flags: needinfo?(pekka.lahtiharju)

Could you recommend how and which tools should be used if we do this. We prefer Linux and Python systems.

ZLint (https://github.com/zmap/zlint) and ZCertificate are the tools I'd use for this, as they support batch processing of data well and emit JSON that you could then post-process using Python.

Just to clarify: should we run both zlint and zcertificate or is it enough to run zlint to be safe?

zcertificate will run zlint for you and has different output that may or may not fit your workflow better. You don't need to use both.

Pekka: please update this bug when all unexpired Telia certificates have been linted.

Telia has now done mass zlint scan for all its active publicly trusted SSL certificates. Mass scan revealed 9 new error certificates like this:

a) 2 CN IP value incorrectly in SAN DNS field (e_dnsname_not_valid_tld)(created in 2016)
b) 1 FQDN value incorrectly in SAN rfc822 field (e_ext_san_rfc822_name_present,e_subject_common_name_not_from_san)(created in 2016)
c) 1 CN SAN had leading space (e_dnsname_bad_character_in_label,e_subject_common_name_not_from_sa)(created in 2016)
d) 1 CN FQDN without domain part (e_dnsname_not_valid_tld)(created in 1Q2017)
e) 1 invalid wildcard format (e_dnsname_left_label_wildcard_correct)(created in 1Q2018)
f) 3 invalid OU value "-" (e_subject_contains_noninformational_value)(created in 2017-2Q2018)

Seven error certificates are revoked now and two are in replacement process. Telia has verified that only error e) was still possible (bug fix pending). All other bugs had been fixed before the scan.

a) 2 CN IP value incorrectly in SAN DNS field (e_dnsname_not_valid_tld)(created in 2016)

Are these certificates that have the same issue as this original incident report in this bug that were not detected/revoked previously?

It would be very useful if you could file separate incident reports for each distinct issue.

The two new IP errors on comment 10 were similar and were incorrectly dropped from our database scan results. Not sure why but perhaps because those had also normal valid SAN names in addition to invalid IP values. Fix to prevent those is anyway the same and was done in January 2017.

Now all existing active Telia SSL certificates are OK against zlint. Fixes are in use to prevent those same errors to happen again (e not yet in production). If any new lint error should happen we would detect that soon because of our regular weekly lint checking. Next we try to do lint verification daily or per certificate to react faster to possible errors.

Pekka: are you planning to file one or more incident reports covering the certificates described in comment #10(b)-(f)? Jonathan has kindly requested separate reports for each problem. I am most concerned that all the required information is provided, including the complete certificates and root cause analysis.

Flags: needinfo?(pekka.lahtiharju)

I created today four new incident reports covering error categories b),d),e),f). I believe that categories a) and c) were reported before because: New category c) certificate is basically the same that was in my previous report "Telia: Misissued certificate - invalid dnsName". I added details of the new space issue to that report. New category a) certificates are covered in this bz.

Flags: needinfo?(pekka.lahtiharju)

Serial numbers of the other affected certificates are:
C5BC673506CE07BC885DC32B1244A6DF
A3BC79A93D8013A2C7F7845C71115175
64B9FE86003EC4C4C6C0BCCE936D3BDA
58E4D161A728867C1AA872BA9EA3E989
AB0842CD66B6527ED2435E987D286DA5
0AD986CBA0DC598974F036AE012EF5D0
01B671D39EBE7042E9E9E39B2EAA133F
A8937999ED44E209C3181E20ED07E68B
E8C44962E84947BDB5534EFC33981632

These may include private data but Telia may give details of these for reasonable purposes if separately requested. In practice they have exactly the same error that can be seen in the three CT logged certificates.

Speaking only as a community member: can you please log these, where able, and describe the specific circumstances that prevent you from disclosing the rest?

I try to follow agreements at the time. I hesitate to log our old certificates because at the time of issuance our CPS and Customer agreement didn't mention that Telia will publish customer data or certificates. Previously in 2016 our CPS wording was: "All issued certificates are stored in the local database of the CA system. Certificates may also be published to other repositories if it is a part of the Telia CA Service or agreed with a Customer.". I'm afraid that text won't mean publishing is OK unless we agree that with the related Customers. Probably it wouldn't hurt but we try to keep our promises in CPS. Telia changed the agreement/CPS wording just before we started to use CT pre-certificate logging in Spring 2018. Now agreements allow us to publish all to CT logs. The already published ones were owned by Telia or created after the agreement change so I was able to disclose those.

Do you have some specific reason to get and publish our actual old revoked certificates? What is the benefit of that? Part of those were already found by CT agents so the error type can already be examined. I could deliver those privately to you for private investigation purposes if you want.

Flags: needinfo?(wthayer)

While it is highly recommended to do so, Mozilla does not require CAs to disclose specific misissued certificates.

It appears that all questions have been answered and remediation is complete.

Status: ASSIGNED → RESOLVED
Closed: 3 months ago
Flags: needinfo?(wthayer)
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.