Complete incident report:
- How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.
Telia was informed by firstname.lastname@example.org on Tuesday, January 29, 2019 7:54:07 PM (UTC+02:00) about three invalid certificates that were issued in 2016. Problem was that IP value was stored incorrectly into DNS field. He was using Telia's problem reporting email address email@example.com to report these problems.
- A timeline of the actions your CA took in response.
Tuesday, January 29, 2019 7:54:07 PM (UTC+02:00): Problem report received
Wednesday, January 30, 2019 10:00 AM (UTC+02:00): Telia Security Board had meeting that confirmed the problem to be invalid dns name (IP address in DNS field; value was in SAN DNS instead of SAN IP because of an old Telia CA bug). Board estimated this problem against BR 18.104.22.168 and estimated that this wasn't posing any immediate security threat. This issue was BR revoke reason 7 which has requirement "CA SHOULD revoke a certificate within 24 hours and MUST revoke a Certificate within 5 days". These three certificates had been used more than two years and were still in active use. Telia decided to replace those ASAP without doing emergency shutdown/revocation to the related services. Also Board stated that this particular problem had been previously fixed (fix date was January 9, 2017) so that this can't happen again.
Thursday, January 31, 2019: Server operators were able to replace the problematic reported certificates and all three were revoked.
Friday, February 1, 2019: Telia did CA database check to verify if there are any other illegal certificates like these. 9 others were found and replacement for those was immediately started. 6 were revoked on Friday, 3 remaining for later revoke. These nine were also created in 2016 because of the same bug in CA at the time. Note: These nine are in private networks that may not be visible to CT agents. Preliminary incident report was created on Friday.
Monday, February 4, 2019: a new incident report (this) was created. The remaining illegal certificates will be revoked.
- Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.
All problematic certificates were issued in 2016. At the time there was a bug that copied IP formatted CN-value to SAN-DNS field instead of SAN-IP. Since 2017 this error hasn't happened and can't happen.
- A summary of the problematic certificates.
3+9 certificates had incorrect value (IP Address) in SAN DNS field. All were created in 2016 either by "TeliaSonera Gateway CA v2" or by "TeliaSonera Server CA v2". Otherwise certificates were OK, in use and browsers accepted those.
- The complete certificate data for the problematic certificates.
Full data of reported illegal certificates is:
Serial numbers of the other affected certificates are:
These may include private data but Telia may give details of these for reasonable purposes if separately requested. In practice they have exactly the same error that can be seen in the three CT logged certificates.
- Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
It is still a common practice that SSL customers won't add SAN fields to CSR but instead the server name is added to CN field. Telia accepts such requests but in that case Telia automatically copies the value from CN to appropriate SAN field. It was originally incorrectly implemented and only SAN DNS was used. The IP address field was introduced to Telia CN-to-SAN code in early 2017. Before that in this special scenario the Telia SSL IP certificates may be formatted incorrectly. Because Telia, browsers, auditors or community didn't detect these before now these 12 incorrect certificates were in live production until now. Any/all of those parties could/should have detected these before now. No security threat known by Telia was posed to anybody but this was clearly against BR. The same would be detected by Telia systems (check next chapter).
- List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.
- All 12 this kind of illegal certificates were/will be replaced and revoked within 5 days from discovery. Mostly they were revoked within 24h.
Telia has prevented similar to happen in the future by utilizing this:
- This particular issue was fixed 9 January 2017. Error was possible only before that.
- Now there are much more comprehensive Telia tests including lots of automatic test cases. New tests would probably disclose similar problem in the new Telia systems.
- Regular weekly lint verification to all Telia SSL certificates. All Telia certificates are now verified against lint by Telia CA to disclose any irregularity.
- CT logging. Telia is logging all pre-certificates to CT to let CT community find out if there are any issues.