Telia did mass lint scan to all its SSL certificates on Friday, February 8, 2019 to find all old certificates that are invalid. This mass scan revealed 9 previously unknown invalid certificates in 6 different error categories:
a) 2 CN IP value incorrectly in SAN DNS field (e_dnsname_not_valid_tld)(created in 2016)
b) 1 FQDN value incorrectly in SAN rfc822 field (e_ext_san_rfc822_name_present,e_subject_common_name_not_from_san)(created in 2016)
c) 1 CN SAN had leading space (e_dnsname_bad_character_in_label,e_subject_common_name_not_from_sa)(created in 2016)
d) 1 CN FQDN without domain part (e_dnsname_not_valid_tld)(created in 1Q2017)
e) 1 invalid wildcard format (e_dnsname_left_label_wildcard_correct)(created in 1Q2018)
f) 3 invalid OU value "-" (e_subject_contains_noninformational_value)(created in 2017-2Q2018)
Category d) incident report is in this bz. Others are in other incident reports created by Telia.
- How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.
Telia did mass lint scan to all its SSL certificates on Friday, February 8, 2019.
- A timeline of the actions your CA took in response.
Friday, February 8, 2019: Discovery by Telia itself
Friday, February 8, 2019: Preliminary analysis of the issue by Telia Security Board (e_dnsname_not_valid_tld). Not critical but invalid according to BR 188.8.131.52, rule7 thus decision was: "Should be revoked within 24 hours and MUST revoke a Certificate within 5 days"
Monday, February 11, 2019: Quick analysis that confirmed that similar error can't happen in current Telia systems.
February 11-15: Root cause analysis: Telia has added FQDN verification code that prevents values without TLD when such requirement came from BR several years ago. Normally prevention has worked but not in this particular case. We can see from the logs from March 30,2017 that code had detected this invalidity: "Wrong domains found; info=ev001eventprod". But nevertheless code then incorrectly accepted the CSR anyway. Next similar trial without TLD on the logs was correctly rejected few months later. Our hypothesis is that there was a bug related to error handling that was fixed soon after the accident. Because at the time there wasn't any reasonable way to run mass scan to our database we didn't scan older errors after the fix. Now Telia has a new CA software and mass database scans and also mass lint scans are possible and done after each major fix.
Friday, February 15, 2019: Revoked by Telia CA. Renewal of the certificate was started right after we found this problem. Renewal wasn't possible immediately so we let the invalid certificate live to maximum 5 days and then Telia CA operators revoked it and reported to me it was revoked. However, the original revocation failed and we only thought it was revoked in time. When I double-checked the status of all nine lint invalid certificates I found that the original revocation of this one was failed and operators revoked this one again successfully. Then I was able to verify it. To avoid this kind of delay in the future we decided to modify our process so that we verify all lint revocations within 5 days.
- Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.
Telia hasn't reproduced this error after this one. Can't happen again.
- A summary of the problematic certificates.
- The complete certificate data for the problematic certificates.
Only one of this kind exists. Affected certificate has CN+SAN value "ev001eventprod" and has serial 26649485c0947f748284e9d4c6bcfe8f and was created March 30, 2017 for three years. It can't be found from CT logs but Telia can give details of it if asked.
- Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
Check root cause analysis above why this happened. This wasn't detected before by Telia, browsers, auditors or community because a) it caused no problems, b) it wasn't logged to CT, c) Telia didn't do lint check for old certificates before now. If similar problem would happen now the issue would be detected because of regular lint checks by Telia.
- List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.
Telia has already resolved this like described above.