Telia: Delayed revocation of seven (7) certificates related to incident 1896108
Categories
(CA Program :: CA Certificate Compliance, task)
Tracking
(Not tracked)
People
(Reporter: antti.backman, Assigned: antti.backman)
References
(Blocks 1 open bug)
Details
(Whiteboard: [ca-compliance] [leaf-revocation-delay] Next update 2025-03-03)
Incident Report
Summary
This report is supplementing incident reporting on this Telia CA incident report in Mozilla bugzilla and indicating Telia CA's failure to revoke misissued certificates as required by CA/Browser Forum Baseline Requirements for the Issuance and Management of Publicly‐Trusted TLS Server Certificates section 4.9.1.1 (12) and corresponding section in Telia Certificate Policy and Certification Practice Statement for Telia Server Certificates
.
Impact
Seven (7) end entity certificates failed to be revoked in 5 days due to issue described in the referenced incident report.
Timeline
Details of the timeline of events presented in the referenced incident report.
Telia CA continues to report on this incident until all affected certificates are revoked.
Root Cause Analysis
Services having the affected certificates installed require pre-planned change procedure to replace certificate and that could not be completed between Friday the 10th and Tuesday the 14th of May (00:17 UTC) and therefore the existing certificates could not be revoked without undue harm to the subscriber operations.
As reported on the primary incident, considering the fact that information on the affected certficates is unambiguous, Telia CA would not forcefully revoke the affected certificates, but continues to work with the certificate holders to complete the revocation as soon as possible.
Lessons Learned
What went well
- 6 certificates were successfully revoked within 5 days.
What didn't go well
- 7 certificates still remain to be revoked.
Where we got lucky
- The information in the certificates was still unambiguous indicating the correct and validated Country of the Subject.
Action Items
Action Item | Kind | Due Date |
---|---|---|
Revocation of misissued certificates by Telia CA and update on this incident of the completion of revocation. | Action (revocation) | Right after subscribers have renewed their certificates. |
Continuous follow-up / reminding of the renewal requirement of the affected certificate holders. | Action (follow-up / remind) | As long as last of the misissued certificates have been revoked |
Appendix
Details of affected certificates
Precertificate-list of end-entity certificates to be revoked:
- https://crt.sh/?id=11839878201 (Not Before: Jan 23 07:37:09 2024 GMT)
- https://crt.sh/?id=11840688066 (Not Before: Jan 23 09:18:34 2024 GMT)
- https://crt.sh/?id=11841544979 (Not Before: Jan 23 11:00:27 2024 GMT)
- https://crt.sh/?id=11842684205 (Not Before: Jan 23 13:23:44 2024 GMT)
- https://crt.sh/?id=11842698882 (Not Before: Jan 23 13:27:11 2024 GMT)
- https://crt.sh/?id=11842711133 (Not Before: Jan 23 13:28:11 2024 GMT)
- https://crt.sh/?id=11851399778 (Not Before: Jan 24 09:00:41 2024 GMT)
Assignee | ||
Updated•9 months ago
|
Updated•9 months ago
|
Assignee | ||
Comment 1•9 months ago
|
||
This is to update that following certificates have been revoked:
- https://crt.sh/?id=11842684205 (2024-05-15 04:54:35 UTC)
- https://crt.sh/?id=11842698882 (2024-05-15 04:56:56 UTC)
- https://crt.sh/?id=11842711133 (2024-05-15 04:58:44 UTC)
Telia continues to follow-up and report updates on this incident until all affected certificates are revoked. Next update at latest by Next Update.
Comment 2•9 months ago
|
||
(In reply to Antti Backman from comment #0)
Root Cause Analysis
Services having the affected certificates installed require pre-planned change procedure to replace certificate and that could not be completed between Friday the 10th and Tuesday the 14th of May (00:17 UTC) and therefore the existing certificates could not be revoked without undue harm to the subscriber operations.
As reported on the primary incident, considering the fact that information on the affected certficates is unambiguous, Telia CA would not forcefully revoke the affected certificates, but continues to work with the certificate holders to complete the revocation as soon as possible.
I believe it would help if there was some additional breakdown on the types of Subscribers and specific difficulties/challenges that were reported to Telia CA for this delayed revocation. Some good guidance could be obtained from a recent similar bug https://bugzilla.mozilla.org/show_bug.cgi?id=1896053. The more details shared, the better the community will understand specific situations and work on specific solutions/improvements.
Assignee | ||
Comment 3•9 months ago
|
||
Thank you Dimitris for your comment / feedback, in response to your request:
- General delaying factor is what we could say unfortunate timing of the issue report.
As we received the researcher’s report on the early morning hours local time the 9th of May, that day was national / public holiday in the countries where the certificate holders reside -> when we were in contact with the certificate holders, we got back initial responses that they had limited capacity available until the Monday 13th as some of the people were taking long weekend off, thus could not act immediately.
This is obviously unfortunate, but something that is to our opinion out of CA’s control.
For certificate holders that in fact could replace their certificates with short notice, we were able to do so by end of Monday the 13th (except for one insubordinate certificate holder, see below).
- Certificate holder unwilling to revoke.
In this case the subscriber / certificate holder has just refused our request to replace the certificate, arguing the issue be so minor that replacement would not be an urgent issue to them. We have been in contact with the person multiple times to explain why it is mandatory to replace.
In this case, we would take our lessons learned and try to better the background information of why it is so important to replace and revoke certificates when communicating with the certificate holders even if the issue might not be that severe.
Anyhow this is something that one may encounter and for a CA to forcefully execute its capability to revoke certificate unilaterally causing immediate harm to certificate holder is obviously something we should not do lightly.
Given the nature of the issue with certificate data, we chose to give the certificate holder some time to digest our request. But as we have not seen expected progress, we’ll take on more strict approach to get the particular certificate revoked.
- Nature of the services that the affected certificates are protecting
To change any component, security feature like certificate, the change must be managed through strict protocol / procedure. As explained in the general timing related issue, those service teams were only able to fully address the change from Monday the 13th onward.
When the procedural / administrative part of the change passes, those services (even they are built with redundancy) cannot be taken down during daytime local. Separate service window during the night time must be planned and required resources reserved to perform the service.
Consequently, these procedures could not be completed by the 5 days deadline early morning local time the 14th of May in the countries where the people capable to manage the change reside.
Telia continues to follow-up and report updates on this incident until all affected certificates are revoked. Next update at latest by Next Update.
"Certificate holder unwilling to revoke."
Your responsibility is to all the users of the webPKI, the relying partys, users of browsers and clients worldwide. NOT your 'subscribers' or customers or 'holders'.
Also, all certificate have 'Telia' in the name. They are for YOUR kompany. Not revoking or replacing these certificates is crazy.
The fact that you say you did not revoke because "certificate holder unwilling to revoke" - that says you should not be a trusted CA.
Mozilla MUST begin the process to distrust Telia as a CA.
Comment 5•9 months ago
|
||
Dear JR Moir,
Did you consider that the wording may have been unlucky because the person who wrote it is not native english speaking?
I don't understand where this call for a letter-by-letter strict interpretation of the BRs is coming from. From my perspective, the WebPKI ecosystem is not at danger of falling apart. But of course, I'm biased as a CA pays my salary. ;)
Rgds
Roman
Assignee | ||
Comment 6•9 months ago
|
||
In response to JR Moir,
As we tried to explain, the initial response from the certificate holder was as we have openly described, to illustrate the different reactions by individual Subscribers may have in this type of cases.
Further we have been in constant contact with the respective subscriber and in fact today after we took the explained "stricter approach" and the subscriber has started the required replacement process.
Even if they indeed are internal subscribers, we treat them with same respect as any other subscriber of Telia CA. Given that the information on the certificate is unambiguous and only matter of incorrect case, forceful revocation of the certificate would have had unnecessary impact on the given service protected by the certificate.
To Roman, thank you for your comment and yes I do see that it may have been somewhat clumsy language on our side as we are not native english speaking.
Telia continues to follow-up and report updates on this incident until all affected certificates are revoked. Next update at latest by Next Update.
Comment 7•9 months ago
|
||
(In reply to Antti Backman from comment #3)
Antti,
I must say that this comment fundamentally misunderstands the nature of the relationship between a public CA and its Subscribers and the CA’s responsibilities to the WebPKI. It is vital for not only Telia but every CA with at least one delayed revocation issue to internalize the idea that your primary “customer” is not your paying Subscriber but all the world’s internet users.
We public CAs have the extraordinary privilege that we are allowed to be gatekeepers for the global, public trust model modern computing systems, and therefore humanity, depend on. This is an incredibly rare privilege available to a mere few dozen organizations in the entire world. And it is a privilege that can be taken away at any time.
Let’s look at some of your specific statements and break down how they are misguided.
when we were in contact with the certificate holders, we got back initial responses that they had limited capacity available until the Monday 13th as some of the people were taking long weekend off, thus could not act immediately.
Yikes, what a way to start. What you are saying here is that this Subscriber (which comment 4 rightly points out is your own company) is woefully inadequate to run an IT operation. If a long weekend is sufficient to fundamentally compromise this company’s security and resiliency, then that bodes ill for ideas like uptime and remaining breach-free. This is a smear on any company’s reputation and is especially bad for a public CA, of all things.
I have to suspect that had you proceeded with revocation, then the relevant stakeholders would have discovered that they could, in fact, replace these certificates without a catastrophic outage. I imagine, rather, that they simply didn’t want to be bothered. And by indulging that attitude, you enable it.
This is obviously unfortunate, but something that is to our opinion out of CA’s control.
Of course not. This statement is objectively false on its face. It is 100% in the CA’s control to proceed with the revocation according to the guidelines the CA has promised to follow.
In this case the subscriber / certificate holder has just refused our request to replace the certificate, arguing the issue be so minor that replacement would not be an urgent issue to them. We have been in contact with the person multiple times to explain why it is mandatory to replace.
No, no, sorry but no. The Subscriber does not have the option to refuse the mandatory revocation, just as you do not have the option to neglect it. The fact that you use a word like request is indicative of basic misunderstanding. This is not a request. Revocation is a mandatory action for you to perform, and if the Subscriber wants to remain operational, it will be mandatory for this Subscriber to replace the revoked certificates with others from you or someone else.
unilaterally causing immediate harm to certificate holder is obviously something we should not do lightly.
You seem to labor under the misimpression that you have decision making power here. You do not.
But for the sake of discussion, could you explain what harm the certificate holder would have faced that you feel outweighed the needs of the world’s Relying Parties?
To change any component, security feature like certificate, the change must be managed through strict protocol / procedure. As explained in the general timing related issue, those service teams were only able to fully address the change from Monday the 13th onward.
If your Subscriber is incapable of dealing with a very standard, and very small, Certificate Agility event, then your Subscriber is inadequate to its IT operational requirements. That is not your problem, even if it is the same company. Your problem is to obey your obligations as a public CA, and the Subscriber’s problem is to figure out how to operate as a modern IT organization.
When the procedural / administrative part of the change passes, those services (even they are built with redundancy) cannot be taken down during daytime local.
This is not your problem to solve. Tens of thousands of enterprises around the globe manage to perform certificate operations despite the demands of round-the-clock availability. Your Subscriber can and must find a way to do what these many thousands of other companies do routinely.
Consequently, these procedures could not be completed by the 5 days deadline
Your use of could here is inappropriate. Absolutely Telia could. Telia chose not to. These are different things.
Can you explain what would have happened if the Subscriber had suffered a private key compromise, in which case you would have had to revoke the certificates within 24 hours?
And finally, from comment 6
Given that the information on the certificate is unambiguous and only matter of incorrect case, forceful revocation of the certificate would have had unnecessary impact on the given service protected by the certificate.
The Mozilla policy, which so many CAs cling to despite the fact that Mozilla is only one of the root programs they must comply with, explicitly clarifies that claiming that an issuance error does not constitute a security concern is inadequate justification for delayed revocation.
Let me reinforce that my comments here apply to every single CA with an incident for the willful delay of mandatory revocation. If every CA with a current delayed revocation issue is monitoring this bug, which they are all supposed to be, then they need to assimilate the points in this message and apply them to every revocation event from here on out.
(reply to: https://bugzilla.mozilla.org/show_bug.cgi?id=1896553#c5)
I did not consider translation to be a problem, English is not my native language also.
"I don't understand where this call for a letter-by-letter strict interpretation of the BRs is coming from."
A CA is required to follow the guidelines in order to be trusted as a CA. If a CA cannot follow simple parts of those guidelines, such as revoking a certificate in a set time - what other things might they also choose to ignore?
If the rules are 'not right' then discussion and ballot to change them. Do not ignore them with weak excuses.
"From my perspective, the WebPKI ecosystem is not at danger of falling apart."
It is good your opinion does not matter. Others may think that CAs ignore simple parts of required guidelines indicates they are not to be trusted.
(reply to: https://bugzilla.mozilla.org/show_bug.cgi?id=1896553#c6)
Thank-you, Antti. It is good that you now seem to be OK to make subscribers do what you should have asked them to do some time ago.
Perhaps the public involvement and comments made you do what you simply did not want to before?
This is unrelated to this bug but Roman I'm deeply concerned with what you've mentioned here:
I don't understand where this call for a letter-by-letter strict interpretation of the BRs is coming from. From my perspective, the WebPKI ecosystem is not at danger of falling apart. But of course, I'm biased as a CA pays my salary. ;)
That's...the requirement for being a CA? The unfortunate implication here is that swisssign has a different interpretation of what the rules for being a CA are. That implication is genuinely concerning.
the WebPKI ecosystem is not at danger of falling apart
The only thing keeping this ecosystem from another diginotar incident is strict compliance with the rules set by the BRs. I've worked in two CAs in my life (Let's Encrypt, and Google Trust Services), and I do not share your view on the dangers WebPKI is facing. The entire ecosystem for WebPKI is always just a few steps from falling apart.
Antti, one of the requirements for incident response is action items so that this incident doesn't repeat itself into the future. In this incident response, I do not see any such action items that will prevent a delayed revocation in the future.
Does Telia have any intentions to ensure future delayed revocation incidents do not happen, and that Telia does not delay revocations where necessary in the future? If so, please explain with action items how you're going to accomplish that.
Assignee | ||
Comment 10•9 months ago
|
||
This is to update that following certificate is revoked:
https://crt.sh/?id=11840688066
To Amir,
Antti, one of the requirements for incident response is action items so that this incident doesn't repeat itself into the future. In this incident response, I do not see any such action items that will prevent a delayed revocation in the future.
We're posting full incident report 2024-05-17 for Bug 1896108 and in that we're addressing action items related to your request.
To Jr Moir,
Thank-you, Antti. It is good that you now seem to be OK to make subscribers do what you should have asked them to do some time ago.
Perhaps the public involvement and comments made you do what you simply did not want to before?
Thank you for your feedback, we welcome all suggestions, comments as an opportunity to improve and better our practices.
Telia continues to follow-up and report updates on this incident until all affected certificates are revoked. Next update at latest by Next Update.
Assignee | ||
Comment 11•9 months ago
|
||
Tim,
In response to your extensive comments:
I must say that this comment fundamentally misunderstands the nature of the relationship between a public CA and its Subscribers and the CA’s responsibilities to the WebPKI. It is vital for not only Telia but every CA with at least one delayed revocation issue to internalize the idea that your primary “customer” is not your paying Subscriber but all the world’s internet users.
We public CAs have the extraordinary privilege that we are allowed to be gatekeepers for the global, public trust model modern computing systems, and therefore humanity, depend on. This is an incredibly rare privilege available to a mere few dozen organizations in the entire world. And it is a privilege that can be taken away at any time.
We're fully aware of our responsiblity as a CA in relation to the WebPKI and the information provided with my earlier response had no intention to provide excuses justifying delayed revocation. I was just responding to the requestor about topics / issues we face with subscribers having challenges to replace certificates within regulated time frames. As a CA we recognize the privileged position we have and be sure that we work hard on day-to-day basis to make sure we fulfil and adhere to the requirements set forth for a Public CA. I feel like based on your comments that my intention did not come through with the response.
The justification and security assessment we conducted when reviewing and responding to the external issue report took very careful consideration of the security impact on WebPKI, reying parties and certificate consumers. Telia CA promptly addressed the identified issue as described in the main incident Bug ID 1896108 (further to be extended with final / full incident report 2024-05-17). With that said, we acknowlege our responsiblity to execute revocation as required by external requirements, we want to make this absolutely clear. We see that this incident on itself is proof of understanding our responsiblity to report on as per external requirements.
With the initial incident report provided we also indicated the initial responses we got back from the subscribers, there's a likely possibility that some of the revocation could be delayed, again this is not make delayed revocation justified by any means. We submitted this incident report as required and are continuously working with the affected subscribers to be able to revoke the remaining three (3) certificates as soon as possible. We also feel that the continuous progress to revoke the affected certificates serves as proof us taking on required steps to resolve the incident.
As the community also recognize the fact that incidents do happen at times and this public communication is excellent opportunity to the CA concerned as well as for the whole community to learn from each other and further strenghten the security of WebPKI together. We Telia CA take all responses to our incidents welcome and see those as opportinity to improve our practices to avoid such incidents from recurring.
We would like to make note on comments / opinions on Bug ID 1896108 for the main incident, as there seems to be different view points in the community, if the underlying issue with issued certificates concerned in this incident would infact be in violation of the requirements. We concluded in our review and our interpretation took the position that unfortunately the issue should be interpreted as a violation of TLS BR. We see that this interpretation is also indication of our understanding of the responsiblity to the WebPKI as a Public CA and shows the seriousness we address any incident being it at any level signficancy.
Yikes, what a way to start. What you are saying here is that this Subscriber (which comment 4 rightly points out is your own company) is woefully inadequate to run an IT operation. If a long weekend is sufficient to fundamentally compromise this company’s security and resiliency, then that bodes ill for ideas like uptime and remaining breach-free. This is a smear on any company’s reputation and is especially bad for a public CA, of all things.
Yes Telia Certifcate Authority is part of Telia Company, which is serving 25 million customers across the Nordic and Baltic regions with essential digital infrastructure and digital services that are fundamental enablers of the digital societies we live in. Telia is the telecommunications leader in the region, the leading Nordic media house, and the leader in ICT in both Finland and the Baltics.
While we're part of the enterprise, Telia CA is not controlling the IT operations other than those directly supporting the CA. How the IT operations supporting telecommunication networks and services is governed, is outside the scope of Telia CA, and is the responsiblity of other Telia corporate functions. I apologise if I have misunderstood you're comment, but to me it looks as if you refer the whole Telia Company equally to Telia CA, which is not the case and I hope I have been able to clarify that topic with this comment.
I have to suspect that had you proceeded with revocation, then the relevant stakeholders would have discovered that they could, in fact, replace these certificates without a catastrophic outage. I imagine, rather, that they simply didn’t want to be bothered. And by indulging that attitude, you enable it.
I believe that the fact we have revoked 6 of the 13 certficates identified in the Bug ID 1896108 before the 5 days deadline, shows clear indication of the intention and appropriate actions taken with the subscribers to find possiblity to revoke affected certificates in due time.
Of course not. This statement is objectively false on its face. It is 100% in the CA’s control to proceed with the revocation according to the guidelines the CA has promised to follow.
No, no, sorry but no. The Subscriber does not have the option to refuse the mandatory revocation, just as you do not have the option to neglect it. The fact that you use a word like request is indicative of basic misunderstanding. This is not a request. Revocation is a mandatory action for you to perform, and if the Subscriber wants to remain operational, it will be mandatory for this Subscriber to replace the revoked certificates with others from you or someone else.
Agree, and therefore we have had to post this delayed revocation incident. But as it was requested to further elaborate topics, responses and issues CA might face over time and cosenquently resulting delayed revocation incidents, these are issues we have faced in relation to handling this incident.
You seem to labor under the misimpression that you have decision making power here. You do not.
But for the sake of discussion, could you explain what harm the certificate holder would have faced that you feel outweighed the needs of the world’s Relying Parties?
The services secured by the certificates that suffer delayed revocation are protecting nationwide critical telecommunication network of which societies are dependent on. To excecute forceful revocation of the certificates which are, yes to our interpretation violating CA/B TLS BR as indicated in the Bug ID 1896108, suffering from character case issue, but still containing unambiguous and internationally recognized value of the country in concern 'se' as opposed to 'SE', would have had more severe impact.
If your Subscriber is incapable of dealing with a very standard, and very small, Certificate Agility event, then your Subscriber is inadequate to its IT operational requirements. That is not your problem, even if it is the same company. Your problem is to obey your obligations as a public CA, and the Subscriber’s problem is to figure out how to operate as a modern IT organization.
This is not your problem to solve. Tens of thousands of enterprises around the globe manage to perform certificate operations despite the demands of round-the-clock availability. Your Subscriber can and must find a way to do what these many thousands of other companies do routinely.
Agree, we take this feedback with us and look into our possiblities to help our subscribers internal and external to be prepared on cases like this in the future.
Your use of could here is inappropriate. Absolutely Telia could. Telia chose not to. These are different things.
We're sorry if we chose the wrong word here, what the comment explained is the issues / topics we face from CA's perspective. Not implying that Telia CA would not been able to revoke the certificates or making any excuses to justify delayed revocation.
Can you explain what would have happened if the Subscriber had suffered a private key compromise, in which case you would have had to revoke the certificates within 24 hours?
This comment is purely speculative, significant security issue like private key compromise would have initiated appropriate and adequate actions to revoke certificates within 24 hours.
And finally, from comment 6
The Mozilla policy, which so many CAs cling to despite the fact that Mozilla is only one of the root programs they must comply with, explicitly clarifies that claiming that an issuance error does not constitute a security concern is inadequate justification for delayed revocation.
Agree, and we hope we have made that clear with this response as well, thus unfortunately having to deal with this incident. Be assured that we take all comments with adequate seriousness and make sure going forward to avoid this type of incident recurring on our behalf.
Telia continues to follow-up and report updates on this incident until all affected certificates are revoked. Next update at latest by Next Update.
Assignee | ||
Updated•9 months ago
|
Comment 12•9 months ago
|
||
This comment is purely speculative, significant security issue like private key compromise would have initiated appropriate and adequate actions to revoke certificates within 24 hours.
I disagree with this. If the reason being cited for not revoking within 120 hours is that:
The services secured by the certificates that suffer delayed revocation are protecting nationwide critical telecommunication network of which societies are dependent on.
Then what's going to happen to the aforementioned societies if a key compromise has happened?
This is I think what the main problem is. If you put your subscribers ahead of the WebPKI requirements now, there is simply not a single reason we have to believe you wouldn't do that if the incident was a much more serious issue.
Lets look at it another way. Let's say there is a CA that never puts their subscribers ahead of the rules of the WebPKI. This CA is objectively in a better security model than a CA that does put its subscribers ahead of the rules of WebPKI. However, that objectively better CA will not get as many subscribers compared to the objectively worse CA.
Over time, this means that WebPKI is going to degrade more and more as other CAs realize that to get customers (and, at the end of the day, money), they need to start being an objectively worse CA. Realistically what this means is either 1) CAs need to not let this degradation of services to take place, or 2) root programs need to start protecting WebPKI by removing misbehaving CAs. I suspect one of these options sounds more favorable to CAs.
Also,
We're posting full incident report 2024-05-17 for Bug 1896108 and in that we're addressing action items related to your request.
That incident report is an entirely different incident from this. Yet it did lead to this, but it is not the root cause of failure to revoke in the appropriate time. So I'm going to push back and say that I expect action items that specifically address how Telia will ensure they won't have a delayed revocation in the future to be published on this bug.
We also still have not seen a per-subscriber breakdown of why they revocation is being delayed. So I expect Telia to also follow that requirement by the Mozilla Root Program.
Assignee | ||
Comment 13•9 months ago
|
||
To Amir,
We will update requested information on this incident on Next Update.
Updated•9 months ago
|
Assignee | ||
Comment 14•9 months ago
|
||
Following certificate is revoked: https://crt.sh/?id=11839878201, at 2024-05-21 12:08:58
Action Items
Action Item | Kind | Due Date |
---|---|---|
Revocation of misissued certificates by Telia CA and update on this incident of the completion of revocation. | Action (revocation) | Right after subscribers have renewed their certificates. |
Ensure and enforce existing revocation policy and practice defined in Telia TLS CP/CPS to ensure adherence to the revocation requirements to avoid future delayed revocation incidents for all applicable Trusted Role persons in Telia CA. | Action (prevent) | 2024-05-31 |
Internal to Telia, include revocation instructions in the internal periodic certificate lifecycle control policy, with requirement to explicitly review and understand the importance of timely performance of revocations as requested by Telia CA. Control to be implemented by Telia Corporate Security, schedule not in control of Telia CA. To be updated on this incident when deployed. | Action (improve, prevent) | |
Subscriber info / news in the certificate management portal of required revocation practices to better the awareness of necessary actions in case certificates must be revoked as per requirements. | Action (inform, prevent) | From 2024-06 until end of Q3 2024, permanent notification. |
Per certificate breakdown of delayed revocation reasons
Following certificates were delayed in revocation because the services secured by the certificates that suffer delayed revocation are protecting nationwide critical telecommunication network of which societies are dependent on.
https://crt.sh/?id=11840688066 (Not Before: Jan 23 09:18:34 2024 GMT)
https://crt.sh/?id=11841544979 (Not Before: Jan 23 11:00:27 2024 GMT)
https://crt.sh/?id=11842684205 (Not Before: Jan 23 13:23:44 2024 GMT)
https://crt.sh/?id=11842698882 (Not Before: Jan 23 13:27:11 2024 GMT)
https://crt.sh/?id=11842711133 (Not Before: Jan 23 13:28:11 2024 GMT)
https://crt.sh/?id=11851399778 (Not Before: Jan 24 09:00:41 2024 GMT)
Susbcriber initially disputed the request by Telia CA to replace the certificate and enable revocation.
https://crt.sh/?id=11839878201 (Not Before: Jan 23 07:37:09 2024 GMT)
Certificates are pending revocation
Telia CA and subscriber are working on to replace the certifcates in following days to come and revoke following certificates.
https://crt.sh/?id=11841544979
https://crt.sh/?id=11851399778
Next update by Next Update.
Assignee | ||
Updated•9 months ago
|
Assignee | ||
Comment 15•9 months ago
|
||
This is to update that pending revocation of these certificates have completed:
https://crt.sh/?id=11851399778, 2024-05-23 08:30:57 UTC
https://crt.sh/?id=11841544979, 2024-05-23 08:32:40 UTC
Action Item:
"Revocation of misissued certificates by Telia CA and update on this incident of the completion of revocation."
is then completed.
Comment 16•9 months ago
|
||
Why is a webPKI certificate - which has requirements for revocation in 1 or 5 days - being used for "protecting nationwide critical telecommunication network of which societies are dependent on." and cannot be replaced in those times?
Did Telia (CA) or Telia (Subscriber) make that mistake?
Assignee | ||
Comment 17•9 months ago
|
||
Hi JR,
As a CA we do not control how subscriber (whether internal or external) decide to deploy the certificate(s), but we'll see your point. We'll look into opportunity to have discussion with the peers internally, to see if there's places where there could be changes considered to provide same level of security where the public trust may not be needed.
We would see that the use of publicly trusted TLS certificates to protect services in these type of use cases isn't necessarily a bad choice.
Comment 18•9 months ago
|
||
(In reply to Antti Backman from comment #17)
Hi JR,
As a CA we do not control how subscriber (whether internal or external) decide to deploy the certificate(s), but we'll see your point.
That may be true, but as CA you do have the legally-protected ability to immediately revoke certificates as required by the BRs. All your subscribers must be legally bound to an agreement that they “acknowledge and accept” such revocation can occur, as part of the BR requirements in 9.6.3, item 8.
Is Telia itself not legally bound to accept that revocation, in this odd self-dealing case? Why would Telia assert that it would accept immediate revocation, when required by the BRs, if that is not in fact the case?
Assignee | ||
Comment 19•9 months ago
|
||
Hi Mike,
Yes the same subscriber obligations apply to Telia as a subscriber as for any other subscriber and acceptance of Terms is required from the Telia subscribers.
Assignee | ||
Comment 20•9 months ago
|
||
This is to update that this action item is completed today:
"Subscriber info / news in the certificate management portal of required revocation practices to better the awareness of necessary actions in case certificates must be revoked as per requirements."
Information notice / bulletin available publicly through our TLS Single certificate order
Next update by Next Update
Assignee | ||
Updated•9 months ago
|
Assignee | ||
Comment 21•9 months ago
|
||
This is to update that remaining open action times on this incident have been completed.
- Ensure and enforce existing revocation policy and practice defined in Telia TLS CP/CPS to ensure adherence to the revocation requirements to avoid future delayed revocation incidents for all applicable Trusted Role persons in Telia CA.
- Internal to Telia, include revocation instructions in the internal periodic certificate lifecycle control policy, with requirement to explicitly review and understand the importance of timely performance of revocations as requested by Telia CA. Control to be implemented by Telia Corporate Security, schedule not in control of Telia CA. To be updated on this incident when deployed.
Next update by Next Update
Assignee | ||
Comment 22•9 months ago
|
||
No updates at this update, Telia continues to follow up this incident.
Next update by Next Update
Assignee | ||
Updated•9 months ago
|
Assignee | ||
Comment 23•8 months ago
|
||
No updates at this update, Telia continues to follow up this incident.
Next update by Next Update
Assignee | ||
Updated•8 months ago
|
Assignee | ||
Comment 24•8 months ago
|
||
No updates at this update, Telia continues to follow up this incident.
As all action items have been completed about a month ago, we kindly ask if this incident could be closed.
Comment 25•8 months ago
|
||
I will schedule for this to be closed on or about next Wed. 10-July-2024.
Comment 26•7 months ago
|
||
(In reply to Antti Backman from comment #24)
As all action items have been completed about a month ago, we kindly ask if this incident could be closed.
Antii,
You still have not dealt with the problem that your Root Cause Analysis is incorrect and that you have no Action Items to address the real root cause. I explained at length this failure in comment 7. Your reply in comment 11 failed to acknowledge your ultimate responsibility for ensuring that revocations occur on time, and other commentors’ subsequent feedback in comment 12 and comment 18 haven’t created any change in your position.
Mozilla gave us a good roadmap for dealing with the deliberate decision to miss revocation in bug 1889062 comment 15, where among other things Mozilla prescribes,
- detailed changes to policies and procedures to ensure timely revocation, including new guidelines, checklists, and approval processes; and
- monitoring and auditing to ensure compliance with such policies and procedures and to identify any lapses quickly
I believe you need to publish a Root Causes Analysis that acknowledges your CA’s sole responsibility for on-time revocation and credible Action Items that will address this problem, such as those given above, before this issue can be closed.
Ben, given the grilling that Entrust and several other CAs have received recently in relation to the exact same sort of bug, I have to push back on your comment 25. I recommend leaving this issue open until Telia deals with these matters.
Assignee | ||
Comment 27•7 months ago
|
||
Hi Tim,
We'll address the topics in the next weekly update on Friday the 12th.
Assignee | ||
Comment 28•7 months ago
|
||
In response to comment #26:
Root Cause Analysis
your Root Cause Analysis is incorrect
Supplementing and clarifying the initial Root Cause analysis provided in the incident report, root cause leading to this incident was that based on the explained risk assessment and judgment we did not revoke the affected certificates in time, which was incorrect (and bad) decision by us consquently leading to this incident.
Action Items to address the real root cause
you have no Action Items to address the real root cause
Action Item listed on comment #14 and repeated below for reference, we have addressed
- detailed changes to policies and procedures to ensure timely revocation, including new guidelines, checklists, and approval processes; and
- monitoring and auditing to ensure compliance with such policies and procedures and to identify any lapses quickly
After review of the details on this incident, we can see that the language could have been more explicit to clearly indicate that as part of below Action Item we reviewed and supplemented our internal instructions in relation to revocation handling. The updates were supplemented to Certificate Problem handling guidelines and other processes possibly requiring timely revocation of certificates. By these updates we ensure that should issues requiring certificate revocation occur in the future, we avoid missing the revocation deadlines as required by Root Program policies and BRs and enforced over to our subscribers by our subscriber agreement.
Action Item | Kind | Due Date |
---|---|---|
Ensure and enforce existing revocation policy and practice defined in Telia TLS CP/CPS to ensure adherence to the revocation requirements to avoid future delayed revocation incidents for all applicable Trusted Role persons in Telia CA. | Action (prevent) | 2024-05-31 |
Acknowledge of responsiblity
We as a publicly trusted CA are fully aware and acknowledge our responsiblity for ensuring revocations occur on time. We have taken seriously the hard lessons learned through the expensive scrutiny and comments by the community on this incident.
Telia continues to follow-up this incident and provides updates on weekly basis.
Comment 29•7 months ago
|
||
(In reply to Antti Backman from comment #28)
Perfectly expressed, Antti. Right now, with so many comments falling on deaf ears, it is refreshing to witness genuine receptiveness and positive change.
This is exactly the response I was hoping to see. One of the main purposes of Bugzilla incident reporting is for CAs to learn from each other and improve their practices. Other CAs are digging in their heels and denying or ignoring their responsibilities. You have listened, considered, and improved. I credit Telia for its humility and open-mindedness. These public statements are very powerful, with recent events illustrating the potential consequences when a CA ignores clear commitments it made in the past.
A lot of CAs are having trouble making the intellectual leap we have witnessed in Telia, and I will be holding your comment 28 up as an example for other CAs to follow.
Assignee | ||
Comment 30•7 months ago
|
||
(In reply to Tim Callan from comment #29)
We thank you for your appreciative feedback.
This is also to give our weekly update, no updates this week.
Telia continues to follow-up this incident and provides updates on weekly basis.
Assignee | ||
Comment 31•7 months ago
|
||
This is our weekly update, no updates this week.
Telia continues to follow-up this incident and provides updates on weekly basis.
Assignee | ||
Comment 32•7 months ago
|
||
This is our weekly update, no updates this week.
Telia continues to follow-up this incident and provides updates on weekly basis.
Assignee | ||
Comment 33•6 months ago
|
||
This is our weekly update, no updates this week.
Telia continues to follow-up this incident and provides updates on weekly basis.
Assignee | ||
Comment 34•6 months ago
|
||
This is our weekly update, no updates this week.
Telia continues to follow-up this incident and provides updates on weekly basis.
Assignee | ||
Comment 35•6 months ago
|
||
This is our weekly update, no updates this week.
We have not received further comments on this incident for about a month now, so we kindly request this incident to be closed.
Assignee | ||
Comment 36•6 months ago
|
||
This is our weekly update, no updates this week.
We have not received further comments on this incident for a month now, so we kindly request this incident to be closed.
Comment 37•6 months ago
|
||
I'll leave this flagged for my attention next week.
Assignee | ||
Comment 38•6 months ago
|
||
This is our weekly update, no updates this week and now further updates expected as all action items completed and we have not received any comments, questions for over a month now.
Updated•6 months ago
|
Assignee | ||
Comment 39•5 months ago
|
||
Hi Ben,
We kindly request this incident to be closed. It's been over 2 months since last comment on this incident and we have complete all action items.
Comment 40•5 months ago
|
||
I will try and address this bug along with several other delayed revocation bugs prior to October 1, 2024.
Assignee | ||
Comment 41•5 months ago
|
||
Thank you Ben,
Telia CA will be monitoring this incident and provide our next update by Next Update set by you on this bug (if the bug is not closed before that).
Assignee | ||
Comment 42•5 months ago
|
||
This is our update on Next Update, no updates at this update. Requesting the Next Update to be set the 15th of October 2024 if this incident is not closed prior the 15th of October.
Updated•5 months ago
|
Assignee | ||
Comment 43•4 months ago
|
||
This is our update on Next Update, no updates at this update. Requesting the Next Update to be set the 30th of November 2024 if this incident is not closed prior that date.
Assignee | ||
Comment 44•4 months ago
|
||
Hi Ben,
Would it be possible to have this incident closed?
Updated•4 months ago
|
Comment 45•3 months ago
|
||
We continue work on incident-reporting and compliance requirements aimed at reducing delayed revocation, so this bug will remain open until at least February 1, 2025. Meanwhile, CAs should review https://github.com/mozilla/www.ccadb.org/pull/186.
Assignee | ||
Comment 46•21 days ago
|
||
This is our update on Next Update, no updates.
Requesting the Next Update to be set the 1st of March 2025, if this incident is not closed prior that date.
Updated•20 days ago
|
Updated•16 days ago
|
Assignee | ||
Comment 47•12 days ago
|
||
Incident Report Closure Summary
- Incident Description: This report is Telia CA's failure to revoke misissued certificates as required by CA/Browser Forum Baseline Requirements for the Issuance and Management of Publicly‐Trusted TLS Server Certificates section 4.9.1.1 (12) and corresponding section in
Telia Certificate Policy and Certification Practice Statement for Telia Server Certificates
. - Incident Root Cause(s): Root cause leading to this incident was that based on the explained risk assessment and judgment we did not revoke the affected certificates in time, which was incorrect decision by us consquently leading to this incident.
- Remediation Description: Action Item we reviewed and supplemented our internal instructions in relation to revocation handling. The updates were supplemented to Certificate Problem handling guidelines and other processes possibly requiring timely revocation of certificates. By these updates we ensure that should issues requiring certificate revocation occur in the future, we avoid missing the revocation deadlines as required by Root Program policies and BRs and enforced over to our subscribers by our subscriber agreement.
- Commitment Summary: We as a publicly trusted CA are fully aware and acknowledge our responsiblity for ensuring revocations occur on time. We have taken seriously the hard lessons learned through the scrutiny and comments by the community on this incident.
Comment 49•12 days ago
|
||
Sectigo supports closing this bug. We would like to remind the community that Telia handled it correctly. Telia listened to feedback from the community, acknowledged that its original policy had been in error, and committed to change.
We presently have an environment where, unfortunately, too many CAs still refuse to meet these basic requirements. I would like to encourage all CAs to follow the example Telia sets here.
Updated•6 days ago
|
Description
•