Digicert: Delayed Revocation for bug 1894560
Categories
(CA Program :: CA Certificate Compliance, task)
Tracking
(Not tracked)
People
(Reporter: jeremy.rowley, Assigned: dcbugzillaresponse)
References
(Blocks 1 open bug)
Details
(Whiteboard: [ca-compliance] [leaf-revocation-delay])
Attachments
(2 files, 2 obsolete files)
User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36
Steps to reproduce:
Per our updated incident post (https://bugzilla.mozilla.org/show_bug.cgi?id=1894560), DigiCert is not revoking all affected certificates within the 5-day timeframe required by Section 4.9.1 of the Baseline Requirements. This is a preliminary report on why and will be followed up with a full listing of delayed certificates, proposed revocation dates, reasons why revocation is delayed, and information about the automation being used by our customers. We will be revoking a significant number on May 11th with the remainder according to a proposed timeline.
There are several reasons that DigiCert is filing a delayed revocation incident report.
First, per 4.9.1.1, the CAB Forum Baseline Requirements requires revocation within five days if “the CA is made aware that the Certificate was not issued in accordance with these Requirements or the CA’s Certificate Policy or Certification Practice Statement (CRLReason #4, superseded);”. As explained in the full incident report, DigiCert was made aware that there were certificates with incorrect capitalization in the businessCategory attribute back in September 2023. Although the certificates were categorized as false positives, there is a reading of the requirements that would start the five-day clock back then. For full transparency, none of the certificates were revoked five days after we were originally alerted in 2023 because it was not considered a compliance issue.
Second, our data engineering team pulled a list of impacted certificates on April 29, 2024. When reviewing the data pull, errors were discovered that made us doubt the data, both from a completeness and accuracy viewpoint. Revocation kickoff did not occur at that time. Instead, we repaired the query error and gathered a complete list of certificates. DigiCert had a complete list of data and set out revocation notices at 16:00 UTC on May 6th,. We began the five-day clock at that time. The revocation email informed all impacted customers that revocation would occur on May 11th at 16:00 UTC.
Third, several customers have notified us with concerns about the revocation, including customers with automation. There are several contributing factors to this. For example, there are several holidays around the world, including Japan and Europe. Another issue is trying to explain why the guidelines require revocation even if X.520 says the field is case insensitive. Several customers believed this notification to be fake due to the difficulty in understanding something like this would require a five-day replacement window. Third, some installs require a restart, which even automated deployments are reluctant to perform outside of their change management window or while a disaster recovery exercise is ongoing. Despite these obstacles, we are only delaying revocation for instances that meet the Mozilla minimum standard for revocation delays:
“Mozilla recognizes that in some exceptional circumstances, revoking the affected certificates within the prescribed deadline may cause significant harm, such as when the certificate is used in critical infrastructure and cannot be safely replaced prior to the revocation deadline, or when the volume of revocations in a short period of time would result in a large cumulative impact to the web. However, Mozilla does not grant exceptions to the BR revocation requirements. It is our position that your CA is ultimately responsible for deciding if the harm caused by following the requirements of the Baseline Requirements outweighs the risks that are passed on to individuals who rely on the web PKI by choosing not to meet this requirement.”
Impacted operators have mentioned that exceptional circumstances exist including a large cumulative effect on the web, the outage of critical infrastructure, and a potential risk to human safety.
Each certificate user requiring a delay in revocation is required to answer three questions. First, what is the harm to the community caused by revoking the certificates? Second, why is it not possible to replace the certificate within the five-day timeline? Third, when can the certificate be reasonably revoked. The reasons for the delay vary, but I can break them down into:
a) Too short of time given a holiday/vacation/process.
We reject these delay requests insufficient reason for a delay under the Mozilla policy.
b) Issues with new chain
Per the article at https://wiki.mozilla.org/CA/Root_CA_Lifecycles, the Digicert G1 roots are being removed from the Mozilla trust store in less than a year. Many of these certificates use the G1 root. We have updated our systems to issue certificates automatically from our G2 and G5 roots. A lot of systems were not updated for the G2 series of roots. This is causing technical delays in rotating the certificates and breaking some instances of automation.
c) Deployment of automation.
Several large enterprises are currently deploying automation solutions. These users requested additional time to complete the automation so they can replace their corpus of certificates automatically. Given that part of five-day limitation is to encourage crypto-agility, we believe allowing this as an exceptional circumstance furthers the browsers’ goal of increasing crypto-agility. Some users have deployed automation but not across their entire ecosystem. For example, one customer was able to instantly replace 30% of their certificates but needed more time for the remainder due to technical limitations on those systems. Each required a manual touch after issuance.
d) Legal requirements.
A few users have legal requirements (not contacts) that require them to inform their governing body before replacing the certificate. These are always national banks. Although this is a process problem that limits crypto-agility, we believe violating a country’s law counts as exceptional circumstances.
e) Third party administration.
Many companies requesting a delay have a single PKI team that manages certificates and distributes the certificates to entities within their organization. These teams do not install the certificate but help coordinate the effort. Because of the collaboration required and the short notice, these companies cannot meet the five-day timeline. We rejected these requests unless it related to causing an outage in critical infrastructure.
f) Human Safety.
Several customers manage PKI for hospital and hospital devices. These operators asked for a delay to ensure that patients and patient devices would not be impacted. This companies required additional safety testing before deployment. This seemed reasonable considering that human safety should be the number one priority.
g) Outages in critical infrastructure.
The last category are users who are part of a country’s critical infrastructure with a large number of certificates. Due to the number of certificates, there would be large-scale internet outages if these certificates were revoked on May 11th. These customers reported needing to take an outage to replace the certificates.
Please be aware that we have already performed the identity validation required under the EV Guidelines for each impacted customer. Identity vetting is not a reason for the delay, and a DV certificate would cause the same delay as an EV certificate in these cases.
DigiCert is currently organizing the list of delayed certificates, the specified reasons they cannot be revoked, and a proposed timeline for revocation. The list will be posted as an update to this bug on Saturday May 11th as we expect some of the users requesting a delay to drop off this list. With the current plan, most certificates will be revoked by May 19th with the long tail taking the end or middle of July 2024. Certificate holders replacing a large number are still working out the exact timing. Any users delaying revocation that do not have automation are being offered our ACME automation and asked to commit to automation of their certificates.
|
||
Updated•2 years ago
|
I do want to say, even as a preliminary incident, this is a great example of what a CA that issues EV/OV certs should do when they're facing a delayed revocation.
|
||
Updated•2 years ago
|
|
Reporter | |
Comment 2•2 years ago
|
||
We revoked 7138 certificates this morning meeting the five day requirement assuming a start time of when we had serial numbers. This leaves 3788 delayed past the required five day requirement. I've attached a list of still active certificates, their proposed timeline, the reason they are delayed (mapping as best I can to the Mozilla policy around delayed revocation), and whether that customer has automation in place. Many "Unknown automation" customers have automation but found they could readily replace an expiring certificate but not an unrevoked certificate. We are unsure if this was a customer knowledge issue or a limitation in some of the available, third-party automation solutions. Next week we will be reaching out to all "Automation unknown" customers to confirm whether they have automation and what obstacles they had in using that automation to replace the certificates.
We did ask customers for a burn-down list by serial number. We did not receive the burn down list in most cases. We will be working with customers to accelerate the final revocation date. We will post updated lists weekly here on what remains to be revoked along with updated timelines as those timelines become available. I hope to have better burn down charts by the next substantial update.
|
|
Reporter | |
Comment 3•2 years ago
|
||
|
|
Reporter | |
Comment 4•2 years ago
|
||
Hi all - I'm working on a list of remaining certificates by serial number that has the dates on when we should be revoking and better explanations for the proposed dates. I should be done with this file next week.
We did revoke an additional 1090 certificates this week, which leaves 2696 certificates left to replace.
|
|
Reporter | |
Comment 5•2 years ago
|
||
We've made steady progress on revoking the remaining certificates. Here is some information on the remainder. This is all self-reported data from impacted subscribers.
29% of the remaining certificates have automation. The delays are due to third party issues, regulatory issues, or issues with their devices and automation.
26% of the remaining certificates remaining have automation but the automation broke during the replacement for some reason. The majority seem to be a sensor failure of some kind.
13% of the certificates have an unknown automation status.
11% of the remaining certificates have automation but the automation could not handle replacement of non-expiring certificates. The automation was deployed for renewal of certificates.
10% of the remaining certificates were being fitted with automation and were in the middle of deployment before the incident.
6% had no automation of the ceritifcates.
The remaining were miscellenous items.
Here are the reasons listed for the delay. I tried to keep the reasons aligned for the Mozilla policy reasons for delay but added some additional details when available.
Critical infrastructure that will cause significant internet outages 54.76%
Focused on automation so it can support deployment 25.77%
Critical infrastructure where they require third party testing 7.37%
Requires third party coordination 3.97%
Revoking as certificates are replaced. Getting updated timeline 1.70%
Requires report to Financial Services agency and external party assistance 1.65%
Legal restriction in Chinese law requires sign-off from government 1.07%
Requires roll-back testing and report to Financial Services Agency 0.89%
Authorization required to update cert 0.80%
Requires a change by a third-party 0.58%
Remaining certificates require cross-collaboration with third parties for the installation. 0.54%
Requires government approval and deployment on app 0.18%
Requires an update to app, which is currently being blocked by Google 0.13%
Requires third party deployment 0.13%
Requires mobile update 0.13%
Pinned in mobile app and requires a rebuild 0.09%
Disaster recovery exercise under way 0.09%
Ran into issue with automated deployment. Working on issue while replacing 0.09%
Issue with installation and third party dependency 0.04%
I will quote Digicert's Certificate Policy S1.4.2: Prohibited Certificate Uses:
DigiCert strongly discourages key pinning and shall not consider it a sufficient reason to delay revocation. DigiCert continually researches and implements technological processes in order to detect pinned applications and other prohibited uses so we can counsel customers on the way pinning impacts the agility of the Web PKI (e.g., rotation of intermediate certificates). Customers should also take care in not mixing certificates trusted for the web with non-web PKI. Any certificates trusted by the browsers must comply with all requirements of all applicable browser root policies, including revocation periods of 24 hours and 5 days as asserted in the relevant policies, obligations, and requirements of this CP and the CPS.
Digicert's Certificate Terms of Use S6: Certificate License notes:
DigiCert strongly discourages certificate or key pinning, using Certificates trusted for the web with non-web PKI, or any other use of Certificates that would make it difficult for Customer to meet the revocation timelines or other requirements of the CPS, and any such use will not be considered a sufficient reason to delay revocation.
It certainly sounds like cert-pinning is amongst the delayed revocation batch. Could you clarify what 'critical infrastructure' actually means, what is meant by 'significant internet outages', and for who? Reading the S1.4.2 reasons it certainly sounds like you've explicitly prohibited any certificate usage that would move into a delayed revocation. How are you justifying any later re-issuance for these subscribers while abiding by your certificate policy?
|
|
Reporter | |
Comment 7•2 years ago
|
||
Cert pinning was not considered a sufficient reason for delayed revocation in this bug. Anyone asking for a delay because of pinning was denied and included in the initial batch of revocations executed on May 11th. The reasons for the delay were provided in the revocation update excel file and did not include pinning.
The stats I posted above were gathered after the initial delay and is updated information that we've been pushing to get related to burndown charts. I wanted to know what automation each customer had in place and specific reasons additional delays were required. Whenever pinning is mentioned, we found those an unacceptable reason and denied any additional delay in revocation. We've been consistent with our policy against pinning because any time pinning is mentioned, we revoke the certificate. All remaining certificates are delayed based on the reasons stated in the Mozilla guidelines for revocation:
Mozilla recognizes that in some exceptional circumstances, revoking the affected certificates within the prescribed deadline may cause significant harm, such as when the certificate is used in critical infrastructure and cannot be safely replaced prior to the revocation deadline, or when the volume of revocations in a short period of time would result in a large cumulative impact to the web. However, Mozilla does not grant exceptions to the BR revocation requirements. It is our position that your CA is ultimately responsible for deciding if the harm caused by following the requirements of the Baseline Requirements outweighs the risks that are passed on to individuals who rely on the web PKI by choosing not to meet this requirement.
What exactly constitutes "cannot be safely replaced prior to the revocation deadline, or when the volume of revocations in a short period of time would result in a large cumulative impact to the web" is vague. Without further guidance in the policy itself, I relied on the CISA definition (https://www.cisa.gov/topics/critical-infrastructure-security-and-resilience) for critical infrastructure: "Critical Infrastructure are those assets, systems, and networks that provide functions necessary for our way of life. There are 16 critical infrastructure sectors that are part of a complex, interconnected ecosystem and any threat to these sectors could have potentially debilitating national security, economic, and public health or safety consequences. "
CISA lists 16 categories which include finance, food, government, transportation, and others. Entities seeking a delay had to qualify based on one of the CISA categories. I didn't have a way to measure the second part of the statement - whether an outage would be "unsafe" or there would be a cumulative impact on the web so had to rely on what the subscriber reported as the potential outcome for immediate revocation. What is interesting is how many of these entities require some sort of government approval before replacing a certificate. Most of the customers can quickly replace a certificate once the approval is complete but the approval can take quite some time.
The reason I mentioned cert pinning as you noted was due to comment 5:
Here are the reasons listed for the delay. I tried to keep the reasons aligned for the Mozilla policy reasons for delay but added some additional details when available.
Pinned in mobile app and requires a rebuild 0.09%
I appreciate you following CISA's framework for critical infrastructure. The reason I ask is that it's not clear at all in the breakdown other than it being self-reported. As you state Mozilla provide exceptions, but they are the only program to do so, and have a list of requirements above and beyond self-reporting. This incident is still missing a final report per CCADB's incident reporting template, and it has been over 2 weeks since the preliminary report was published.
That there is an inherent issue in applying the revocation periods is also why I mentioned Digicert's Certificate Policy S1.4.2: Prohibited Certificate Uses:
Any certificates trusted by the browsers must comply with all requirements of all applicable browser root policies, including revocation periods of 24 hours and 5 days as asserted in the relevant policies, obligations, and requirements of this CP and the CPS.
Given you have subscribers who have stated that are not able to comply with these requirements then surely any re-issuance is prohibited by your own terms?
|
|
Reporter | |
Comment 9•2 years ago
|
||
Ah - you're right that I never posted the incident report in the required format. Apologies for that. Here is the properly formatted issuance report.
Incident Report
Summary
DigiCert delayed revocation of certificates identified in bug 1894560. The delay was caused by the number of certificates impacted. Although we were able to revoke 7138 certificates within the five days (starting when we confirmed the serial numbers), that left 3788 that missed the May 11th deadline.
Impact
The impact is that there were 3788 certificates remaining after May 11th that had "Private organization" or "Government entity" listed as the business category.
Timeline
2024-05-06 16:00 - Final data on problematic certificates collected.
2024-05-07 4:17 - Emails sent to customers notifying of revocation along with date and expectations.
2024-05-05 10:48 - Started receiving escalations that the May 11th timeline would be impossible to hit. Calls initiated about whether revocation was actually required under the CABF requirements. Started tracking potential delays. No extensions or exceptions permitted and all pinning related requests were rejected.
2024-05-10 21:45 - Created list of certificates being delayed along with reasons.
2024-05-11 14:26 - Revocation tool begins processing requested revocations.
2024-05-11 15:25 - Revocation completes on all expected delayed revocation list.
2024-05-14 14:05 - Created an RCA for general distribution explaining what happened.
From there, certificates are revoked as they are replaced.
Root Cause Analysis
The number of certificates requiring revocation was overwhelming and the core reason that customers could not replace within the required 5 day timeline. The reasons provided customers were added to the delayed revocation list included the following:
a) Issues with new chain
Per the article at https://wiki.mozilla.org/CA/Root_CA_Lifecycles, the Digicert G1 roots are being removed from the Mozilla trust store in less than a year. Many of these certificates use the G1 root. We have updated our systems to issue certificates automatically from our G2 and G5 roots. A lot of systems were not updated for the G2 series of roots. This is causing technical delays in rotating the certificates and breaking some instances of automation.
b) Deployment of automation.
Several large enterprises are currently deploying automation solutions. These users requested additional time to complete the automation so they can replace their corpus of certificates automatically. Given that part of five-day limitation is to encourage crypto-agility, we believe allowing this as an exceptional circumstance furthers the browsers’ goal of increasing crypto-agility. Some users have deployed automation but not across their entire ecosystem. For example, one customer was able to instantly replace 30% of their certificates but needed more time for the remainder due to technical limitations on those systems. Each required a manual touch after issuance.
c) Legal requirements.
A few users have legal requirements (not contacts) that require them to inform their governing body before replacing the certificate. These are always national banks. Although this is a process problem that limits crypto-agility, we believe violating a country’s law counts as exceptional circumstances.
d) Third party administration.
Many companies requesting a delay have a single PKI team that manages certificates and distributes the certificates to entities within their organization. These teams do not install the certificate but help coordinate the effort. Because of the collaboration required and the short notice, these companies cannot meet the five-day timeline. We rejected these requests unless it related to causing an outage in critical infrastructure.
e) Human Safety.
Several customers manage PKI for hospital and hospital devices. These operators asked for a delay to ensure that patients and patient devices would not be impacted. This companies required additional safety testing before deployment. This seemed reasonable considering that human safety should be the number one priority.
f) Outages in critical infrastructure.
The last category are users who are part of a country’s critical infrastructure with a large number of certificates. Due to the number of certificates, there would be large-scale internet outages if these certificates were revoked on May 11th. These customers reported needing to take an outage to replace the certificates.
Neither validation nor pinning was a reason for the delayed revocation.
Lessons Learned
Revoking a large number of certificates proved difficult, especially when government entities are in play that do not understand why a certificate would be revoked for a capitalization issue. The other lesson learned is that the automation available is generally insufficient. A large number of customers have automation but that automation either broke or could not handle revocation outside of certificate expiration. A net positive for the industry that automation is being adopted but there is still a way to go. One thing we are looking at is how we can better help customers with that automation.
What went well
We revoked over 7000 certificates by the deadline and are aggressively pushing for faster revocation. Automation worked for a majority of customers.
What didn't go well
There was a delay in notification to customers, which led to a delay in customers noticing that revocation was required.
Several countries had holidays at the same time as the revocations which caused them to miss notices.
Where we got lucky
We had a strict policy against delays for pinning.
A lot of customers already use automation
Action Items
| Action Item | Kind | Due Date |
|Revoke all problematic certs. | Remediate | 2024-07-31 |
Appendix
Details of affected certificates
See previously provided list and updated list.
|
|
Reporter | |
Comment 10•2 years ago
|
||
|
|
Reporter | |
Comment 11•2 years ago
|
||
Given you have subscribers who have stated that are not able to comply with these requirements then surely any re-issuance is prohibited by your own terms?
Re-issuance isn't prohibited under the CP. However, the failure of subscribers to abide by the CP is one reason we are asking for the automation information and a never-again plan . We want to do know what subscribers are going to do going forward to meet the 5-day requirement.
|
||
Comment 12•2 years ago
|
||
What is the purpose of that line in your Prohibited Certificate Uses section if not to be very clear that a certificate is not allowed to be issued if a potential subscriber is not capable of handling revocation within 24h or 5d? If you are re-issuing to subscribers while currently knowing of their inability to comply, then what purpose does that prohibition serve?
While I appreciate the idea of a never-again plan you mentioned situations outside of the subscriber's control, how is this compatible with the above?
I am embedding a breakdown of the current revocation periods so it is more visible to everyone:
| Date | Certs |
|---|---|
| 2024-05-31 | 62 |
| 2024-06-04 | 9 |
| 2024-06-10 | 11 |
| 2024-06-14 | 2 |
| 2024-06-15 | 33 |
| 2024-06-18 | 1 |
| 2024-06-22 | 4 |
| 2024-06-30 | 463 |
| 2024-07-31 | 971 |
|
||
Comment 13•2 years ago
|
||
Hi Jeremy,
Thanks for the detailed breakdown of the kinds of circumstances in which revocation was delayed. However, the purpose of a "delayed revocation" incident is to see what steps the CA can take / is taking to help ensure that a future misissuance incident does not result in another delayed revocation incident.
The action item listed is, in my opinion, a remediation item for https://bugzilla.mozilla.org/show_bug.cgi?id=1894560.
What action items is Digicert taking to ensure that there are fewer or no delayed revocations next time?
|
|
Reporter | |
Comment 14•2 years ago
|
||
What is the purpose of that line in your Prohibited Certificate Uses section if not to be very clear that a certificate is not allowed to be issued if a potential subscriber is not capable of handling revocation within 24h or 5d? If you are re-issuing to subscribers while currently knowing of their inability to comply, then what purpose does that prohibition serve?
To ensure people know the rules and explain why delays in revocation are not allowed outside of the scope set by the Mozilla policy. Plus it prevents legal retaliation if we do revoke a cert. That language also helped us revoke over 7000 certificates within the 5 day timeframe - everything that did not fall under the posted Mozilla delayed revocation process.
One note on the breakdown is that 577 of the July 31st certs had automation that broke while trying to replace the certificates. They'll be able to replace all of their certificates at once as soon as the automation works. The July 31st deadline is assuming the automation won't work, and the customer needs to replace the certificates manually.
What action items is Digicert taking to ensure that there are fewer or no delayed revocations next time?
This one is tough. First, we found that most customers had automation. Although we are continuing to push automation to all customers and discourage pinning, I don't think automation or pinning played a huge role in the delay. For examples on us pushing automation:
https://itbrief.com.au/story/digicert-s-certificate-management-platform-is-now-on-multiple-cas
https://www.forbes.com/sites/forbestechcouncil/2023/09/27/why-you-need-a-unified-approach-to-digital-trust/?sh=6dd18b76737f
https://www.digicert.com/campaign/trust-lifecycle-manager?s_kwcid=AL!15928!3!677530218645!b!!g!!digicert%20pki&utm_source=google&utm_medium=paid-search&utm_campaign=bofu-tlm&toc=7014z000001JjlXAAS&mkwid=sA0shLCCZ_pcrid_677530218645_pdv_c_pmt_b_pkw_digicert%20pki_slid__product__pgrid_157632642431_ptaid_kwd-2227581634743&gad_source=1&gclid=CjwKCAjwmYCzBhA6EiwAxFwfgA3EvvXnzzVJK1538v10OTF5J9izkR6rvRRbf0I9H__Wa2ZnRWjl3BoC-JAQAvD_BwE
During this incident, we found that the blocker in replacing certificates is not automation for ordering and deployment of certificates. Instead, its the extensive sign off processes required before using that automation to deploy the certificate. We will continue to push the automation angle (including ACME and others that we support), but I'm not sure what to do about the sign-off process inherent in more regulated communities. Right now, I've been talking to each of the delayed customers about how they can streamline their process. I've been asking the question, "What would you do if a key was compromised?" I think a lot of them would have an outage.
|
|
||
Comment 15•2 years ago
|
||
(In reply to Jeremy Rowley from comment #14)
What is the purpose of that line in your Prohibited Certificate Uses section if not to be very clear that a certificate is not allowed to be issued if a potential subscriber is not capable of handling revocation within 24h or 5d? If you are re-issuing to subscribers while currently knowing of their inability to comply, then what purpose does that prohibition serve?
To ensure people know the rules and explain why delays in revocation are not allowed outside of the scope set by the Mozilla policy. Plus it prevents legal retaliation if we do revoke a cert. That language also helped us revoke over 7000 certificates within the 5 day timeframe - everything that did not fall under the posted Mozilla delayed revocation process.
One note on the breakdown is that 577 of the July 31st certs had automation that broke while trying to replace the certificates. They'll be able to replace all of their certificates at once as soon as the automation works. The July 31st deadline is assuming the automation won't work, and the customer needs to replace the certificates manually.
So to be clear it is Digicert's view that the Prohibited Certificate Usage clause is not about prohibiting certificate issuance at all?
All that continued issuance to these subscribers is doing is creating an agreement that the subscriber can continue not fixing things and that Digicert will not perform their role as a CA and revoke by the 24h and 5d deadlines. If it is indeed Digicert's attempt to use this clause as legal protection, then they need to start consistently enforcing it or they won't be able to rely on it at all.
What action items is Digicert taking to ensure that there are fewer or no delayed revocations next time?
This one is tough. First, we found that most customers had automation. Although we are continuing to push automation to all customers and discourage pinning, I don't think automation or pinning played a huge role in the delay. For examples on us pushing automation:
https://itbrief.com.au/story/digicert-s-certificate-management-platform-is-now-on-multiple-cas
https://www.forbes.com/sites/forbestechcouncil/2023/09/27/why-you-need-a-unified-approach-to-digital-trust/?sh=6dd18b76737f
https://www.digicert.com/campaign/trust-lifecycle-manager?s_kwcid=AL!15928!3!677530218645!b!!g!!digicert%20pki&utm_source=google&utm_medium=paid-search&utm_campaign=bofu-tlm&toc=7014z000001JjlXAAS&mkwid=sA0shLCCZ_pcrid_677530218645_pdv_c_pmt_b_pkw_digicert%20pki_slid__product__pgrid_157632642431_ptaid_kwd-2227581634743&gad_source=1&gclid=CjwKCAjwmYCzBhA6EiwAxFwfgA3EvvXnzzVJK1538v10OTF5J9izkR6rvRRbf0I9H__Wa2ZnRWjl3BoC-JAQAvD_BwEDuring this incident, we found that the blocker in replacing certificates is not automation for ordering and deployment of certificates. Instead, its the extensive sign off processes required before using that automation to deploy the certificate. We will continue to push the automation angle (including ACME and others that we support), but I'm not sure what to do about the sign-off process inherent in more regulated communities. Right now, I've been talking to each of the delayed customers about how they can streamline their process. I've been asking the question, "What would you do if a key was compromised?" I think a lot of them would have an outage.
We can agree that if a subscriber states they're going to have an outage if you revoke, then they are incapable of handling revocation within 24h or 5d?
My reason to put this forward is not only consistency in application of your policies, but also that these subscribers need to have a firm decision put on them in order to change. If we are stating that it'll be over 2 months for subscribers to even accept revocation, then that is materially incompatible with the policy.
With regards to the more regulated communities they are not operating within a vacuum. Have them speak to their regulator, which they must be doing regularly, and mention that these constraints are creating security and safety nightmares. The regulations can be changed directly, or specific exceptions can be approved outwith the cycle depending on the regulation analyst replying.
|
|
Reporter | |
Comment 16•2 years ago
|
||
So to be clear it is Digicert's view that the Prohibited Certificate Usage clause is not about prohibiting certificate issuance at all?
No - our view is that subscribers should not use certificates under the circumstances stated in that section. I think your umbrage is with the lack of a statement around what happens to subscribers who violate this policy, which is that they will have their certificates revoked in accordance with the Mozilla policy. You're missing the "Any certificates trusted by the browsers must comply with all requirements of all applicable browser root policies." This includes the delayed revocation policy, which the browsers explained was an expected part of the process during the Portsmouth CABF meeting.
All that continued issuance to these subscribers is doing is creating an agreement that the subscriber can continue not fixing things and that Digicert will not perform their role as a CA and revoke by the 24h and 5d deadlines. If it is indeed Digicert's attempt to use this clause as legal protection, then they need to start consistently enforcing it or they won't be able to rely on it at all.
There's no privy of contract here so the ramifications of any agreement issues are best handled by the parties involved. Note that a waiver in one case does not constitute a waiver in all cases per 9.16.4.
We can agree that if a subscriber states they're going to have an outage if you revoke, then they are incapable of handling revocation within 24h or 5d?
No. We expect all of our customers to abide by the current stated Mozilla policies. The Mozilla policy requires 24h/5 days for revocation, not for replacement of the certificate. Mozilla has a public policy that allows for delayed revocation in certain circumstances. We expect subscribers to follow that policy as well as the other policies.
My reason to put this forward is not only consistency in application of your policies, but also that these subscribers need to have a firm decision put on them in order to change. If we are stating that it'll be over 2 months for subscribers to even accept revocation, then that is materially incompatible with the policy.
Sure - you could petition Mozilla to remove the delayed revocation section. Alternatively, if Mozilla puts out a policy that delayed revocations have a maximum timeframe of an additional 5/10/30 days, we'd follow that too.
With regards to the more regulated communities they are not operating within a vacuum. Have them speak to their regulator, which they must be doing regularly, and mention that these constraints are creating security and safety nightmares. The regulations can be changed directly, or specific exceptions can be approved outwith the cycle depending on the regulation analyst replying.
We do this. In fact, two days ago I spoke to regulators about what a terrible idea pinning is. I don't think that will stop them from requiring it. I've encouraged all subscribers that I've spoken with to change their practices and avoid policy restrictions that prevent them from replacing certificates. We've recommended that they speak to regulators about not requiring approval of something as mundane as revoking a certificate. We've warned them that the Mozilla delayed revocation policy is subject to change and they should not rely on that, nor should they expect to qualify under the process if there is an additional incident. I speak at any conference I'm invited to about the dangers of pinning and the need for cryptoagility.
|
||
Comment 17•2 years ago
|
||
(In reply to Jeremy Rowley from comment #16)
Mozilla has a public policy that allows for delayed revocation in certain circumstances. We expect subscribers to follow that policy as well as the other policies.
Sure - you could petition Mozilla to remove the delayed revocation section. Alternatively, if Mozilla puts out a policy that delayed revocations have a maximum timeframe of an additional 5/10/30 days, we'd follow that too.
I have seen this brought up a few times recently, and it is a misinterpretation of the written policy. This is the article you are referring to https://wiki.mozilla.org/CA/Responding_To_An_Incident#Revocation
Please quote where in that article Mozilla allows delayed revocation. What that article says is that Mozilla imposes additional expectations on CAs of transparency and disclosure. The article actually says "Mozilla does not grant exceptions to the BR revocation requirements."
|
||
Comment 18•2 years ago
|
||
(In reply to Jeremy Rowley from comment #16)
You're missing the "Any certificates trusted by the browsers must comply with all requirements of all applicable browser root policies." This includes the delayed revocation policy, which the browsers explained was an expected part of the process during the Portsmouth CABF meeting.
Perhaps you have me at a disadvantage, Jeremy, because I was at the Portsmouth face-to-face and I’m not recalling the conversation in which the browsers explained that they considered delayed revocation to be part of the expected process. We all are aware of Mozilla’s published policy, which has been much discussed on this forum over the past three months. Can you remind me specifically what other browsers said about their expectations regarding delayed revocation and who said it?
I’ll likewise invite the browsers to chime in directly, if they would like, and clarify their expectations around the 24/120-hour revocations mandated in the BRs.
|
|
Reporter | |
Comment 19•2 years ago
|
||
1340 remaining
|
|
Reporter | |
Comment 20•2 years ago
|
||
I have seen this brought up a few times recently, and it is a misinterpretation of the written policy. This is the article you are referring to https://wiki.mozilla.org/CA/Responding_To_An_Incident#Revocation
"Mozilla recognizes that in some exceptional circumstances, revoking the affected certificates within the prescribed deadline may cause significant harm, such as when the certificate is used in critical infrastructure and cannot be safely replaced prior to the revocation deadline, or when the volume of revocations in a short period of time would result in a large cumulative impact to the web....It is our position that your CA is ultimately responsible for deciding if the harm caused by following the requirements of the Baseline Requirements outweighs the risks that are passed on to individuals who rely on the web PKI by choosing not to meet this requirement. "
So yeah, its in there - at least the allowance for CAs to make that call. Or are you implying that Mozilla policy doesn't actually allow CAs to be ultimately responsible for deciding the balance of harm? Mozilla really should delete this language from its public statement if it doesn't want CAs making delayed revocation decisions.
Perhaps you have me at a disadvantage, Jeremy, because I was at the Portsmouth face-to-face and I’m not recalling the conversation in which the browsers explained that they considered delayed revocation to be part of the expected process. We all are aware of Mozilla’s published policy, which has been much discussed on this forum over the past three months. Can you remind me specifically what other browsers said about their expectations regarding delayed revocation and who said it?
It was during the revocation discussion after lunch. I said that the 5 day rule should be changed as its too short, which is causing delayed revocation bugs. The browsers said this is normal and they'd rather have the bug posted than change the revocation windows. I followed up again, asking roughly the same thing, as I was surprised by the answer. Inigio was at the front since he was leading the discussion but both Ryan and Clint answered the question. Obviously I'm paraphrasing what I recall since the meeting minutes during that discussion are non-existence. There are references to the discussion later in the minutes though. The Server Cert WG chair really needs to enforce better minute taking.
|
|
||
Comment 21•2 years ago
|
||
(In reply to Jeremy Rowley from comment #16)
So to be clear it is Digicert's view that the Prohibited Certificate Usage clause is not about prohibiting certificate issuance at all?
No - our view is that subscribers should not use certificates under the circumstances stated in that section. I think your umbrage is with the lack of a statement around what happens to subscribers who violate this policy, which is that they will have their certificates revoked in accordance with the Mozilla policy. You're missing the "Any certificates trusted by the browsers must comply with all requirements of all applicable browser root policies." This includes the delayed revocation policy, which the browsers explained was an expected part of the process during the Portsmouth CABF meeting.
The circumstances stated are:
Any certificates trusted by the browsers must comply with all requirements of all applicable browser root policies, including revocation periods of 24 hours and 5 days as asserted in the relevant policies, obligations, and requirements of this CP and the CPS.
My 'umbrage' as you attempt to ascribe is holding Digicert against their own Certificate Policy. A Root Program's expectation that some delayed revocation events will occur should not be conflated with it being an accepted normal event during all revocation incidents. My point is if you are choosing to re-issue to subscribers who are incapable of holding to the 5-day window, then you are not solving the original delayed revocation incident as-is and are going to cause this to re-occur in the future. Does this make sense?
All that continued issuance to these subscribers is doing is creating an agreement that the subscriber can continue not fixing things and that DigiCert will not perform their role as a CA and revoke by the 24h and 5d deadlines. If it is indeed Digicert's attempt to use this clause as legal protection, then they need to start consistently enforcing it or they won't be able to rely on it at all.
There's no privy of contract here so the ramifications of any agreement issues are best handled by the parties involved. Note that a waiver in one case does not constitute a waiver in all cases per 9.16.4.
As part of being a public CA DigiCert asserts itself to be uphold by its own Certificate Policy and as a Relying Party I, like much of the internet, are a party to the CPS attached to all WebPKI certificates DigiCert issues. To clarify you are quoting 9.16.4:
9.16.4 Enforcement (attorneys' fees and waiver of rights)
DigiCert may seek indemnification and attorneys' fees from a party for damages, losses, and expenses related to that party's conduct. DigiCert’s failure to enforce a provision of this CP does not waive DigiCert’s right to enforce the same provision later or right to enforce any other provision of this CP. To be effective, waivers must be in writing and signed by DigiCert.
My statement is not about DigiCert's choice of enforcement voiding the rest of the contract. It is about the Prohibited Certificates Usage policy in itself being a statement of intent by DigiCert to all parties that these practices are expressly prohibited by DigiCert. All I am looking for is clarification that any issuance to subscribers is brought with an assurance that no future revocation will go beyond 5 days as both the Baseline Requirements, and more expressly your Certificate Policy demands.
I hope you understand that statements by certain subscribers within this incident are not providing confidence on this matter?
We can agree that if a subscriber states they're going to have an outage if you revoke, then they are incapable of handling revocation within 24h or 5d?
No. We expect all of our customers to abide by the current stated Mozilla policies. The Mozilla policy requires 24h/5 days for revocation, not for replacement of the certificate. Mozilla has a public policy that allows for delayed revocation in certain circumstances. We expect subscribers to follow that policy as well as the other policies.
If your answer is 'No', then the subscriber is capable of handling revocation and thus you are revoking all outstanding certificates by the end of business day then? It is not for subscribers to follow Root Program policies, that is the role of the CA. You are also beholden to more than one Root Program's policies.
My reason to put this forward is not only consistency in application of your policies, but also that these subscribers need to have a firm decision put on them in order to change. If we are stating that it'll be over 2 months for subscribers to even accept revocation, then that is materially incompatible with the policy.
Sure - you could petition Mozilla to remove the delayed revocation section. Alternatively, if Mozilla puts out a policy that delayed revocations have a maximum timeframe of an additional 5/10/30 days, we'd follow that too.
Would this be the same way DigiCert are following the maximum timeframe for revocation at 24h or 5 days? The delayed revocation incident is not a catch-all for a CA to hit and then drop all attempts at revocation in a timely manner.
With regards to the more regulated communities they are not operating within a vacuum. Have them speak to their regulator, which they must be doing regularly, and mention that these constraints are creating security and safety nightmares. The regulations can be changed directly, or specific exceptions can be approved outwith the cycle depending on the regulation analyst replying.
We do this. In fact, two days ago I spoke to regulators about what a terrible idea pinning is. I don't think that will stop them from requiring it. I've encouraged all subscribers that I've spoken with to change their practices and avoid policy restrictions that prevent them from replacing certificates. We've recommended that they speak to regulators about not requiring approval of something as mundane as revoking a certificate. We've warned them that the Mozilla delayed revocation policy is subject to change and they should not rely on that, nor should they expect to qualify under the process if there is an additional incident. I speak at any conference I'm invited to about the dangers of pinning and the need for cryptoagility.
I'm glad you are actively speaking to regulators to push this, but it is also not on them to abide by the Root Program's policies. If you have a subscriber who is beholden to a regulator who is not listening then you are actively creating harm by re-issuing them certificates whilst knowing they are not capable of handling revocation in a timely manner. If the subscriber had no CA offering them certificates then a stronger incentive would exist for the regulator to change their policies.
(In reply to Jeremy Rowley from comment #20)
I have seen this brought up a few times recently, and it is a misinterpretation of the written policy. This is the article you are referring to https://wiki.mozilla.org/CA/Responding_To_An_Incident#Revocation
"Mozilla recognizes that in some exceptional circumstances, revoking the affected certificates within the prescribed deadline may cause significant harm, such as when the certificate is used in critical infrastructure and cannot be safely replaced prior to the revocation deadline, or when the volume of revocations in a short period of time would result in a large cumulative impact to the web....It is our position that your CA is ultimately responsible for deciding if the harm caused by following the requirements of the Baseline Requirements outweighs the risks that are passed on to individuals who rely on the web PKI by choosing not to meet this requirement. "
So yeah, its in there - at least the allowance for CAs to make that call. Or are you implying that Mozilla policy doesn't actually allow CAs to be ultimately responsible for deciding the balance of harm? Mozilla really should delete this language from its public statement if it doesn't want CAs making delayed revocation decisions.
I would appreciate if DigiCert kept up to date with all delayed revocation incidents as this has been discussed to death lately. DigiCert are reading a single line in paragraphs and then not reading further and it elides a lack of understanding in the delayed revocation incident process itself. There is also the choice of ellipses over a line that was quoted at you:
However, Mozilla does not grant exceptions to the BR revocation requirements.
By expressly choosing to not include this you are showing a disregard for simple discussion. However, moving on there is far more to that paragraph if you would be willing to listen:
If your CA will not be revoking the certificates within the time period required by the BRs, our expectations are that:
- A separate incident report will be filed in Bugzilla.
- The decision and rationale for delaying revocation will be disclosed in the form of a preliminary incident report immediately; preferably before the BR-mandated revocation deadline. The rationale must include detailed and substantiated explanations for why the situation is exceptional. Responses similar to “we do not deem this non-compliant certificate to be a security risk” are not acceptable. When revocation is delayed at the request of specific Subscribers, the rationale must be provided on a per-Subscriber basis.
- Any decision to not comply with the timeline specified in the Baseline Requirements must also be accompanied by a clear timeline describing if and when the problematic certificates will be revoked or expire naturally, and supported by the rationale to delay revocation.
- The issue will need to be listed as a finding in your CA’s next BR audit statement.
- Your CA will work with your auditor (and supervisory body, as appropriate) and the Root Store(s) that your CA participates in to ensure your analysis of the risk and plan of remediation is acceptable.
- You will perform an analysis to determine the factors that prevented timely revocation of the certificates, and include a set of remediation actions in the final incident report that aim to prevent future revocation delays.
If your CA will not be revoking the problematic certificates as required by the BRs, then we recommend that you also contact the other root programs that your CA participates in to acknowledge this non-compliance and discuss what expectations their Root Programs have with respect to these certificates.
I have added the numbers for clarity purposes. To further clarify as this has recently became a misunderstanding by CAs:
When revocation is delayed at the request of specific Subscribers, the rationale must be provided on a per-Subscriber basis.
This was already expressly provided to you in the question you are responding to. To date we have subscriber breakdowns per-certificate which is not what is asked. We will also need evidence of factors 5 and 6. To date there are no action items in this incident to address the delayed revocation incident itself to ensure this does not happen in the future. That is why the discussion keeps returning to these re-issuances.
Perhaps you have me at a disadvantage, Jeremy, because I was at the Portsmouth face-to-face and I’m not recalling the conversation in which the browsers explained that they considered delayed revocation to be part of the expected process. We all are aware of Mozilla’s published policy, which has been much discussed on this forum over the past three months. Can you remind me specifically what other browsers said about their expectations regarding delayed revocation and who said it?
It was during the revocation discussion after lunch. I said that the 5 day rule should be changed as its too short, which is causing delayed revocation bugs. The browsers said this is normal and they'd rather have the bug posted than change the revocation windows. I followed up again, asking roughly the same thing, as I was surprised by the answer. Inigio was at the front since he was leading the discussion but both Ryan and Clint answered the question. Obviously I'm paraphrasing what I recall since the meeting minutes during that discussion are non-existence. There are references to the discussion later in the minutes though. The Server Cert WG chair really needs to enforce better minute taking.
Those sound like entirely reasonable answers as presented, what was the surprise about?
The revocation timeline is not about what is most convenient for the CA, it is about making sure that the certificates are out of the system in a timely manner. In the course of the revocation deadlines existing there will need to be a set procedure for a CA going past that required timeframes, but the issue is that this should not become the norm. In much the same way that a distrust event is an 'expected' part of the process, a delayed revocation incident is an 'expected' part of the process that a CA should be doing everything it can to stop from happening. I trust this explanation clarifies matters?
(In reply to Jeremy Rowley from comment #19)
Created attachment 9407121 [details]
Remaining Certs - June 12 2024.xlsx1340 remaining
Breakdown of the current revocation deadlines:
| Date | Certs |
|---|---|
| 2024-06-14 | 2 |
| 2024-06-15 | 5 |
| 2024-06-17 | 37 |
| 2024-06-18 | 1 |
| 2024-06-22 | 4 |
| 2024-06-30 | 351 |
| 2024-07-31 | 940 |
| Total | 1340 |
I will speak positively here: 42 certificates have moved from the previous deadline of 2024-07-31 to revocation deadlines of the latest being 2024-06-17. A further 1443 certificates with a deadline after today in a previous report have since been revoked. That's good to see.
Now the bad news. Two certificates have moved back 28 days. These were originally set to be revoked 2024-05-17. What happened?
Five more certificates have been similarly pushed back. One was due to expire 2024-05-18, and is now expiring 2024-06-18. The other four have had their deadlines changed from 2024-06-30 to 2024-07-31. What happened?
Additionally I am seeing that these are still not showing on crt.sh?
|
|
||
Comment 22•1 year ago
|
||
(In reply to Jeremy Rowley from comment #20)
So yeah, its in there - at least the allowance for CAs to make that call. Or are you implying that Mozilla policy doesn't actually allow CAs to be ultimately responsible for deciding the balance of harm? Mozilla really should delete this language from its public statement if it doesn't want CAs making delayed revocation decisions.
Describing that the CA "is ultimately responsible" is just describing reality. CAs control the signing keys and are responsible for their own operations. CAs that make that call, such as in this bug, are not meeting the requirements in the BRs and in their stated CPS as explicitly stated by Mozilla: "However, Mozilla does not grant exceptions to the BR revocation requirements." That's the same as if the CAs made any other call in their operations that didn't meet requirements. It's an incident, and CAs should be endeavouring to take necessary actions to prevent incidents and ensure for the benefit of relying parties that requirements are always met.
On the other hand, Mozilla does have additional expectations on top of the BRs for this type of incident, which Wayne already enumerated, and those expectations haven't been met either.
|
|
||
Comment 23•1 year ago
|
||
(In reply to Jeremy Rowley from comment #20)
It was during the revocation discussion after lunch. I said that the 5 day rule should be changed as its too short, which is causing delayed revocation bugs. The browsers said this is normal and they'd rather have the bug posted than change the revocation windows.
That was the browsers shrugging off a flimsy argument by CAs that longer revocation periods would prevent Bugzilla from being clogged with pointless delayed revocation incidents. You can’t stretch it into conceptual approval of late revocations by the root programs. And of course, that conversation was about moving certain misissuance types, of which yours in this incident would have been one, to a fifteen-day revocation period. As such its relevance to failure-to-revoke incidents extending to 30, 60, or 90 days is questionable at best.
If we are entering statements from root programs into the record, let’s include Chromium’s prepared presentation on this topic at the most recent CABF face-to-face on May 28, 2024. The presentation at one point says (emphasis in original),
We consider recent trends observed in Bugzilla unacceptable. Examples include:
- Mis-issuance prevented by linting
- Delayed revocations becoming routine rather than exceptional
Persistent and willing non-compliance has no place in this ecosystem.
This strikes me as a highly relevant, timely, and clear statement about Chromium’s attitude toward delayed revocations and is useful in interpreting the large number of late revocation bugs active today.
|
||
Comment 24•1 year ago
|
||
I’m sure the root programs would agree with Tim here. Also, if you’re unsure, ask the root programs to clarify. I’m sure they would be happy to.
Either way this revocation handling document does not exist for any of the other root programs anyway.
|
||
Comment 25•1 year ago
•
|
||
Beyond what I hope was a clear update communicating Chrome’s position at F2F 62 (slides), I went back and reviewed my minutes from F2F 60 to review the discussion described by Jeremy in Comment 20.
In reviewing them, my talking points focused on:
- The absence of data that concludes an extended revocation window is an appropriate or necessary response for the ecosystem. (during this discussion, members of the community suggested a 15-day window was appropriate, and we still disagree with that proposal given observed behavior in recent bugs).
- How linting might prevent the issues necessitating revocation. This holds true, today - and was further corroborated given the numerous profile conformance issues observed in the Spring. Almost all of the revocation delay bugs could have been prevented through freely available linting tools, and should have been prevented by more thorough examinations of profile alignment with the BRs.
- The importance of automation, specifically ACME and ARI. Not only to help us move past revocation delays, but also be better prepared for the next Heartbleed-style event.
- The opportunity for Private PKIs to address customer use cases that are not a good fit for the Web PKI.
I believe we’ve been consistent in our perspective.
|
|
Reporter | |
Comment 26•1 year ago
|
||
Thank you for all for your feedback and questions so far. We greatly appreciate the input and direction on this delayed revocation bug. Based on the comments, we are revising the timeline to ensure that all certificates are revoked by the end of June and already informed subscribers about the results of this discussion. A new schedule is attached. The certificates are being installed now.
Would this be the same way DigiCert are following the maximum timeframe for revocation at 24h or 5 days? The delayed revocation incident is not a catch-all for a CA to hit and then drop all attempts at revocation in a timely manner.
We are following the delayed revocation procedure as outlined in this bug. If our analysis of the timeline based on the reasons for the delay is incorrect, then we welcome additional feedback. Based on the feedback, we already revised the revocation timelines. If the Root Programs believe the revised timeline is still too long based on the analysis provided, then we would be happy to rework the timeline again.
To make sure we’ve followed the procedure correctly:
- A separate incident report will be filed in Bugzilla.
The incident report is this bug.
- The decision and rationale for delaying revocation will be disclosed in the form of a preliminary incident report immediately; preferably before the BR-mandated revocation deadline. The rationale must include detailed and substantiated explanations for why the situation is exceptional. Responses similar to “we do not deem this non-compliant certificate to be a security risk” are not acceptable. When revocation is delayed at the request of specific Subscribers, the rationale must be provided on a per-Subscriber basis.
This is complete. We provided the rationale for the reasons and why the delay happened in the opening post of this bug, aligning those reasons with the Mozilla statement.
- Any decision to not comply with the timeline specified in the Baseline Requirements must also be accompanied by a clear timeline describing if and when the problematic certificates will be revoked or expire naturally, and supported by the rationale to delay revocation.
We have provided this information in the initial attachment to this bug.
- The issue will need to be listed as a finding in your CA’s next BR audit statement.
Yes. As required, this finding will be listed in our next audit report.
- Your CA will work with your auditor (and supervisory body, as appropriate) and the Root Store(s) that your CA participates in to ensure your analysis of the risk and plan of remediation is acceptable.
We are working with the root stores and our auditors through this bug and in separate discussions. We did reach out to the root programs before filing this incident report. We appreciate all of the feedback so far. We acknowledge Ryan’s post and additional information provided in his update.
- You will perform an analysis to determine the factors that prevented timely revocation of the certificates and include a set of remediation actions in the final incident report that aim to prevent future revocation delays.
Reasons provided by customers for delaying revocation include pinning, requirements around regulatory approval, automation breaking, delays due to a mandatory DR test, app updates that are pending on Google approval, and revocation of critical infrastructure that will cause a widespread outage. DigiCert does not have access to customer systems and cannot evaluate these representations against Mozilla requirements. We are rejecting reasons (like pinning) that do not meet the Mozilla standards, and we already fully support automated certificate actions.
If your CA will not be revoking the problematic certificates as required by the BRs, then we recommend that you also contact the other root programs that your CA participates in to acknowledge this non-compliance and discuss what expectations their Root Programs have with respect to these certificates.
We have been in contact with the other relevant Root Programs.
Now the bad news. Two certificates have moved back 28 days. These were originally set to be revoked 2024-05-17. What happened?
At the time the list was published, these were the dates provided by the customer, although they were still tentative. Later, we received an updated reason and a new date.
Five more certificates have been similarly pushed back. One was due to expire 2024-05-18, and is now expiring 2024-06-18. The other four have had their deadlines changed from 2024-06-30 to 2024-07-31. What happened?
These were originally listed as delayed due to critical infrastructure. We set this date based on that reason but heard back that the government approval will require more time. The new date was set based on when government approval would occur.
On the other hand, Mozilla does have additional expectations on top of the BRs for this type of incident, which Wayne already enumerated, and those expectations haven't been met either.
Hopefully the above points address any deficiencies. We are happy to provide additional information if they do not.
|
|
Reporter | |
Comment 27•1 year ago
|
||
Remaining certificates: 487
|
||
Comment 28•1 year ago
|
||
(In reply to Jeremy Rowley from comment #26).
Five more certificates have been similarly pushed back. One was due to expire 2024-05-18, and is now expiring 2024-06-18. The other four have had their deadlines changed from 2024-06-30 to 2024-07-31. What happened?
These were originally listed as delayed due to critical infrastructure. We set this date based on that reason but heard back that the government approval will require more time. The new date was set based on when government approval would occur.
If these subscribers require months of time to get government approval to replace their certificates, should they be using webPKI?
Given that these subscribers have chosen to use webPKI certificates knowing that they are limited by (some unspecified) government approval that prevents them from handling a timely revocation in case of miss issuance (or a security risk?) in a timely manner according to the BRs, is it responsible to allow these government approvals as a reason for delayed revocation? Surely the subscribers must be aware that if you use webPKI there are scenarios that require prompt revocation, the requirement for government approval for these subscribers cannot be a surprise either. I would argue that these subscribers were not using webPKI certificates in good faith and as such you should not delay revocation.
|
|
||
Comment 29•1 year ago
|
||
Even putting aside the subjective responsibility it is still not compatible with the Prohibited Certificate Usage terms, but I've stated that enough times at this point.
Without going into the each answer provided I do think it is telling that now each cert has a revocation date of 06-22 or 06-26. We still haven't learned why those new revocation dates into late July appeared, nor what made these government approval issues vanish. At most, they were tentative dates stated by the subscriber and presumably no pushback occurred until the past week? It is hard to reconcile otherwise.
I hope this is a useful example as to why you need to be far more stern about the revocation terms stated to subscribers going forward.
|
|
Reporter | |
Comment 30•1 year ago
|
||
All certificates are now revoked.
If these subscribers require months of time to get government approval to replace their certificates, should they be using webPKI?
Yes. Online banking and healthcare services are an important part of the WebPKI. Its possible that in the future X9 or another group will take over that role. For now, banks and healthcare providers have their websites on the traditional web.
Given that these subscribers have chosen to use webPKI certificates knowing that they are limited by (some unspecified) government approval that prevents them from handling a timely revocation in case of miss issuance (or a security risk?) in a timely manner according to the BRs, is it responsible to allow these government approvals as a reason for delayed revocation?
Only the browsers can answer this question.
Without going into the each answer provided I do think it is telling that now each cert has a revocation date of 06-22 or 06-26. We still haven't learned why those new revocation dates into late July appeared, nor what made these government approval issues vanish. At most, they were tentative dates stated by the subscriber and presumably no pushback occurred until the past week? It is hard to reconcile otherwise.
The majority of certs were gated on revocation because of a broken automation system. The bug was identified and fixed, which accelerated replacement of the remaining certificates. Certificates delayed for other reasons were revoked on Google’s response to delay. We interpreted Google’s message as requiring immediate revocation. This revocation, unfortunately, did result in significant outages and potential fines.
|
|
||
Comment 31•1 year ago
|
||
(In reply to Jeremy Rowley from comment #30)
All certificates are now revoked.
If these subscribers require months of time to get government approval to replace their certificates, should they be using webPKI?
Yes. Online banking and healthcare services are an important part of the WebPKI. Its possible that in the future X9 or another group will take over that role. For now, banks and healthcare providers have their websites on the traditional web.
Are these subscribers technically able to handle revocation and reissuance within the deadlines but chose to delay because of these government approvals?
Given that these subscribers have chosen to use webPKI certificates knowing that they are limited by (some unspecified) government approval that prevents them from handling a timely revocation in case of miss issuance (or a security risk?) in a timely manner according to the BRs, is it responsible to allow these government approvals as a reason for delayed revocation?
Only the browsers can answer this question.
Going forward, will Digicert allow a subscriber to delay the revocations of their certificates in the case of mississuance by claiming government approval is needed? I think the answer is implied further down in your quoted reply, but I would like to be sure.
Without going into the each answer provided I do think it is telling that now each cert has a revocation date of 06-22 or 06-26. We still haven't learned why those new revocation dates into late July appeared, nor what made these government approval issues vanish. At most, they were tentative dates stated by the subscriber and presumably no pushback occurred until the past week? It is hard to reconcile otherwise.
The majority of certs were gated on revocation because of a broken automation system. The bug was identified and fixed, which accelerated replacement of the remaining certificates. Certificates delayed for other reasons were revoked on Google’s response to delay. We interpreted Google’s message as requiring immediate revocation. This revocation, unfortunately, did result in significant outages and potential fines.
|
|
Reporter | |
Comment 32•1 year ago
|
||
Are these subscribers technically able to handle revocation and reissuance within the deadlines but chose to delay because of these government approvals?
Yes. Most of our customers have some form of automation in place. We've had extensive conversations with those that do not have automation about supporting automation asap.
Going forward, will Digicert allow a subscriber to delay the revocations of their certificates in the case of mis-issuance by claiming government approval is needed? I think the answer is implied further down in your quoted reply, but I would like to be sure.
No, based on the responses here and the MDSP discussion, I would not permit a delay based on government approval going forward. I think Google definitely made that clear. I do hope the browsers codify this into a better policy as knowing every CAs bugs and what's posted on official vs. personal opinion is difficult. Personally, I would love to see a complete overhaul of the revocation process.
|
||
Comment 33•1 year ago
|
||
(In reply to Jeremy Rowley from comment #26)
To make sure we’ve followed the procedure correctly:
- The decision and rationale for delaying revocation will be disclosed in the form of a preliminary incident report immediately; preferably before the BR-mandated revocation deadline. The rationale must include detailed and substantiated explanations for why the situation is exceptional. Responses similar to “we do not deem this non-compliant certificate to be a security risk” are not acceptable. When revocation is delayed at the request of specific Subscribers, the rationale must be provided on a per-Subscriber basis.
This is complete. We provided the rationale for the reasons and why the delay happened in the opening post of this bug, aligning those reasons with the Mozilla statement.
Apologies, Jeremy, but I do not see the required per-subscriber detail in your initial post, but instead a summarization of themes. I think the Mozilla policy is quite clear on this matter; do you disagree? I would expect that per-subscriber rationale would involve one explanation for each distinct subscriber that requested and was granted a delay. I think that such a rationale would necessarily include both a description of the specific harms that would have been caused to the web ecosystem if revocation had been performed promptly, and at least an overview of how Digicert was evaluating the impact on the web PKI of delayed revocation when contrasting them. (The latter is likely quite similar across subscribers, though I would expect services that are not publicly accessible to have a somewhat different calculus from those that are.)
This is important even if the certs have all been revoked by now, because it helps the community better understand how CAs are deciding that situations are exceptional, give concrete feedback on those analyses, and learn from each other’s experiences. It can also help identify places where the policies would benefit from clarification or additional detail.
|
|
Reporter | |
Comment 34•1 year ago
|
||
Hey Mike - the reasons were provided by certificate in the attachment uploaded of the certificates being delayed. We provided it on a per certificate basis as the reason depended often on the certificate, not the subscriber, but you can easily find the subscriber information there as well. I figured by certificate met the requirement and more since a by certificate analysis was more comprehensive than just per subscriber.
I realized after you posted that obsoleting the files would have unintentionally hidden the reason information. I'll see if I can un-obsolete the file that shows the reason for the delay by certificate. If not, I'll upload the list of reasons again as a new attachment.
|
|
||
Updated•1 year ago
|
|
|
Reporter | |
Comment 36•1 year ago
|
||
Looks like you did the same thing I did. You should see the file now with the reasons listed. Happy to answer questions about it. The big one was ADP, which had automation but it unfortunately broke. Took them until last week to get it fixed.
|
|
Reporter | |
Comment 37•1 year ago
|
||
Looks like you did the same thing I did. You should see the file now with the reasons listed. Happy to answer questions about it. The big one was ADP, which had automation but it unfortunately broke. Took them until last week to get it fixed.
|
|
Reporter | |
Comment 38•1 year ago
|
||
Any more questions? If not, can this bug be closed?
|
|
||
Comment 39•1 year ago
|
||
I'll close this on or about Wed. 10-July-2024, unless there are additional questions or issues to discuss.
|
||
Comment 40•1 year ago
|
||
I don't think we can close. I don't see real action-item or more detailed root-cause.
Entrust were just distrusted for almost the same thing here, and these revokes were done only when pressure applied.
No commitment to fix the problems.
Also say 'Online banking and healthcare services are an important part of the WebPKI." - only if they can revoke and re-place in 5 days or less. Otherwise, no. We need action from Digicert what they do here.
If we do not see commitment to assure future revoke will happen in time and steps to fix what Digicert think are the root-cause, then we cannot close this I think.
|
|
||
Comment 41•1 year ago
|
||
We do need to see more detail on the action items, so far it has been solely to revoke the certificates. This was highlighted in Comment 13, and then discussed in Comment 14 but we have no updated list of action items to date to ensure no delayed revocation occurs going forward for these subscribers.
|
|
Reporter | |
Comment 42•1 year ago
|
||
I don't think we can close. I don't see real action-item or more detailed root-cause.
The root cause was provided in the initial analysis. Wide-spread revocation would lead to critical infrastructure outages and required government approval before changes. Based on the discussion on MDSP, required government approvals is not a sufficient reason for delay and we have updated our incident response plans based on this. Action items to prevent mis-issuance were described in the other bug report.
Entrust were just distrusted for almost the same thing here, and these revokes were done only when pressure applied.
This is incorrect. We revoked 7.000 of the 11,000 certificates within the 5 days (4 days after sending notice). Most of the remaining were replaced within two weeks. Roughly 1500 remained by the time Google posted its thoughts on the timeline. External pressure from the community had nothing to do with our decision to revoke, but we did value the feedback as figuring out what constituted critical infrastructure or an acceptable timeline wasn't clear from the Mozilla page. We will closely monitor and adhere to suggested improvements from the root programs. We look forward to whatever revisions come from the discussion on MDSP.
If we do not see commitment to assure future revoke will happen in time and steps to fix what Digicert think are the root-cause, then we cannot close this I think.
We are committed to following the requirements of the browsers, which includes following their written procedures and processes for revocation and delayed revocation.
We do need to see more detail on the action items, so far it has been solely to revoke the certificates. This was highlighted in Comment 13, and then discussed in Comment 14 but we have no updated list of action items to date to ensure no delayed revocation occurs going forward for these subscribers.
Some of the actions we have taken include:
• Requesting that customers use our automation solutions, including ACME,
• Reiterated that crypto-agility is essential and reminding people that pinning is not permitted,
• Posted that government approval is not a sufficient reason to delay revocation,
• Provided customers additional information on the CABF requirements by email, phone, webinar and other methods, and
• Updating our policy around revocation and delayed revocation with additional evaluation of whether a delay is both for critical infrastructure and constitutes “exceptional circumstances” per the Google post. All revocations will meet the expectations of the browsers as laid out in their written documentation. We appreciate any additional ideas from the community as well.
We are working on updating our ACME service to include ARI, but that’s not necessarily relevant for this project. However, it will help replace certificates faster.
|
|
Reporter | |
Comment 43•1 year ago
|
||
Tim posted additional ideas on how to improve revocation to MDSP. I don't have further updates for this bug though.
|
|
||
Comment 44•1 year ago
|
||
I think we can close this with the understanding that CAs, root stores, and stakeholders need to all address the delayed revocation issues with a comprehensive solution. As I've said numerous times, the issue is not going to go away by itself and that we need to address it head on.
|
|
||
Comment 45•1 year ago
|
||
I plan to close this bug on or around Monday, July 29, 2024. If there are any significant concerns or inadequacies that have not yet been addressed, please raise them before then.
|
|
||
Comment 46•1 year ago
|
||
(In reply to Ben Wilson from comment #44)
I think we can close this with the understanding that CAs, root stores, and stakeholders all need to all address the delayed revocation issues with a comprehensive solution. As I've said numerous times, the issue is not going to go away by itself and that we need to address it head on.
Ben,
Before we close it, I do have some questions specific to this incident. I very much agree with the sentiment that delayed revocation is an unresolved problem in the WebPKI. Sectigo applauds and intends to support efforts to achieve a clear, consistent resolution that leads to reliable compliance among CAs.
But for now, let’s speak specifically about this delrev incident:
Jeremy, I see you circling in on the idea of DigiCert taking sole responsibility for completing revocations on time, but I’m struggling to find a clear commitment. Comment 42 states,
Based on the discussion on MDSP, required government approvals is not a sufficient reason for delay and we have updated our incident response plans based on this.
It’s good for you to remove this excuse for late revocation from the roster, but this is far from a blanket commitment not to willfully delay revocation. Likewise, you state,
We are committed to following the requirements of the browsers, which includes following their written procedures and processes for revocation and delayed revocation.
This is reminiscent of Entrust’s repeated statements such as “we are guided by the TLS Baseline Requirements and Mozilla’s Responding to an Incident,” which appear to have been attempts at posing as promises not to delay certificate revocation without actually making this promise.
Just in the last few days we have seen convincing commitments not to deliberately delay revocation from Telia in bug 1896553 comment 28 and Chunghwa Telecom in bug 1903066 comment 54.
Question 1: I see the above as DigiCert explicitly choosing not to make any commitment against deliberately delaying revocation in the future. In case I am misinterpreting your stance, I want to give you every opportunity to make that commitment in plain writing without equivocation. Is DigiCert prepared to follow the lead of CAs such as Telia and Chunghwa Telecom by making a firm, public commitment never again to deliberately delay revocation of misissued certificates?
Comment 42 contains this passage:
Entrust were just distrusted for almost the same thing here, and these revokes were done only when pressure applied.
This is incorrect. We revoked 7.000 of the 11,000 certificates within the 5 days (4 days after sending notice). Most of the remaining were replaced within two weeks.
I disagree with the suggestion that this was good work on DigiCert’s part. You failed to meet expectations on more than a third of these certificates, and the last of them was not revoked until 50 days after DigiCert confirmed misissuance.
Question 2: What is DigiCert’s true, unvarnished attitude toward this delayed revocation? Do you view it as an accomplishment that a mere 4,000 of 11,000 revocations were late? Do you view completing a five-day revocation for a given certificate in fourteen days as success?
|
|
||
Comment 47•1 year ago
|
||
I also disagree with closing this bug until proper action items are set that will prevent a repeat of this delayed revocation.
So far all the action items from #42 are not measurable and really amount to we said X to Y.
Also, ARI is partially relevant here so feel free to throw that in as an action item so it can then be tracked.
|
||
Comment 48•1 year ago
•
|
||
Legal restriction in Chinese law requires sign-off from government 1.07%
Could you elaborate details on this, e.g. citing the relevant law?
Please note that the Cyberspace Administration of China is not a government entity but a party organ of the Communist Party.
|
||
Comment 49•1 year ago
|
||
(In reply to Stephan Verbücheln from comment #48)
Legal restriction in Chinese law requires sign-off from government 1.07%
Could you elaborate details on this, e.g. citing the relevant law?
Please note that the Cyberspace Administration of China is not a government entity but a party organ of the Communist Party.
(Post Personally)
I would like to cite Chapter V, Security and Openness of Government Data, of the Data Security Law of the People's Republic of China:
Article 40 Where a state organ entrusts others to construct or maintain e-government systems, or to store or process government data, the state organ shall go through strict approval procedures, and shall supervise the entrusted party in the performance of data security protection obligations. The entrusted party shall perform its data security protection obligations in accordance with the provisions of laws, regulations, and contracts signed, and shall not retain, use, divulge, or provide others with government data without authorization
If deploying a SSL/TLS certificate is considered part of the endeavors to maintain e-government systems, I believe a sign-off is required by this clause for both the deployment and replacement of an SSL/TLS certificate.
I do not think the laws of any jurisdictions will specify the details to specifically say that replacement of digital certificate will go through a sign-off procedure. The law typically provides a framework, while the detailed implementation rules are formulated by the relevant departments or agencies.
|
|
||
Comment 50•1 year ago
|
||
If deploying a SSL/TLS certificate is considered part of the endeavors to maintain e-government systems, I believe a sign-off is required by this clause for both the deployment and replacement of an SSL/TLS certificate.
In that case one could argue that issuing to such a subscriber is already a misissuance, as pointed out by Mike Shaver in the GDCA discussion.
https://bugzilla.mozilla.org/show_bug.cgi?id=1889062#c9
https://bugzilla.mozilla.org/show_bug.cgi?id=1889062#c11
Also note that Article 40 states that “the entrusted party shall perform its data security protection obligations in accordance with the [...] contracts signed”. Since revocation policy is part of the contract, it is not obvious why this should prevent execution of contractually mandated revocations.
|
||
Comment 51•1 year ago
|
||
Thanks for the additional questions. I just wanted to provide an update that we're still in the process of working on responses.
-Tim
|
|
||
Comment 52•1 year ago
|
||
(In reply to Stephan Verbücheln from comment #50)
If deploying a SSL/TLS certificate is considered part of the endeavors to maintain e-government systems, I believe a sign-off is required by this clause for both the deployment and replacement of an SSL/TLS certificate.
In that case one could argue that issuing to such a subscriber is already a misissuance, as pointed out by Mike Shaver in the GDCA discussion.
https://bugzilla.mozilla.org/show_bug.cgi?id=1889062#c9
https://bugzilla.mozilla.org/show_bug.cgi?id=1889062#c11
(Post Personally)
I am not sure how exactly the sign-off procedures are implemented within these applicants, but it is the CA’s responsibilities to make sure to get the acknowledgment from subscribers about the circumstances where revocation with mandatory timeline is required.
One possibility is that the applicants did not anticipate that they could be informed about a certificate revocation and asked to replace a certificate within 24 hours or five days and therefore did not take it seriously before accepting the terms and conditions from the CAs, where possibilities of mandatory revocation by the CA was properly communicated. And when the circumstances of the revocation became a reality, they immediately regretted their initial commitment and were unwilling to complete the certificate replacement and revocation within the specified timeframe.
Also note that Article 40 states that “the entrusted party shall perform its data security protection obligations in accordance with the [...] contracts signed”. Since revocation policy is part of the contract, it is not obvious why this should prevent execution of contractually mandated revocations.
Yes I agree that such a contract should not prevent the execution of certificates revocations, instead, it provides the basis for the CA to execute mandatory revocations.
|
|
||
Comment 53•1 year ago
|
||
Tim,
I feel like all this discussion about whose opinions do or do not have varnish on them, and how you personally view, interpret, and mis-restate our various responses, and the "opportunities" you're giving us to make various declarations ... none of this helps move the discussion forward or help improve the security of the internet, so I'm going to politely decline to participate.
The CA/Browser Forum and Mozilla Dev Security Policy are wonderful places to have discussions about revocation philosophy and what the rules should be moving forward. We've had plenty of interactions and discussions there, including in Bergamo and the lunch we had in DC, and we will continue to do so.
If there are more questions on this specific incident, I am more than happy to answer them. But I think keeping discussions properly scoped to the correct topic is important, too.
Forward and up,
-Tim
|
|
||
Comment 54•1 year ago
|
||
Still keeping an eye on this one, but spending more time on the more recent ones.
-Tim
|
|
||
Comment 55•1 year ago
|
||
We have nothing further on this bug.
-Tim
|
||
Comment 56•1 year ago
|
||
(In reply to amir from comment #47)
I also disagree with closing this bug until proper action items are set that will prevent a repeat of this delayed revocation.
So far all the action items from #42 are not measurable and really amount to we said X to Y.
Also, ARI is partially relevant here so feel free to throw that in as an action item so it can then be tracked.
As per https://bugzilla.mozilla.org/show_bug.cgi?id=1910805 we are in the process of deploying ARI.
(In reply to Stephan Verbücheln from comment #48)
Legal restriction in Chinese law requires sign-off from government 1.07%
Could you elaborate details on this, e.g. citing the relevant law?
Please note that the Cyberspace Administration of China is not a government entity but a party organ of the Communist Party.
Thanks to Eric for already posting this up
https://bugzilla.mozilla.org/show_bug.cgi?id=1896053#c49
|
|
||
Comment 57•1 year ago
|
||
Still watching this one, but are we ready to close it out?
|
|
||
Updated•1 year ago
|
|
|
||
Updated•1 year ago
|
|
|
||
Updated•1 year ago
|
|
|
||
Comment 58•1 year ago
|
||
We continue work on incident-reporting and compliance requirements aimed at reducing delayed revocation, so this bug will remain open until at least February 1, 2025. Meanwhile, CAs should review https://github.com/mozilla/www.ccadb.org/pull/186.
|
|
||
Comment 59•1 year ago
|
||
Ben, thanks for your hard work trying to come up with improvements in this area, I know this is a particularly thorny and controversial topic and really appreciate your perspectives and proposals.
|
Assignee | |
Comment 60•1 year ago
|
||
Testing out the new DigiCert email address, as requested by root programs.
|
|
Assignee | |
Comment 61•1 year ago
|
||
Test succeeded.
Going forward, we will be posting updates as DigiCert, as suggested by a proposed update to CCADB policy. This will make it clear these are official DigiCert responses.
|
|
||
Updated•1 year ago
|
|
|
Assignee | |
Comment 62•1 year ago
|
||
This one is ready for a closing statement which we will post this week.
|
|
||
Updated•1 year ago
|
|
|
Assignee | |
Comment 63•1 year ago
|
||
Digicert completed all action items listed for this bug.
Action Items
| Action Item | Kind | Due Date |
|---|---|---|
| Revoke all impacted certificates | Remediation | Completed |
| Inform customers about our ACME services, emphasized the need for crypto-agility, and reminded customers that pinning is never an acceptable reason for delayed revocation | Prevention | Completed |
| Started a thread on MDSP about improving delayed revocation and improving the revocation process | Prevention | Completed |
| Deployment of ARI | Prevention | Completed |
| Updated internal policies to clarify that delayed revocation for exceptional circumstances is unacceptable | Prevention | Completed |
The primary discussion on this thread was about when delayed revocation constituted exceptional circumstances and whether delayed revocation is ever necessary. One of the primary takeaways is that Mozilla is clarifying its policy on delayed revocation. I think this discussion has also clarified some of the mis-interpretations of the current Mozilla policy that we’ve seen in relying parties and CAs. Specifically, this bug clarified that exceptional circumstances are not a reason for delayed revocation.
Incident Report Closure Summary
Incident Description:
Certificates issued with an incorrect business category took longer than 5 days to revoke as some of these revocations were considered “exceptional circumstances”. Exceptional circumstances, as defined by DigiCert, included critical infrastructure and systems where Chinese law prohibited revocation. This bug was filed before the delayed revocation occurred in accordance with the Mozilla incident process.
Incident Root Cause(s):
The root cause was that subscribers could not replace their certificates within the required 5-day window. DigiCert and these subscribers both, incorrectly, believed there was a process for delayed revocation described on the Mozilla page for responding to incidents. Mozilla has since updated this page. The discussion on this bug has also clarified any confusion around whether delayed revocation is acceptable, even in exceptional circumstances (it is not).
Remediation Description:
DigiCert provided active communication to its customers around automated solutions, including ACME, clarified to its customers that delayed revocation is never allowed, and promoted the use of ARI. DigiCert implemented ARI in its offering and revoked all impacted certificates.
Commitment Summary:
We have updated our internal incident management processes to remove any reference to critical infrastructure. We implemented ARI for customers to use in automatically replacing certificates impacted by an incident. Finally, we started a thread that is leading to clarification of the expectations around revocation.
@Ben Wilson – As all action items are completed, will you please close this bug?
|
|
||
Comment 64•1 year ago
|
||
@Digicert I’m curious about how you dealt with the Chinese law prohibiting revocation, is this no longer an issue because of some automation? Or are you prioritizing your commitments to webPKI over those laws?
There have been similar problems for other CAs, for example citing laws preventing them from taking an updated CP/CPS into effect immediately.
|
|
Assignee | |
Comment 65•1 year ago
|
||
Mozilla policy is very clear that CAs are expected to follow the Baseline Requirements, and the Baseline Requirements are very clear that CAs need to follow the law:
"9.15 Compliance with applicable law
The CA SHALL issue Certificates and operate its PKI in accordance with all law applicable to its business and the Certificates it issues in every jurisdiction in which it operates."
This is the way it needs to be. Neither Mozilla, nor any other organization, should be advocating for organizations to ignore the law. Our commitment to the WebPKI is why we take these laws seriously.
|
|
||
Comment 66•1 year ago
|
||
I'm not sure where any potential advocacy to not comply with laws in the relevant jurisdiction occurred to merit that response.
To explain my understanding of Comment 64's question in the Incident Report Closure Summary, the Incident Description notes:
Certificates issued with an incorrect business category took longer than 5 days to revoke as some of these revocations were considered “exceptional circumstances”. Exceptional circumstances, as defined by DigiCert, included critical infrastructure and systems where Chinese law prohibited revocation.
Given the above explicit description was provided by DigiCert, it seems that DigiCert's final understanding of this incident is that the relevant local law prohibits revocation. This is contrary to Comments 49 and 52, although only 49 was noted as in-line with DigiCert's presumptive answer in Comment 56. The lack of direct answers by DigiCert to the questions does make both DigiCert's internal understanding and outward intentions going forward rather hard to parse.
Comment 52 does make note that the law, as this incident's interpretation seemingly reads, isn't prohibiting revocation. However DigiCert's stance on this to date is unclear, but apparently contradictory with the closing remarks. The lack of any clarity on procedures going forward to handle revocation with regards to pre-existing legal challenges.
To be clear this isn't a question of issuance into an previously unknown legal scenario, but rather ongoing wilful issuance where DigiCert is asserting that future revocation is prohibited. How this aligns with WebPKI commitments is questionable, and Mozilla's policy does not seem to have been written with this in mind given its publication date.
The sole remediation item noted is automation, so I hope that Comment 64's statement makes more sense with more context surrounding this incident. When an incident like this occurs in the future, what does DigiCert intend to do, and how have they actively planned ahead to ensure this does not turn into a delrev incident?
The comments here are not looking for legal opinions of jurisdiction and final authorities, but the reality of how issuance will operate going forward in the most practical sense. I hope that clarifies the situation currently.
|
|
||
Comment 67•1 year ago
|
||
(In reply to DigiCert from comment #63)
Sectigo would like to acknowledge DigiCert for reversing its position and accepting that Mozilla’s unambiguous policy against delayed revocation cannot be reinterpreted to mean the opposite of what it says. We particularly note three statements:
… exceptional circumstances are not a reason for delayed revocation.
… whether delayed revocation is acceptable, even in exceptional circumstances (it is not).
DigiCert … clarified to its customers that delayed revocation is never allowed…
We hope these statements are a blanket commitment from DigiCert never again to knowingly delay revocation beyond the BR mandated deadlines. We do wish it had been stated that plainly, with no future room for claiming otherwise. So we’ll ask the question directly.
Question 1: Yes or no, is DigiCert committing to never again knowingly delay BR-mandated certificate revocation?
One of the primary takeaways is that Mozilla is clarifying its policy on delayed revocation. I think this discussion has also clarified some of the mis-interpretations of the current Mozilla policy that we’ve seen in relying parties and CAs. Specifically, this bug clarified that exceptional circumstances are not a reason for delayed revocation.
We have to push back on this mischaracterization of the central point of this and twenty-odd other Bugzilla bugs since the beginning of 2024. We don’t accept that Mozilla’s policy was at any point unclear or that any meaningful clarification had to or did occur. We feel that Mozilla doubled down with what should have been unnecessary language to hamstring indefensible and dishonest justifications for simply choosing not to follow clearly articulated rules on which the BRs and all major root programs agreed.
This is not simple nit-picking or rubbing your face in things. This is a key point. If DigiCert maintains that Mozilla’s published policy for most of 2024 did, in fact, justify the willful ignorance of requirements by the BRs and other root programs and that it is possible for a single root program to make a declaration that nullifies requirements from both the BRs and other root programs, then that is an untenable position, and we cannot trust DigiCert not to play that card again in the future whenever it’s convenient. This appears to be what you are claiming.
To put it another way, the quoted passage above suggests that, were Mozilla to return to its previous policy language, DigiCert would return to its previous position regarding delayed revocation. That would be unacceptable. I will point to the dozens of comments by many community members on delrev bugs against DigiCert, Entrust, and other CAs going back to early 2024 articulating the absurdity of your earlier position. These are not new ideas.
Question 2: We contend that published Mozilla policy going back to before DigiCert’s delrev events of 2024 has been clear in that it left no room for CAs to choose not to revoke certificates according to BR language. Yes or no, does DigiCert agree?
Question 3: We contend that a single root program cannot unilaterally remove requirements by other root programs or the BRs through its own policy declarations. Yes or no, does DigiCert agree?
Finally, we continue to support Mozilla’s concept of attaching consequences to domains receiving revocation delays in the case of misissuance, especially reduction in maximum term. We encourage ongoing work on such a policy.
|
|
Assignee | |
Comment 68•1 year ago
|
||
@Wayne https://bugzilla.mozilla.org/show_bug.cgi?id=1896053#c64 very clearly asked whether we were prioritizing the WebPKI over applicable law, so we were answering that question.
|
|
||
Comment 69•1 year ago
|
||
I had a few more questions in that comment that I would appreciate answered.
|
|
Assignee | |
Comment 70•1 year ago
|
||
Wayne, I can ask legal what their current opinion is on these laws is.
|
|
Assignee | |
Comment 71•1 year ago
|
||
Q1: Our CPS states that we follow the BRs.
Q2: I think it is clear from previous answers that Sectigo and DigiCert disagree on this point. I'm not sure what purpose asking again serves.
Q3: This one is actually subtle, as one root program has recently proposed that its root program requirements do in fact supersede the Baseline Requirements. We asked for clarification in our response, as it would be a disaster if all root programs started asserting this.
Since the obligation to follow the BRs actually flows down from root program requirements, it is in fact true that any root program can state requirements that conflict with the BRs. This actually causes a significant amount of CA pain and confusion when it happens, even accidentally, which is why we often respectfully ask the root programs not to override the BRs, even though they are capable.
There is even a historical example of all four root programs simultaneously declaring a troublesome BR requirement inoperative. We're a bit surprised Sectigo is asserting a position that is at odds with reality and history.
|
|
||
Updated•1 year ago
|
|
|
Assignee | |
Comment 72•1 year ago
|
||
Legal is still looking into the exact parameters of the issue with Chinese law, and what to do going forward. We agree that we'd rather not be issuing certificates that can't legally be revoked.
|
|
||
Comment 73•1 year ago
|
||
(In reply to DigiCert from comment #72)
Legal is still looking into the exact parameters of the issue with Chinese law, and what to do going forward. We agree that we'd rather not be issuing certificates that can't legally be revoked.
To recap: In comment 56, DigiCert appeared to acknowledge as correct Eric's description of the relevant Chinese law in comment 49. And in comment 52, Eric agreed with Stephan that if the contract between the CA and its Chinese subscribers permits the CA to revoke within TLS BR mandated deadlines, then it is not illegal for the CA to do just that.
If the DigiCert Legal team's current review arrives at the conclusion that revocation of Chinese subscriber certificates within TLS BR mandated deadlines is not illegal, then I think it's clear that TLS BR 9.1.5 (Compliance with applicable law), which you referenced in comment 65, has no bearing. To delay revocation beyond TLS BR mandated deadlines under these circumstances, simply because a Chinese subscriber is unable to obtain and install a replacement certificate in a timely manner, means that the CA is prioritizing their subscriber's wishes above the CA's responsibility to adhere to the TLS BRs / root program policies. Doing this for lots of Chinese subscribers would clearly fall short of both Mozilla's current guidance that "Delayed revocation is a measure of last resort and MUST NOT be used routinely" and Mozilla's previous guidance that anticipated delayed revocation incidents only "in some exceptional circumstances".
The only other possible outcome would be that DigiCert's Legal team stands by DigiCert's previous statement in comment 63 that "Exceptional circumstances, as defined by DigiCert, included critical infrastructure and systems where Chinese law prohibited revocation". In this scenario I can certainly see why you would consider TLS BR 9.1.5 to be applicable, but if that's the case then it seems obvious to me that you would also need to follow TLS BR 9.16.3 (Severability), which says (emphasis mine):
"In the event of a conflict between these Requirements and a law, regulation or government order (hereinafter 'Law') of any jurisdiction in which a CA operates or issues certificates, a CA MAY modify any conflicting requirement to the minimum extent necessary to make the requirement valid and legal in the jurisdiction. This applies only to operations or certificate issuances that are subject to that Law. In such event, the CA SHALL immediately (and prior to issuing a certificate under the modified requirement) include in Section 9.16.3 of the CA's CPS a detailed reference to the Law requiring a modification of these Requirements under this section, and the specific modification to these Requirements implemented by the CA. The CA MUST also (prior to issuing a certificate under the modified requirement) notify the CA/Browser Forum of the relevant information newly added to its CPS by sending a message to questions@cabforum.org and receiving confirmation that it has been posted to the Public Mailing List and is indexed in the Public Mail Archives available at https://cabforum.org/pipermail/public/ (or such other email addresses and links as the Forum may designate), so that the CA/Browser Forum may consider possible revisions to these Requirements accordingly. Any modification to CA practice enabled under this section MUST be discontinued if and when the Law no longer applies, or these Requirements are modified to make it possible to comply with both them and the Law simultaneously. An appropriate change in practice, modification to the CA's CPS and a notice to the CA/Browser Forum, as outlined above, MUST be made within 90 days."
I don't see any reference to any Chinese law in Section 9.16.3 of the current version of the DigiCert CPS, and I also don't recall seeing any notification to CABForum on this matter. So if DigiCert plans to delay revocation in the future for any Chinese subscriber certificates in the belief that timely revocation is forbidden under TLS BR 9.1.5 due to Chinese Law, then I think you need to follow the requirements in TLS BR 9.16.3 before you proceed with that plan.
Also, given that comment 63 indicates that DigiCert has delayed revocation on those grounds in the past, I think you now need to file a new incident bug to explain why you didn't follow the requirements in TLS BR 9.16.3 before doing so.
|
|
Assignee | |
Comment 74•1 year ago
|
||
In response to Rob Stradling in comment 73
The only other possible outcome would be that DigiCert's Legal team stands by DigiCert's previous statement in comment 63 that "Exceptional circumstances, as defined by DigiCert, included critical infrastructure and systems where Chinese law prohibited revocation". In this scenario I can certainly see why you would consider TLS BR 9.1.5 to be applicable, but if that's the case then it seems obvious to me that you would also need to follow TLS BR 9.16.3 (Severability), which says (emphasis mine):
We wanted to raise the community’s attention to the Chinese law as it surprised us during our response to this bug. Even though the delay was categorized under exceptional circumstances because the institution was a bank that alleged outages across China if the cert was revoked, as part of our commitment to transparency, we flagged the Chinese law specifically to make the community aware of all of the circumstances surrounding the revocation. As explained in the initial bug report, the primary reason for delayed revocation was simply critical infrastructure and exceptional circumstances.
I don't see any reference to any Chinese law in Section 9.16.3 of the current version of the DigiCert CPS, and I also don't recall seeing any notification to CABForum on this matter. So if DigiCert plans to delay revocation in the future for any Chinese subscriber certificates in the belief that timely revocation is forbidden under TLS BR 9.1.5 due to Chinese Law, then I think you need to follow the requirements in TLS BR 9.16.3 before you proceed with that plan.
We don’t plan on delaying revocation in the future. Thank you for the reference to TLS BR 9.16.3. The Chinese law and its impact on issuance in China should indeed be discussed at the CAB Forum.
Also, given that comment 63 indicates that DigiCert has delayed revocation on those grounds in the past, I think you now need to file a new incident bug to explain why you didn't follow the requirements in TLS BR 9.16.3 before doing so.
DigiCert has not delayed revocation on the basis of law in the past. We were unaware of the Chinese law before this bug occurred. We will raise the Chinese law at the CAB Forum and welcome the community to highlight other laws of this nature.
|
|
||
Comment 75•1 year ago
|
||
(In reply to Rob Stradling from comment #73)
(In reply to DigiCert from comment #72)
Legal is still looking into the exact parameters of the issue with Chinese law, and what to do going forward. We agree that we'd rather not be issuing certificates that can't legally be revoked.
To recap: In comment 56, DigiCert appeared to acknowledge as correct Eric's description of the relevant Chinese law in comment 49. And in comment 52, Eric agreed with Stephan that if the contract between the CA and its Chinese subscribers permits the CA to revoke within TLS BR mandated deadlines, then it is not illegal for the CA to do just that.
(Post in a personal capacity)
To further assess whether mandatory revocation of SSL certificates violates relevant laws and regulations in China, I believe it is critical to review the content of applicable legal frameworks and administrative rules in question.
While comment 49 mentioned the Data Security Law of the People's Republic of China, in my view, the Provisions on the Security Management of Internet Government Affairs Applications—formulated under the "Cybersecurity Law of the People's Republic of China", the "Data Security Law of the People's Republic of China", the "Personal Information Protection Law of the People's Republic of China", and the "Implementation Measures for the Responsibility System for Party Committee (Party Group) Network Security Work"—are likely more directly relevant to government entities in managing SSL certificates lifecycle as subscribers. These Provisions specify network and data security obligations for government agencies in e-government operations. Key provisions include:
Chapter IV: Network and Data Security
Article 17: The establishment of internet government affairs applications shall implement the tiered network security protection system and the national requirements for the management of cryptography applications, carry out grading and filing and grading assessment efforts in accordance with relevant standards and specifications, implement measures for rectification and reinforcement of security establishments, and prevent network and data security risks.
The portal websites of central and state organs, local Party and government organs at the prefectural or municipal level or above, as well as the websites of organs and public institutions that carry important business applications, internet e-mail systems, and so forth, shall comply with the requirements of Level 3 of the Classified Protection of Network Security.
Article 18: State organs and public institutions shall, on their own or by retaining a third-party network security service establishment with corresponding qualifications, conduct security inspections and assessments of internet government application networks and data security at least once a year.
Internet government affairs application system upgrades, new functions, and the introduction of new technologies and applications shall conduct security testing and assessments before they go online.
The Provisions require that critical government websites and systems comply with the requirements designed for Level 3 of the Classified Protection of Network Security, referencing section 8.1.2.2 of
GBT 22239 2019 Information Security technology-Baseline for classified protection of cybersecurity, Level 3 requirements for communication security include:
a) Data integrity during communication must be ensured through cryptographic or authentication technologies;
b) Data confidentiality during communication must be ensured through cryptographic technologies.
GB/T 22239-2019 explicitly requires encryption for e-government communications, and deploying SSL certificates on government websites is a practical and auditable method to meet this standard, forming a key criterion for third-party assessors to verify Level 3 cybersecurity compliance.
Therefore, while existing laws and regulations do not explicitly prohibit SSL certificate revocation, revoking certificates could render government entities non-compliant with Level 3 cybersecurity requirements specified by the Provisions on the Security Management of Internet Government Affairs Applications.
Conclusion
CAs revoking SSL certificates for government entities does not inherently violate any Chinese laws or regulations. Government entities relying on SSL certificates for Level 3 cybersecurity compliance must: (a) Collaborate with third-party assessors to ensure uninterrupted adherence to GB/T 22239-2019 encryption standards; (b) Proactively manage certificate lifecycle agility, including contingency plans for emergency revocation scenarios.
Additional Observations
CA Accountability: Some other CAs (no specific entity implied) inadequately communicated revocation obligations to subscribers, and in some cases, they even preemptively justify non-revocation without trying to work with government entities (or other types of subscribers) to facilitate compliance at all.
Government Posturing: When revocation becomes unavoidable (e.g. due to community scrutiny), government entities often resist by asserting—without legal basis—that revocation is "illegal", a claim some CAs accept uncritically.
Thanks.
|
|
||
Comment 76•1 year ago
|
||
(In reply to DigiCert from comment #70)
Wayne, I can ask legal what their current opinion is on these laws is.
(In reply to DigiCert from comment #72)
Legal is still looking into the exact parameters of the issue with Chinese law, and what to do going forward. We agree that we'd rather not be issuing certificates that can't legally be revoked.
It has been 7 days with no relevant answers to my questions, or a timeframe for when a response will occur.
To be clear my questions were about the continued issuance into an environment where has DigiCert attested that revocation is prohibited. Even ignoring the BRs by DigiCert's own CPS this isn't allowed as I mentioned in comment #6 almost 9 months ago:
I will quote Digicert's Certificate Policy S1.4.2: Prohibited Certificate Uses:
DigiCert strongly discourages key pinning and shall not consider it a sufficient reason to delay revocation. DigiCert continually researches and implements technological processes in order to detect pinned applications and other prohibited uses so we can counsel customers on the way pinning impacts the agility of the Web PKI (e.g., rotation of intermediate certificates). Customers should also take care in not mixing certificates trusted for the web with non-web PKI. Any certificates trusted by the browsers must comply with all requirements of all applicable browser root policies, including revocation periods of 24 hours and 5 days as asserted in the relevant policies, obligations, and requirements of this CP and the CPS.
It is crystal clear as per DigiCert's own policies that issuance is prohibited where parties can't comply with 24 hours and 5 days revocation.
Given that a statement from DigiCert's legal team does not seem to be coming anytime soon, perhaps we can generalize the problem further. My question was primarily what DigiCert will be doing going forward. To that end, please consider the following scenarios:
- Scenario 1: Certificate is issued to Subscriber A, who does not know they are incapable of handling revocation.
- Scenario 2: Certificate is issued to Subscriber B, who does know they are incapable of handling revocation but has not told their CA.
- Scenario 3: Certificate is issued to Subscriber C, who has told the CA that they are incapable of handling revocation - but through oversight the certificate gets issued regardless.
An incident occurs requiring revocation within 5 days. By the end of the 5 days each Subscriber has not responded to the CA.
- What would DigiCert think would be a suitable response per subscriber?
- If knowledge from another CA incident made it clear that Subscriber B is incapable of handling revocations, what would DigiCert's reaction look like?
- Likewise, if an internal review found the paperwork for Subscriber C that was overlooked at the time, what would DigiCert's response be?
For clarity even in Scenario 3 where a CA has made a mistake, that happens. The important part is to accept mistakes can happen and acknowledge them when they do so that everyone can learn. We then build on the procedures to try and remove the risk as much as possible.
To be extremely clear there isn't any hidden trick in any question I pose, the answers should be straightforward. My primary concern so far is any potential wilful issuance where revocation would be considered impossible by the CA. As Comment 72 makes clear:
We agree that we'd rather not be issuing certificates that can't legally be revoked.
But what I'm curious about is what DigiCert will be doing in practice going forward, rather than preferences.
|
|
Assignee | |
Comment 77•1 year ago
|
||
In response to comment 75:
Eric, thanks for the breakdown. We were also very surprised when subscribers asserted that Chinese law prevents revocation. We included this reason for awareness as it was given by a bank customer. Please note that we did not seek or obtain permission from the Chinese government prior to revocation.
In response to comment 76:
Wayne, in light of our internal assessment, we do not agree that Chinese law prevents revocation so will not take any action regarding issuance to Chinese entities. Should a case arise where local law does in fact hinder or prohibit revocation, we will certainly evaluate that situation and take appropriate action, which could include cessation of issuance to that jurisdiction.
|
|
||
Comment 78•1 year ago
|
||
(In reply to DigiCert from comment #77)
In response to comment 76:
Wayne, in light of our internal assessment, we do not agree that Chinese law prevents revocation so will not take any action regarding issuance to Chinese entities. Should a case arise where local law does in fact hinder or prohibit revocation, we will certainly evaluate that situation and take appropriate action, which could include cessation of issuance to that jurisdiction.
I made my comment explicitly not regarding any potential Chinese laws, as jurisdiction is incidental to the point. I even made sure to frame my questions so that local laws were not a factor. I surmise that the answer to all of my questions are 'it depends'?
|
|
||
Comment 79•1 year ago
|
||
I'd like to thank DigiCert for clarifying in comment 77 that its current view is that Chinese law does not prevent CAs from revoking within BR mandated deadlines. However, in this bug we are conducting a post-mortem of DigiCert's actions and views of 9 months ago, and I believe that there are still questions that need to be answered and actions that need to be taken.
(In reply to DigiCert from comment #74)
In response to Rob Stradling in comment 73
Also, given that comment 63 indicates that DigiCert has delayed revocation on those grounds in the past, I think you now need to file a new incident bug to explain why you didn't follow the requirements in TLS BR 9.16.3 before doing so.
DigiCert has not delayed revocation on the basis of law in the past. We were unaware of the Chinese law before this bug occurred.
The delayed revocations covered by this incident bug occurred "in the past", around 9 months ago. In comment 5 you reported that 1.07% of those delayed revocations were due to "Legal restriction in Chinese law requires sign-off from government", and in your Incident Report Closure Summary in comment 63 you stated that "systems where Chinese law prohibited revocation" were amongst the "exceptional circumstances" for which you allowed delayed revocation to occur for this incident.
If those statements in comment 5 and comment 63 are accurate, then your claim in comment 74 that "DigiCert has not delayed revocation on the basis of law in the past" must be untrue.
We wanted to raise the community’s attention to the Chinese law as it surprised us during our response to this bug. Even though the delay was categorized under exceptional circumstances because the institution was a bank that alleged outages across China if the cert was revoked, as part of our commitment to transparency, we flagged the Chinese law specifically to make the community aware of all of the circumstances surrounding the revocation. As explained in the initial bug report, the primary reason for delayed revocation was simply critical infrastructure and exceptional circumstances.
The "exceptional circumstances" language that appeared in Mozilla's previous delayed revocation guidance is not a 'get out of jail free card'. It's not part of the MRSP, it's not part of the TLS BRs, and it's not part of any other root store policies. Does DigiCert really believe that if a TLS BR violation is "categorized under exceptional circumstances" then that TLS BR violation can be swept under the carpet? I sincerely hope not!
https://www.ccadb.org/cas/incident-report says:
"There should be a single incident report for each distinct matter, and CA Owners should submit an additional, separate incident report when...In the process of researching one incident, another incident with a distinct root cause and/or remediation is discovered."
When you delayed revocation on the grounds that "Chinese law prohibited revocation" (as you stated you did in comment 5 and comment 63), you did so without complying with the requirements of TLS BR 9.16.3. I'm presuming you weren't aware of this noncompliance until I posted comment 73. Nevertheless, it is "another incident with a distinct root cause", and so DigiCert "should submit an additional, separate incident report".
Q1: Will DigiCert open a new incident bug to report this noncompliance with TLS BR 9.16.3?
Missing a deadline for a requirement doesn't mean that the CA is absolved of its obligation to follow that requirement. Therefore:
Q2: Will DigiCert publish a delayed update to section 9.16.3 of its CPS and contact the CA/Browser Forum as per the requirements outlined in the first two paragraphs of TLS BR 9.16.3, to retrospectively document the revocations that were delayed in this incident because, in DigiCert's view at the time, "Chinese law prohibited revocation"?
Finally, noting that DigiCert now takes the view that Chinese law does not prohibit revocation:
Q3: Within 90 days, will DigiCert publish a second CPS update and contact the CA/Browser Forum again as per the requirements outlined in the final paragraph of TLS BR 9.16.3 (quoted below)?
"Any modification to CA practice enabled under this section MUST be discontinued if and when the Law no longer applies, or these Requirements are modified to make it possible to comply with both them and the Law simultaneously. An appropriate change in practice, modification to the CA's CPS and a notice to the CA/Browser Forum, as outlined above, MUST be made within 90 days."
|
|
Assignee | |
Comment 80•1 year ago
|
||
In response to Wayne from comment 78
Wayne - our answer was not geographic specific. We were answering both the facts in this case and your hypothetical region. Our comment 'Should a case arise where local law does in fact hinder or prohibit revocation, we will certainly evaluate that situation and take appropriate action, which could include cessation of issuance to that jurisdiction.' is not specific to China.
|
|
Assignee | |
Comment 81•1 year ago
|
||
In response to Rob Stradling from comment 79
The delayed revocations covered by this incident bug occurred "in the past", around 9 months ago. In comment 5 you reported that 1.07% of those delayed revocations were due to "Legal restriction in Chinese law requires sign-off from government", and in your Incident Report Closure Summary in comment 63 you stated that "systems where Chinese law prohibited revocation" were amongst the "exceptional circumstances" for which you allowed delayed revocation to occur for this incident.
If those statements in comment 5 and comment 63 are accurate, then your claim in comment 74 that "DigiCert has not delayed revocation on the basis of law in the past" must be untrue.
We are surprised and disappointed by the implication that we have not been truthful in the course of answering questions on this bug or in any other correspondence with the community. As we stated very clearly in our response to eric in comment 77, and we will reiterate here for you, we did not seek nor obtain permission from the Chinese government prior to revocation. There was an allegation by a subscriber that Chinese law prevented us from revoking. At the time that went into the same bucket as all other reports by subscribers of “exceptional circumstances” and was reported on this bug for that reason. All of the subscribers in China where revocation was delayed were also financial institutions, which meant they fell under the categories of critical infrastructure DigiCert previously used.
Q1: Will DigiCert open a new incident bug to report this noncompliance with TLS BR 9.16.3?
No. The remedies outlined in Section 9.16.3 are required prior to issuance of another certificate in a jurisdiction affected by such law. A review was conducted in connection with the delayed revocation in this instance, but our ultimate determination was that Chinese law did not prevent revocation, making Section 9.16.3 irrelevant in this case. So, there was no violation of BR Section 9.16.3.
Q2: Will DigiCert publish a delayed update to section 9.16.3 of its CPS and contact the CA/Browser Forum as per the requirements outlined in the first two paragraphs of TLS BR 9.16.3, to retrospectively document the revocations that were delayed in this incident because, in DigiCert's view at the time, "Chinese law prohibited revocation"?
No, as it is not applicable.
Q3: Within 90 days, will DigiCert publish a second CPS update and contact the CA/Browser Forum again as per the requirements outlined in the final paragraph of TLS BR 9.16.3 (quoted below)?
No, as the requirements are not applicable and CPS modifications are not needed to comply with the current BRs as written.
|
|
||
Comment 82•1 year ago
|
||
(In reply to DigiCert from comment #71)
I continue not to understand DigiCert’s resistance to giving straight answers to simple factual questions.
Q1: Our CPS states that we follow the BRs.
You didn’t provide a straight answer to question 1. Please provide a straight answer starting with yes or no.
Question 4: Between May 6 and May 11, 2024 did DigiCert’s CPS state that you followed the BRs?
Question 5: If yes to Question 4, what is the meaningful change in policy and expected behavior you intend to communicate with your response to question 1? Please be specific about how the community can take this statement as a meaningful remediation measure for the repeated choice to delay mandatory revocation in discussion on this bug and bug 1910805.
Q2: I think it is clear from previous answers that Sectigo and DigiCert disagree on this point. I'm not sure what purpose asking again serves.
On multiple occasions in the past year as I have attempted to clarify partial or vague responses from DigiCert on substantive points, you have made public statements suggesting that somehow this effort is false or deceptive, statements such as “how you personally view, interpret, and mis-restate our various responses,” “Again, please do not put words in my mouth. I do not appreciate this,” and “At this point, I think you are just making stuff up.” While I am able to go back to each of the online statements that received these responses to demonstrate how they are neither misstated nor made up, the purpose of this question and those like it was and is to mitigate this sort of accusation in the future. If DigiCert continues its habit of failing to provide concrete and clear answers to straightforward questions of import, then the community members following this conversation will have no choice but to “interpret” DigiCert’s responses. You have stated that you dislike such interpretation. This question and others like it give you the opportunity to prevent that, which on this occasion you have declined.
(And yes, I do recall based on comment 53 that you have stated you don’t want these “opportunities.” If it helps, don’t think of them as opportunities for DigiCert but rather for the rest of the community in its attempts to unpack DigiCert’s actions, policies, and practices.)
Based on the above response I will offer to those community members that DigiCert appears to disagree with my contention that Mozilla policy from at least early 2024 did not leave room for willing delay of revocation at that time. Even though the evidence for this point has been heaped on many bugs in post after post for the past year, we will probably have to do it again on this thread. But that will be another post at another time. For now, we know DigiCert’s position and will use that moving forward. Similarly to what I stated in bug 1910322 comment 73, if DigiCert in the future would like to offer a direct statement of its position on this matter, I’m sure we all will welcome it.
Q3: This one is actually subtle, as one root program has recently proposed that its root program requirements do in fact supersede the Baseline Requirements. We asked for clarification in our response, as it would be a disaster if all root programs started asserting this.
Since the obligation to follow the BRs actually flows down from root program requirements, it is in fact true that any root program can state requirements that conflict with the BRs. This actually causes a significant amount of CA pain and confusion when it happens, even accidentally, which is why we often respectfully ask the root programs not to override the BRs, even though they are capable.
Once again, you didn’t say yes or no, but let’s see what we can tease out of these two paragraphs. I will remind the reader that the original question 3 in comment 67 was,
Question 3: We contend that a single root program cannot unilaterally remove requirements by other root programs or the BRs through its own policy declarations. Yes or no, does DigiCert agree?
Note that your response from comment 71 does not address if you feel a single root program can unilaterally remove requirements by other root programs. I would appreciate an answer to that question.
You do, however, appear to be contending that a single root program can unilaterally remove a BR requirement without the cooperation of other root programs. This is a strange assertion to make. The BR requirements are captured in whichever of the several documents is relevant to the case we’re talking about, which in this case is the TLSBRs. These are the baseline requirements. Each of the major root programs has a requirement in its own program that CAs must follow the BRs. Even if a single root program were to waive that requirement in part or in whole, that would have no bearing on the requirements the other root programs would continue to carry.
This is a very easy concept to understand. If multiple authoritative sources place requirements on a single object, that object’s requirements are the set of all requirements from at least one of these sources. Think of it as a Boolean Or, not a Boolean And. To claim that the requirements are the set of strictures consistently leveed by all authoritative sources is absurd and indefensible.
It’s easy to come up with thought experiments to demonstrate the indefensibility of this claim. Let’s suppose that one root program were to declare in its program requirements that DCV was no longer mandatory for TLS certificate issuance. Does any of us believe the other root programs would calmly accept it if CAs began issuing TLS certificates in the absence of DCV?
Question 6: Hypothetically, if one of the major root programs were to publish in its guidelines that DCV was no longer required, would DigiCert interpret that exception as applicable to the TLS certificates it issues on its generally available public roots?
There is even a historical example of all four root programs simultaneously declaring a troublesome BR requirement inoperative. We're a bit surprised Sectigo is asserting a position that is at odds with reality and history.
This is not unilateral action, which should be readily apparent. Whatever conclusions we may draw from this coordinated browser action, they are apart from the matter we are discussing here.
|
|
Assignee | |
Comment 83•1 year ago
|
||
We answered question 1. We are not going to reword our answer to your loaded question just because you don't like our answer.
In response to questions 4 and 5, we’ve stated our previous interpretation. The community has indicated that, whether or not one agrees with that interpretation, and whether or not it was ever valid, it is not valid today, and as we have always tried to do, we will abide by the WebPKI community’s standards.
Our statement that, “one root program has recently proposed that its root program requirements do in fact supersede the Baseline Requirements,” is a clear statement of fact and it is not in dispute. It is, however, along with the statement of yours to which it was a response, not particularly germane to this discussion so we see no reason to debate the issue further on this bug.
|
|
Assignee | |
Comment 84•1 year ago
|
||
There has been some lively discussion since we posted our Incident Closure Summary in comment 63 to which we have provided answers to the best of our ability. As none of this ensuing discussion has resulted in the need for modifications to that Summary, nor to any additional action items needed from DigiCert, we again request closure of this bug.
|
|
||
Updated•1 year ago
|
|
|
||
Comment 85•1 year ago
|
||
(In reply to DigiCert from comment 83)
We answered question 1. We are not going to reword our answer to your loaded question just because you don't like our answer.
As a reminder, question 1 from comment 67 is,
Question 1: Yes or no, is DigiCert committing to never again knowingly delay BR-mandated certificate revocation?
Your response in comment 71 was not an answer to the question. You said,
Q1: Our CPS states that we follow the BRs.
This statement about your CPS is not an answer to the question that was asked. This is obvious. Therefore, your statement that you answered question 1 is false. The community still requires an answer to question 1.
Question 7: Considering that these are clearly different topics, why do you contend that you have answered the question? Please explain your position.
I also don’t see how this question is “loaded.” It is a simple factual question that is easily answered.
Question 8: Please explain specifically how Question 1 is “loaded.”
You did not fully answer question 3. Comment 82 specifies,
Note that your response from comment 71 does not address if you feel a single root program can unilaterally remove requirements by other root programs. I would appreciate an answer to that question.
I still would appreciate an answer to this question.
You did not answer question 6. In bug 1910322 comment 73, I suggested that DigiCert should check its own bugs for unanswered questions before asking to close them.
Question 9: When you asked to close this bug in comment 84, were you aware that question 3 was not fully answered? Were you aware of the follow up in comment 82?
Question 10: When you asked to close this bug in comment 84, were you aware that question 6 had gone entirely unanswered?
|
|
||
Comment 86•1 year ago
|
||
(In reply to DigiCert from comment #81)
We are surprised and disappointed by the implication that we have not been truthful in the course of answering questions on this bug or in any other correspondence with the community.
I am taking great care to not allege or imply anything that cannot be defended simply by applying cold, hard logic to DigiCert's previous statements.
As we stated very clearly in our response to eric in comment 77, and we will reiterate here for you, we did not seek nor obtain permission from the Chinese government prior to revocation.
I believe you, but your statement here is beside the point.
Regardless of whether or not DigiCert sought or obtained "permission from the Chinese government prior to revocation", DigiCert did delay revocation for a number of certificates beyond the BR mandated deadline due to a belief at that time that a "Legal restriction in Chinese law requires sign-off from government" because the certificates were being used on "systems where Chinese law prohibited revocation". I am not making this up. These are DigiCert's own words.
There was an allegation by a subscriber that Chinese law prevented us from revoking. At the time that went into the same bucket as all other reports by subscribers of “exceptional circumstances” and was reported on this bug for that reason. All of the subscribers in China where revocation was delayed were also financial institutions, which meant they fell under the categories of critical infrastructure DigiCert previously used.
May I remind you, again, that the concept of "exceptional circumstances" only appeared in the previous version of Mozilla's guidance for responding to delayed revocation incidents. The TLS BRs and root program policies did not and do not permit any form of non-compliance due to "exceptional circumstances". I understand you are stating that you believed at the time in the "exceptional circumstances" justification for delayed revocation, but I'm also aware that in bug 1910805 comment 73 you have subsequently acknowledged that,
there is no such thing as exceptional circumstances.
Q4: How does this realisation relate to your contentions on this bug? Do you want to revise your position?
Q1: Will DigiCert open a new incident bug to report this noncompliance with TLS BR 9.16.3?
No. The remedies outlined in Section 9.16.3 are required prior to issuance of another certificate in a jurisdiction affected by such law.
It is true that the "remedies outlined in Section 9.16.3 are required prior to issuance of another certificate in a jurisdiction affected by such law", but again, this is beside the point.
The first paragraph of TLS BR 9.16.3 says (emphasis mine):
"In the event of a conflict between these Requirements and a law, regulation or government order (hereinafter 'Law') of any jurisdiction in which a CA operates or issues certificates, a CA MAY modify any conflicting requirement to the minimum extent necessary to make the requirement valid and legal in the jurisdiction. This applies only to operations or certificate issuances that are subject to that Law. In such event, the CA SHALL immediately (and prior to issuing a certificate under the modified requirement) include in Section 9.16.3 of the CA's CPS a detailed reference to the Law requiring a modification of these Requirements under this section, and the specific modification to these Requirements implemented by the CA."
Clearly the parenthetical remark "(and prior to issuing a certificate under the modified requirement)" does not apply to revocation. However, "the CA SHALL immediately" does apply to revocation, because revocation is a function of a CA's "operations" under "these Requirements". This means that when you believed that "Chinese law prohibited revocation" and as a result chose to "modify a[ny] conflicting requirement" by delaying revocation of a number of certificates, you had an obligation "immediately" to modify your CPS.
A review was conducted in connection with the delayed revocation in this instance, but our ultimate determination was that Chinese law did not prevent revocation, making Section 9.16.3 irrelevant in this case. So, there was no violation of BR Section 9.16.3.
That review was still ongoing at the time of comment 72 (2025-02-04), and you announced the outcome of the review in comment 77 (2025-02-13). But yet again, this is all beside the point.
We are talking about actions that DigiCert took, and actions that DigiCert did not take, 9 months earlier in May 2024 during this delayed revocation incident. At that time, in DigiCert's own words from comment 5 and comment 63, DigiCert delayed revocation beyond the TLS BR mandated deadline for 1.07% of 10,926 certificates on the grounds that "Legal restriction in Chinese law requires sign-off from government" because those certificates were used on "systems where Chinese law prohibited revocation". Despite the clear applicability of TLS BR 9.16.3 to this situation (as I laid out in comment 73), DigiCert did not perform the actions required by TLS BR 9.16.3.
That, right there, is an incident.
The outcome of DigiCert's review 9 months later does not retrospectively cause this to cease to have been an incident. DigiCert's apparent unawareness of the applicable requirements in TLS BR 9.16.3 prior to comment 73 (posted 2025-02-07) does not mean that it wasn't an incident. The appeal to "exceptional circumstances" does not mean that it wasn't an incident.
DigiCert: Having read and digested this comment, please re-read comment 79 and reconsider your answers to my questions Q1, Q2, and Q3.
|
|
Assignee | |
Comment 87•1 year ago
|
||
In response to Tim Callan in comment 85:
Question 1: Yes or no, is DigiCert committing to never again knowingly delay BR-mandated certificate revocation?
As per our revised Action Items on bug 1910805, which applies here as well:
“Clarify messaging to customers that DigiCert will not delay revocation beyond the timeframes required by the Baseline Requirements and that there is no such thing as exceptional circumstances.”
As questions 7 and 8 are primarily based upon the semantics of our previous answer to question 1, and we hope now that we have given you a clear, definitive answer to question 1, we don’t believe that Sectigo, Digicert or the community as a whole will derive benefit from further debate about semantics. As these questions do not directly pertain to the topic of this bug we will politely decline to answer.
Note that your response from comment 71 does not address if you feel a single root program can unilaterally remove requirements by other root programs. I would appreciate an answer to that question.
In the draft version of the Chrome Root Program Policy version 1.6, section 1:
"In some cases, this policy strengthens requirements described in the Baseline Requirements. In all cases of conflict, this policy MUST take precedence."
This sentence was removed from the final version but was present at the time of our response. This sentence made it clear that browser policy trumps all other policies.
Additionally, the Mozilla Root Store Policy, version 3.0, section 2.3 currently states:
"In the event of inconsistency between this policy's requirements and either the S/MIME BRs or TLS BRs, this policy's requirements take precedence."
Again, very clear from Mozilla that their policy overrides the CAB/Forum BR policy.
When you asked to close this bug in comment 84, were you aware that question 3 was not fully answered? Were you aware of the follow up in comment 82?
Question 3 was already answered. We made a clear statement regarding trust store operators and their policies. We’ve clearly shown, using direct quotations from their policies, the factual nature of that statement.
When you asked to close this bug in comment 84, were you aware that question 6 had gone entirely unanswered?
We do not engage in hypotheticals, and you did not ask a question related to the current bug.
In response to Rob Stradling in comment 86:
You’ve emphasized a lot of phrases but missed emphasizing the most important statement: “In the event of a conflict between these Requirements and a law…” You assert that our review of the law and it’s outcome is “beside the point.” However, it is impossible to determine whether or not a conflict exists without a review, and our review concluded there isn’t a conflict between the Requirements and the law. Because there isn’t a conflict, there is no action required. Because no conflict exists, the rest of the requirements are moot.
|
|
||
Comment 88•1 year ago
|
||
(In reply to DigiCert from comment #87)
Note that your response from comment 71 does not address if you feel a single root program can unilaterally remove requirements by other root programs. I would appreciate an answer to that question.
In the draft version of the Chrome Root Program Policy version 1.6, section 1:
"In some cases, this policy strengthens requirements described in the Baseline Requirements. In all cases of conflict, this policy MUST take precedence."
This sentence was removed from the final version but was present at the time of our response. This sentence made it clear that browser policy trumps all other policies.
Additionally, the Mozilla Root Store Policy, version 3.0, section 2.3 currently states:
"In the event of inconsistency between this policy's requirements and either the S/MIME BRs or TLS BRs, this policy's requirements take precedence."
Again, very clear from Mozilla that their policy overrides the CAB/Forum BR policy.
The question asked was if DigiCert believed that a single root program could unilaterally remove requirements. The Chrome draft cited clearly states that it strengthens the BRs, strengthening is the opposite of weakening which the removal of requirements would be.
The Mozilla policy quoted is not as clear, but we’ve had incidents in the past year where a CA has had a CP/CPS that was inconsistent with the BRs in that it was stricter, and included the same type of disclaimer only in reverse. That the BRs took precedence in the case of any inconsistency. But the community conclusion was that they had to follow their own stricter rules even if the BRs were more lenient. I feel confident in stating that interpreting Mozilla’s policy to allow for the unilateral weakening or removal of any requirement is absurd.
I would welcome if either Chrome or Mozilla would care to correct me if I’m wrong.
DigiCert, since the question was explicitly about the removal of requirements, do you stand by your answer?
|
|
||
Comment 89•1 year ago
|
||
(In reply to DigiCert from comment #87)
As per our revised Action Items on bug 1910805, which applies here as well:
“Clarify messaging to customers that DigiCert will not delay revocation beyond the timeframes required by the Baseline Requirements and that there is no such thing as exceptional circumstances.”
(The emphasis above is mine.) After months of avoiding the question, DigiCert has quietly committed not to deliberately delay revocation. I’m happy to see DigiCert eventually arrive at the right end state. Note that this is not just a commitment against using the “exceptional circumstances” excuse. It is a commitment not to delay revocation at all.
Congratulations to DigiCert finally for making this public commitment. The WebPKI community will hold you to this promise.
|
|
Assignee | |
Comment 90•1 year ago
|
||
In response to comment 88
DigiCert, since the question was explicitly about the removal of requirements, do you stand by your answer?
Yes. The trust store operators have stated plainly that the Baseline Requirements are not binding on them and that they can take any action in their own policies that they see fit.
|
|
||
Comment 91•1 year ago
|
||
(In reply to DigiCert from comment #90)
In response to comment 88
DigiCert, since the question was explicitly about the removal of requirements, do you stand by your answer?
Yes. The trust store operators have stated plainly that the Baseline Requirements are not binding on them and that they can take any action in their own policies that they see fit.
I agree, the root programs aren't bound by either the BRs or the requirements of other root programs. But again, that was not the question that was asked. Let me remind you of the context of the original question. It was about that; if Mozilla's previous policy allowed for delayed revocation (which we all agree that it did not) then would delayed revocation be allowed for a CA? In conflict with the fact that delayed revocation is (and was) not allowed by the BRs or any (other) root program policy.
Your own most recent public trust CP/CPS states "This CP/CPS governs certificates issued under the following policies, guidelines and standards" where the Mozilla Root Store Policy is only one of many. If one of the the governing documents lacks a requirement that is present in any of the others, then that does not make that requirement invalid. Yes, a trust store operator can take any action they fit regarding their own policies, but a CA needs to comply with the union of all applicable requirements.
Let me try to illustrate this with a simple example.
- My CA has a CP/CPS governing certificates issued according to Policy Pi and Guideline Gamma .
- Policy Pi has the following requirements:
- Certificate lifetime is limited to 420 days
- Delayed revocation is fine, actually
- Guideline Gamma has the following requirements:
- Certificate lifetime is limited to 42 days
- Compromised or mississued certificates MUST be revoked within 48 hours
Then the requirements of Policy Pi are in effect redundant since they are all weaker than those of Guideline Gamma. If there were conflicts (like Policy Pi requiring a minimum lifetime of 100 days) that is a paradox, which would make it impossible to issue valid certificates according to the CP/CPS.
So, again. Could you please answer the question?
Tim Callan in comment 67
Question 3: We contend that a single root program cannot unilaterally remove requirements by other root programs or the BRs through its own policy declarations. Yes or no, does DigiCert agree?
|
|
Assignee | |
Comment 92•1 year ago
|
||
In response to comment 91:
If the trust store operators are not bound by the BR, and can do as they see fit with their own policies, then it's very easy to imagine a scenario wherein two different trust store operators might enact policies in conflict with one another, thereby requiring a CA to violate one of the other of those policies. We've now answered this line of questioning 10 times in 10 different ways, and as it is entirely dealing in hypotheticals we will not engage in this line of reasoning further. It is irrelevant to the topic of this bug which is delayed revocation.
|
|
||
Comment 93•1 year ago
|
||
(In reply to DigiCert from comment #92)
In response to comment 91:
If the trust store operators are not bound by the BR, and can do as they see fit with their own policies, then it's very easy to imagine a scenario wherein two different trust store operators might enact policies in conflict with one another, thereby requiring a CA to violate one of the other of those policies. We've now answered this line of questioning 10 times in 10 different ways, and as it is entirely dealing in hypotheticals we will not engage in this line of reasoning further. It is irrelevant to the topic of this bug which is delayed revocation.
This is very exhausting, you keep answering that root programs are free to introduce conflicting policies. But that is not relevant to the question. The question, which IS very relevant to the topic of this bug, stems from that: even if Mozilla’s previous policy allowed delayed revocation in ”exceptional circumstances” (removing the requirement of timely revocation) then the requirement of timely revocation from the BRs and other root programs remain. Ergo: ”We contend that a single root program cannot unilaterally remove requirements by other root programs or the BRs through its own policy declarations”
It is important for the community to know whether DigiCert agrees with this or not. If I were a root program, then I would look very suspiciously on a CA that claims that other root programs could hand out ”get out of requirements”-cards
Since you keep answering some other question, let me address your answer. Which I also find worrying.
If you have a CP/CPS that includes requirements from two policies and those two policies have conflicts, introducing a paradox then the solution is not to violate one or the other.
If it is impossible to comply with your CP/CPS then you stop issuing. And if that impossibility comes from trying to include requirements from two conflicting policies then instead you need to create two separate CP/CPSs each being governed by one of the policies.
This obviously wouldn’t be a good thing, fracturing webPKI. But I trust that the root programs will be good stewards of public trust and continue to cooperate.
I don’t think this is that hypothetical, it is also very easy to imagine a root program deciding to introduce some new agreed upon requirement earlier than the other root programs. I would be surprised if there isn’t a historical precedent this, but the closest thing I can think of right now is how the root programs have rolled out enforcing certificate transparency on different timelines for their respective browsers.
But we aren’t talking about conflicting requirements, we are talking about one root program policy being able to REMOVE requirements. Removing a requirement will not introduce a conflict. This should be obvious. If DigiCert does not agree that one root program CANNOT unilaterally remove requirements of the BRs or other root programs. And DigiCert continued to avoid stating that they agree with Tim Callans original contention. Then the conclusion is that DigiCerts CP/CPS is impossible to actually verify and understand. And that they only follow a the intersecting subset of “applicable requirements” instead of all of them.
|
|
||
Comment 94•1 year ago
|
||
(In response to comment 87)
As questions 7 and 8 are primarily based upon the semantics of our previous answer to question 1, and we hope now that we have given you a clear, definitive answer to question 1, we don’t believe that Sectigo, Digicert or the community as a whole will derive benefit from further debate about semantics. As these questions do not directly pertain to the topic of this bug we will politely decline to answer.
Question 7 pointed out that DigiCert’s repeated statements about its CPS and its espousal of dedication to the BRs was independent of the question of whether or not DigiCert would commit not to deliberately delay revocation again. As we see from comment 89, DigiCert eventually slipped that commitment into a statement about customer communication, so we don’t need to discuss that commitment any longer.
However, question 7 is not about the actual potential commitment. Question 1 was dedicated to that. Question 7 is about why DigiCert claimed in comment 83 that it had answered the question when simply reading the question and the reply is enough to establish that it has not.
This question remains relevant. The Bugzilla process depends on CAs presenting information accurately and in good faith. DigiCert made a claim that on its face is false, where the falseness is easily determined by even a casual observer. I would like to understand how that occurred. I would hope DigiCert would want, also, to get to the bottom of such occurrences and prevent them from happening in the future.
Question 8 was,
Question 8: Please explain specifically how Question 1 is “loaded.”
This question, too, remains relevant. In the apparent attempt to discredit both the question and the questioner, DigiCert wrote,
We answered question 1. We are not going to reword our answer to your loaded question just because you don't like our answer.
After another review of question 1, I continue to see no way in which it is loaded. I am careful to be fair and factual in my Bugzilla posts, so I would like DigiCert to explain in what way it feels this question is problematic. That the subject matter of comment 1 is now resolved per comment 89 is independent of the fact that DigiCert made this accusation in service to its attempt not to answer.
So please, I must insist, explain what you meant when you described question 1 as loaded.
|
|
Assignee | |
Comment 95•1 year ago
|
||
This is very exhausting, you keep answering that root programs are free to introduce conflicting policies. But that is not relevant to the question. The question, which IS very relevant to the topic of this bug, stems from that: even if Mozilla’s previous policy allowed delayed revocation in ”exceptional circumstances” (removing the requirement of timely revocation) then the requirement of timely revocation from the BRs and other root programs remain. Ergo: “ We contend that a single root program cannot unilaterally remove requirements by other root programs or the BRs through its own policy declarations”
This is not a question. We are answering questions relevant to our compliance with the current Mozilla and Chrome root programs. Questions about compliance with other root store programs are addressed as those programs see fit. We do not believe pitting one root program against another is a productive discussion
It is important for the community to know whether DigiCert agrees with this or not. If I were a root program, then I would look very suspiciously on a CA that claims that other root programs could hand out” get out of requirements”-cards
Root programs are free to set their own policies. Answering hypotheticals is not productive to an RCA.
If you have a CP/CPS that includes requirements from two policies and those two policies have conflicts, introducing a paradox then the solution is not to violate one or the other.
We are not aware of an actual conflict or paradox in root program requirements. DigiCert’s CP/CPS is compliant with the Root Store policies utilizing the DigiCert roots.
If it is impossible to comply with your CP/CPS then you stop issuing. And if that impossibility comes from trying to include requirements from two conflicting policies then instead you need to create two separate CP/CPSs each being governed by one of the policies.
This is not currently the situation on this Bugzilla report.
I don’t think this is that hypothetical, it is also very easy to imagine a root program deciding to introduce some new agreed upon requirement earlier than the other root programs. I would be surprised if there isn’t a historical precedent this, but the closest thing I can think of right now is how the root programs have rolled out enforcing certificate transparency on different timelines for their respective browsers.
This is a hypothetical argument. There are not conflicts in the Root program policies impacting this Bugzilla report.
But we aren’t talking about conflicting requirements, we are talking about one root program policy being able to REMOVE requirements. Removing a requirement will not introduce a conflict. This should be obvious. If DigiCert does not agree that one root program CANNOT unilaterally remove requirements of the BRs or other root programs. And DigiCert continued to avoid stating that they agree with Tim Callans original contention. Then the conclusion is that DigiCert’s CP/CPS is impossible to actually verify and understand. And that they only follow the intersecting subset of “applicable requirements” instead of all of them.
This is a hypothetical argument. This hypothetical is not relevant to delayed revocation, which is the topic of this Bugzilla.
However, question 7 is not about the actual potential commitment. Question 1 was dedicated to that. Question 7 is about why DigiCert claimed in comment 83 that it had answered the question when simply reading the question and the reply is enough to establish that it has not.
This question remains relevant. The Bugzilla process depends on CAs presenting information accurately and in good faith. DigiCert made a claim that on its face is false, where the falseness is easily determined by even a casual observer. I would like to understand how that occurred. I would hope DigiCert would want, also, to get to the bottom of such occurrences and prevent them from happening in the future.
We do not believe this discussion will be relevant to the topic of delayed revocation. Mozilla can moderate the Bugzilla discussion and information posted herein as it sees fit.
Question 8: Please explain specifically how Question 1 is “loaded.”
This question, too, remains relevant. In the apparent attempt to discredit both the question and the questioner, DigiCert wrote,
We answered question 1. We are not going to reword our answer to your loaded question just because you don't like our answer.
After another review of question 1, I continue to see no way in which it is loaded. I am careful to be fair and factual in my Bugzilla posts, so I would like DigiCert to explain in what way it feels this question is problematic. That the subject matter of comment 1 is now resolved per comment 89 is independent of the fact that DigiCert made this accusation in service to its attempt not to answer.
So please, I must insist, explain what you meant when you described question 1 as loaded.
We do not believe this question is relevant to the discussion on delayed revocation. At this point, we believe all questions relevant to the delayed revocation were answered. If you’d like to start a different thread about the accuracy of information, then we suggest that be a forum post instead of part of this incident report.
|
|
||
Comment 96•1 year ago
|
||
DigiCert, I must please ask you to treat both me and Mr. Callan with the respect we deserve. Accusations of ”loaded” questions is not constructive, and refusing to clarify your objections regarding a certain question you deem ”loaded” is neither constructive or transparent. Please (re-)familiarize yourselves with both the Mozilla and CCADB Code of Conduct.
The CCADB Incident Reporting Guidelines invites everyone to participate and among other things ask clarifying questions. The guidelines states that a CA MUST answer questions.
It asks all of us to keep our comments relevant and constructive, to follow the CoC. I believe the question to be relevant, I believe I’m being polite, since DigiCert keeps misunderstanding I try to explain my reasoning.
Two months ago Tim Callan asked a very simple question. You have still not answered it. I have repeatedly tried remind you to answer the question but have only been met with evasive non-answers, or at best answers to other questions. I want an answer to this question. I believe it to be relevant, that is why I keep asking it. If you believe that my comments are violating any guidelines, code of conduct, etiquette, or similar rules you are free to tag it for review by the administrators or report other relevant stakeholders. This persistent stonewalling is insulting, I should not have to go to these extreme lengths to try and get an answer to a very simple yes-or-no question.
In the spirit of trying to keep things constructive, here is why I believe the question to be relevant:
In this incident DigiCert (earlier comments in this bug were made by Jeremy Rowley representing DigiCert, since they are now using a common account it feels correct to treat all of his comments as DigiCert’s) has stated:
Mozilla has a public policy that allows for delayed revocation in certain circumstances.
Even if that was true it would not ”allow delayed revocation” since a CA is not only bound to follow Mozilla’s policy, but also by other requirements. Applicable requirements in the language of the DigiCert’s own CP/CPS. And those other requirements (such as the TLS BRs or other root programs) do not allow for delayed revocation.
DigiCert has since understood that they were wrong in their interpretation of Mozilla’s policy. That is good. But if DigiCert makes a similar mistake again, would they let a belief that if one policy allows something then that overrides other policies forbidding it? We don’t know, because DigiCert refuses to answer the question.
Since a key purpose of bugzilla incidents is for the owning CA to demonstrate why and how something won’t happen again this bug should not be closed until DigiCert has provided an convincing answer. This is also why it’s not productive to dismiss questions as hypothetical, all future incidents are hypothetical. And to prevent them we need to address hypotheticals. DigiCert, please understand that neither Mr. Callan or I am asking you how you would handle conflicting requirements by root programs, this would an existential crisis for the webPKI and not something I expect you to have an answer to. This is about one policy saying: ”absolutely no X” and another saying: ”X is fine, actually”. It is possible to comply with: ”No delayed revocation” and ”Delayed revocation is fine in these circumstances” by not delaying revocation.
Finally I want to remind DigiCert that another purpose of bugs is to build and maintain trust. Please take this opportunity!
|
|
Assignee | |
Comment 97•1 year ago
|
||
In response to comment 96
We don’t feel that we have treated either you or Mr. Callan disrespectfully, however if we have inadvertently done so, you have our sincere apologies.
Two months ago Tim Callan asked a very simple question.
Yes, that question was, from comment 67;
Question 3: We contend that a single root program cannot unilaterally remove requirements by other root programs or the BRs through its own policy declarations. Yes or no, does DigiCert agree?
Our answer at that time should have been that the question requires delving into all sorts of hypotheses which are unrelated to the topic of this bug which is delayed revocation, and specifically DigiCert’s decision to delay revocation in this specific instance.
You bring up a statement posted by Jeremy Rowley early on in this bug:
Mozilla has a public policy that allows for delayed revocation in certain circumstances.
We believe that we’ve already addressed this issue, but we also believe that addressing it clearly is what really gets to the heart of Mr. Callan’s and your questions, without the need for hypotheticals. In that spirit, in case we have not been clear enough, Mr. Rowley and DigiCert’s reasoning at that time was in error. We have now stated plainly that there are no “exceptional circumstances” and that we will not delay revocation in the future.
DigiCert has since understood that they were wrong in their interpretation of Mozilla’s policy. That is good. But if DigiCert makes a similar mistake again, would they let a belief that if one policy allows something then that overrides other policies forbidding it? We don’t know, because DigiCert refuses to answer the question.
Since a key purpose of bugzilla incidents is for the owning CA to demonstrate why and how something won’t happen again this bug should not be closed until DigiCert has provided an convincing answer.
Without delving into any of the hypotheticals necessitated by the question about trust store policies, it is clear now that CAs MUST adhere to the letter of the law in all instances and rigorously adhere to written policy regardless of circumstances and heedless of any other perceived potential harm.
|
|
||
Comment 98•1 year ago
|
||
(In reply to DigiCert from comment #97)
In response to comment 96
We don’t feel that we have treated either you or Mr. Callan disrespectfully, however if we have inadvertently done so, you have our sincere apologies.
Thank you DigiCert
Two months ago Tim Callan asked a very simple question.
Yes, that question was, from comment 67;
Question 3: We contend that a single root program cannot unilaterally remove requirements by other root programs or the BRs through its own policy declarations. Yes or no, does DigiCert agree?
Our answer at that time should have been that the question requires delving into all sorts of hypotheses which are unrelated to the topic of this bug which is delayed revocation, and specifically DigiCert’s decision to delay revocation in this specific instance.
You bring up a statement posted by Jeremy Rowley early on in this bug:
Mozilla has a public policy that allows for delayed revocation in certain circumstances.
We believe that we’ve already addressed this issue, but we also believe that addressing it clearly is what really gets to the heart of Mr. Callan’s and your questions, without the need for hypotheticals. In that spirit, in case we have not been clear enough, Mr. Rowley and DigiCert’s reasoning at that time was in error. We have now stated plainly that there are no “exceptional circumstances” and that we will not delay revocation in the future.
DigiCert has since understood that they were wrong in their interpretation of Mozilla’s policy. That is good. But if DigiCert makes a similar mistake again, would they let a belief that if one policy allows something then that overrides other policies forbidding it? We don’t know, because DigiCert refuses to answer the question.
Since a key purpose of bugzilla incidents is for the owning CA to demonstrate why and how something won’t happen again this bug should not be closed until DigiCert has provided an convincing answer.Without delving into any of the hypotheticals necessitated by the question about trust store policies, it is clear now that CAs MUST adhere to the letter of the law in all instances and rigorously adhere to written policy regardless of circumstances and heedless of any other perceived potential harm.
And thank you for this clear answer.
For clarity I personally believe that there are circumstances where it is appropriate to break the rules to prevent harm. But what we have seen regarding delayed revocation has been CAs treating test environments as critical infrastructure, or the core problem was that the system was issued a webPKI certificate at all. Please don’t use webPKI certificates for things that cannot handle quick reissuance without risking significant harm! There has been a problem of CAs not taking responsibility, of saying ”we cannot revoke these certificates right now, because doing so would risk human lives” while their terms of use state: ”you cannot use certificate issued by us if revoking them would risk human lives”.
I believe this is what the previous language regarding ”exceptional circumstances” in Mozilla’s policy was about. I still believe that the reasoning in it was valid. But if a CA were to rely on it, an incident that would still need to be created wherein they must demonstrate what happened, why it happened, and why it won’t happen again.
|
||
Comment 99•1 year ago
|
||
For clarity I personally believe that there are circumstances where it is appropriate to break the rules to prevent harm. But what we have seen regarding delayed revocation has been CAs treating test environments as critical infrastructure, or the core problem was that the system was issued a webPKI certificate at all.
What circumstances would those be? Life and death is certainly not because you shouldn't use public certs for life and death circumstances? I guess a natural disaster that wipes out the infrastructure required to execute the revocation? I agree with DigiCert - there are no such thing as exceptional circumstances... which is good as that language was removed from the Mozilla policy. Although the Mozilla policy isn't quite a bright-lined as I would hope (ie - no delays for any reason), removal of that language was a significant step forward.
Please don’t use webPKI certificates for things that cannot handle quick reissuance without risking significant harm!
Definitely agree.
There has been a problem of CAs not taking responsibility, of saying ”we cannot revoke these certificates right now, because doing so would risk human lives” while their terms of use state: ”you cannot use certificate issued by us if revoking them would risk human lives”.
For me, it was the confusion of what "exceptional circumstances" were. Re-reading this bug and all other delayed revocation bugs convinced me that there is no such thing. DigiCert arrived at the same conclusion, and I believe Tim Callan would agree.
I believe this is what the previous language regarding ”exceptional circumstances” in Mozilla’s policy was about. I still believe that the reasoning in it was valid.
Perhaps, but it was clear as mud, and I'm glad the language is gone.
But if a CA were to rely on it, an incident that would still need to be created wherein they must demonstrate what happened, why it happened, and why it won’t happen again.
I don't think this was ever in dispute and why this bug was originally filed. Figuring out a never again plan is pretty hard to do unless you simply say that reasons for delaying revocation do not exist.
|
|
||
Comment 100•1 year ago
|
||
(In reply to Jeremy from comment #99)
For clarity I personally believe that there are circumstances where it is appropriate to break the rules to prevent harm. But what we have seen regarding delayed revocation has been CAs treating test environments as critical infrastructure, or the core problem was that the system was issued a webPKI certificate at all.
What circumstances would those be? Life and death is certainly not because you shouldn't use public certs for life and death circumstances? I guess a natural disaster that wipes out the infrastructure required to execute the revocation? I agree with DigiCert - there are no such thing as exceptional circumstances... which is good as that language was removed from the Mozilla policy. Although the Mozilla policy isn't quite a bright-lined as I would hope (ie - no delays for any reason), removal of that language was a significant step forward.
This is likely a reference to Chunghwa Telecom's claims in this incident: https://bugzilla.mozilla.org/show_bug.cgi?id=1903066#c15
They have backtracked since but this should act as a strong reason why CAs should double-check subscriber delayed revocation reasons. Doubly so whether a subscriber should get any new certificates issued even during a mis-issuance revocation scenario. Those particular claims are about the only time I could see everyone agreeing to allow for exceptional circumstances, outside of anyone's policies actually allowing it, retroactively for what I hope are obvious reasons. In 'normal' incidents though, not something anyone should concern themselves over.
|
|
||
Comment 101•1 year ago
|
||
Jeremy, in my attempt to clarify my view on things I see that I wasn’t as clear as I hoped to be. My apologies, I was indeed referring circumstances such as those in the incident linked by Wayne.
I felt the to explain my view because I don’t think that CAs must:
adhere to written policy regardless of circumstances and heedless of any other perceived potential harm.
We both agree that webPKI certificates shouldn’t be used for life and death circumstances. But if they end up there, then I’m more interested in how we can prevent that from happening again instead of potentially jeopardizing human lives. And of course delayed revocation might end up jeopardizing those very lives that could be at stake.
What prompted me to start participating in this community is that I find it very much not to be heedless. Instead it is a very calm and reasoned oasis, for which I am thankful for the opportunity to join.
|
|
Assignee | |
Comment 102•1 year ago
|
||
Please note that as an action item on Bug 1957499, we are reviewing this Bug and all of our other open Bugs to ensure that we have fully answered each posted question. Our goal is to complete that review by the end of the week and to post any outstanding responses as quickly as possible thereafter.
Regarding the discussion that’s taken place here since our last post, we do not see any open questions for DigiCert.
However, we have heard the message clearly that use cases that are not fully compatible with the webPKI rules should seek non-browser PKI hierarchies. This is one reason why DigiCert has worked extensively with certain user communities to move to non-browser PKI hierarchies; the latest example being our work with ASC X9 to build a private hierarchy suited to the needs of the financial services industry.
|
|
Assignee | |
Comment 103•1 year ago
|
||
In reviewing this Bug for unanswered questions, we found five questions from Tim Callan that we either did not answer or where the answer was not clearly associated with the question.
Question 6: Hypothetically, if one of the major root programs were to publish in its guidelines that DCV was no longer required, would DigiCert interpret that exception as applicable to the TLS certificates it issues on its generally available public roots?
Answer: comment 83;
Our statement that, “one root program has recently proposed that its root program requirements do in fact supersede the Baseline Requirements,” is a clear statement of fact and it is not in dispute. It is, however, along with the statement of yours to which it was a response, not particularly germane to this discussion so we see no reason to debate the issue further on this bug.
This comment was the answer to Question 6. Apologies for not expressly stating that in the response.
As stated in comment 87 we are reluctant to engage in hypothetical discussions that are not relevant to this Bug . We believe that comment 97 gives an answer to this line of questions.
Question 1: Yes or no, is DigiCert committing to never again knowingly delay BR-mandated certificate revocation?
Your response in comment 71 was not an answer to the question. You said,
Q1: Our CPS states that we follow the BRs.
This statement about your CPS is not an answer to the question that was asked. This is obvious. Therefore, your statement that you answered question 1 is false. The community still requires an answer to question 1.
Question 7: Considering that these are clearly different topics, why do you contend that you have answered the question? Please explain your position.
Answer: We believe this question was clearly answered. The BRs state the timeframes required for revocation; we stated that we will follow the BRs. We provided a clearer answer in comment 87 which you acknowledged in comment 89.
Question 8: Please explain specifically how Question 1 is “loaded.”
Answer: We believe the framing of the question was loaded. The repeated insistence that our answer was not an answer, when we already answered the question, seems framed to unfairly cast us in a negative light.
Question 9: When you asked to close this bug in comment 84, were you aware that question 3 was not fully answered? Were you aware of the follow up in comment 82?
Answer: Question 3 was answered multiple times. We have given our definitive response to the questions dealing with trust store policies in comment 97
Question 10: When you asked to close this bug in comment 84, were you aware that question 6 had gone entirely unanswered?
Answer: Question 6 was answered (see above). We have given our definitive response to the trust store question in comment 97.
|
|
Assignee | |
Comment 104•1 year ago
|
||
We believe that we have addressed all questions and are preparing a revised Closure Summary, which we intend to post sometime next week. in the meantime, we will continue to monitor for any additional questions or comments from the community.
|
|
||
Comment 105•1 year ago
|
||
(In reply to DigiCert from comment #87)
In response to Rob Stradling in comment 86:
You’ve emphasized a lot of phrases but missed emphasizing the most important statement: “In the event of a conflict between these Requirements and a law…” You assert that our review of the law and it’s outcome is “beside the point.” However, it is impossible to determine whether or not a conflict exists without a review, and our review concluded there isn’t a conflict between the Requirements and the law. Because there isn’t a conflict, there is no action required. Because no conflict exists, the rest of the requirements are moot.
Mozilla's original guidance on "exceptional circumstances", which was the current version of that guidance when these delayed revocations occurred and when Jeremy quoted it in comment 0, outlined Mozilla's "position that your CA is ultimately responsible for deciding if the harm caused by following the requirements of the Baseline Requirements outweighs the risks that are passed on to individuals who rely on the web PKI by choosing not to meet this requirement." (emphasis mine)
Per DigiCert's own words in comment 63, "Exceptional circumstances, as defined by DigiCert, included critical infrastructure and systems where Chinese law prohibited revocation." (emphasis mine)
Therefore, in order to have defined those "Exceptional circumstances", DigiCert must have decided that the harm caused by timely revocation of Chinese subscriber certificates would outweigh the risks of delayed revocation that would be passed on to relying parties. And having reached that decision, DigiCert must have chosen to not meet the BR requirement for timely revocation of these subscriber certificates.
That sort of risk analysis and contemplation of deliberate noncompliance sounds very much like...a review! And this review reached the conclusion (again using DigiCert's own words from comment 63) that "Chinese law prohibited revocation".
Whether or not DigiCert's legal team was involved in this initial review is beside the point. A CA is not only responsible for decisions made by, or actions explicitly blessed by, its legal department. A CA is responsible for all of its decisions and actions.
As we know from comment 77, a later review that definitely did involve DigiCert's legal team arrived at a different conclusion. However, this does not alter the fact that an initial review must also have occurred.
I believe the rationale for requiring further action from DigiCert that I've laid out in comment 73, comment 79, comment 86, and this comment, are sound and can "be defended simply by applying cold, hard logic to DigiCert's previous statements" (quoting myself from comment 86). However, since DigiCert has stuck to its position so far, I am not expecting a different response this time. Anyone else following this bug can form their own opinion. I have nothing further to say on this particular matter.
|
|
Assignee | |
Comment 106•1 year ago
|
||
Thanks for your ongoing interest, Rob. We have answered all previous questions. As this Comment 105 has no further questions, we reiterate that DigiCert acknowledges our understanding of Mozilla’s then-guidance relating to “exceptional circumstances” in the case of revocations was not acceptable. This discussion has led to a clarification of Mozilla’s language and policy in this regard. DigiCert acknowledges that there are no circumstances where delayed revocation from the BR is acceptable.
|
|
Assignee | |
Comment 107•1 year ago
|
||
Report Closure Summary
Incident description: Certificates issued with an incorrect business category took longer than 5 days to revoke as some of these revocations were considered “exceptional circumstances”. The Bug included extensive discussion of the circumstances surrounding this revocation, while also leading to a clarification of Mozilla’s policy language relating to revocation, see Section 6.1.3. Specifically, this Bug and the corresponding policy update clarified that “exceptional circumstances” are not a reason for delayed revocation.
Incident Root Cause(s): This incident required the revocation and replacement of more than 10,000 certificates. Given the span of comments on this Bug, and the variety of topic addressed in the discussion, we focus on the root causes that some Subscribers could not replace their certificates within the required 5-day window and DigiCert believed that there was a valid process for delayed revocation based on the Mozilla webpage for responding to incidents:
“Mozilla recognizes that in some exceptional circumstances, revoking the affected certificates within the prescribed deadline may cause significant harm, such as when the certificate is used in critical infrastructure and cannot be safely replaced prior to the revocation deadline, or when the volume of revocations in a short period of time would result in a large cumulative impact to the web. However, Mozilla does not grant exceptions to the BR revocation requirements. It is our position that your CA is ultimately responsible for deciding if the harm caused by following the requirements of the Baseline Requirements outweighs the risks that are passed on to individuals who rely on the web PKI by choosing not to meet this requirement.”
As this Mozilla page was widely known, Subscribers also believed there was leeway to argue that their situation constituted the “exceptional circumstances” described by Mozilla, leading to a significant wave of requests to DigiCert for delayed revocation. This added to the confusion of the mass revocation event, including some of the side discussions that have arisen in this Bug.
Mozilla has since updated that webpage and its policy to clarify that:
Mozilla’s goal is to ensure that revocation occurs as swiftly as possible while maintaining the overall security and stability of the web. Mozilla does not grant exceptions to the revocation requirements of the TLS BRs.
The discussion on this Bug has also removed any confusion that may have existed in the industry whether delayed revocation is acceptable, even in exceptional circumstances. DigiCert acknowledges that revocation must always occur within the relevant windows defined in the TLS BR.
Remediation description:
Once clarity was achieved that delayed revocation was not allowed, DigiCert enacted its revocation procedures, and DigiCert systems and processes performed as planned.
Following up from the event, DigiCert made the following commitments:
-
Implemented ARI in its ACME implementations. DigiCert is committed to PKI modernization and automation, (noting that the Chrome Policy was also updated during the duration of this Bug seeking such efforts across all new CAs).
-
Undertaking Mass Revocation planning in accordance with the requirements of the updated Mozilla Policy.
-
DigiCert released its pkilint linting framework in 2023 as open source. Since this Bug, DigiCert has added a large corpus of additional internal lints, tailored to its specific issuance practices, seeking 100% coverage of linting of public trust certificates (not just TLS).
-
DigiCert has increased the depth and frequency of its customer communications on the issue of TLS automation, ACME, and ARI. These communications have gained additional momentum with the passage of Ballot SC-081v3, which introduces staged reduction in the validity of certificates between now and 2029. In many cases, the clarity of the deadlines is important to customers who need to seek budgets to implement TLS automation.
-
As described in other Bugs, DigiCert has refreshed its incident handling processes to expedite the decision making and response in incidents, as well as the proper handling of Bugs such as this.
-
DigiCert has clarified its policies and messaging both internally and with customers that delayed revocation is not an option, and the best path for both CA and Subscriber is to implement agility through automation.
All Action Items previously disclosed in this report have been completed as described, Accordingly, we request closure of this bug.
|
|
||
Updated•1 year ago
|
|
|
Assignee | |
Comment 108•1 year ago
|
||
We have posted our closing summary. As we have no additional comments at this time and there have been no additional comments or questions from the community we request that this bug be closed, or please set an appropriate nextUpdate date.
|
||
Updated•1 year ago
|
|
||
Comment 109•1 year ago
|
||
This is a final call for comments or questions on this Incident Report.
Otherwise, this bug will be closed on approximately 2025-05-08.
|
|
||
Updated•1 year ago
|
|
|
Assignee | |
Comment 110•1 year ago
|
||
The Closure Summary has been posted and the tentative closure date has been set. We have no additional updates.
|
|
||
Updated•1 year ago
|
|
|
||
Updated•11 months ago
|
Description
•