Closed Bug 1563573 Opened 5 years ago Closed 5 years ago

DigiCert: Failure to disclose Unconstrained Intermediate within 7 Days

Categories

(CA Program :: CA Certificate Compliance, task)

Product:
CA Program
Component:
CA Certificate Compliance
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: ryan.sleevi, Assigned: brenda.bernal)

Details

(Whiteboard: [ca-compliance] [disclosure-failure])

DigiCert has again failed to properly disclose intermediates to CCADB within the time period required.

TLS Server Certificates:

E-Mail Certificates:

The error is an exact repeat of the incident a year prior.

Please provide an Incident Report explaining why this failure happened and what steps are being taken to ensure this does not happen again. Please provide a binding commitment by DigiCert that DigiCert understands and commits to not having this problem occur again.

Flags: needinfo?(brenda.bernal)

Ryan, Thank you for your report. Our review shows that all the reported CAs were disclosed to CCADB but there was an issue with the completeness of the audit and CPS information. We are doing an additional check and will provide an update on Monday with an incident report.

Flags: needinfo?(brenda.bernal)

Brenda:

In addition to pursuing an update as to why these were not disclosed, I am concerned about the quality of DigiCert's existing disclosures.

In the course of looking through Bug 1556906, I happened to notice that certificates like https://crt.sh/?id=1027088214 exist, which state that they are covered by DigiCert's audit (Audit Same as Parent checked in CCADB), when from certificates like https://crt.sh/?id=5250464, they are audited separately and by a separate firm. Note that these two certificates share the same SPKI.

This shows up with other CAs. For example, https://bug1489232.bmoattachments.org/attachment.cgi?id=9006992 is from EY, dated for the period ending 2018-04-15, and states Apple IST CA 2, 3, 4, 5, 6, 7, and 8 are all in the scope of the same audit. However, in the disclosure for https://crt.sh/?id=1027088055 , DigiCert claims that this is in scope of the audit of 2018-04-20, at http://www.cpacanada.ca/GenericHandlers/AptifyAttachmentHandler.ashx?AttachmentID=221125 - except no such evidence supports this.

The reasonable conclusion I'd reach is that DigiCert staff is not consistently aware of the requirements requiring disclosure, or how to appropriately use CCADB. This becomes an issue with respect to improper disclosures of independently operated sub-CAs, and is a serious concern in understanding both risk and accountability.

My request is thus:

  • For every root certificate owned or operated by DigiCert
    • Go through every Intermediate certificate that is disclosed in CCADB
    • If DigiCert does not operate that intermediate, disclose the CP/CPS and audit statements for that CA
    • If DigiCert does operate that intermediate:
      • If it existed prior to the completion of your previous audit period, it is listed within the scope of the audit.
      • If it was created on or after the completion of the previous audit period, but was conducted according to procedures specified in your CP/CPS and using infrastructure previously audited, that such audit information is included
      • If it was created or operated under a new CP/CPS, that the Root Key Generation Ceremony Report or the Point-in-Time Assertion Letter are disclosed
    • Use "Audit Same as Parent" / "CP/CPS Same as Parent" if the information is redundant to the parent certificate
    • Ensure "Audit is Same as Parent" and "CP/CPS Same as Parent" is not checked for sub-CAs unless that is truly the case
Flags: needinfo?(brenda.bernal)

Hey Ryan,

I greatly appreciate you raising this because the items you asking for point out a gap in the Mozilla policy. We are, of course, willing and happy to go through each ICA disclosure and ensure the records are accurate. However, before we do so, we’d like to see an official Mozilla policy clarification about how to mark/disclose new ICAs as this is the ultimate source of confusion on this report.

The original ICAs listed were all disclosed in CCADB, but the audits were not marked “Same as Parent”. Although the ICAs were operated in DigiCert’s infrastructure, the ICAs were not audited yet, although they will be covered in the next audit. You pointed me to this discussion: (https://groups.google.com/forum/#!msg/mozilla.dev.security.policy/CAaC2a2HMiQ/IKimeW4NBgAJ). However, this discussion does not detail exactly what you should do with CCADB between when the ICA is created and when the audit happens.

The help text for the "Same as Parent" checkbox in CCDAB says "Click this box if audit of parent cert includes this cert". This is clearly incorrect as the audit of the parent cert does not include this cert. By default we usually check this box. However, we have an employee who is worried that doing so provides mis-leading information since new ICAs are clearly not audited the same as parent until the next audit cycle. I think they are correct.

From the Mozilla policy, section 5.3.2:
“We recognize that technically constraining intermediate certificates as described above may not be practical in some cases. All certificates that are capable of being used to issue new certificates, that are not technically constrained, and that directly or transitively chain to a certificate included in Mozilla’s root program:

  • MUST be audited in accordance with Mozilla’s Root Store Policy. If the CA has a currently valid audit report at the time of creation of the certificate, then the new certificate MUST appear on the CA's next periodic audit reports.
  • MUST be publicly disclosed in the CCADB by the CA that has their certificate included in Mozilla’s root program. The CA with a certificate included in Mozilla’s root program MUST disclose this information within a week of certificate creation, and before any such subordinate CA is allowed to issue certificates. All disclosure MUST be made freely available and without additional requirements, including, but not limited to, registration, legal agreements, or restrictions on redistribution of the certificates in whole or in part.”

The policy says the CA must disclose the ICA on the next periodic audit report, not the current audit report. Uploading the ICA and checking the box the way Google does it (marking it the same as parent, for example GTS Y1 issued 11/1/2018 claiming same as parent: https://ccadb.force.com/0011J00001K3m7X?srPos=0&srKp=001) is potentially misleading and misrepresenting when the audit occurs.
Therefore, I think the Mozilla policy needs to be updated to clarify the following:

  1. Should the audit box be checked for same as parent before the audit happens?
  2. If so, should there be a note to indicate that the audit has not yet happened?

As for the Apple ICAs, this isn’t a quality issue. The difference between the audit listings is the limitations in CCADB issue and a lack of Mozilla policy on what you do with audit same as parent in the gap between when the ICA is created and when the new audit is conducted. This is the very same issue that caused the original but to flag the ICAs as non-disclosed. We’re happy to fix the listings if we can get clarity around how the ICAs should be properly listed in the interim period between the creation and audit.

Taking the Apple ICAs as an example, our option was to:

  1. Use the previous Apple audit. However, the previous audit report doesn’t specify this ICA. We felt this was the most devious option since the ICA was clearly not listed in the previous Apple audit.
  2. Put nothing, but that gets the CA flagged by the needs audit report and means the ICA (per this bug) was not actually disclosed in the 7-day requirement.
  3. Flag the ICA as same as parent. We listed the ICA this way to indicate it would be listed in the next audit.
    CCADB does not give an option to indicate this information correctly. We chose option 3. Option 1 could have been a better option, but none of them accurately represent the real situation.

In the meantime, we’ll file an incident report in the interest of transparency on what happened and then clarify the incident report with section 7 once we get more information from Mozilla on the go-forward policy.

Jeremy

As requested, here is the incident report requested:

Incident Report

1.How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

A Bugzilla was opened by Ryan Sleevi on Thursday, July 4, 2019 noting “Failure to disclose Unconstrained Intermediate within 7 Days”

2.A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

4-July-2019: Bug opened for Digicert, reporter was Ryan Sleevi
4-July-2019: Investigation started with CAs identified and confirmed that all CAs were already uploaded in CCADB. See details below in section 4.
5-July-2019 to 8-July 2019: Internal discussion to confirm events and document next steps.
8-July -2019: Posted this incident report. Jeremy also posted on this bug asking Mozilla for policy clarification.

3.Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

We are refining the process for CCADB updates internally as such that there is consistency in how intermediates are uploaded, along with their audit information and CPS reference. However, this is contingent upon Mozilla clarification on the policy.

4.A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

First certificate: 9-October-2018
Last certficate: 2-July-2019

Here are the list (as reported) of the certificates with their CCADB links when they were posted.

TLS Server Certificates:

https://crt.sh/?sha256=03ce9bc71b91fdb7cb3c5235cae0701cb486bbd628d4aade5841fc5f0aa37a46&opt=mozilladisclosure
CN: DigiCert CN RSA CA G1
CCADB Link: https://ccadb.force.com/0011J00001P5hRSQAZ

https://crt.sh/?sha256=23ddf08b22373d86158eb9c99fdb5366b19804560531372dd20dce3fb766f56c&opt=mozilladisclosure
CN: GeoTrust CN RSA CA G1
CCADB Link:https://ccadb.force.com/0011J00001P5hUHQAZ

https://crt.sh/?sha256=f614ae2b101484f15f66f4dda56ee6eb422728f179d94399ede19ac1d85a8bd3&opt=mozilladisclosure
CN: Secure Site CA G2
CCADB Link:https://ccadb.force.com/0011J00001P5hRrQAJ

https://crt.sh/?sha256=02fed3ba2e6a7843a318a981bc847061fd282d9e8847ffa9f54d785b6b81d6f3&opt=mozilladisclosure
CN: Secure Site Pro CA G2
CCADB Link:https://ccadb.force.com/0011J00001P5hSpQAJ

https://crt.sh/?sha256=da3be2b6a6d9715c1295a42be526e0001d10e5d7540f06e7631b34e644934848&opt=mozilladisclosure
CN: Secure Site Pro ECC CA G2
CCADB Link:https://ccadb.force.com/0011J00001P5hUgQAJ

https://crt.sh/?sha256=7867aae905eb8d55635ffebbbf8cf05a63b9b390665de2367ac1073e92913728&opt=mozilladisclosure
CN: Thawte CN RSA CA G1
CCADB Link:https://ccadb.force.com/0011J00001P5hTOQAZ

https://crt.sh/?sha256=70357b9e56d3fb3c6c009c38c7181454c462908dfbce6d54d60dfe1e506e14fd&opt=mozilladisclosure
CN: TrustAsia ECC OV TLS Pro CA G2
CCADB Link:https://ccadb.force.com/0011J00001P5hV5QAJ

https://crt.sh/?sha256=dbdfa91acc4db8ad83fcc7978e35d62f6e32e55108273c8ec998e3133580d664&opt=mozilladisclosure
CN: TrustAsia OV TLS Pro CA G2
CCADB Link:https://ccadb.force.com/0011J00001P5hSLQAZ

https://crt.sh/?sha256=b131905cc7221270613b529ac9e786aa230abfe154a0acbe452bc350bd1efe4b&opt=mozilladisclosure

CN: DigiCert CN RSA EV CA G1
CCADB Link:https://ccadb.force.com/0011J00001P5hMSQAZ

https://crt.sh/?sha256=c87ce03affb5de6319c219971f2ed2d8f6f5389e2d53b22dd2c562a5c9827fc0&opt=mozilladisclosure
CN: GeoTrust EV CN RSA G1
CCADB Link:https://ccadb.force.com/0011J00001P5hOsQAJ

https://crt.sh/?sha256=b6eb4f8ad197073fe5203f8fcebfd5c509cde9ca3aa65ec59d203831424192d4&opt=mozilladisclosure
CN: Secure Site Extended Validation CA G2
CCADB Link:https://ccadb.force.com/0011J00001P5hN1QAJ

https://crt.sh/?sha256=40d69920ea05456fa0117de608b8a8013790b542097e343ec1cbce2dfb9713b0&opt=mozilladisclosure
CN: Secure Site Pro Extended Validation CA G2
CCADB Link:https://ccadb.force.com/0011J00001P5hQFQAZ

https://crt.sh/?sha256=70dc86f9f7750b74b1dec8cd352ec25837c36e640f7148e08464ef5901e5a589&opt=mozilladisclosure
CN: Secure Site Pro Extended Validation ECC CA G2
CCADB Link:https://ccadb.force.com/0011J00001P5hO4QAJ

https://crt.sh/?sha256=ada188f830c313f6046488ec341f1ed4af793c6dc28c58600445dfbeb4163746&opt=mozilladisclosure
CN: Thawte CN RSA EV CA G1
CCADB Link:https://ccadb.force.com/0011J00001P5hPvQAJ

https://crt.sh/?sha256=c0dd060ff8ce556f35683d564e0e66b2907178808693f83f3aa642326bf6d0c8&opt=mozilladisclosure
CN: TrustAsia ECC EV TLS Pro CA G2
CCADB Link:https://ccadb.force.com/0011J00001P5hNfQAJ

https://crt.sh/?sha256=0d51b6cdcf34a9f22144787f3fd920dc8800fb60490bfc8d289a1953e0fbda4d&opt=mozilladisclosure
CN: TrustAsia EV TLS Pro CA G2
CCADB Link:https://ccadb.force.com/0011J00001P5hPWQAZ

E-Mail Certificates:

https://crt.sh/?sha256=1b5c35ccad3d081e9a902ee278490a6436d23f6fcd1149e087b511ab969016aa&opt=mozilladisclosure
CN: DigitalSign CA - G4
CCADB Link:https://ccadb.force.com/0011J00001K65NhQAJ

https://crt.sh/?sha256=4f8137882b6d7b7f1abbcfded0082f0e5cdec870ba3923ef84d08dde42e43749&opt=mozilladisclosure
CN: Alcon Public Online CA
CCADB Link:https://ccadb.force.com/0011J00001PT636QAD

https://crt.sh/?sha256=e33076ffb5267df419f72de06c46e2c6ca44ff4383b5316cf7d6e628e98d9bfd&opt=mozilladisclosure
CN: Saudi Telecom Company Email CA
CCADB Link:https://ccadb.force.com/0011J00001PT63fQAD

https://crt.sh/?sha256=0f3284b306ac422986c8f560ea46226fd8425519a6f61aa9c55ab508274b82d2&opt=mozilladisclosure
CN: Solenis Public Online CA
CCADB Link:https://ccadb.force.com/0011J00001PT63GQAT

https://crt.sh/?sha256=d4f4357c103b034382c5a0dbf454a9c72799a28cab32cfcc28f80aa5c7b142a4&opt=mozilladisclosure
CN: CrossCert Class 1 Consumer Individual Subscriber CA - G3
CCADB Link:https://ccadb.force.com/0011J00001P5iZNQAZ

https://crt.sh/?sha256=7ad6072874552470f8c3b78f75cd7e0a04e2a17bda057b2c5e992fce3c5bab9c&opt=mozilladisclosure
CN: CrossCert Class 2 Managed PKI Individual Subscriber CA - G3
CCADB Link:https://ccadb.force.com/0011J00001P5ia6QAB

5.The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

Please see 4) above with the details.

6.Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

PKI Operations uploaded these ICAs based on the current Mozilla root policy. We typically mark all ICAs we operate with “audit same as parent” with the understanding that the new ICAs would be covered in the next audit period and covered by our Digicert CPS. However, in this case, PKI Operations did not mark these ICAs same as parent (mentioned above) according to how we typically do. Based on their interpretation of the policy, they felt this was the most appropriate option.

7.List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

We have raised the question to Mozilla on this specific policy situation of how we mark up the ICAs for audits as indicated in Jeremy’s post. We will elaborate further on our next steps once we receive an update.

Flags: needinfo?(brenda.bernal)

(In reply to Jeremy Rowley from comment #3)

I greatly appreciate you raising this because the items you asking for point out a gap in the Mozilla policy. We are, of course, willing and happy to go through each ICA disclosure and ensure the records are accurate. However, before we do so, we’d like to see an official Mozilla policy clarification about how to mark/disclose new ICAs as this is the ultimate source of confusion on this report.

The original ICAs listed were all disclosed in CCADB, but the audits were not marked “Same as Parent”. Although the ICAs were operated in DigiCert’s infrastructure, the ICAs were not audited yet, although they will be covered in the next audit. You pointed me to this discussion: (https://groups.google.com/forum/#!msg/mozilla.dev.security.policy/CAaC2a2HMiQ/IKimeW4NBgAJ). However, this discussion does not detail exactly what you should do with CCADB between when the ICA is created and when the audit happens.

I'll admit, I'm not terribly happy with this response. It seems a return to Symantec's process of rules lawyering. I'm all for clarification, but I think if a CA is going to argue that there's confusion, they should have raised this.

For example, a CA that had been following m.d.s.p. and the CA/Browser Forum would, for example, know that this very topic was touched on during last year's 2.6 update, in the context of Policy Issue #32. You might see, for example, that at the linked thread you mentioned, that Wayne's original posting stated

the TSP can select "CP/CPS same as parent" and "Audits same as parent" in CCADB to indicate that the same policies apply to the new subordinate as to the root.

I'm not sure where the perceived gap is, given that it was specifically addressed in the message you mentioned.

The help text for the "Same as Parent" checkbox in CCDAB says "Click this box if audit of parent cert includes this cert". This is clearly incorrect as the audit of the parent cert does not include this cert. By default we usually check this box. However, we have an employee who is worried that doing so provides mis-leading information since new ICAs are clearly not audited the same as parent until the next audit cycle. I think they are correct.

As part of your incident response, it would be useful to know why this issue wasn't raised or addressed previously.

The policy says the CA must disclose the ICA on the next periodic audit report, not the current audit report. Uploading the ICA and checking the box the way Google does it (marking it the same as parent, for example GTS Y1 issued 11/1/2018 claiming same as parent: https://ccadb.force.com/0011J00001K3m7X?srPos=0&srKp=001) is potentially misleading and misrepresenting when the audit occurs.

Could you explain, perhaps, what you believe an audit covers? I am deeply troubled by the suggestion that audits apply to specific certificates, and I'm quite concerned if CAs are confused by that. An audit is an assessment of an organization's ability to execute according to a stated set of procedures (a CP/CPS), based on evidence observed. The scope of that evidence would consider existing keys, and their associated certificates.

For example, it would be absolutely laughable if a CA were to argue it was perfectly OK use their CA key to sign a misissued certificate, setting the same issuer name and key of their 'in-scope' certificate, but then to claim it wasn't misissuance because what 'actually' signed it was some 'out-of-scope' version. While that seems extreme, that's logically what you're claiming here, especially as demonstrated below, so you can understand why I am quite concerned.

Therefore, I think the Mozilla policy needs to be updated to clarify the following:

  1. Should the audit box be checked for same as parent before the audit happens?

This was previously addressed.

  1. If so, should there be a note to indicate that the audit has not yet happened?

As for the Apple ICAs, this isn’t a quality issue. The difference between the audit listings is the limitations in CCADB issue and a lack of Mozilla policy on what you do with audit same as parent in the gap between when the ICA is created and when the new audit is conducted. This is the very same issue that caused the original but to flag the ICAs as non-disclosed. We’re happy to fix the listings if we can get clarity around how the ICAs should be properly listed in the interim period between the creation and audit.

I'm sorry, but I find this answer unacceptable. The reason is because you signed the exact same issuer name and key. As mentioned above, what you're functionally claiming is that this certificate is somehow 'out of scope' of the CA's audit, even though all certificates previously issued verify just as well to the newly issued certificate, and conversely, all certificates issued by the 'new' certificate are just as valid under the old. I specifically picked these certificates, because they demonstrate this case of an extant key pair and subject name.

Taking the Apple ICAs as an example, our option was to:

  1. Use the previous Apple audit. However, the previous audit report doesn’t specify this ICA. We felt this was the most devious option since the ICA was clearly not listed in the previous Apple audit.
  2. Put nothing, but that gets the CA flagged by the needs audit report and means the ICA (per this bug) was not actually disclosed in the 7-day requirement.
  3. Flag the ICA as same as parent. We listed the ICA this way to indicate it would be listed in the next audit.
    CCADB does not give an option to indicate this information correctly. We chose option 3. Option 1 could have been a better option, but none of them accurately represent the real situation.

Are you suggesting that you deliberated all of these decisions and choose the path you did? And that your Compliance team reviewed these and agreed that the best course of action was not to raise a question or concern, despite your perceived ambiguity, and instead do what you think was best? And even though this very topic had been previously been discussed?

I want to be clear, there's only one other time I'm aware of where such a claim has been made, and that was in the root inclusion of new DarkMatter roots, and such a response was very unfavorably received. If there is confusion or ambiguity, it should be raised.

Here's what I'm most concerned about: The choices in disclosure that DigiCert has made actively mislead the community about the entities in possession of private keys capable of TLS issuance. The improper disclosures, as such, cause real and demonstrable harm, in that they fail to distinguish independent organizations. I was shocked to see it for Apple, because I knew Apple was in possession of their private key - because it's listed right their in their audit. And there is absolutely no way that the key is in DigiCert's facilities. Whatever perceived ambiguity may exist, that choice was quite literally the worst choice and most misleading and harm-inducing.

Beyond that, I'm concerned because these were trivially detectable at https://crt.sh/mozilla-disclosures . You can still see that as of 2019-07-09, three of those sub-CAs are still not resolved, one of them which has been on that list since 2019-04-30, the other two since 2019-06-25, and all three mentioned in Comment #0. Given that DigiCert has already had issues where certificates were listed on that dashboard for some time, and Mozilla had to chase down Digicert management, I'm concerned that nothing in the intervening year was taken away to improve that process, and I'm worried that the takeaway now may be to mislead the public, as was done with Apple, in order to squelch such warnings. While ALV for intermediates in CCADB would detect such obvious misstatements, I'm worried that at any point it was seen as acceptable as stating Apple's "new" intermediate - which instantly made trusted (to the Baltimore root) every past certificate ever issued by their "old" one - was somehow seen as the best option. It really doesn't make any sense, and thus, I'm worried for all disclosures.

Flags: needinfo?(jeremy.rowley)

(In reply to Brenda Bernal from comment #4)

7.List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

We have raised the question to Mozilla on this specific policy situation of how we mark up the ICAs for audits as indicated in Jeremy’s post. We will elaborate further on our next steps once we receive an update.

In addition to the questions asked to Jeremy, a very specific question is why after DigiCert's management was informed last year about compliance violations, which were and are trivially obtainable via https://crt.sh/mozilla-disclosures , that DigiCert's management did not incorporate any steps to review this.

Similarly, given that there have been multiple incident reports filed over the past two years, collectively using this page, DigiCert's management did not take proactive steps to recognize it as a potential risk to their operations, and ensure compliance activities of a similar nature. This is not some new or magical page - it's been the basis for multiple issues, and every CA involved in disclosing certificates absolutely should be considering it as part of their playbook. I understand that it wasn't, and I understand that it will likely be a welcome suggestion for improvement - but I want to make sure y'all take a look at the systemic factors here. Regularly looking at the list is a remediation; my goal is to understand what sort of mitigations for future incidents can be taken from this. I can think of a number of ideas here, but I'm want to give y'all the opportunity to demonstrate you're on top of this first.

Flags: needinfo?(brenda.bernal)

Ignoring most of the response for now (will respond to it later), the summary of what happened is essentially multiple people read the Mozilla policy and have a different interpretation of the correct way to handle what to do with ICAs. The tooltip on the "Same as parent" led to further confusion. The primary gap this showed in our process is that we don't have a good two-person process to ensure quality control over CCADB listings. We are fixing this immediately.

Currently, CCADB uploads are managed by our PKI ops team and Compliance team, somewhat independently. This means that the values set (like the "same as parent") depend partly on the person reading the requirements. We do have compliance oversight, but this particular check is not one of them.

To ensure better quality, we have restructured the process so that our PKI Ops team is the the team providing the initial data to CCADB. The compliance team will be checking each upload to ensure the appropriate audit is selected and the information is correctly entered. This division makes the most sense as the PKI Ops team knows everything issued from the CA while the compliance team better understands the policy nuances.

On a more general level, I've asked Brenda to also identify any other areas where there is a single area where we should have multi-controls in place that include better compliance oversight and a checklist process. She is going to provide me the list this week so we can create new processes in each area. You are correct that most of the recent issues we've had are process related, not necessarily technical. The domain validation scope issue, the CAA issue, and the some-state issue were all knowledge-based failures where the two-party control failed because one of the parties simply didn't have the knowledge/diligence necessary to prevent the bad action from happening.

Flags: needinfo?(jeremy.rowley)

I'll admit, I'm not terribly happy with this response. It seems a return to Symantec's process of rules lawyering. I'm all for clarification, but I think if a CA is going to argue that there's confusion, they should have raised this.

I don't think we knew there was going to be confusion until it happened. Generally, our instruction to people is to look at the policy. If there is confusion over the policy, then ask the compliance team for clarification. Here, the policy is pretty clear (from the alternative perspective):

  1. Upload the ICA.
  2. Upload the audit information when the audit report became available.

Our policy should probably change to always involve compliance review since we're no longer at the point where everyone can be involved directly with the CAB forum.

For example, a CA that had been following m.d.s.p. and the CA/Browser Forum would, for example, know that this very topic was touched on during last year's 2.6 update, in the context of Policy Issue #32. You might see, for example, that at the linked thread you mentioned, that Wayne's original posting stated the TSP can select "CP/CPS same as parent" and "Audits same as parent" in CCADB to indicate that the same policies apply to the new subordinate as to the root.
I'm not sure where the perceived gap is, given that it was specifically addressed in the message you mentioned.

  1. We do follow the mdsp thread, but this discussion didn't actually relate to CCADB at the time from what I remembered. Instead, the focus was on what to do about audit reports, not on disclosing intermediates.
  2. I still don't read the requirement the way you are interpreting it. A CA can check it off this way, but I didn't think the post made checking this box required. The post would have used the term "must" or "is required".

That's semantics though and doesn't matter much if the real crux is does Mozilla

The help text for the "Same as Parent" checkbox in CCDAB says "Click this box if audit of parent cert includes this cert". This is clearly incorrect as the audit of the parent cert does not include this cert. By default we usually check this box. However, we have an employee who is worried that doing so provides mis-leading information since new ICAs are clearly not audited the same as parent until the next audit cycle. I think they are correct.

As part of your incident response, it would be useful to know why this issue wasn't raised or addressed previously.

I should have added more explanation around this. This particular rationale was why the box wasn't checked during the latest uploaded by the new-ish person. I think it's a good reason for why the checkbox was not checked but your mileage may vary. I do think it's a good reason to clarify the policy on whether to check the box or not so we can go through the list of ICAs and ensure they all mean the same thing.

The policy says the CA must disclose the ICA on the next periodic audit report, not the current audit report. Uploading the ICA and checking the box the way Google does it (marking it the same as parent, for example GTS Y1 issued 11/1/2018 claiming same as parent: https://ccadb.force.com/0011J00001K3m7X?srPos=0&srKp=001) is potentially misleading and misrepresenting when the audit occurs.

Could you explain, perhaps, what you believe an audit covers? I am deeply troubled by the suggestion that audits apply to specific certificates, and I'm quite concerned if CAs are confused by that. An audit is an assessment of an organization's ability to execute according to a stated set of procedures (a CP/CPS), based on evidence observed. The scope of that evidence would consider existing keys, and their associated certificates.

For example, it would be absolutely laughable if a CA were to argue it was perfectly OK use their CA key to sign a misissued certificate, setting the same issuer name and key of their 'in-scope' certificate, but then to claim it wasn't misissuance because what 'actually' signed it was some 'out-of-scope' version. While that seems extreme, that's logically what you're claiming here, especially as demonstrated below, so you can understand why I am quite concerned.

The audit covers the entire CA of course. The only thing that gave me pause in our previous post was asking for proof that the audit covers the ICAs. The proof that the audit covers the ICAs is the audit letter that lists the ICAs. I guess what's tripping me up is that you're asking us to point to proof that can't exist until we get an updated audit letter from the auditor. That doesn't happen until the next audit cycle. The ICA is covered by the current audit in that the ICA is hosted and operated under the same CPS and in the same infrastructure as all the other ICAs though, meaning everything except the ICA listing is the same.

Therefore, I think the Mozilla policy needs to be updated to clarify the following:
Should the audit box be checked for same as parent before the audit happens?
This was previously addressed.
Was it though? I think the answer is "yes, check that dang check box" but it'd be nice to have it said.

Re: Apple
I'll have to talk to people tomorrow. I don't have actual knowledge of what's going on there. It'll take a bit of digging to find out exactly what went into the decision making process. I know we host some, and some are on-prem.

Beyond that, I'm concerned because these were trivially detectable at https://crt.sh/mozilla-disclosures . You can still see that as of 2019-07-09, three of those sub-CAs are still not resolved, one of them which has been on that list since 2019-04-30, the other two since 2019-06-25, and all three mentioned in Comment #0. Given that DigiCert has already had issues where certificates were listed on that dashboard for some time, and Mozilla had to chase down Digicert management, I'm concerned that nothing in the intervening year was taken away to improve that process, and I'm worried that the takeaway now may be to mislead the public, as was done with Apple, in order to squelch such warnings. While ALV for intermediates in CCADB would detect such obvious misstatements, I'm worried that at any point it was seen as acceptable as stating Apple's "new" intermediate - which instantly made trusted (to the Baltimore root) every past certificate ever issued by their "old" one - was somehow seen as the best option. It really doesn't make any sense, and thus, I'm worried for all disclosures.

I'll take a look at this too. I know there was a conversation about this before it ever happened. I'll have to dig that up. We've done several things to improve reporting of ICAs (like making sure they were actually uploaded). The next thing we are doing (as mentioned in the previous comment) is splitting up responsibility. Our PKI OPs team (led by Ben Wilson) is responsible for creating all issuing certs. They will also be responsible for uploading all issuing CAs. Before creating any issuing cert, compliance will review the profile and sign off on it. This will give compliance actual knowledge of what is created. After a key ceremony, compliance gets a copy of the script and the issuing CAs. We then compare what was uploaded with the expected browser representations and CPS documents. This make sense as we manage the audits and documentation. Ben's team may not know which audits control which CAs. Doing so ensures you have two unique departments reviewing and approving each issuing CA before creation and approving each CCADB entry.

To ensure each current entry is correct, the compliance team will go through the CCADB listings and 1) ensure the correct CPS is listed and 2) ensure the audit entry is correctly listed. We have consolidated oversight of all compliance related activities to my team, including Quovadis. Going through the listings will make sure there is consistency in what we upload, how we report, and the communication to you guys.

CC'ing the other DigiCert (In reply to Jeremy Rowley from comment #7)

Ignoring most of the response for now (will respond to it later), the summary of what happened is essentially multiple people read the Mozilla policy and have a different interpretation of the correct way to handle what to do with ICAs. The tooltip on the "Same as parent" led to further confusion. The primary gap this showed in our process is that we don't have a good two-person process to ensure quality control over CCADB listings. We are fixing this immediately.

Thanks. Trying to make sure I distill this correctly:

The Root Cause is that your Compliance and PKI Ops team lacked a formal policy regarding the expectations around disclosures. As a consequence, individual members of the team were left to interpret the requirements individually, and this resulted in inconsistent and incomplete disclosures.

The Remediation is that the Compliance team (Brenda) will use the new understanding and examine existing disclosures to ensure correctness.

The Mitigations going forward are that:

  • DigiCert will develop a common policy regarding what to do for disclosure under a variety of cases
  • DigiCert will implement two-person controls to ensure that the policy is correctly adhered to
    • The PKI Ops team, which is responsible for creating certificates, will ensure all new certificates are entered to CCADB
    • The Compliance team, which is responsible for policy compliance, will ensure all the relevant policy information is disclosed appropriately
  • DigiCert is holistically looking at controls
    • To identify any remaining controls that rely on either Single Person or Single Organization (e.g. PKI Ops) action, to examine the possibility of making them Multi-party controls
    • To identify multi-party controls that rely on consistency and shared knowledge between both/all parties
    • Developing Checklists, to be overseen by Compliance, to allow for consistency and review

(In reply to Jeremy Rowley from comment #8)

Thanks. I look forward for the update re: Apple, which was just fortuitous discovery.

The process you described makes it clearer about where things went wrong, and the described change does sound like an appropriate way of ensuring that the Compliance team is involved in all decision making. Combined with the previous response, it sounds like the steps will have Compliance directly involved in a variety of sensitive operations, and ensure that Compliance is responsible for the development of Checklists and Training for routine operations.

This touches on some of the past issues, as you mentioned, where Compliance was not involved, and whether good intentioned or not, mistakes were made. I'm not trying to blame the other teams, but I'm wanting to make sure that there's a consistency in the organization structure and operations that either ensures all teams are equally versed in Compliance, or that Compliance is intimately involved in all decisions, and ultimately accountable for them. This may require additional work or effort to grow the Compliance team to adequately support the organization, but given the risks posed, this seems justified if necessary.

Looking forward to mitigations, I want to make sure y'all take a look at the system of issuance and operations, and make sure Compliance is involved in all such decisions, if that's how you structure your organization. Code reviews, checklists, operational changes (such as the transport of key material or service outages/migrations), etc - basically, a risk-based analysis and ensuring that either Compliance can distill its knowledge into repeatable procedures (i.e. checklists) or can be directly involved in development or review.

Okay - investigating Apple, I think we found the root of all evil (at least in this bug) is still confusion about what to do in each of the cases. Apple was legit that people though you should check "Same as parent" (based on the email thread you cited) for all issuing CAs until the next audit. This made the audit look like it belonged to DigiCert even though it didn't. Obviously this is incorrect. We should have caught this earlier, and we need a better process for looking where different individuals are doing things differently, especially when it comes to process questions. We have a good idea on how to improve things substantially that I think is pretty innovative. I'll give details on that in the next update after I work out a couple of details.

Here's our current proposal on how we are going to provide quality control over these (based on Kathleen's and your post to MDSP) and the rest of the remediation process.

  1. If the issuing CA is operated by DigiCert, we will select same as parent.
  2. If the issuing CA is operated by a third party, the previous audit for the third party should be provided and a note should be added that explains the issuing CA was created after the audit report issued and that the next audit report must list the ICA.
  3. If the issuing CA is technically constrained, the ICA will be uploaded but a note will be added to mark the ICA as technically constrained. The audit box will not be checked.

Here's the flow we are putting in place for all future issuing CA creation:

  1. A JIRA must be created for every ICA creation activity. A key ceremony may not be scheduled by the team until the JIRA exists.
  2. The PKI Ops team (led by Ben Wilson) sends the customer approved naming doc to compliance (xxx@compliance.com) for approval. This naming doc must also be to the JIRA. The PKI Ops team schedules the ceremony and includes the ceremony date in the JIRA and approval email.
  3. If Compliance approves, approval is sent to the PKI Ops email to start ICA creation and marked in JIRA.
  4. The key ceremony is held on the specified date. During the key ceremony, as part of the script, approval by compliance is confirmed.
  5. Once created, PKI Ops sends the newly created ICA to compliance for approval before releasing the ICA to product or customers. Compliance reviews the cert and confirms the ICA is created as per the approved naming document.
  6. If Compliance approval is granted, confirming the naming document matches the certificate contents, the ICA is integrated into the system/released to the customer. If the naming document and certificate do not match, the ICA is rejected and a new key ceremony is initiated to make the corrections. The remaining steps happen regardless of whether compliance approves the certificate or not. Afterall, the certificate is created and live regardless of whether it was used in production.
    a) If the certificate contains contents unacceptable to the BRs, the Mozilla root policy, or any other requirements, the compliance team must escalate to Jeremy within 4 hours. An incident report will be filed within 24 hours. Any questions about contents must be escalated to Jeremy.
  7. Regardless of whether compliance approve is granted the following occurs, all created certs are logged in CCADB for auditing purposes. Their revocations are also logged.5.PKI Ops adds newly created ICA to CCADB (including appropriate audit requirements)a.If managed by DigiCert Logging follows the rules specified for CCADB uploading [the three rules above].
  8. Compliance addsthe ICA to the appropriate audit schedule and CPS review schedule.
  9. PKI Ops emails newly signed ICAs to compliance each Thursday by COB (for all locations). An email is sent whether there is activity or not. If no ICAs were created, the email should indicate that no ICAs were created. Compliance will return an email on Friday on status of checking and what corrections to be made to the CCADB listing (if any). A return email will be sent even if all items are correct. Two persons from each team are identified for this notification process. a) Compliance Primary: XXXXXX Compliance Secondary: XXXXXXX a) PKI Ops Primary: XXXXXX b) PKI Ops Secondary: XXXXXX X
  10. Compliance performs weekly checks on CCADB listings and https://crt.sh/mozilla-disclosures to ensure all ICAs are accurate and accounted for.

We're going to run this process for a month and report back.

At the same time we are kicking off a complete review of all CCADB listings starting tonight. I'm not sure how long it will take. We should have a good velocity after a couple days of measuring the rate at which we get through them.

Additional details:

  1. We finished fixing the Apple ICAs
  2. We marked the audits on the three remaining ICAs where the audit was still not marked (per the rules I posted above)
  3. We are starting our review with the external issuing CAs first and then will review the ones hosted by DigiCert.

I'm slowly trying to drive compliance towards a checklist driven system. Not sure if you've read "The Checklist Manifesto", but I may make it required reading on the compliance team. It's the model I'd like to follow in a lot of the operations.

The other thing we are implementing is embedding the compliance team directly in the scrum team. Each scrum team will have a dedicated compliance person that will be part of the stand-up, sprint planning etc. This will give compliance a forefront view into what is happen and ensure compliance is central focus in the product build. I'm pretty exciting about this and can't wait to share how it goes. We're going to start giving the team scrum training and will launch test teams during the first sprints in August.

Do you have a timeframe for the "review with the external issuing CAs first and then will review the ones hosted by DigiCert."?

Flags: needinfo?(jeremy.rowley)

We finished reviewing the external CAs over the weekend. We are estimating the internally hosted ones will take about 3 wweks.

Flags: needinfo?(jeremy.rowley)

This has been a good exercise. We've reviewed 1099 so far. Of these, two issues were spotted.

1965F68BEA95015988026F43E9F82EB3
7DA7F01891CC48F3B42BB9E6E9D4B188

These were marked same as parent and are not same as parent. They are controlled by Adacom. However, they were not listed in the last Adacom audit. These issuing CAs are only capable only of issuing email certificates. They are being revoked shortly (within the next couple of days).

Jeremy: Thanks for the update. I'm glad you caught those certificates were not listed. Was that due to human review, or was that the result of CCADB's ALV executing once you'd corrected the disclosure, and CCADB informing you about the lack of inclusion?

I ask, because I'd be concerned if this oversight was only discovered as a result of disclosing into CCADB. While I'm glad secondary controls caught it (if they did), I think it'd be useful to understand why primary controls failed (if they did). As it's ambiguous as to what caused you to discover the oversight, I'm uncertain as to whether that's something that the final incident report should address.

The incorrect ICA listing was caught because of the human review you asked us to conduct on this bug. The ICAs were disclosed in CCDAB but marked same as parent instead of being listed as belonging under the Adacom CPS. The issue is identical to the Apple ICAs you identified earlier (same interpretation of the requirements). The consistent policy we identified above will remove this issue. Do you want us to file a final bug report that includes these ones as well and any others we find?

Jeremy: I was focused on this part from Comment #15:

However, they were not listed in the last Adacom audit.

I’m trying to understand if you received the audit, and only now noticed they were missing, in the course of performing the review.

We received Adacom's audit on 5/17/2019. These are from 2010 and 2012 so they should have been included in the parent's audit well before that. Something doesn't make sense here. Let me investigate. Sorry - looks like I don't have the full story.

Oh - I know this one. The issuing CAs are s/mime only (not enabled in Mozilla for TLS). They were re-signed in 2018 and uploaded for same as parent (improperly) . We received the audit from Adacom on 5/17/2019. I don't know why they were not included in the 2019 audit report or how they were missed before performing this review. I'll have to dig deeper to find out and report back.

Ignore my previous post. I was wrong on which CAs they were and what happened. (I misunderstood, thinking we did some signings in 2018 but it was a mass upload of certs). Brenda did the investigation.

Here's the timeline for what went down with these issuing CAs:

  1. 2010, Feb - Symantec creates the KIBS Qualified Certificate Services CA (7d a7 f0 18 91 cc 48 f3 b4 2b b9 e6 e9 d4 b1 88)
    This one is hosted by KIBS Verba CA and issued from a root enabled only for smime certs

  2. 2012, Sept - Symantec creates the KIBS Certificate Services CA (19 65 f6 8b ea 95 01 59 88 02 6f 43 e9 f8 2e b3)
    Symantec hosts this one and issues it from a root only enabled for smime. This one is covered by the Symantec audits.

  3. 2016, Feb - The last end entity certificate from KIBS Certificate Services CA (the one hosted by Symantec) issued.

  4. 2016, Dec - The last end entity certificate from KIBS Qualified Certificate Services issued. This is the on prem CA.

NOTE: The end entity certificate lifecycles for certs issued from either CA are 1-3 years, meaning they are essentially all expired.

  1. 2017, April - Mozilla policy was clarified that all issuing CAs including the email trust bit required an audit and needed to be uploaded to CCADB. At some point, Symantec uploads these records. Adacom was operated under a different business unit than the TLS business. We are unsure how these records were uploaded. They were marked "same as parent" which was true for the one hosted by Symantec but false for the one hosted by DigiCert.

  2. 2017, Nov - DigiCert acquires Symantec.

  3. 2018, Feb - Symantec and DigiCert records are merged in CCADB, resulting in a loss of historic data. The records now show as submitted by: Poonam Bhargava - Mozilla SFDC consultant.

  4. 2018, June - Received our first Adacom audit; however, the ICAs were not listed. We did not know that these were omitted because a) this was the first audit conducted of Adacom by brenda's team, the ICA was marked same as parent and we didn't realize that the audit report was missing one.

  5. 2019-May: DigiCert requested scope information from ADACOM for the upcoming audit; ADACOM responded that their audit was already completed, and they started audit earlier than the previous years. Previous audit end date was 21st of May.

  6. 2019, June: ETSI Audit Report from Adacom sent to Digicert, which was missing the two KIBS CAs.

  7. 2019, July - Bugzilla opened on “Failed to disclose intermediate”. Internal review raised these two ICAs in question about whether they were covered in an audit (email from Steve to Martin/Mike L/Brenda) – Note this was before Ryan opened the bug on Failure to properly disclose intermediates. Digicert reached out to ADACOM to request information about why the two KIBS CAs were not in their audit list and were disclosed in CCADB. (followed up on the 4th and the 8th). ADACOM responded (contact person was on leave) regarding our email inquiry on 2-July about KIBS in audit. ADACOM confirmed the KIBS CAs were not in use and that they can be revoked. ADACOM also said they would update their audit reports to include the issuing CA. Completed investigation on the rest of CCADB and identified the anomalies around “audit same as parent” marked CAs, including the 2 KIBS CAs

  8. 2019, July 18 - KIBS CA’s revoked

Conclusions:

  • Human error between how these ICAs were originally loaded in CCADB and us not catching this earlier
  • We need to enforce 3rd party CA/RA policy at all times – including getting a continuous audit cycle for unrevoked ICAs disclosed
  • Scoping of audits for external subCAs must match what we have in CCADB (this presumes we’ve caught all the errors). Steve and Ben did the first and second pass review, but recommend we have Christina and Martin tie out also on the Affiliate ICA data to be absolutely certain.
  •   The riskiest ones for not having correct audit information listed in CCADB are from the legacy Symantec side. These will be completely resolved when the sMIME distrust is complete.

Update: The ADACOM amended audit letter has been posted here: https://bug1503610.bmoattachments.org/attachment.cgi?id=9081022. Please advise if there are any further questions.

Flags: needinfo?(brenda.bernal)

Hi Wayne: I have posted a question on the forum discussion regarding a number of items listed on: https://crt.sh/mozilla-disclosures. Primarily, I've asked for clarification on our cross-roots that are appearing on the report that are disclosed and included in their own audit reports.

Specifically for the items you've noted above, let me provide the following clarifications/explanations:

  1. https://crt.sh/?sha256=feb915628c979e06f171805d6d2702c0270420bd46bf6011d03623fc932454fc&opt=mozilladisclosure - We had posted an update here regarding the status of the Verizon audit: https://bugzilla.mozilla.org/attachment.cgi?id=9081020.

  2. We have reached out to Apple on the below items, and they are looking into these. We will provide an update as soon as we can:
    https://crt.sh/?sha256=3db76d1dd7d3a759dccc3f8fa7f68675c080cb095e4881063a6b850fdd68b8bc&opt=mozilladisclosure
    https://crt.sh/?sha256=17f96609ac6ad0a2d6ab0a21b2d1b5b2946bd04dbf120703d1def6fb62f4b661&opt=mozilladisclosure
    https://crt.sh/?sha256=904fb5a437754b1b32b80ebae7416db63d05f56a9939720b7c8e3dcc54f6a3d1&opt=mozilladisclosure

  3. https://crt.sh/?sha256=9e230bb73827b3f7b3f75bf231661dfd40786a1b9ac256f1aa1d752b506a2ccb&opt=mozilladisclosure - this is the ABB Intermediate that is technically constrained but we plan to revoke the Issuer: CA 5 per our update on bug https://bugzilla.mozilla.org/show_bug.cgi?id=1456655

  4. https://crt.sh/?sha256=c40d6ab75f7e5d11cf7b0404ac7e4321a7a8b0ddf7631328f3d7cc80d6644e2b&opt=mozilladisclosure - this is covered in a Webtrust audit as noted in their Standard Audit record link on this page.

We would like to better understand how some of these items (cross-roots in audits and the NRC CA listed in 4 above) came to appear on the disclosures report to help in reducing the number of false-positives.

Flags: needinfo?(brenda.bernal)

(In reply to Brenda Bernal from comment #24)

Hi Wayne: I have posted a question on the forum discussion regarding a number of items listed on: https://crt.sh/mozilla-disclosures. Primarily, I've asked for clarification on our cross-roots that are appearing on the report that are disclosed and included in their own audit reports.

I want to acknowledge we've received your request for clarification, and we appreciate the detailed analysis you've provided.

  1. We have reached out to Apple on the below items, and they are looking into these. We will provide an update as soon as we can:
    https://crt.sh/?sha256=3db76d1dd7d3a759dccc3f8fa7f68675c080cb095e4881063a6b850fdd68b8bc&opt=mozilladisclosure
    https://crt.sh/?sha256=17f96609ac6ad0a2d6ab0a21b2d1b5b2946bd04dbf120703d1def6fb62f4b661&opt=mozilladisclosure
    https://crt.sh/?sha256=904fb5a437754b1b32b80ebae7416db63d05f56a9939720b7c8e3dcc54f6a3d1&opt=mozilladisclosure

Thanks. Please provide weekly updates on the progress here.

  1. https://crt.sh/?sha256=c40d6ab75f7e5d11cf7b0404ac7e4321a7a8b0ddf7631328f3d7cc80d6644e2b&opt=mozilladisclosure - this is covered in a Webtrust audit as noted in their Standard Audit record link on this page.

Just to make sure I'm parsing this correctly:

And this is because all TLS-issuing certificate paths have expired or been revoked.

Is that correct? Just making sure I parsed what "Webtrust audit" meant, as I originally thought you were referring to the BRs :)

We would like to better understand how some of these items (cross-roots in audits and the NRC CA listed in 4 above) came to appear on the disclosures report to help in reducing the number of false-positives.

Agreed. I appreciate your explanations, as they help identify and resolve areas of confusion in policy or in analyzing the disclosures.

Flags: needinfo?(brenda.bernal)

Hi Ryan, You are correct; this is a Webtrust for CA only as the trust path is through a root that is enabled for email/clientauth.

Flags: needinfo?(brenda.bernal)

For 2) above on Comment 25, we are in discussions with Apple on next steps and will get back to you shortly once we have a definitive plan.

Update on the Apple CAs as noted in 2) above Comment 25 - Apple is working with their auditors to amend their audit reports to include these CAs. Please refer to https://bugzilla.mozilla.org/show_bug.cgi?id=1575125 for the separate incident report opened to track this incident.

Brenda: I don't think Bug 1575125 is a substitute for DigiCert's own failure to supervise this, which I think we can continue to use this bug for.

If I'm understanding correctly, from the details provided by Apple in Bug 1575125:

  • 2014-05: Mozilla sent a CA communication requiring all unconstrained CAs are required to be technically constrained or disclosed (i.e. with the appropriate audits)
  • 2014-06: Symantec/GeoTrust cross-signs Apple IST CA 5, 6, 7
  • 2014 - 2017: Symantec fails to abide by Mozilla's policy with respect to disclosing Apple IST CA 5, 6, 7, which are technically capable of TLS issuance but neither technically constrained nor audited
  • 2017-10: DigiCert acquires Symantec's Legacy PKI
  • 2017-10: Mozilla agrees to that integration
    • Note: Mozilla raised concern "if Symantec processes appeared to displace DigiCert processes" and if it "were to appear as if Symantec were the controlling CA organization in the merger"
  • 2017-11: DigiCert files Bug 1417771 to track the failures of Symantec to disclose the appropriate audits, and fails to detect this issue when integrating Symantec's infrastructure
  • 2018-06: DigiCert receives the first set of audits for IST CA 5, 6, 7 for the periods 2017-04 through 2018-04, and fails to detect the issue
  • 2019-06: DigiCert receives the second set of audits for IST CA 5, 6, 7 for the periods 2018-04 through 2019-04, and fails to detect the issue
  • 2019-07: DigiCert is informed of this
  • 2019-08: DigiCert fails to take action.
  • 2019-08: Apple proposes that DigiCert and Mozilla accept an amended audit for the period 2018-04 through 2019-04, despite there being no WTBR audit from 2014-06 to present.

Is that timeline correct? Have I materially misstated anything?

I'm passing this to Wayne, because I am seriously concerned about DigiCert's failure to detect this through the Symantec integration, followed by the repeated failure to detect this through two subsequent audits. This is part of the quintessential behaviour that led to Symantec being distrusted, and the failure to handle this with the utmost urgency and seriousness gives me serious pause as to whether Symantec processes have indeed displaced DigiCert processes.

Flags: needinfo?(wthayer)
Flags: needinfo?(brenda.bernal)

Hi Ryan,
Yes, I agree that this is not a substitute for addressing DigiCert’s issue on this bug. We have been discussing the remedy with Apple which they’ve captured quite well in their incident report. We understand that our responsibility as the Root CA is to continue to monitor their progress with remediation, and we will post an update here as necessary.

We can keep this bug open until you/Wayne feel we have satisfactorily resolved all points.

While I can’t speak to the events that you’ve noted above in 2014, I will address the events from 2016 onwards since that is the part of the time-line I am aware of. I wasn’t involved before then.

Correct, we missed detecting that Apple had WTCA audits for IST CA 5, 6, 7 but they lacked BR audits for the periods 2017-04 through 2018-04. These CAs were commonly referred to as the email CAs, because they have their own CPS separated from the Apple TLS CPS. An incorrect interpretation bias and a year over year pattern of excluding these Apple CAs from BR audits prior to the Mozilla policy change that implemented the "technically capable" language has resulted in us failing to correct this violation. Post-merger, post-reorganization, the Compliance function that is now in place has been working diligently to implement good controls. We review external subCA audits and have implemented multi-reviewer checks to ensure different sets of eyes catch policy non-compliance anomalies. In this case, the recent Apple IST audits again did not cover these ICAs in a BR audit due to reviews affected by the same bias. While we have been discussing remediation of this omission with Apple since July when we confirmed the gap, this proves that multi-reviewer attention is mandatory for all audits under our management. In failing to detect the omission and probably filing an incident report shortly after the gap was confirmed, we evaluated additional controls beyond just making progress with implementing multi-reviewer checks on audit scope. This audit scope issue also has brought up the need to further scrutinize additional metadata to confirm accuracy, beginning with the root trust bits from which each ICA chains. In the case of Apple, those ICAs chain to GeoTrust Roots that were trusted for email and serverAuth. Although Apple’s intent and our collective oversight is that the ICAs in question were only intended for email usage, they’ve inherited the trust bits from the Root. We are challenging that legacy knowledge about this PKI by relying on source data (e.g. Mozilla trust store bits) to drive decision making and analysis.

In a nutshell:
1)We mapped out all the hierarchical structure of CAs across our entire public PKI.
2)We mapped all the policies that are tied to the CAs, including audit requirements.
3)We also flagged all the required audits the CAs undergo annually.
4) This information is reviewed and updated on an ongoing basis by our Compliance team.

We are pleased to see the new derived trust bits feature in CCADB and we're building this same idea into our internal root and ICA management software that will be used as our source of truth for audit scoping and disclosures in CCADB.

On your point above, the Compliance processes and procedures that we’ve put in place are not displacing DigiCert’s processes with what Symantec had in place. We actually have been implementing a lot of goodness with control procedures since the acquisition closed. Our SubCA management – 3rd party program is net new, and we have only begun to fully implement it across the former Symantec business. You can read the full text here: https://bugzilla.mozilla.org/show_bug.cgi?id=1566162#c13. I understand that your preference would be for us to post this type of information on mdsp, which I don’t have a problem with and I can. I’ve sought that guidance as well on the posting.

Some background: When Digicert acquired the Symantec Legacy PKI business, this included the User Auth (UA) business that was part of a different business unit than the SSL business (Website Security). Symantec UA had ownership of the externally managed SubCAs who issued S/MIME and client auth but not TLS. It did not have a compliance function so their governance / oversight on compliance matters was non-existent. In the 2017-2018 audit concluded under Digicert’s ownership of all Symantec roots/ICAs, that was the first time all hosted CAs that required an audit per Mozilla policy were included in Webtrust audits and uploaded to CCADB, as referenced in the Bug 1417771. Our stricter management of the audit review for the externally operated SubCAs has been in motion since. It is our ongoing procedure to review and work with subCAs to file proper audits; in some cases, we’ve found issues/discrepancies which we correct with these parties. This is still a work-in-progress but we are making marked improvements from the lack of audit review and governance under Symantec UA.

We have evaluated the repeated patterns of issues from the past, and have identified areas of automation that will reduce human bias or misinterpretations and decrease potential for errors. Compliance is the top priority for DigiCert; we have consistently implemented procedural changes along the way that require multiple reviews of information and remove single points of failure as much as possible. Examples of areas we’ve covered include ICA creation from key ceremonies, CCADB updates, audit reviews, and incident reviews /reporting. I am happy to write more about these efforts to generalize how the CA broader community can understand where we started, where we are now, and how we go from here.

Overall, DigiCert has a proactive compliance mind-set and a culture of openness. We remind everyone to always freely report any and all issues they come across so we can address them as quickly as possible. We welcome your input on where else you feel we need to address.

Flags: needinfo?(brenda.bernal)

Thanks for highlighting https://bugzilla.mozilla.org/show_bug.cgi?id=1566162#c13 . I don't mind it being posted here vs m.d.s.p., as long as we've got a public discussion.

In that context, it was mentioned that "Since the beginning of 2019" that y'all "required scoping review of audits with DigiCert before they commence". Did that happen here with Apple? If so, is there a clear understanding of what went wrong here?

I highlight it, because while I appreciate DigiCert is trying to make improvements, it feels like for every two steps forward, we later find out there was also one step back - that is, that things which sound like they would be substantive to address the issues end up overlooking the stuff you'd expect to be caught. For example, that response gave me much more hope that things had been turning around, and that's why I'm dismayed at the timeline from Comment #29 or the proposal in Bug 1575125. For example, Bug 1566162 had given me hope that DigiCert understood and appreciated the importance of revoking anything out of compliance, especially with respect to Legacy Symantec Infrastructure, but Bug 1575125 makes me think that both DigiCert and the subscriber (in this case, Apple) are suggesting they should be grandfathered in on a 5 year old policy, because Symantec materially failed in its controls, and DigiCert didn't catch it in a timely fashion.

Here's my worry with the current solution outlined in Comment #30: Given issues like this and Bug 1566162, one of my worries is that in some time following, we'll hear something like "Oh, an employee forgot to update the trust bits, so we forgot we needed to audit according to X. This was a one-off human error" (i.e. a failure in point 4), and we'll all sort of chuckle, then sigh, and then sort of quietly sob as we realize it's a functionally a repeat of the issue.

I get that to err is human, and mistakes happen. I appreciate that examination of CCADB as a cross-reference here, which is useful. What's the timeline for that?

From this issue, I'm looking for a few things:

  • A clear statement from DigiCert about what it plans to do about the unaudited, long non-compliant CAs. I understand the sub-CA has a proposal. What's DigiCert's proposal, since DigiCert is accountable?
  • An examination, perhaps as a separate doc, that looks at what things might go wrong with your proposed plan to address it, and how you have existing controls that might address it.

For example:

  1. We mapped out all the hierarchical structure of CAs across our entire public PKI.
  • Risk: A new sub-CA is introduced and this list is not updated
  • Mitigations:
    • A playbook exists requires disclosure every time this list is updated
    • This shall be examined by two different people to confirm it was done correctly
    • In addition, an automated tool will examine CCADB and CT Logs to ensure no omissions here
    • ... etc
  • Risk: We have overlooked a CA within our existing set
  • Mitigation:
    • We'll be publishing our list via mechanism X and soliciting feedback
    • We'll be releasing a tool that shows how we cross-reference this against CT Logs and CCADB, to get public review that we're comprehensive
      ... etc

I realize that this is a lot more detailed than has been requested of other CAs here. That's largely because there's this pattern of good ideas and not always the best execution. The most useful thing here is to try and get input to make sure you're not overlooking things or missing things, and a great way to do that, while also improving trust in your CA in spite of this spate of incidents, is to be open and transparent. It may be that DigiCert wants to solely rely on the Compliance team and a four-eyes principle. I think that's riskier, but it's certainly a choice DigiCert can make. The downside is that it makes it much harder to be understanding or charitable about repeat issues. I'd rather have DigiCert lead by example here, rather than have to clarify and clean-up after-the-fact. :)

Flags: needinfo?(brenda.bernal)

Just FYI - We're working on a reply to this. It's going to take a bit to gather the information in a way that I want to present it. I think it'll be best if split the information among multiple comments as we form it correctly.

Here is our update on a few efforts under this incident:
Apple has committed and are on track to completing their audits of prior periods by September 30, 2019. On the question of whether scoping was done with Apple to identify these anomalies upfront, this did not occur as we rolled out a new version of our third party policy in April 2019. At that point, Apple’s audit was already underway.

We have added the following language to our third party policy, “All Third-Party CA/RA organizations MUST review with DigiCert the scope of CAs and Operations under audit before scope is finalized with their respective auditors.” This was included as an updated policy language as renewals occurred. In early June 2019 we communicated widely this new policy amendment to our external SubCAs. As for unaudited CAs, they will be covered under an audit, or be addressed through discontinued use of that CA and likely revocation. Our policy is clear on the matter – audit reports must be provided as long as the SubCA is valid and even if revoked during the audit period. This would include SubCAs that have ceased issuance of certificates in the audit period and remain unrevoked. Along these lines, DigiCert has also completed an internal audit of all CA’s with 3 separate sets of eyes for reviews. Additionally, DigiCert is working with a WebTrust accredited audit firm to independently review the entire DigiCert PKI master mapping that we have developed, which includes all internal and external operated CAs. We plan to use this method to compare and validate the data from our reviews, and close out any gaps found. Due to the size of this effort we are reviewing CAs in a number of stages:
• Stage 1 - Our external SubCAs – tentatively targeting by September 13
• Stage 2 - DigiCert hosted CAs (which includes legacy Symantec/VeriSign)
• Stage 3 – QuoVadis CAs and roots

We will provide the next update with timelines for the later stages.

We certainly hope that beyond just making marked progress, we are able to put the data inconsistency issues behind us with the steps we are taking. Having a repeat issue would certainly not get a chuckle out of us, more of a heavy sigh…we are committed to making sure our data is good and we take actions based on that good information.

Flags: needinfo?(brenda.bernal)

Brenda,

I do not see any path forward for this sub-CA that is reasonable and demonstrates compliance.

I understand that DigiCert has had a long relationship with Apple. I understand that Apple is, well, Apple. I understand that Apple had audits for other parts of its infrastructure. However, I cannot see any remotely reasonable path that allows an sub-CA to escape multiple years of audits and then say "whoopsie", and there is zero path that the community can reasonably regain trust in these CAs after the fact.

There is no way a reasonable root program can, in any way, apply a consistent standard to all CAs and allow such a situation. Please revoke these intermediates immediately. Please understand that this sort of situation is unacceptable, and it's one that's been communicated to DigiCert before

Flags: needinfo?(brenda.bernal)

Since this is coming through Mozilla, could we put it on OneCRL instead of revoking? Or is the lack of audits dating back to pre-acquisition related to sMIME sufficient that the CAs have to be revoked?

Regardless, we will still provide the other information requested about the PKI hierarchy for the rest of the operations.

For these three subCAs, there appears to be no TLS issuance, and they are valid for S/MIME, so OneCRL is neither a prudent or sufficient form of revocation.

Flags: needinfo?(wthayer)

Right, but the compliance issue is the CA failed to obtain a BR audit starting back when they were controlled by Symantec. They've had a valid and continuous Webtrust audit. Adding the CA to OneCRL cures the lack of a BR audit by essentially constraining the CAs to s/MIME. OneCRL effectively remediates the non-compliance, which is that the Sub CA should have been technically constrained to sMIME when originally created.

The idea of adding these Apple intermediates to OneCRL to remediate the problem by constraining them to S/MIME does make sense to me, and there is some precedent for this, e.g. bug #1398240.

I am concerned about this idea being interpreted as OneCRL being a more generally valid remediation mechanism beyond this specific scenario, because (as I suggested above) it is not. However, I also expect this specific scenario to be repeated as we look more closely at subCA audits and discover other subCAs with inadequate audits resulting from invalid "legacy" interpretations of our policy, just as we have in the past seen arguments that certificates aren't "intended for" some use even though they are "capable of".

Jeremy: If desired, Mozilla will consider a request to remediate the BR audit issue with these certificates by adding them to OneCRL. Please confirm that this is the approach DigiCert and Apple would like to take.

Flags: needinfo?(jeremy.rowley)

Yes please - adding to OneCRL would be great. Acknowledged that this shouldn't be general remediation mechanism.

Flags: needinfo?(jeremy.rowley)

Wayne: At what point do you expect that "legacy" interpretations of the policy should be corrected? Given that, for five years, DigiCert has affirmatively responded that they understood the expectations, specifically September 2018, January 2018, November 2017, April 2017 and May 2014, in addition to countless clarifications on m.d.s.p., it's unclear when Mozilla would view this as no longer acceptable or ambiguous. As noted in Comment #0, been something that DigiCert has been repeatedly informed as to expectations.

Flags: needinfo?(wthayer)

If I can just add some context/background,

As it relates to the Apple SubCAs, these were part of legacy Symantec hierarchy that was acquired by DigiCert in November 2017. They were covered in a WTCA audit since 2017. They were missed as technically capable of issuing TLS as they were always referred to as email CAs and have only issued S/MIME. There was a missed understanding of the inherited trust from the Root to which they chain. This was the point of their prior period audit which would confirm that these SubCAs have not issued TLS. These showed up as needing a BR audit on the crt.sh/disclosures list from Rob Stradling which kicked off a deeper dive and remediation planning.

As mentioned in our post here (https://bugzilla.mozilla.org/show_bug.cgi?id=1563573#c30), we have to break away from the incorrect interpretation bias and not rely on legacy knowledge about this PKI by relying on source data (e.g. Mozilla trust store bits) to drive decision making and analysis. This is exactly what we are aiming to do.

The question is not about misunderstanding the requirement, but how we deal with the remediation which we will address shortly here.

Flags: needinfo?(brenda.bernal)

From Mozilla's perspective, the following two actions are sufficient regarding the Apple IST 5,6,7 certs:

  1. Add them to OneCRL (in progress)
  2. Provide an Incident Report (bug #1575125)

Thanks,
Kathleen

PS: This is consistent with how we have been handling this type of situation so far, and I believe we are going to uncover some more of these typese of situations as we continue to extend automation and checks to the intermediate certificate level. This is indeed an area that I have been working on, but I think we still need a bit more time before being able to draw the hard line of requiring actual revocation of this type of cert for all CAs.

(In reply to Ryan Sleevi from comment #40)

Wayne: At what point do you expect that "legacy" interpretations of the policy should be corrected? Given that, for five years, DigiCert has affirmatively responded that they understood the expectations, specifically September 2018, January 2018, November 2017, April 2017 and May 2014, in addition to countless clarifications on m.d.s.p., it's unclear when Mozilla would view this as no longer acceptable or ambiguous. As noted in Comment #0, been something that DigiCert has been repeatedly informed as to expectations.

To be clear, subCAs lacking proper audits is a situation that is not, and has not been acceptable for some time. As Kathleen noted, we are unfortunately uncovering more of these problems as we automate intermediate certificate audit letter validation, and I don't think we'll be able to confidently say that we have detected all of these problems until that is completed.

What I think is in dispute here are the acceptable ways to remediate the problems. I think we're in agreement that missing audits going more than a year back can't be "fixed". That does lead to the conclusion that the intermediate certificate(s) in question must be revoked. However, if the only missing audit covers TLS, then it's reasonable to only revoke the certificate for TLS usage, which is of course exactly what OneCRL accomplishes.

Flags: needinfo?(wthayer)

Update: Apple IST CA 6 and 7 were revoked on Sept 12th. Here are the links:
https://crt.sh/?id=19602724
https://crt.sh/?id=19602741

Apple IST CA 5 will be revoked next week.

An update on Comment 33 above, here's our progress:
• Stage 1 - Our external SubCAs – completed; data mapping aligns to our info in CCADB and our internal source database
• Stage 2 - DigiCert hosted CAs (which includes legacy Symantec/VeriSign) - targeting Sept 27
• Stage 3 – QuoVadis CAs and roots - targeting Oct 4

An update on Comment 33 above, here's our next progress report:
• Stage 2 - DigiCert hosted CAs (which includes legacy Symantec/VeriSign) - completed; we have had the 3rd party audit firm review the data we've compiled for this portion of the Master PKI list; due to the size of this review, we are correlating their notes with our information confirming that the there are no gaps with our master data source.
• Stage 3 – QuoVadis CAs and roots - still targeting completion by Oct 4 - although some initial review results are being addressed in a separate bug (https://bugzilla.mozilla.org/show_bug.cgi?id=1581597)

An update on Comment 33 above, here's our next progress report:
• Stage 3 – QuoVadis CAs and roots - Completed a deep reconciliation of our EBJCA system, CCADB extract and WTCA (based on SHA hashes) as part of this overall PKI effort. We continue to work through some of the earlier discoveries in our bug mentioned in Comment 46.

Update: For transparency, we are disclosing that we had 3 ICAs that were created on October 16, 2019 and were not loaded in CCADB until today, October 31 2019. These ICAS were not released to the customer and have not issued any end entity certificates. I will publish the crt.sh links for the revoked ICAs once they become available. In the meantime, here are the serial information for each ICA:

DigiCert Extended Validation CA-2 0B6BD03B0892E8FCB4CCEF5CF1143ECB
DigiCert SHA2 Extended Validation Server CA-2 022C59485E390D8B829F68218C63C463
DigiCert Global CA-2 G2 0D23474192830C8D307AF9FDEB7B3127

We are still investigating why the process step was missed for loading these to CCADB within the 7 day required disclosure time frame. We are automating the key ceremony process as much as possible as this was identified as high risk area during the recent Mozilla discussion.

(In reply to Brenda Bernal from comment #48)

Update: For transparency, we are disclosing that we had 3 ICAs that were created on October 16, 2019 and were not loaded in CCADB until today, October 31 2019.

Brenda: this sounds like a different issue (failure to disclose within a week versus incomplete/incorrect disclosure) than the subject of this bug. If that's correct, I'd suggest you open a separate bug and provide a full incident report.

Flags: needinfo?(brenda.bernal)

Hi Wayne, I believe this is the same issue as Ryan reported originally here (https://bugzilla.mozilla.org/show_bug.cgi?id=1563573#c0) plus the fundamental issue is the same. The root cause is a failure to implement the remediation specified herein by the DigiCert PKI team in Lehi, which lead to the lapse in reporting. We thought we should address it here because the root cause is interconnected with the same resolution, although we want to supplement our proposal to resolve the issue with additional technical controls. We've mentioned on the discussion forum that the manual nature of key ceremonies are one of the biggest risks most CAs face since they lack a lot of the routine processes found in regular cert issuance. We feel this needs to be remediated with controls enhancements. Although we believe the controls previously proposed would have been sufficient to prevent a lapse in reporting if implemented properly, we want to take additional steps to eliminate manual processes and will report on that as well. We believe this, including the automation we have in mind, can help all CAs achieve better and more consistent reporting. I think Jeremy has already reached out to Kathleen to find out how to establish integration with CCADB so that all CAs can be automatically uploaded when a key ceremony is complete. Once we can do that with the key ceremony script and combine that with a linter, the human elements responsible for a majority of our issues (I think we count 35 among the CAs) can be eliminated..

Moreover, we've noted that we need to ensure that new policies and practices are fully implemented. Although we do have internal audits to catch these types of findings and trainers responsible for the policy propagation, there's a conundrum on what to do when not all teams in every location are following the policy. We had caught the lapse in process during an internal audit. However, the root problem is one location did not fully implement all parts of the new policy/procedure since we rolled out as Jeremy described here: https://bugzilla.mozilla.org/show_bug.cgi?id=1563573#c7. Because of this, our bugzilla report needs to talk about how we are going to ensure policies and procedures are effectively implemented and monitored. As mentioned, the remediation is the same, so this next portion of the report would be to address how we ensure implementation and monitoring of the already disclosed new policy, and with wider use of automation.

Flags: needinfo?(brenda.bernal)

Brenda: my understanding is that comment #0 refers to incomplete disclosures (cert was added to CCADB within 7 days but all of the necessary information was not provided or was incorrect), and that the 3 certs in comment #49 were not disclosed within one week. Are you saying that I'm wrong about one of these two, or are you saying in comment #50 that they are the same issue? I'm arguing that "I did it wrong" is not the same as "I forgot to do it when I was supposed to".

Flags: needinfo?(brenda.bernal)

Wayne: I viewed the two issues having the same root cause (lack of full implementation of the proper governance process) which is why I thought we could combine. I understand that there are distinct reasons with each instance but my thought process for combining was more focused on how do we holistically solve for these issues with the same framework design for greater oversight and automation.

Flags: needinfo?(brenda.bernal)

The root cause on these items is the dependency on staff to remember to upload this information. History has shown (based on the JOI issue and this repeat problem) that we cannot rely on something so manual as generic key ceremonies. We are currently working on code that will eliminate as many manual steps as possible. Unfortunately, with both key ceremonies and HSMs being stored offline, we can't fully automate the process - just parts of it.

DigiCert has an internal database similar CCADB. This database contains both private and public certs. One of the first things we are doing is scripting a notification system that will compare our internal DB with CCADB records. If anything is missing between the two, a notification is sent to both the PKI ops team and the compliance team. This is much more than our current approach, which is to rely on two people to review each key ceremony and post the certs as part of the post-key ceremony clean up. We do recognize there are additional steps that we can take to automate key ceremonies so we're going to continue to explore those after we get the notifications working. We'd love suggestions from the community.

Jeremy: is there a timeline for implementing the "scripting notification system"?

Flags: needinfo?(jeremy.rowley)

The code is actually already deployed - we're just doing testing now. We have a key ceremony next week and will use that to make sure it works.

Flags: needinfo?(jeremy.rowley)

We had a key ceremony but people uploaded teh ICAs too fast to test the code. It runs once a day and the ICAs were uploaded before it ran. We're going to try it again with the next key ceremony.

Jeremy: Not to be a pain, but Comment #56 sounds a bit like "The key ceremony didn't run as scripted".

Now, I may be misunderstanding, and that it was scripted to run and not test the control. But Comment #55 said you were going to make sure it works then, so I would have thought that 'wait' was part of the key ceremony script?

I know key ceremonies are hard and full of high stakes, but my hope is that the clear scripts and dry runs put everyone at ease, so I'm not sure why there was an issue with testing here, which seems like the same systemic root cause as what got us here in the first place.

Flags: needinfo?(jeremy.rowley)

Sorry, to be more accurate - the key ceremony script ran just fine. What we couldn't test is the code we deployed to check whether everything was uploaded from our CCADB-like internal database to the actual CCADB. The code runs nightly as a check to determine whether something was not uploaded after a key ceremony. We wanted to test failure (where we didn't upload a CA cert after the key ceremony) but couldn't do so during the last key ceremony. The process was followed and the CA cert was uploaded before script ran. Thus, we couldn't make sure that the failure case was working. We want to test that before closing the bug.

Flags: needinfo?(jeremy.rowley)

Sure, but I thought your last key ceremony procedure was scripted to test the failure, based on Comment #55. Was that not the case? When is the next key ceremony?

We never added the test to the script unfortunately. We only talked about delaying upload long enough for the script to run, but never wrote it into the actual key ceremony script. I've been trying to find the next key ceremony for public certs, but we don't have one scheduled right now.

Jeremy: Have you been able to test the script yet? Is this bug part of your triage review for updates?

Flags: needinfo?(jeremy.rowley)

The bug is part of our triage, but we haven't had a key ceremony. We've tested the script, but not with a live key ceremony (just general testing).
I wanted to test with a live key ceremony before we closed the bug completely. The next ceremony is currently scheduled for tomorrow (was scheduled for today). I'm hoping to close this right after that happens.

Flags: needinfo?(jeremy.rowley)

We finally had another key ceremony today and the script worked great. I think we're ready to close this bug.

It appears that all questions have been answered and remediation is complete.

Status: ASSIGNED → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
Product: NSS → CA Program
Whiteboard: [ca-compliance] → [ca-compliance] [disclosure-failure]
You need to log in before you can comment on or make changes to this bug.