Closed Bug 1626868 Opened 4 years ago Closed 4 years ago

WISeKey: Issuance of certificate with two potentially conflicting policy OIDs

Categories

(CA Program :: CA Certificate Compliance, task)

task
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: pfuentes, Assigned: pfuentes)

Details

(Whiteboard: [ca-compliance] [dv-misissuance])

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_2) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0.4 Safari/605.1.15

Steps to reproduce:

Hello.
We don't consider this a clear misissuance, but we are disclosing it for the shake of transparency.

1.How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.
28.3.2020 - 15:30 CET: We received a mail sent to our notifications address cps@wisekey.com
This mail made us aware of a possible misissuance in a certificate issued with two Policy OIDs that were potentially conflicting.

  1. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.
    28.3.2020 - 15:30 CET: The notification is sent, making reference to one certificate:
    https://crt.sh/?id=1719241697
    28.3.2020 - 20:00 CET: We analyze the issue. Determining that there's a misconfiguration in the certificate template for DV certificates. We immediatelly amend the template and we search for other occurrences, detecting other 3 internally-used certificates with the same issue. We don't consider this a real misissuance at this point, but we treat as it was, starting the revocation process.
    30.3.2020 - 14:00 CET: The internal certificates are confirmed to be revoked
    31.3.2020 - 19:30 CET: The initially reported certificate is revoked

  2. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.
    This problem is not recurrent and we checked that it didn't happen beyond the detected cases, not having found a similar issue.
    The necessary steps to avoid similar issues have been defined. See item #7 for details.

  3. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.
    There are four certificates issued with this problem. All are DV certificates.

  4. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.
    https://crt.sh/?id=1719241697
    https://crt.sh/?id=2603823719
    https://crt.sh/?id=2604570473
    https://crt.sh/?id=2604532300

  5. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
    When we configured initally the certificate template in the CA software, we added two Policy OIDs in OV and DV certificates:

  • The appropriate OID defined by the CABF for the type of certificate (DV or OV)
  • A generic custom OID to designate our SSL certificates (single OID for all non-EV SSL certificate types, that would apply to OV and DV certificates)
    Some time later we decided to define separate custom OIDs for DV and OV certificates. Defining a new OID specifically for DV.
    We missed to adapt the DV template, so it was left with two OIDs that could be considered conflicting after updating the CPS.

We didn't detect it ourselves because our linters check only the CABF OIDs to determine the certificate type, but not our custom OID.

I'd like to point out that in our CPS we have a clause to resolve potential conflicts with the CABF BR. In section 2.2.1 “Statement of compliance with CA/Browser Forum Requirements”, we stipulate that:
“In the case of discrepancy of any certification practices with the stipulations of the CAB/Forum requirements, it must be understood that those requirements must prevail to the CP and CPS documents."

This means that in this particular case for two potentially conflicting OIDs, it must be understood that the CABF OID clearly prevails over our custom OID. This means that the certificate must be univocally understood as DV, which indeed matches the certificate type. This is why we don't consider this a clear missuance but a quality issue. In any case, we decided to treat it as a misissuance, revoking the certificates and making this incident report.

  1. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.
    We reviewed the rest of certificate templates to detect similar issues, not finding any.
    To avoid the iteration of this problem, we added a note in our internal process to ensure that a change in the OID list doesn't have impact in existing certificate templates already configured in the CA systems.
    We don't foresee further action.
Summary: WISeKey Incident report. Issuance of certificate with two potentially conflicting policy OIDs → WISeKey: Issuance of certificate with two potentially conflicting policy OIDs

Pedro: can you expand your answer to questions 6 and 7 to provide more detail on how this happened and how it will be prevented in the future? Why weren't existing profiles considered when the DV-specific OID was defined and added to the CPS? What is the internal process for changing an OID?

Assignee: wthayer → pfuentes
Status: UNCONFIRMED → ASSIGNED
Type: defect → task
Ever confirmed: true
Flags: needinfo?(pfuentes)
Whiteboard: [ca-compliance]

(In reply to Wayne Thayer from comment #1)

Pedro: can you expand your answer to questions 6 and 7 to provide more detail on how this happened and how it will be prevented in the future? Why weren't existing profiles considered when the DV-specific OID was defined and added to the CPS? What is the internal process for changing an OID?

Hello Wayne,
we weren't issuing DV certificates before, and that's why we didn't have the specific OID to differentiate DV and OV, so initially we used the one defined by CABF.

The OIDs are defined now by the OISTE Foundation, which is the owner of the Roots, and published in the Root CPS and CP. WISeKey, as subCA is bound to adopt these OIDs.

When we did all the refactoring of the CP/CPS implied by the change of ownership of the roots this also motivated a review of the OID inventory. Unfortunately we missed to change this DV template after it.

As mitigation measure. As explained in the report, we introduce now an step in our procedure of CPS review, so in case there's any change of the OID there's a formal need to review possible impact in existing certificate profiles.

Flags: needinfo?(pfuentes)

(From Comment #0)

This is why we don't consider this a clear missuance but a quality issue. In any case, we decided to treat it as a misissuance, revoking the certificates and making this incident report.

For clarity sake, I think we consider any divergence from the CP/CPS to be a misissuance, and so definitely appreciate that you decided to err on the side of revoking. While I realize you have the clause noting the BRs supercede, the "misissuance" is that you asserted a policy OID but did not follow the practices in the CP/CPS described for that policy OID. The CABF OID helps to identify this, for relying parties, but the root failure was the "quality issue" which lead to a misissuance.

(From Comment #2)

As mitigation measure. As explained in the report, we introduce now an step in our procedure of CPS review, so in case there's any change of the OID there's a formal need to review possible impact in existing certificate profiles.

I can appreciate that this treats the symptom of the problem (related to OIDs), but I'm concerned it overlooks the systemic root cause. The root cause would suggest that a process failure/breakdown happened in which a change to the CP/CPS was made, and the system was not re-evaluated. OIDs are just one of a myriad ways in which a CP/CPS can be changed, and yet have impact or require changes on a certificate profile and/or line practices.

I think it's useful to understand, and to analyze, what systemic controls are in place to ensure that changes to a CP/CPS change are entirely reflected in the system, and, similarly, that changes to the system are accurately presented and reflected in the CP/CPS. This applies to profiles and OIDs, but more broadly, to all of the CA practices.

Understanding how changes to the CP/CPS are made and reviewed, for example, would help bring clarity here.

Flags: needinfo?(pfuentes)

Hello Ryan,
as always, thanks for your comments.
My answers below.
Best,
Pedro

(In reply to Ryan Sleevi from comment #3)

(From Comment #0)
(...) but the root failure was the "quality issue" which lead to a misissuance.

Yes, we assumed it and that's why we made revocations and this incident report, even if I think that the certificates were still valid as DV certificates.

(From Comment #2)
(...) Understanding how changes to the CP/CPS are made and reviewed, for example, would help bring clarity here.

This is documented in the CPS, explaining the role of the "Policy Approval Authority" (PAA) and the rules to accept minor and major version changes... but I must humbly tell you that I don't think this was the root cause, because the CPS update process itself was respected... What we missed is to follow a formal and properly stipulated "post-CPS update" impact assessment to ensure that any configuration change motivated by the new CPS (in this case the OID list update) was fully considered, applied and properly documented. This is what I responded to Wayne as new mitigation control.
What happened was a rare and unfortunate case (due basically to the fact that at that moment the DV template was disabled in our system, as we weren't issuing this type of certs), but we think that the new item in our change management process will avoid this case or similar cases to repeat.

Please let me know if you have further concerns. Even if I hate to report incidents, these exchanges always help us to improve...

Flags: needinfo?(pfuentes)

(In reply to Pedro Fuentes from comment #4)

(From Comment #2)
(...) Understanding how changes to the CP/CPS are made and reviewed, for example, would help bring clarity here.

This is documented in the CPS, explaining the role of the "Policy Approval Authority" (PAA) and the rules to accept minor and major version changes... but I must humbly tell you that I don't think this was the root cause, because the CPS update process itself was respected... What we missed is to follow a formal and properly stipulated "post-CPS update" impact assessment to ensure that any configuration change motivated by the new CPS (in this case the OID list update) was fully considered, applied and properly documented. This is what I responded to Wayne as new mitigation control.
What happened was a rare and unfortunate case (due basically to the fact that at that moment the DV template was disabled in our system, as we weren't issuing this type of certs), but we think that the new item in our change management process will avoid this case or similar cases to repeat.

I guess I was thinking of this more holistically. That is, I understand the role of the PAA that you have documented, but I think you rightfully highlighted was the lack of a "post-CPS update" assessment and change.

To me, those seem like all part of the same process, in that the CP/CPS and the operational practices go hand in hand, because the former describes the latter.

For example, I was thinking of the CP/CPS update practices to include:

  • The process of evaluating that, before any change is made to a system (e.g. a profile change, an operational change), a review happens to ensure that the change is compliant with the CP/CPS. For example, if a new profile is being proposed, there's a review to ensure it complies with the CP/CPS.
    • If it doesn't, then before that change is made, the CP/CPS is updated and published, and then the change is made, and then after the change is made, a review to make sure the change was what was described
  • The process of evaluating that, before any change to a CP/CPS, any operational changes that need to be made are also made.
    • It sounds like this is what broke down, in which you have the process for reviewing the update process from the PAA, but nothing to reconcile that when those changes are made, the system and configuration is examined to make sure it complies

That's why I was trying to get to understanding how the CP/CPS changes are made, since presumably they're made when configuration/practices change, and now with your new mitigation, configuration/practices change when they do.

This is also a potentially complex area for interpretation, and that's why I was trying to understand WISeKey's approach. That is, consider a CP that says "Our certificates look like X", and you'd like them to look like Y.

One interpretation is:

  • Configure certificates from looking like X to look like Y
  • Invoke the CP/CPS update process

However, if these aren't done simultaneously, you could have certificates looking like Y before the CP/CPS said they would look like Y.

Another interpretation is:

  • Update the CP/CPS to say they look like Y
  • Make the configuration change

Similarly, if these aren't done simultaneously, you could end up in a situation where the CP/CPS says they look like Y, but they still look like X.

These are extreme examples, mind you, and we've seen a variety of approaches, such as halting issuance while the two are out of sync, or using separate CPS URLs and policy OIDs, to making updates to CP/CPSes that first say "X", then say "X or Y", and then say "Y".

These all broadly fit the class of figuring out how the CP/CPS and the configuration align, and that's why I was trying to understand how WISeKey manages this.

Flags: needinfo?(pfuentes)

Hello Ryan,
let me try to add some more infos on this topic...

(In reply to Ryan Sleevi from comment #5)
(...)

Another interpretation is:

  • Update the CP/CPS to say they look like Y
  • Make the configuration change

The short answer is that when a change in the certificate profile is required by a change in the policy or the practices, then this is the process to follow: First the document update, then the configuration change.

(...)

These all broadly fit the class of figuring out how the CP/CPS and the configuration align, and that's why I was trying to understand how WISeKey manages this.

Actually, if you want me to answer more "holistically", we should consider the complexity of our trust model and documentation framework...
If you remember, we recently transferred the ownership of the Roots to the "OISTE Foundation", and that had multiple implications, also in the way that CP and CPS are defined and maintained.
Before, there was WISeKey, and we had a single CPS document... That was a quite common situation compared to other CAs.
Now, we have:

  • The OISTE Foundation, owner of the Roots and ruler of the Trust Model
  • WISeKey, that acts as a subordinated CA under the OISTE Roots, and adhering to the Trust Model

When redefining the Trust Model to adopt this reality, I had to find a way to structure the documentation framework in a way that matched the trust model, being the chosen model:

  • OISTE defines the CP (separate CP documents for each type of certificates, considering right now CP for Personal, CP for SSL and CP for IoT objects)
  • OISTE defines a "Root CPS", that includes the stipulations of the Roots and the rules to be applied to participants in the Trust Model, being WISeKey the main participant
  • WISeKey defines its "Subordinate CPS", where discloses the practices related to how the stipulations of the Root CPS and the different CP are meet

This has several implications on how the documentation framework changes over time and the different combinations we could find, like:

  • OISTE could change its CPS or CP, in a way that implies a change in WISeKey's CPS
  • OISTE could change the CPS or CP, in a way that WISeKey doesn't need to change its CPS

Imagine that OISTE changes the CP for SSL, changing the maximum validity period (something probably happening soon), but WISeKey's CPS stipulates that the maximum validity period is the one specified in the OISTE CP for SSL... In this situation, it can happen that WISeKey didn't need to change its CPS, but it's still required to conduct a technical activity which is imposed by the change of a document issued by OISTE.

Before it was easier to manage, as we had a simple CPS maintained by WISeKey. Now the process is more complex due to the split of the trust model and we had to build new procedures and methods.

In our case, the PAA of OISTE that regulates the Root CPS and the CPs has members of WISeKey, and vice-versa, so we are always always involved and aware of any change in the documentation framework, but what we lacked is a formal procedure to trigger a technical assessment when ANY document of the framework is modified, to detect the possible impact in existing configurations, that were correct before the change but that could present issues after it.

Flags: needinfo?(pfuentes)

Thanks for the detailed explanation.

So to make sure I understand, when you originally said:

(In reply to Pedro Fuentes from comment #2)

As mitigation measure. As explained in the report, we introduce now an step in our procedure of CPS review, so in case there's any change of the OID there's a formal need to review possible impact in existing certificate profiles.

This is more formally a process that if either the OISTE or WISeKey CP/CPSes changes, there's a formal evaluation of changes needing to be made to any configuration, and any configuration changes are similarly reflected?

Flags: needinfo?(pfuentes)

Hello Ryan

(In reply to Ryan Sleevi from comment #7)

This is more formally a process that if either the OISTE or WISeKey CP/CPSes changes, there's a formal evaluation of changes needing to be made to any configuration, and any configuration changes are similarly reflected?

Yes, but let me rephrase...
The new formal item in the process, triggered after the modification of either the OISTE or WISeKey CP/CPSes, is to do an assessment to identify any required configuration change. Then each detected intervention is executed separately as per our change management process, which includes the typical approval and documentation activities.

Flags: needinfo?(pfuentes)

Thanks for clarifying!

Wayne: I wasn't sure how you felt about the added details with regards to your question in Comment #1, but I don't have any further follow-up questions.

I'm concerned that this wasn't already part of the process, as it seems like it should have been critical to do, both when same organization or with the split. It's good that it's been corrected, at least.

Flags: needinfo?(wthayer)

My questions have also been answered, and it appears that remediation is complete.

Status: ASSIGNED → RESOLVED
Closed: 4 years ago
Flags: needinfo?(wthayer)
Resolution: --- → FIXED
Product: NSS → CA Program
Whiteboard: [ca-compliance] → [ca-compliance] [dv-misissuance]
You need to log in before you can comment on or make changes to this bug.