1558552 - SwissSign: CP/CPS certificate profile issue

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Mike Guenther]

Attachment: Certifcates.xls

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:60.0) Gecko/20100101 Firefox/60.0

Steps to reproduce:

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

In Bug 1551364 "SwissSign: "Some-State" in stateOrProvinceName", Ryan Sleevi reported potential issues in relation to OIDs in the CP/CPS of SwissSign Gold PKI (https://bugzilla.mozilla.org/show_bug.cgi?id=1551364#c2).

In the subsequent internal analysis SwissSign found the following:
• The SwissSign Gold CP/CPS with OID 2.16.756.1.89.1.2.1.7 is missing on the online archive under https://repository.swisssign.com and is only available in the internal archive.

• For the SwissSign Gold CP/CPS with OID 2.16.756.1.89.1.2.1.7, the certificate profile for end user certificates for G22 in section 7.1.2 f) was still 2.16.756.1.89.1.2.1.6 instead of 2.16.756.1.89.1.2.1.7
• For the SwissSign Gold CP/CPS with OID 2.16.756.1.89.1.2.1.8, the certificate profile for end user certificates for G22 in section 7.1.2 f) was still 2.16.756.1.89.1.2.1.6 instead of 2.16.756.1.89.1.2.1.8

• By applying the certificates profiles out the CP/CPS correctly, SwissSign issued certificates with a reference to an OID that did not match the applicable valid CP/CPS for these certificates. Concretely:

• between 14.09.2017 - 12.10.2017 under the wrong OID 2.16.756.1.89.1.2.1.6 instead of 2.16.756.1.89.1.2.1.7.
• between 12.10.2017 - 16.07.2018 under the wrong OID 2.16.756.1.89.1.2.1.6 instead of 2.16.756.1.89.1.2.1.8.
  • Note: No EV certificates are affected by this issue

Because of the above findings we additionally checked the SwissSign Silver product line which resulted in similar findings
• The SwissSign Silver CP/CPS with OID 2.16.756.1.89.1.3.1.8 is missing on the online archive under https://repository.swisssign.com and is only available in the internal archive.

• For the SwissSign Silver CP/CPS with OID 2.16.756.1.89.1.3.1.7, the certificate profile for end user certificates for G22 in section 7.1.2.6 was still 2.16.756.1.89.1.3.1.6 instead of 2.16.756.1.89.1.3.1.7.
• For the SwissSign Silver CP/CPS with OID 2.16.756.1.89.1.3.1.8,

• the title page showed for this CP/CPS the OID is 2.16.756.1.89.1.3.7 instead of 2.16.756.1.89.1.3.8 in contrast to section 1.2
• the certificate profile for end user certificates for G22 in section 7.1.2.6 was still 2.16.756.1.89.1.3.1.6 instead of 2.16.756.1.89.1.3.1.8
  • By applying the certificates profiles out the CP/CPS correctly, SwissSign issued certificates with a reference to an OID that did not match the applicable valid CP/CPS for these certificates. Concretely:
• between 15.09.2017 - 16.10.2017 under the wrong OID 2.16.756.1.89.1.3.1.6 instead of 2.16.756.1.89.1.3.1.7.
• between 16.10.2017 - 28.06.2018 under the wrong OID 2.16.756.1.89.1.3.1.6 instead of 2.16.756.1.89.1.3.1.8.

As part of the risk impact analysis, all changes in the concerned CP/CPS were revisited. All changes were related either to a change in regulations not directly impacting the issued certificate (i.e. CT Log and CAA) respectively to an audit finding requiring a statement on used key length and algorithms for the subscriber’s QCSD in the CP/CPS. Therefore, the risk analysis did not show any security impact for our customers nor for the community by further usage of these certificates. The detailed impact was shared with TÜV Trust IT to confirm our interpretation. We did not identify any relevant changes that would put the customer at risk by usage of the impacted certificates.

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

[May-14-2019 Post Ryan Sleevi] Notification of potential OID divergence in one of the CP/CPS Gold documents
[May-15-2019] Start of validation of potential issue
[May-17-2019] SwissSign validates the issue in the 2 CP/CPS Gold mentioned above.
[May-20-2019] Confirmation of issue on a partial set of certificates and decision to initiate a full analysis. Risk impact analysis recorded a formal issue triggering a revocation of the concerned certificates, no security impact for our customers nor for the community by further usage of these certificates was identified.
[May-21-2019] Information of auditor (TüV Trust IT) about issue.
[May-23-2019] Management Board acknowledged the issue and the associated risks.
[May-27-2019] First results of analysis syndicated and validated on Gold product line. Decision to extend analysis to Silver product line.
[May-29-2019] Results confirmed for Gold product line scope, information to the management board of the additional extend, information of the auditor about the need to post a misissuance report to Bugzilla.
[June-04-2019] Confirmation that Silver product line is also in scope of issue, Information of auditor about the additional scope
[June-07-2019] Proposal for staged revocation plan of the concerned 72430 certificates until end of 2019 agreed SwissSign internally, discussed with auditor for agreement
[June-11-2019] Post on Bugzilla

3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

Yes, the currently issued certificates have the correct OID. All CP/CPs after July 16, 2018 on Gold product line and after June 28, 2018 on Silver product line have the correct OID in the certificate profile in section 7.1.2.

4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

Our two related roots for SwissSign's SSL and S/MIME certificate products are concerned:
The silver product line issued

1. between 15.09.2017 - 16.10.2017 under the wrong OID 2.16.756.1.89.1.3.1.6 instead of 2.16.756.1.89.1.3.1.7.
2. between 16.10.2017 - 28.06.2018 under the wrong OID 2.16.756.1.89.1.3.1.6 instead of 2.16.756.1.89.1.3.1.8.
   The gold product line issued
3. between 14.09.2017 - 12.10.2017 under the wrong OID 2.16.756.1.89.1.2.1.6 instead of 2.16.756.1.89.1.2.1.7.
4. between 12.10.2017 - 16.07.2018 under the wrong OID 2.16.756.1.89.1.2.1.6 instead of 2.16.756.1.89.1.2.1.8.

In total, 72430 certificates as per June 11, 2019 are concerned.

5. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

In the attached table, we only list the SSL certificates, the S/MIME ones are excluded for privacy reasons.

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

Section 7.1 (Certificate Profiles) in CP/CPS had the wrong OID for leafs certificates.
Our analysis was not able to pinpoint a single event/task to explain the issue. As the update of the CP/CPS is a tedious manual process, we assume that the four eyes check updating the OID-parameter failed due to human error.
We therefore looked at the entire process of managing the CP/CPS and identified the following weaknesses:

• Change Control on the variable parameters is difficult to implement in the process as it has limited support in the used tooling (i.e. word editor). Variable parameters such as the OID in the certificate profile are prone to copy-paste errors in the process.
• Dual control tasks are not part of an end-to-end checklist which makes human error more likely.

7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

SwissSign will execute a staged revocation of the concerned 72430 certificates until end of 2019.
SwissSign has already taken steps to address the introduced weaknesses. We will parameterize future CP/CPS in a way that parameters such as OID are centrally set in a template with no update possibility in the individual CP/CPS. The parameter fields will be constraint by rules for valid changes. In addition, we also enhance our current end-to-end checklist for the different process steps with special attention to dual control tasks.

The text above will also be posted at mozilla.dev.security.policy

Comment 1

[Mike Guenther]

Link to MDSP: https://groups.google.com/forum/#!topic/mozilla.dev.security.policy/83HOFvRBZEs (also with better readable formatting)

Comment 2

[BugBot [:suhaib / :marco/ :calixte]]

The priority flag is not set for this bug.
:kwilson, could you have a look please?

For more information, please visit auto_nag documentation.

Comment 3

[Wayne Thayer]

Mike: thank you for this report, and for posting it to m.d.s.p.

Despite this incident, I really like the fact that SwissSign attempts to clearly link a certificate to the specific applicable policies.

Mozilla has a specific set of expectations for CAs when they choose to violate the BR revocation requirements. These expectations are listed at https://wiki.mozilla.org/CA/Responding_To_An_Incident#Revocation Please review this and provide the information, including an explanation for why this situation is exceptional, and the results of you analysis of how these delays will be prevented in the future.

Also, please update this bug regularly on the progress being made with revocations.

Comment 4

[Mike Guenther]

Dear Wayne
We gladly provide more information on the decision to follow this exceptional approach in revocation for the misissued certificates. First of all, we want to stress that we only considered this approach because the misissuance does not affect the secure usage of the affected certificates as explained in our first post.

Furthermore, the timeline has the following reasoning:

• There are around 7'000 customers affected.
• We feel obliged to give additional support to our customers in this matter as the fault was made on our side.
• Several clients affected need to replace a large number of certificates (in the hundreds and thousands) in one go.
• We know that the certificates are used in servers and other infrastructure components which needs expert knowhow for replacement on the customers side.
• Our risk analysis resulted that a sudden revocation would do more damage to our customers compared to a longer revocation period.

As our auditors were involved from the beginning they are aware of the issue. The remediation status as well the measures for avoiding such issue in the future will be verified in the upcoming onsite audit in September 2019 and the issue will be listed accordingly in the next annual BR audit statement.

The detailed revocation plan is regularly updated according to the needs and capacity of our customers (e.g. Swiss Summer holidays).

I will give regular updates on our revocation efforts.
Thanks Mike

Comment 5

[Ryan Sleevi]

Mike: Thanks for the update, but I think there's still very important details being omitted here. I think the most important thing to highlight is this:

• Any decision to not comply with the timeline specified in the Baseline Requirements must also be accompanied by a clear timeline describing if and when the problematic certificates will be revoked or expire naturally, and supported by the rationale to delay revocation.

Similarly, I want to make sure to draw attention to this other expectation:

• That you will perform an analysis to determine the factors that prevented timely revocation of the certificates, and include a set of remediation actions in the final incident report that aim to prevent future revocation delays.

From your response, it's unclear that an actual timeline is provided - or if the expectation is that you won't provide a timeline until September, or if it remains "End of 2019", per Comment #0. Similarly, it's unclear what the plan to address those issues highlighted in Comment #4. That is, if SwissSign were to make another mistake, would it again delay revocation because the fault was on their side? What steps are being taken to ensure that clients are prepared to replace large number of certificates - such as thousands - within the BR allocated timeframe?

Your responses as to the issues don't seem that distinct from, say, Bug 1534295 or Bug 1515564. A critical part of understanding when a CA violates the BRs is what, systemically, they're doing to prevent further violations. As Wayne (in Comment #3) and I (in https://bugzilla.mozilla.org/show_bug.cgi?id=1551364#c2 ) mentioned, SwissSign's use of specific OIDs in policies actually made it much easier to understand and evaluate y'alls issuance practice, and I think that's very much a mitigating factor in terms of the incident cause and response to the policy OID matter. With respect to the (related) BR violation of the revocation period, we want to make sure we understand what the CA is doing to prevent future revocation delays - in effect, similar to the process we went through in https://bugzilla.mozilla.org/show_bug.cgi?id=1551364#c2 / https://bugzilla.mozilla.org/show_bug.cgi?id=1551364#c3

If you examine the related DigiCert discussion, which lead to clarifications of the incident handling, you can see suggestions on approaches. Those may serve as starting points, but this is an opportunity to demonstrate to the community that meaningful steps are in place to prevent future revocation delays, regardless of the root cause.

Comment 6

[Ryan Sleevi]

Mike: Do you have an update?

Comment 7

[Timo Schmitt]

Hi Ryan

Sorry for the late reply. Mike is out of office. Therefore only a status update of our progress.

As per 16 July 2019 a total of 18294 certificates are corrected.

Mike will post a follow-up with details to your questions at the latest until Tuesday, 23 July 2019.

Thank you and best regards,
Timo

Comment 8

[Ryan Sleevi]

Thanks Timo!

Comment 9

[Mike Guenther]

Sorry for the delay. The information and numbers provided in this comment are in addition to the post of Timo.

After a detailed analysis of all in-scope certificates and customers we now are able to present a binding timeline for the rest (fulfilling one of the expectations for 'exceptional circumstances' revocations).

The following numbers of settled certificates are based on the presumption that our customers use the entire timeframe granted by SwissSign (-> minimal number of certificates settled):
Until 16 August 2019 5'273 certificates
Until 30 August 2019 28'924 certificates
Until 30 September 2019 3'248 certificates

In addition to the dates above our plan includes to let certificates with expiry before 01.01.2020 conclude the standard life-cycle and not to revoke them. These adds up to a total of 16'691 certificates from 17 July to 31 December 2019.

We will give a status update around the dates written above.
Thanks Mike

Comment 10

[Ryan Sleevi]

Wayne: To your evaluation around how well this responds to Comment #5 and the Mozilla Policies.

Comment 11

[Wayne Thayer]

Mike:

(In reply to Mike Guenther from comment #9)

Sorry for the delay. The information and numbers provided in this comment are in addition to the post of Timo.

After a detailed analysis of all in-scope certificates and customers we now are able to present a binding timeline for the rest (fulfilling one of the expectations for 'exceptional circumstances' revocations).

The following numbers of settled certificates are based on the presumption that our customers use the entire timeframe granted by SwissSign (-> minimal number of certificates settled):
Until 16 August 2019 5'273 certificates
Until 30 August 2019 28'924 certificates
Until 30 September 2019 3'248 certificates

In addition to the dates above our plan includes to let certificates with expiry before 01.01.2020 conclude the standard life-cycle and not to revoke them. These adds up to a total of 16'691 certificates from 17 July to 31 December 2019.

Will you please restate the timeline above in terms of how many of the 72K certificates will be revoked at each of these milestones?

We will give a status update around the dates written above.
Thanks Mike

Also, please provide a plan describing how future revocation delays will be prevented, as requested in comments 3 and 5, along with dates for when each part of the plan will be implemented.

Comment 12

[Mike Guenther]

Let me rephrase my last comment concerning timeline

1. At the latest on 16 August 2019: 5'273 certificates are either revoked by the customer or will be revoked by SwissSign
2. At the latest on 30 August 2019: 28'924 certificates are either revoked by the customer or will be revoked by SwissSign
3. At the latest on 30 September 2019: 3'248 certificates are either revoked by the customer or will be revoked by SwissSign

In addition to the revoked certificates above another 16'691 certificates will expire between 17 July and 31 December 2019.

These numbers including the number provided by Timo (18'294) adds up to the total of 72'430 certificates (see comment 0).
I hope this clarifies our revocation plan.

To the aspect of Ryan's comment as well as Wayne's comment
During the handling of this incident SwissSign recognized that our incident plans for mass revocation did not hold up to our quality expectations as we needed more time than planned. While we are technically capable for a mass revocation, we were not fully prepared on an organisational and business perspective level to handle the issue in the necessary time and quality.

As a consequence we developed and are still improving our technical customer support tools as well as organisational aspects for future revocation issues with the lessons learned in the last weeks.

Concerning the readiness of our customers we did get a lot of insight. Some examples
• Customers are often neither aware of the revocation times set by the BR nor are they aware that this is true for all CAs
• Customers sometime do not know the responsible person for a specific certificate (e.g. relevant for ordering a new certificate)
• Customer's management of certificates seems to be more often a manual task than an automated one even if there are 1'000+ certificates to be handled
• Non-SSL certificates are sometimes handled as a one-off action (project mode) and there are no operational plans to replace them on-the-go
• Some customers will be operationally disrupted by our timeline.

As a take-away SwissSign has already started an awareness campaign and will keep creating and distributing additional information to educate our customers of the guidelines provided by the CAB-community. We assume and are confident that this will trigger a big improvement of customer's operational processes for certificate handling. Which will finally result in a BR compliant timely revocation in the future.

All of the results and actions above are a preliminary report. We will provide a final incident report as requested in the Mozilla Wiki with our request for closure of this incident.

All of these points will be discussed with our auditors during their onsite audit.

Comment 13

[Mike Guenther]

The enhanced tools we developed to eliminate the OID error have been run on all of SwissSign's currently valid SSL and S/MIME certificates and revealed unfortunately an additional batch of misissued certificates with a wrong OID.

Concretely, on the silver product line
• the following number have been misissued with OID 2.16.756.1.89.1.3.1.9 instead of 2.16.756.1.89.1.3.1.10 between 19.10.2018 and 17.12.2018:
o 457 SSL certificates
o 19145 S/MIME Certificates
All certificates issued on the silver product line between 18.12.2018 and today have been validated to be issued correctly with regard to the OID, i.e. with OID 2.16.756.1.89.1.3.1.11.

On the gold product
• the following number have been misissued with OID 2.16.756.1.89.1.2.1.6 instead of 2.16.756.1.89.1.2.1.9 between 16.07.2018 and 19.10.2018:
o 78 SSL
o 19322 S/MIME Certificates

• the following number have been misissued with OID 2.16.756.1.89.1.2.1.9 instead of 2.16.756.1.89.1.2.1.10 between 19.10.2018 and 17.12.2018:
o 174 SSL certificates
o 2718 S/MIME Certificates

All certificates issued on the silver and gold product line between 18.12.2018 and today have been validated to be issued correctly with regard to the OID, i.e. with OID 2.16.756.1.89.1.2.1.11 resp. 2.16.756.1.89.1.2.3.11.
No EV certificates are affected.

Therefore, we must unfortunately correct the number of concerned certificates and add another 41’894 as of last Friday.
By decision of SwissSign, we will include the revocation of the additional certificates in the committed timeline, meaning 19’217 will be revoked by end of September 2019, and 22’677 will expire by end of December 2019.

Comment 14

[Wayne Thayer]

Mike: thank you for performing the analysis and reporting these additional misissued certificates. It's great that you found them , but it would also be useful if you can explain how these certificates were missed during the initial investigation?

Also, please continue to update this bug when the three revocation deadlines arrive, and when the final certificate expires.

Comment 15

[Mike Guenther]

As documented in the initial posting we started our investigation based on the finding of Ryan where the CP/CPS OID did not match the certificate OID. This original investigation concentrated on the presumption that the problem is directly connected to the CPS. We conducted spot checks for verification. These spot checks confirmed our presumption. The newly developed tools gave us the complete picture which found the additional certificates.

As we started to analyze the reason for the new certificates we came to the conclusion that our initial statement of point 6 (and therefore also the remediation in point 7) in comment 0 still stands.
The missing parametrization in combination of a non-centrally managed set of information as well as the missing end-to-end checklist enabled these misissuances. We are confident that our approach will prevent similar misissuance.

Comment 16

[Mike Guenther]

On Friday the first revocation deadline was reached which triggers this status update:
As of 19 August 2019, 23:59 CEST a total of 6’176 certificates (affected by the OID-issue) are settled. This number does not include the certificates that are expired.

Another update will be provided at the latest after the next revocation deadline on 30 August 2019.

Comment 17

[Mike Guenther]

On Friday the second revocation deadline was reached which triggers this status update:
As of August 30, 2019, 23:59 CEST a total of 42'264 certificates (vs. committed number of 34'197 certificates; cf. comment # 12 ) affected by the OID-issue are settled. This number does not include the certificates that are expired.

Another update will be provided at the latest after the next revocation deadline on 30 September 2019.

Comment 18

[Nathalie Weiler]

As per October 1, 2019, SwissSign confirms that all certificates (defined in the initial posting and comment 13) with a validity later than January 1, 2020 are revoked.
Another update will be provided at the latest in early January 2020 after all certificates with validity before December 31, 2019 are expired.

Comment 19

[Wayne Thayer]

Nathalie: Please provide an update as promised in comment #18.

Comment 20

[Nathalie Weiler]

As of January 1, 2020 all certificates with the wrong OID and/or with a similar issue (incomplete policy identifiers in certificate ) have been revoked resp. are expired at the latest on Dec 31, 2019.

Comment 21

[Wayne Thayer]

It appears that all questions have been answered and remediation is complete.