1636140 - SwissSign: duplicate serial number

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Andrew Ayer]

The CA "SwissSign Server Silver CA 2014 - G22" has issued two certificates with the serial number 11:da:86:c0:18:8d:1f:fe:33:c9:b2:64:6b:5f:8f:86:36:38:81:08.

The first certificate: https://crt.sh/?sha256=DA6FFC827CFCD83BACBDF245E01EFD9D1D117DA823DDDA3989E0D98532BC861D

The second certificate is presumed to exist based on the existence of this precertificate: https://crt.sh/?sha256=7F85BB7BF46294DD69970560CB1D6BC0F847CEE2E1912E3EB17076FB152DF80A

The certificates differ in the value of the Authority Information Access extension.

Comment 1

[Mike Guenther]

Please assign this bug to me. Timo Schmitt has left SwissSign this week.

Comment 2

[Mike Guenther]

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.
SwissSign took notice of this issue as it was reported in Bugzilla

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

• 20200430 06:43:54 UTC Issuing of Precert: precert
• 20200504 10:30 - 12:00 CEST Change to OCSP-configuration for leaf certificates (see point 6)
• 20200506 09:18:59 UTC Issuing of corresponding cert: cert
• 20200508 07:30 CEST SwissSign took notice of the this Bugzilla and validated the information
• 20200508 07:45 CEST An investigation is started to explore the reason of the mismatch. Additionally we started to look for similar cases.
• 20200508 11:00 CEST Initial Information of our auditors
• 20200508 12:15 CEST First results showed no additional certs
• 20200508 14:30 CEST Reason for mismatch in precert/cert has been identified (see point 6)
• 20200508 15:30 CEST Update to our auditors
• 20200508 19:00 CEST Publishing of this report
• 20200508 ongoing Searching for 'open' precerts issued before the OCSP-configuration change

3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

• We are not issuing any new mismatching precerts/certificates because the mismatch is connected to a specific action taken on 2020054 (see point 6)
• Our first investigation shows only one Precert/certificate was issued with this mismatch
• We are currently checking for other precerts without corresponding certificates and will update this report next week

4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.
Current status of investigation shows no other affected certificates: Total number: 1 precert (20200430), 1 cert (20200506)

5. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

• Precert: https://crt.sh/?sha256=7F85BB7BF46294DD69970560CB1D6BC0F847CEE2E1912E3EB17076FB152DF80A
• Cert: https://crt.sh/?sha256=DA6FFC827CFCD83BACBDF245E01EFD9D1D117DA823DDDA3989E0D98532BC861D

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
SwissSign has several DNS-aliases for OCSP responders (such as http://silver-server-g2.ocsp.swisssign.net/<keyID>) which all connect to http://ocsp.swisssign.net/<keyID> . Any DNS-alias would work with any of our certificates. The status information was and still is always available as required.

The change of the OCSP configuration has been triggered by a recommendation of our auditors during the last audit. As we did not find any good value in the current setup we decided to drop the DNS aliases in the leaf certificates. The DNS-aliases themselves stay in place until the last of valid certificate which include a DNS-alias have been either revoked or expired.

The mistake happened as we missed to check for any existing precerts without the corresponding certificate were still queued at the time of the configuration change. We have not recognized this potential risk in advance.

7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.
In the future we will check for any precerts without corresponding certificates before any change to fields of the leaf certificates such as the OID. The work instructions are being updated accordingly.

Comment 3

[mpalmer]

20200430 06:43:54 UTC Issuing of Precert
20200506 09:18:59 UTC Issuing of corresponding cert

Out of curiosity, why such a long delay between precert and final certificate? Were there CT log issues, or something else?

In the future we will check for any precerts without corresponding certificates before any change to fields of the leaf certificates such as the OID.

What other possible solutions did SwissSign consider to solve this problem? Why were they rejected in favour of the adopted solution?

Also, your response to 7 lacks a "timeline of when your CA expects to accomplish these things".

Comment 4

[Julien Cristau [:jcristau]]

Do you plan on revoking the affected cert?

Comment 5

[Mike Guenther]

Julien
The certificates has been revoked (20200512 09:11:01 UTC)

Matt
There was no problem. Sometimes customers place an order on one day (percert) and execute the order on another day (cert).

The selected solution is effective and efficient and was set in place last Friday. Therefore we did not follow any other possible solutions.

Comment 6

[Ryan Sleevi]

Ben: I wasn't sure if you had any questions

This does seem like a risky implementation, because it requires that the CA effectively:

1. Disable new certificate issuance (inc. pre-certs)
2. Wait until they've drained all pending pre-certs (either revoking them or executing the order)
3. Make the configuration change
4. Re-enable new issuance

Step 2 seems very likely that it will be overlooked, and so alternatives (such as immediately issuing the certificate upon the order being placed, so that issuance and logging are kept as close as possible) seem much more desirable and much less risky, by placing less reliance on playbooks, although perhaps not perfectly.

Comment 7

[Ben Wilson]

Has SwissSign completed the task of updating its work instructions for draining pending pre-certs and/or re-populating the pre-cert queue with the updated end entity profile/content? If so, can it provide a brief overview of the steps that the procedure will follow?

Comment 8

[Mike Guenther]

We updated our work instructions on the same day. As to be expected we came up with the similar steps as Ryan.

In regards to Ryan's step 2: We do not see it as risky as Ryan as updates that could lead to a mismatch between precert/cert and the checking for 'open' precerts are bundled in the same role. The persons responsible will use queries to identify the 'problematic' precerts. These identified precerts will be revoked before the change is conducted.

We are still convinced that the implemented work instructions are easy to follow and effective. This is allowing us to stay operative and make changes to our configuration.

Comment 9

[Ben Wilson]

I'm inclined to close this as resolved if you can provide "a brief overview of the steps that the procedure will follow". Just a brief listing, 5-10 lines, less than 100 words, would be great.

Comment 10

[Andrew Ayer]

I agree with Ryan that this is a risky implementation. I'm particularly concerned that this creates an intermediate state for certificate orders that lies between not-yet-issued and fully-issued. We've seen various CAs have trouble with intermediate order states like this, such as overlooking certificates during searches (https://bugzilla.mozilla.org/show_bug.cgi?id=1524815) and failing to operate revocation services (https://groups.google.com/forum/#!msg/mozilla.dev.security.policy/LC_y8yPDI9Q/NbOmVB77AQAJ).

Maybe SwissSign can find a way to make this design work for them. But I would not consider it best practice for CAs to leave a non-negligible period of time between precertificate and certificate issuance.

Comment 11

[Andrew Ayer]

It is also necessary to consider interactions with CAA when there is a long period of time between precertificate and certificate issuance. Could SwissSign comment whether they check CAA before precertificate issuance, before certificate issuance, or both?

Comment 12

[Mike Guenther]

(In reply to comment #9)

A brief overview of our work instructions (relevant steps only)

1. Task to change a certificate field such as CRL-Distiribution Points/Certificate Policies, AIA, (extended) Key usage is given to the CA configuration team. All changes are approved by SwissSign Compliance.
2. The CA Manager checks if the change impacts certificate of type DV/OV/EV.
   1. If Yes: The CA Manager searches for 'open' PreCerts in our databases.
      i If successful, the PreCert is revoked and the customer informed accordingly.
      ii. The CAM stops the Job Queue so that no new certificate (PreCert/Cert) may be issued
      iii. Another search for 'open' PreCerts. Any new PreCert is also revoked.
      iv. The change is executed.
      v. The Job Queue is resumed.
   2. If no: The change is executed.
3. SwissSign Compliance is informed about the change and all revoked PreCerts.

in reply to comment 8 and comment 10
We understand your concern and share it in part. As part of ongoing improvements a user story for our developers is already being written to assess other solutions which are not dependent on the work instructions. As there is no timeline nor discussed content on the implementation to the user story I did not think it fit to publish this information (based on the Mozilla incident report template 'will be considered a pledge to the community'). This might have been a misjudgement on my side. What we can pledge to today is the work instruction.

in reply to comment 11
I need to check this internally

Comment 13

[Mike Guenther]

Concerning CAA. Just confirmed that we check the CAA before precert issuance. If there is more than 7h between precert and cert the CAA check is redone.