1685557 - Camerfirma: Certificates without CABForum OV Reserved Policy Identifier

Assignee

Description

•

3 years ago

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36

Steps to reproduce:

We have detected an error that affects to some issued certificates.

Those certificates have been issued without the CA/Browser Forum Reserved Policy Identifier (OV - 2.23.140.1.2.2) in the Certificate Policies extension as is required from Sept 30th, 2020.

We have detected this problem due to an error that the new version of zlint throws.

We are investigating the certificates affected with this problem as well as the causes and we will give more details tomorrow in the incident report.

Ben Wilson

Updated

•

3 years ago

Assignee: bwilson → ana.lopes

Status: UNCONFIRMED → ASSIGNED

Type: defect → task

Ever confirmed: true

Whiteboard: [ca-compliance]

Ana Lopes

Assignee

Comment 1

•

3 years ago

Hi all,

Here you are the incident report:

How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

We were aware of the problem on January 7th because the new version of zlint (zlint v3.0) threw some error messages that had never appeared before.

A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

• January 7th zlint started throwing error messages and we started the investigation
• January 7th we detected the cause of the errors
• January 7th stop the issuance of the affected certificate profiles.
• January 7th study the causes of the problem
• January 7th correction of the affected profiles
• January 7th - 8th study to obtain the list of affected certificates and causes of the problem

Whether your CA has stopped, or has not yet stopped, certificate issuance or the process giving rise to the problem or incident. A statement that you have stopped will be considered a pledge to the community; a statement that you have not stopped requires an explanation.

We stopped issuing the affected certificate profiles immediately when we detected the problem (January 7th).

In a case involving certificates, a summary of the problematic certificates. For each problem: the number of certificates, and the date the first and last certificates with that problem were issued. In other incidents that do not involve enumerating the affected certificates (e.g. OCSP failures, audit findings, delayed responses, etc.), please provide other similar statistics, aggregates, and a summary for each type of problem identified. This will help us measure the severity of each problem.

We have detected 286 affected certificates.
The first of them was issued on 2020-09-30 08:33:05 and the last one on 2020-12-23 11:00:36.
You can see the serial numbers in the attached file.

In a case involving certificates, the complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem. In other cases not involving a review of affected certificates, please provide other similar, relevant specifics, if any.

See the attached file.

Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

According to the CA/Browser Forum Reserved Policy Identifier (OV - 2.23.140.1.2.2) in the Certificate Policies extension that is required from Sept 30th, 2020, Camerfirma developed some changes in the configuration file for all the profiles involved in July 2020 and passed them to the Quality Control Department to test them.

The Quality Department tested the configuration file that included the OID and detected errors in the AKI of that configuration file because the file that included the OID had not been the last one and did not include the last corrections that we had made related to the AKI.

The Development Department made the corrections to solve the problem with the AKI, but they did not include the OID to comply with the future requirement by mistake and the Quality Department failed to detect it because they were focus on the AKI correction and overlooked the other requirement.

Due to the fact that nobody could register an error because zlint did not throw any comments and for Quality Department the profiles were apparently corrected, nobody was conscious of the problem until the error message that zlint version 3.0.0 threw on January 7th when Operations Department tried to issue a new certificate using one of the problematic profiles.

This new version of zlint interpretated that OV certificates were EV because of the Certificate Policies extension missing and, despite the fact that this error was not “the real problem” it helped us detect the real one.

After knowing the real cause of the problem, we stopped issuing certificates immediately and started to investigate the affected certificates to revoke them within the deadlines.

List of steps your CA is taking to resolve the situation and ensure that such situation or incident will not be repeated in the future, accompanied with a binding timeline of when your CA expects to accomplish each of these remediation steps.

• Inclusion of the configuration files of the affected OID profiles for OV (January 8th)

• Revocation of the issued certificates that do not comply with the requirement (in progress, all the certificates will be revoked by the end of January 13th)

Apart from those measures that solve the problem, we have studied the real cause that could originate this problem in the change management process that made us overlook such an important thing.

After our study, we have decided to register all different changes that affect to profiles one by one in our tool, perform each change in different releases and always test them separately.

•

3 years ago

This bug report is a complete disappointment. It does not matter much as Camerfirma will hopefully soon be distrusted and very rightfully so.

Complete certificate data is still missing. "upload them to misissued.org [sic]" as was promised starting January 2021 was not done. Instead, as always with Camerfirma, there is some other excuse. Next time, it will, for sure, be blamed on Camerfirma's numerous SubCAs.

For convenience of others, here is at least a last of some random certificates disclosed in this bug: https://misissued.com/batch/197/

(In reply to Eusebio Herrera from comment #6)

Update: We have already revoked all the affected certificates (January 13th)

This is nothing to be proud of, there is a "SHOULD revoke a certificate within 24 hours" in the BR. You should rather explain why you did not reach this SHOULD clause.

But nonetheless, serials of the certificates were disclosed at 2021-01-08 15:51 UTC (they were probably known to Camerfirma before that time, which is not know, though, as Camerfirma's "report" lacks timestamps!). This puts the MUST revoke time at 2021-01-13 15:52 UTC. Among others, the first disclosed certificate (https://crt.sh/?id=3516473139&opt=cablint,ocsp) was revoked at 2021-01-13 16:24:29 UTC.

Thus, revocation HAS been delayed and there was no bug report in advance of this delay.

Ramiro Muñoz Muñoz

Comment 8

•

3 years ago

Hi Paul

Thanks for your constructive answer, and your good whishes for this new year.

Obviously, we had intented to disclose the misissued certificates in misissued.com. This is the first time we use this practice since as far as I know is not mandatory, but in some way helps to improve our transparency. Some technical problems appeared. Certificates should be published in crt.sh to insert in mississued.com, not only precertificates. When we publish in CT, as mandatory, crt.sh is fed with precertificates but not with the certificate. We must make some changes in the code to fix that. Asked how to proceed I decided not to publish in misissued.com as promised since we should do it manually, and instead postponed it and have enough time to modify the code. That was the reason, and obviously we do not intend to blame none of our three SubCAs.

I don't know why there are random certificates published in misissued.com.

We are no proud of revoking the certificates despite a big effort has been done from our operation department with our clients. But this is what the community is expected from us.

I don't know why you say that a 24-hour deadline applies in this bug.

We gave the order to our operation department to revoke before 2021-01-13 (this day included) but no time was informed about the deadline hour so that's why to being revoked 30 minutes later. A new bug to declare this delay will be open asap as the policy states. This is not a big delay, but we must be rigorous as all in this ecosystem try to be.

Ana Lopes

Assignee

Resolution: --- → FIXED

Nobody; OK to take it and work on it

Updated

•

1 year ago

Product: NSS → CA Program

David Lawrence [:dkl]

Updated

•

1 year ago

Whiteboard: [ca-compliance] → [ca-compliance] [ov-misissuance]

serials.txt 3 years ago Ana Lopes 5.58 KB, text/plain		Details
Fingerprints 3 years ago Juan Angel Martin 18.43 KB, text/plain		Details