1586787 - Actalis: Issuance of intermediates after 2019-01-01 that do not comply with Mozilla Policy

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Ryan Sleevi]

During a spot-check of Mozilla Policy Compliance, I discovered that Actalis issued, and then revoked, two intermediates that did not conform with Mozilla Policy 2.6.1

Mozilla Policy 2.6.1 requires (formatted for readability), in Section 5.3:

Intermediate certificates created after January 1, 2019, with the exception of cross-certificates that share a private key with a corresponding root certificate:

• MUST contain an EKU extension; and,
• MUST NOT include the anyExtendedKeyUsage KeyPurposeId; and,
• MUST NOT include both the id-kp-serverAuth and id-kp-emailProtection KeyPurposeIds in the same certificate.

The following certificates were issued and lack an EKU extension:

• https://crt.sh/?id=1283820374
  • Issued 2019-03-13
  • Revoked 2019-03-19, replaced by https://crt.sh/?id=1298407195 which does have an EKU
• https://crt.sh/?id=1287935739
  • Issued 2019-03-12
  • Revoked 2019-03-29, presumably replaced by https://crt.sh/?id=1298407200 which does have an EKU

I could not find a past incident report from Actalis regarding this, and so am requesting one consistent with https://wiki.mozilla.org/CA/Responding_To_An_Incident

Comment 1

[Giorgio Girelli]

Thanks Ryan,

I'm collecting all the information and we will post an incident report consistent with https://wiki.mozilla.org/CA/Responding_To_An_Incident soon.

Giorgio

Comment 2

[Giorgio Girelli]

After a deep internal analisys we posted hereafter the incident report.

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

The problem was detected in March 2019 by our staff upon reviewing recently generated new intermediate CA certificates, during the test phase of the mentioned certificates

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

• 2019-03-12: Intermediate CA certificate DV G1 (id=1287935739) was generated following the procedure then in force.
• 2019-03-13: Intermediate CA certificate EV G2 (id=1287935739) was generated following the procedure then in force.
• 2019-03-18: The internal review found the problem (missing EKU) in EV G2 of March 13 and requested a new generation and and the revocation of the wrong certificate. Consider that this certificate was not considered released in production as the CA was in pre test release phase
• 2019-03-19: EV G2 was re-issued correctly and installed the issuing CA server.
• 2019-03-19: The defective EV G2 was revoked.
• 2019-03-19: The internal review found the same problem (missing EKU) in DV G1 of March 12 and requested a new generation and the revocation of the wrong certificate.
• 2019-03-19: DV G1 was re-issued correctly, but with a different name (DV G2), and installed on the issuing CA server. Since DV G1 was in use, it took a few days before having a clear view of the impact on our customers; this allowed us to coordinate the transition in a controlled manner.
• 2019-03-29: The defective DV G1 was revoked.
• 2019-04-10: We took steps to avoid re-occurrance of that kind of problem (see item 7).

3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

The procedure we had in place also in March 2019 to issue new intermediate CAs assured us that certificates with any seen problem, including the one detected on this two certificates, have been revoked shortly. With actual procedure, as described in item 7, we also avoid certificate issuance with this problem.

4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

Two intermediate CA certificates were involved: one issued on March 12, 2019, the other issued on March 13, 2019.

5. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

• https://crt.sh/?id=1283820374 (issued 2019-03-13, revoked 2019-03-19)
• https://crt.sh/?id=1287935739 (issued 2019-03-12, revoked 2019-03-29)

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

Unlike for EE certificates, our procedure for intermediate CA certificates is carried out on an off-line system, following a complex script, and requires several manual steps. During execution of the CA certificate generation procedures of March 12 and 13, the operator unintentionally selected an incorrect certificate profile. Moreover, the verification that is normally carried out on new CA certificates is based on a checklist that at that time did not explicit ask to check any single parameter as it would be done during the validation phase. The problem was however detected a few days later, upon a second review of the two affected certificates, and corrected promptly.

Considering that in our procedure that certificates was not finally released we did not start the internal procedure to open an incident on the forum as requested. This lack has been already corrected in our internal procedure. See item 7.

7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

The above-mentioned defective intermediates were revoked shortly after their issuance.
We then took the following steps, back in April, to avoid re-occurrance of the problem:

• we reviewed and more aptly renamed the certificate profiles on our root CA system to reduce the risk of selecting the wrong one
• we updated our procedure for the generation of CA certificates
• we updated our checklist for verifying the compliance of CA certificates adding a more detailed list of checks
• we changed our test procedure to revoke immediately wrong certificates even before reissuing, testing and releasing in production the new ones
• we updated our procedure to manage as an incident, to inform also the community, even though the certificates are not released in production

We are also looking for a reliable tool for checking the compliance of intermediate CA certs automatically; so far we have not yet found a satisfactory software. Nor CABlint nor Zlint seems up to the task, for now.

We remain at disposal for farther questions.

Giorgio

Comment 3

[Ryan Sleevi]

Giorgio: Thanks for the detailed incident report.

The timeline is super helpful in understanding the scope and impact. I think one of the most concerning things is that, at least as recently as March of this year, that you didn't treat such issues "as an incident, to inform also the community, even though the certificates are not released in production". The distinction between "released in production" versus "internal" is not one that can be readily made.

I'm curious if you have suggestions on how to improve this, either in policy or language. Understanding the reasoning behind treating the two separately (for example, it might have been derived from other PKI requirements or historic approaches) is useful, because I think it's important that we make sure that no CA makes a similar mistake.

Separately, I'll admit, I'm a little concerned that the risks for manual issuance weren't identified, given the issues we've had over the past year regarding CAs performing manual activities. It had been hoped that every CA is following m.d.s.p. and the incident reports, and thus staying aware of patterns, trends, or issues that can affect them. Beyond this specific incident, has Actalis taken the opportunity to review all the CA incidents in the past year, and look for other areas of confusion or potential opportunities for improvement? There have been a number of issues, and so it's important to make sure that a holistic review is done, and not just this specific one.

In terms of ZLint/Cablint/etc, has Actalis considered contributing a linter? This would seem rather useful as issues are detected. I'll note that Fotis Loukos has done so recently. Has Actalis considered similarly contributing lints to issues it's aware of?

Comment 4

[Giorgio Girelli]

Thanks Sleevi for your considerations,

Indeed we think that it is quite possible for other CAs to make the same mistake, without any intention to violate the Mozilla policy, so we believe it may be worth to clarify in policy that such distinction (between a CA certificate being already in use or still in a test phase) must not be made or must not be taken into account in the context that we are discussing.

We do review the past CA incidents, as well as follow the m.d.s.p., and we do our best to learn from both sources in order to improve and strengthen our processes.

As to contributing a linter: we have considered that, but our staff are not familiar with the programming languages with which those two tools are built. It is easier for us to work on our proprietary software, that we use in parallel with those tools, however we do not exclude in the future to try and give our contribute.

Giorgio

Comment 5

[Wayne Thayer]

It appears that all questions have been answered and remediation is complete.