1717357 - Actalis: Issuance of intermediates after 2020-08-20 that do not comply with Mozilla Policy and the Baseline Requirements

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Ryan Sleevi]

On Mozilla's dev-security-policy list, a thread titled "Intermediate CAs that have both serverauth and Emailprotection extkeyusage" highlighted that Actalis has issued a CA certificate that contains both id-kp-serverAuth and id-kp-emailProtection.

This intermediate certificate is https://crt.sh/?id=3517096458&opt=ocsp and has a commonName value of "AgID CA1" for the organizationName of "Agenzia per l'Italia Digitale".

As part of CA/Browser Forum Ballot SC31, adopted 2020-07-16, the Baseline Requirements v1.7.1 (effective 2020-08-20) have the following requirements in Section 7.1.2.2 (g)

For all other Subordinate CA Certificates, including Technically Constrained Subordinate CA Certificates:

For Subordinate CA Certificates that will be used to issue TLS certificates, the value id-kp-serverAuth [RFC5280] MUST be present. The value id-kp-clientAuth [RFC5280] MAY be present. The values id-kp-emailProtection [RFC5280], id-kp-codeSigning [RFC5280], id-kp-timeStamping [RFC5280], and anyExtendedKeyUsage [RFC5280] MUST NOT be present. Other values SHOULD NOT be present.

The sub-CA certificate Actalis has issued has id-kp-emailProtection in addition to id-kp-clientAuth and id-kp-serverAuth, in violation of the BRs.

In Bug 1586787, filed 2019-10-17, Actalis previously issued a sub-CA in violation of Mozilla's policies around extendedKeyUsage. Namely, Mozilla Policy v2.6.1 includes the following language:

Intermediate certificates created after January 1, 2019, with the exception of cross-certificates that share a private key with a corresponding root certificate:

• MUST NOT include both the id-kp-serverAuth and id-kp-emailProtection KeyPurposeIds in the same certificate.

Actalis previously committed that they had procedures in place to both prevent such issuances and to ensure prompt revocation in such situations.

Please provide an Incident Report that complies with https://wiki.mozilla.org/CA/Responding_To_An_Incident

Comment 1

[Adriano Santoni]

We acknowledge receipt and will provide a feedback soon.

Comment 2

[Adriano Santoni]

Preliminary Incident Report

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in the MDSP mailing list, a Bugzilla bug, or internal self-audit), and the time and date.

This bug was open to us on Sunday 20 June. We read it on Monday 21 June.

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

This will be detailed later on.

3. Whether your CA has stopped, or has not yet stopped, certificate issuance or the process giving rise to the problem or incident. A statement that you have stopped will be considered a pledge to the community; a statement that you have not stopped requires an explanation.

No further Intermediate CA certificates with this problem have been issued, after the one referenced in this bug, and none are planned for the time being, except regeneration of the affected ICA. We will take great care to prevent this type of problem from happening again, and we are already reasoning on how to enhance our process.

4. In a case involving certificates, a summary of the problematic certificates. For each problem: the number of certificates, and the date the first and last certificates with that problem were issued. In other incidents that do not involve enumerating the affected certificates (e.g. OCSP failures, audit findings, delayed responses, etc.), please provide other similar statistics, aggregates, and a summary for each type of problem identified. This will help us measure the severity of each problem.

See next item.

5. In a case involving certificates, the complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem. In other cases not involving a review of affected certificates, please provide other similar, relevant specifics, if any.

To our knowledge, the only affected certificate is the one mentioned by Ryan in this bug's opening:

https://crt.sh/?id=3517096458

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

This is still under investigation.

One thing that we can already mention is that the linter we have integrated into our Root CA - that is ZLint - did not reveal any problems with the EKU extension.

7. List of steps your CA is taking to resolve the situation and ensure that such situation or incident will not be repeated in the future, accompanied with a binding timeline of when your CA expects to accomplish each of these remediation steps.

These are the steps we have taken so far:

• regenerated the affected Intermediate CA certificate, removing serverAuth from EKU and the dNSName elements from Name Constraints;
• disabled the capability of SSL Server certificate issuance from the AgID CA1 certificate management portal;
• re-issued the existing active SSL Server certificates of AgID [1] from a different ICA (an Actalis' one);

As soon as possible, we will post a complete incident report also describing the measures we intend to adopt to ensure that such situation or incident will not be repeated in the future.

We have also started analizing the impact of revocation of the affected ICA.

Very few SSL Server certificates have been issued from this ICA, and they will shortly be revoked. Besides, this ICA is technical constrained so to only allow issuance of SSL Server certificates for domains controlled by AgID [1].

The problem is that this ICA is mainly used to issue S/MIME certificates for all Italian PEC providers [2] who in turn use them to sign all PEC messages. Since PEC messages are essentially S/MIME messages, the ICA is normally embedded into them, as part of the certificate chain. Therefore, after we revoke the affected ICA, it is quite likely that some email clients will report an invalid signature on past PEC messages, due to the embedded ICA being revoked. This may cause big problems to a great many people, in many areas. In Italy, PEC is an officially accepted (and sometimes mandatory) means of communications with the Public Administration. PEC services are widely used in Italy by both Public and Private sectors, in many important processes e.g in the context of Justice, law enforcement, tenders, etc. Disrupting the PEC system amounts to disrupting a critical infrastructure. Only in 2020, more than 2 Billions PEC messages have been exchanged [2]. In view of this, we believe this is an exceptional circumstance wherein revoking the affected certificate within the prescribed deadline may cause significant harm. We are therefore considering postponing the revocation of the affected ICA.

[1] https://www.agid.gov.it/en/agency/about-us

[2] https://it.wikipedia.org/wiki/Posta_elettronica_certificata

[3] https://www.agid.gov.it/it/piattaforme/posta-elettronica-certificata/statistiche-utilizzo-pec

Comment 3

[Ryan Sleevi]

Kathleen: Setting N-I for you. Based on Comment #2, it does sound like a OneCRL action for Firefox+TLS is warranted, while Actalis is still forthcoming with the incident report.

Comment 4

[Seo Suchan]

5. In a case involving certificates, the complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem. In other cases not involving a review of affected certificates, please provide other similar, relevant specifics, if any.

To our knowledge, the only affected certificate is the one mentioned by Ryan in this bug's opening:

https://crt.sh/?id=2620763357 (same name/key but different sign time, 2020/feb) is new enough to be Mozilla policy violation although not new enough for to be BR violation. and it looks like Actalis have some kind every half-year renewal for this CA (https://crt.sh/?caid=62857)
if they couldn't find this, I think we may see another of this in next year.

Comment 5

[Adriano Santoni]

The Actalis task force that, since Monday 21, has been handling this incident, has identified, evaluated and implemented the following measures, in addition to what has already been indicated above:

• All PEC Providers have been urgently requested to install the new ICA (the corrected one we generated on Tuesday 22) by today 25 June, and we are monitoring the execution of this request;

• By Monday 28, we will revoke all active TLS certificates signed by the "AgID CA1" key. We remind you that these are certificates issued exclusively for AgID domains, as imposed by the name constraints within the ICA certificate.

In the meantime, we have been informed that AgID intends to send us a formal request to postpone the revocation of the offending ICA certificate, in consideration of the huge harm to the national PEC system that a revocation within the BR terms (7 days) would entail.

We have no objections to Mozilla actioning OneCRL in Firefox for the offending ICA certificate (the one issued on October 13, 2020, with serial 45:97:32:d8:f3:18:cb:75:93:a2:f4:68:0f:90:ea:d9), to the extent that such action will only affect TLS certificates. However, we ask you to wait at least until Monday 28 June.

We will post another update on Monday 28.

Comment 6

[Adriano Santoni]

Here is an update on the activities we have underway:

• As anticipated, this morning all the TLS (SSL Server) certificates signed by the "AgID CA1" key and not yet expired have been revoked.
• As of today, all PEC providers have installed the new “AgID CA” certificate (the correct one we generated on June 22).
• In consideration of the impact assessment already reported above and a formal communication we received this morning from AgID (wherein they confirm the impact assessment and urge us to delay the revocation), we considered it reasonable and responsible towards all stakeholders to delay the revocation of the offending Sub CA certificate up to a time when the consequences will be tolerable. Consequently, we will soon file a separate bug for this delay, where additional context and explanations will be provided.

As for the root cause of the problem:

At the conclusion of our internal investigation, it emerged that the problem had a twofold cause:

1. In the case of the regeneration of the “AgID CA1” certificate of October 2020, a "fast path" was followed which skipped some checks. In fact, for the case of a Technically Constrained CA we have a simplified procedure that requires limited checks compared to the more general case. This procedure, in the event that it is necessary to regenerate a CA certificate based on a pre-existing Technically Constrained CA profile that remains unchanged except for adding name constraints, does not require a new check of the profile by at least two examiners (belonging to independent teams) but just a single check before issuing the certificate. This procedure has therefore proved to be fallacious, so its immediate revision is necessary.
2. The Zlint tool, which we integrated with our Root CA, did not detect any problems with the EKU extension at the time when we generated the “AgID CA1” certificate of October 2020. Back in 2019, when we were seeking a reliable tool for checking the compliance of intermediate CA certs automatically (see bug 1586787), we evaluated both CABlint and Zlint; neither tool seemed up to the task, at that time, but subsequently we decided to integrate Zlint with our Root CA because - despite some annoying aspects (e.g. some certificate fields are wrong according to RFC5280 but okay according to BR, if you run all supported lints) - it still represented an important risk mitigation measure. However this linter, which in other respects we consider a good one, has proved insufficient to avoid the specific problem that is the subject of this bug. (The same applies to the certlint/cablint tool: this linter also does not detect problems in the EKU). Therefore, the linting tool we use needs to be changed and/or modified.

Another element that may have caused some confusion is section 5.3.1 of the Mozilla Root Store Policy (MRSP):

5.3.1 Technically Constrained

We encourage CAs to technically constrain all intermediate certificates. For a certificate to be considered technically constrained, the certificate MUST include an Extended Key Usage (EKU) extension specifying all extended key usages that the subordinate CA is authorized to issue certificates for.

This sentence was already present, identical, in MRSP v2.5 and has remained unchanged until today.

Since the plural is used in that sentence (all extended key usages), this sounds like – in the case of a Technically Constrained CA – it is allowed for the EKU to contain more than one key usage, otherwise the use of the plural seems weird. We see that this interpretation conflicts with the provisions of the BR (from v1.7.1), however we believe that the current wording of the MRSP 5.3.1 may have been a source of confusion, contributing to the occurrence of the problem.

These are the remedial measures we intend to adopt:

• Revision of our operating procedure so to require in all cases the verification, by two independent teams, of the profile of a CA certificate that is about to be generated, as well as the certificate that is subsequently generated, in light of the most recent version of the Baselines Requirements and Mozilla's Root policy, regardless of whether it is a first issuance or a renewal/modification of a pre-existing CA certificate, and also for technically constrained CAs even if the profile does not require any changes except on name constraints. We will publish this revision on our internal document system by the end of July, and will hold an awareness meeting with all internal stakeholders.
• Enhancement of the Zlint tool aimed at detecting the specific problem wich is the subject of this bug, i.e. the simultaneous presence of serverAuth and emailProtection (or other key usages incompatible with serverAuth) in the EKU extension, as forbidden by Baseline Requirements as well as by Mozilla's Root policy. This enhancement will be contributed to the open-source Zlint project (https://github.com/zmap/zlint), so to also be useful to other CAs. We expect to accomplish this by the end of July.

Comment 7

[Adriano Santoni]

Here is an update on the remedial measures we committed to implement:

• Revision of our operating procedure
  We have revised our operating procedure for the generation of CA certificates, in the section regarding the “Technically Constrained” case, emphasizing that all compliance checks (both pre-issuance and post-issuance) are mandatory even in this particular case, and even if it’s just the regeneration of an existing certificate due to NameConstraints needing an update. This revision is now pending approval and publication to our internal document management system. Next, an awareness meeting will be hold to all internal stakeholders.
  However, we are aware that manual procedures - by their nature - are always subject to oversights and negligence even when entrusted to properly skilled and responsible people, so we do not expect our procedure (alone) to be effective in preventing the type of problem discussed here. With this in mind, although operating procedures and instructions are necessary, we need to rely much more on automated compliance checks (see next item).

• Enhancement of the Zlint tool (integrated with our Root CA)
  We have developed an additional lint for Zlint, specifically for detecting the problem discussed here (incompatible values in the EKU extension, where the serverAuth value is present), and carefully tested it against a range of dummy Subordinate CA certificates (issued by a Test Root CA) featuring various combinations of incompatible EKU values, while being correct on all other respects. Afterwards, we integrated this enhanced version of Zlint with our Root CA and attempted to issue Subordinate CA certificates (from our Test Root CA) affected by the same anomalies, and found that the issuance was blocked by Zlint, as expected. Our testing therefore shown that the new lint we introduced is effective and suited to our needs. We have contributed our lint to the Zlint project (see pull request #619), hoping that it will later be useful to other CAs, possibly with some adjustments.

We will post another update before the end of July, unless we are asked any questions in the meantime.

Comment 8

[Kathleen Wilson]

Adriano, Please confirm that the following two certificates should be added to OneCRL:
https://crt.sh/?id=3517096458
https://crt.sh/?id=2620763357

Are there any others?

Note: Even though these certs are name constrained, I will need to add them to CCADB in order to track their information in OneCRL.

Comment 9

[Adriano Santoni]

Kathleen, in addition to the above, the following three more certs can be added to OneCRL: all of them were regenerations of the same ICA certificate due to updated name constraints:

https://crt.sh/?id=1435438944
https://crt.sh/?id=1708458487
https://crt.sh/?id=2440092184

Comment 10

[Ryan Sleevi]

(In reply to Adriano Santoni from comment #6)

Since the plural is used in that sentence (all extended key usages), this sounds like – in the case of a Technically Constrained CA – it is allowed for the EKU to contain more than one key usage, otherwise the use of the plural seems weird. We see that this interpretation conflicts with the provisions of the BR (from v1.7.1), however we believe that the current wording of the MRSP 5.3.1 may have been a source of confusion, contributing to the occurrence of the problem.

Thanks. I've raised https://github.com/mozilla/pkipolicy/issues/228 for this

I also want to acknowledge and thank Actalis' contributions in https://github.com/zmap/zlint/pull/619 , which turns out to be tricky to solve, but a valuable issue to fix.

Comment 11

[Ryan Sleevi]

Ben: With Comment #7, I believe this is a request by Actalis to provide a next update on 2020-07-31, which seems reasonable.

Bug 1718554 is tracking the revocation delays here.

Comment 12

[Kathleen Wilson]

I have filed bug #1720400 to track adding entries for these intermediate certificates to OneCRL. I plan to do this as part of our next batch of OneCRL updates.

Comment 13

[Adriano Santoni]

With reference to the first bullet in Comment 7: our revised procedure for the generation of CA certificates has been approved and published to our internal document management system. An awareness meeting has then been held with all internal stakeholders to explain the revisions made and to emphasize the importance of following the procedure closely at all times, double-checking everything.

In line with https://bugzilla.mozilla.org/show_bug.cgi?id=1718554, we expect to revoke the affected ICA certificates by 21 September.

We have no further updates, for now.

Comment 14

[Adriano Santoni]

As planned, this morning we have revoked the AgID CA1 certificate issued on 13 Oct 2020 (https://crt.sh/?id=3517096458).

Comment 15

[Ben Wilson]

I believe this bug can now be closed and will do so on Friday, 1-Oct-2021.