1496616 - Consorci AOC: Qualified audit statements

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Wayne Thayer]

Consorci's latest audit statements: https://bugzilla.mozilla.org/show_bug.cgi?id=1425805#c46

They contain the following findings:
1. OCSP responds "good" to unknown certificate - previously reported in bug 1398246
2. Suspended certificates in CRL
3. No evidence of internal vulnerability analyses and
intrusion tests
4. No update procedures for Linux systems

In addition:
- This year's audit statements were delivered to Mozilla more than 2 months late as documented in bug 1425805
- The reports contained multiple errors as documented in bug 1425805
- Findings 1 and 3 on this year's audit were documented on last year's audit as well.

For issues 2-4, please provide the following information:
* When was/will the issue be resolved?
* What was the root cause of the issue?
* What have you done to prevent similar issues from occurring?
* Why was issue #3 not resolved after it was identified on last year's audit?
* Is Consorci planning interim audits to confirm that these issues have been resolved? If so, when will they be conducted?

Comment 1

[Francesc Ferrer]

Attachment: NC1-OCSP_ACC.png

Comment 2

[Francesc Ferrer]

Attachment: NC4-flight_plan_eIDAS.png

Comment 3

[Francesc Ferrer]

(In reply to Wayne Thayer [:wayne] from comment #0)
> Consorci's latest audit statements:
> https://bugzilla.mozilla.org/show_bug.cgi?id=1425805#c46
> 
> They contain the following findings:
> 1. OCSP responds "good" to unknown certificate - previously reported in bug
> 1398246
> 2. Suspended certificates in CRL
> 3. No evidence of internal vulnerability analyses and
> intrusion tests
> 4. No update procedures for Linux systems
> 
> In addition:
> - This year's audit statements were delivered to Mozilla more than 2 months
> late as documented in bug 1425805
> - The reports contained multiple errors as documented in bug 1425805
> - Findings 1 and 3 on this year's audit were documented on last year's audit
> as well.
> 
> For issues 2-4, please provide the following information:
> * When was/will the issue be resolved?
> * What was the root cause of the issue?
> * What have you done to prevent similar issues from occurring?
> * Why was issue #3 not resolved after it was identified on last year's audit?


1. OCSP responds "good" to unknown certificate - previously reported in bug 1398246


We delivered evidence that the issue was resolved to the auditors in the Corrective Actions Plan (May 2018). In fact, in the audit report it was stated that “All minor non-conformities have been corrected and/or scheduled in the corrective action plan of the PSC”.

https://bugzilla.mozilla.org/attachment.cgi?id=9017117

And as stated in “https://bugzilla.mozilla.org/show_bug.cgi?id=1398246#c30”, we have configured the proper OCSP responses for the root CA (CN=EC-ACC), added hardware based high availability. The backup system is also now responding “Unknown” for not issued certificates, according with the requirements.


2. Suspended certificates in CRL

* When was/will the issue be resolved?

We delivered evidence that the issue was resolved to the auditors in the Corrective Actions Plan (May 2018). In fact, in the audit report it was stated that “All minor non-conformities have been corrected and/or scheduled in the corrective action plan of the PSC”.

* What was the root cause of the issue?

EC-SectorPublic CA issues SSL certificates as well as electronic signature certificates (natural persons) and electronic seals certificates (legal person), pursuant eIDAS.

EC-SectorPublic CPS states that:

4.10.17. Maximum suspension period: The maximum suspension period will be 120 natural days. And previously states:

4.10.13. Causes of certificate suspension: [...]
Suspension is prohibited for the following device certificates, which may only be revoked: 
● Secure Socket Certificate (Dispositiu SSL)
● Secure Socket Certificate Extended Validation (Dispositiu SSL EV)
● Electronic Office Certificate (Seu-e nivell mig/substancial)

The minor non-conformity was referring to natural and legal persons certificates suspended for more than 120 days.

There is no SSL certificate suspended in the CRL.

Anyway, the cause was that the cron that checks if a certificate is suspended for more than 120 days didn’t work after a new version deployment of our RA software.

* What have you done to prevent similar issues from occurring?

RA operators do not have the chance to suspend a SSL certificate.

In addition, we added to our new version deployment checklist, to check that the aforementioned cron is working properly. Evidences were sent to auditors in the Corrective Actions Plan in May 2018, unfortunately in Catalan and Spanish:

https://bugzilla.mozilla.org/attachment.cgi?id=9017118


3. No evidence of internal vulnerability analyses and intrusion tests

* When was/will the issue be resolved?

We provided a schedule for solving this issue to the auditors in the Corrective Actions Plan (May 2018). In fact, as in the audit report it is stated that “All minor non-conformities have been corrected and/or scheduled in the corrective action plan of the PSC”. This one was scheduled to be resolved by 10th September 2018 and actions have already been performed. Results have been collected and can be provided to the auditor.

* What was the root cause of the issue?

The root cause was that tests on Consorci AOC’s systems operated by Firmaprofesional were conducted and audited but not tests on Consorci AOC’s owned systems that were scheduled by later 2018. 

* What have you done to prevent similar issues from occurring?

Internal vulnerability analyses and intrusion tests requirements according to BR requirements have been included in the already existing procedures of periodic security analyses in Consorci AOC’s owned systems.

Periodic control meetings between CAOC and Firmaprofesional have been also put into place in order to control the proper development of such tasks and obligations for both parties. 

* Why was issue #3 not resolved after it was identified on last year's audit?

Last year’s audit issue#3 referred to the lack of controls in Consorci AOC’s  systems operated by Firmaprofesional (CA infrastructure operator). These controls were in place for the 2018 audit and, therefore, resolved in 2018 (as stated in the audit, at 6.5.2). 

This year’s audit issue#3 is referring to Consorci AOC’s infrastructure only, on which no internal vulnerability analyses and intrusion tests had been performed for that period according to the audit requirements.


4. No update procedures for Linux systems

* When was/will the issue be resolved?

We provided an schedule for solving this issue to the auditors in the Corrective Actions Plan (May 2018). In fact, as in the audit report it is stated that “All minor non-conformities have been corrected and/or scheduled in the corrective action plan of the PSC”. This one was scheduled to be resolved by 30th October 2018. Some actions have already been performed but some data is still being collected in order to be provided to the auditor.

* What was the root cause of the issue?

Although Consorci AOC’s systems operated by Firmaprofesional did have the update procedures in place, in Consorci AOC’s owned systems infrastructure only update procedures for Windows systems according to the ETSI requirements were in place. This point was not identified when preparing the audits. 

* What have you done to prevent similar issues from occurring?

The required procedure has been created and evidences of its implementation and execution being are collected. 

Periodic control meetings between CAOC and Firmaprofesional have been also put into place in order to control the proper development of such tasks and obligations for both parties. 

> * Is Consorci planning interim audits to confirm that these issues have been
> resolved? If so, when will they be conducted?

Yes. We have asked our practitioner AUREN to conduct a point-in-time audit covering the found issues, before the end of this year in order to confirm that the issues have been resolved. We will provide the specific dates hopefully this week.

Comment 4

[Ryan Sleevi]

Wayne: I think this goes back to you.

Comment 5

[Francesc Ferrer]

Attachment: Interim Report CAOC 201812_v02_en.pdf

Comment 6

[Francesc Ferrer]

Dear All, 

Please find attached the interim report with issue date of December the 20th. 

Looking forward to hear from you soon. 

Francesc,

Comment 7

[Wayne Thayer]

Dear Francesc:

Thank you for providing the AUP report. I am satisfied with it as evidence of remediation except for the noted exceptions of suspended certificates in the EC-SectorPublic CRL. I believe this violates BR section 4.9.13, and it appears that your auditors agree. I understand that these certificates are not intended for TLS, but if they are only restricted by policy, then they are still capable of being used for TLS. Do these certificates contain an extendedKeyUsage extension? If so, what EKUs are asserted?

Even if the end-entity certificates are out-of-scope of the BRs, the CRL is signed by the EC-SectorPublic subordinate CA that is clearly in-scope for the BRs. How and why did Consorci determine that having suspended certificates in this CRL is allowed?

Comment 8

[Francesc Ferrer]

(In reply to Wayne Thayer [:wayne] from comment #7)
> Dear Francesc:
> 
> Thank you for providing the AUP report. I am satisfied with it as evidence
> of remediation except for the noted exceptions of suspended certificates in
> the EC-SectorPublic CRL. I believe this violates BR section 4.9.13, and it
> appears that your auditors agree. I understand that these certificates are
> not intended for TLS, but if they are only restricted by policy, then they
> are still capable of being used for TLS. Do these certificates contain an
> extendedKeyUsage extension? If so, what EKUs are asserted?

Every suspended certificate in the CRL includes extendedKeyUsage extension that enables Email protection and Client Authentication only. This point was indeed checked by the auditor during the audit. 

> 
> Even if the end-entity certificates are out-of-scope of the BRs, the CRL is
> signed by the EC-SectorPublic subordinate CA that is clearly in-scope for
> the BRs. How and why did Consorci determine that having suspended
> certificates in this CRL is allowed?

When the subCA was created back in 2014, a decision based on simplicity put the end-entity signature certificates and end-entity web authentication together into this SubCA scope. As far as we know, at that moment the restriction for not allowing suspended SSL certificates to be in the CRL was not into place. When the restriction for SSL arised, controls were put into place for not allowing SSL certs suspension. This controls have passed already a few audits and are working fine. Therefore, Consorci AOC is asking for legacy-mercy mesures for having non-SSL certificates suspended cert into the EC-SectorPublic's CRL. It is absolutely clear for Consorci that new Roots and SubCAs will need to be technically constrained with purpouse flags and the SSL and signature certs must be separated into diferent subCAs.

Comment 9

[Wayne Thayer]

(In reply to Francesc Ferrer from comment #8)
> When the subCA was created back in 2014, a decision based on simplicity put
> the end-entity signature certificates and end-entity web authentication
> together into this SubCA scope. As far as we know, at that moment the
> restriction for not allowing suspended SSL certificates to be in the CRL was
> not into place. When the restriction for SSL arised, controls were put into
> place for not allowing SSL certs suspension. This controls have passed
> already a few audits and are working fine. Therefore, Consorci AOC is asking
> for legacy-mercy mesures for having non-SSL certificates suspended cert into
> the EC-SectorPublic's CRL. It is absolutely clear for Consorci that new
> Roots and SubCAs will need to be technically constrained with purpouse flags
> and the SSL and signature certs must be separated into diferent subCAs.

Thank you Francesc for this information. I think that this should be discussed on the mozilla.dev.security.policy list. Before I begin that discussion, can you confirm that Consorci has no plans to remediate this problem, such as by moving new TLS issuance to a technically constrained intermediate CA?

Comment 10

[Francesc Ferrer]

(In reply to Wayne Thayer [:wayne] from comment #9)

(In reply to Francesc Ferrer from comment #8)

When the subCA was created back in 2014, a decision based on simplicity put
the end-entity signature certificates and end-entity web authentication
together into this SubCA scope. As far as we know, at that moment the
restriction for not allowing suspended SSL certificates to be in the CRL was
not into place. When the restriction for SSL arised, controls were put into
place for not allowing SSL certs suspension. This controls have passed
already a few audits and are working fine. Therefore, Consorci AOC is asking
for legacy-mercy mesures for having non-SSL certificates suspended cert into
the EC-SectorPublic's CRL. It is absolutely clear for Consorci that new
Roots and SubCAs will need to be technically constrained with purpouse flags
and the SSL and signature certs must be separated into diferent subCAs.

Thank you Francesc for this information. I think that this should be
discussed on the mozilla.dev.security.policy list. Before I begin that
discussion, can you confirm that Consorci has no plans to remediate this
problem, such as by moving new TLS issuance to a technically constrained
intermediate CA?

I can confirm that Consorci AOC do have plans for remediating this problem by creating a new set of roots and subCAs with the state of the art in terms of security and compliance (technically constrained and business scopes divided, in particular) by mid-2019 and that should start new TLS issuance with this new and technically constrainted root and intermediate CA exclusively for web authentication certificates by the end of the year. Another entire hierachy would be dedicated to eSignature and eSeals.

Please bear in mind that Consorci AOC is a public entity/government CA externaly operated and that the actual contract does not directly allow us to issue a new subCA immediately. A new contract is due to start by mid 2019 with the requirements stated above.

Comment 11

[Wayne Thayer]

Given the information provided in comment #10, the remediation for this incident is for Consorci to migrate all issuance to a new hierarchy, and I think that is good. The amount of time required to make that transition while continuing to violate the BRs is a problem. For now, I have marked this bug as requiring an update in January 2020.

Comment 12

[Francesc Ferrer]

Attachment: 2019 AENOR Anexo 2 ETSI 319 411-1 PSC-CAOC_v4.pdf

Comment 13

[Francesc Ferrer]

Attachment: 2019 AENOR Anexo 2 ETSI 319 411-2 PSC-CAOC_v4.pdf

Comment 14

[Ryan Sleevi]

We're now at a midpoint to the migration. Can you confirm the migration is on track as scheduled? Did the new contract, from Comment #10, get successfully executed?

Comment 15

[Francesc Ferrer]

(In reply to Ryan Sleevi from comment #14)

We're now at a midpoint to the migration. Can you confirm the migration is on track as scheduled? Did the new contract, from Comment #10, get successfully executed?

Due to significant changes in the plan stated in Comment#10, an updated version of the plan is provided:

• Due to contracting issues and restrictions, the new contract for the operation of Consorci AOC’s certification services with the new requirements will not start until Jan 2020 instead of mid-2019 as stated before. Please bear in mind that Consorci AOC is a Government entity and, therefore, strict contracting procedures apply.
• Due to the delay stated above Consorci AOC decided a few months ago and before the recent audit, to stop issuing SSL certificates by the end of 2019. Communication to customers was sent a few weeks ago and it can be found here. End-entity SSL Issued certs and its issuing subCAs and infrastructure will still be maintained until all end-entity certs expired.
• In parallel, Consorci AOC is requiring a new set of roots and subCAs in the new contract in order to move all the issuance of eSignature and eSeal certificates in conformance to EIDAS and CCADB Program (for email trust bit enabled only) by mid-2020.

Comment 16

[Francesc Ferrer]

Sorry, this is the link to the communication to clients: https://www.aoc.cat/2019/04/02/consorci-aoc-deixa-demetre-certificats-de-servidor-segur-ssl/

Comment 17

[Wayne Thayer]

Francesc:

The audit statements provided in comments 1 and 13 contain yet more non-conformities. Have these been remediated?

Due to the delay stated above Consorci AOC decided a few months ago and before the recent audit, to stop issuing SSL certificates by the end of 2019. Communication to customers was sent a few weeks ago and it can be found here. End-entity SSL Issued certs and its issuing subCAs and infrastructure will still be maintained until all end-entity certs expired.

Does Consorci intend to achieve and maintain full compliance with the BRs and Mozilla policy until all end-entity certificates expire, presumably in late 2021?

Comment 18

[Francesc Ferrer]

(In reply to Wayne Thayer [:wayne] from comment #17)

Francesc:

The audit statements provided in comments 1 and 13 contain yet more non-conformities. Have these been remediated?

Assuming that you meant the comments corresponding the 2019 audits that are comments #12 (instead of #1) and #13, every non-confomity has been remediated in a plan provided to the auditor and accepted for it before statement's issuance. At the moment, 3 out of the all 12 minor Non-conformities have been already solved.

Due to the delay stated above Consorci AOC decided a few months ago and before the recent audit, to stop issuing SSL certificates by the end of 2019. Communication to customers was sent a few weeks ago and it can be found here. End-entity SSL Issued certs and its issuing subCAs and infrastructure will still be maintained until all end-entity certs expired.

Does Consorci intend to achieve and maintain full compliance with the BRs and Mozilla policy until all end-entity certificates expire, presumably in late 2021?

Yes, as stated in the communication to the costumers (translated by google from the text of the website above and slightly adjusted) : "The certificates issued until December 31, Consorci AOC will ensure the validity of the same during the two years of his term. "

Comment 19

[Ryan Sleevi]

Wayne: Looking at CT, it looks like the last certificate issued by EC-SectorPublic was on 2019-12-27.

I believe Mozilla was looking into the capability of disabling trust on a forward-looking basis (in Bug 1593141), so I wasn't sure if this was a use case for that, but I think it goes back to you as this CA is shutting down for TLS.

Comment 20

[Wayne Thayer]

Francesc: can you confirm that Consorci has stopped issuing TLS certificates from all subCAs signed by the EC-ACC root as of 2019-12-27?

Ryan: I think you're referring to bug 1465613 which affects the entire hierarchy.

Comment 21

[Francesc Ferrer]

(In reply to Wayne Thayer [:wayne] from comment #20)

Francesc: can you confirm that Consorci has stopped issuing TLS certificates from all subCAs signed by the EC-ACC root as of 2019-12-27?

Yes, I can confirm that TLS certificates issuance finished by 2019-12-27 from all subCAs signed by the EC-ACC.

Having confirmed that, Consorci AOC assumes that CCADB trust in the already issued and still valid TLS certificates would last until its expiration in 2021-12-27 at latest. And, in particular, that e-mail trust bit trust for the EC-ACC root is going to remain active. Of course, by complying at all times with the Mozilla Root Store policy. If we are wrong in this assumption and trust configuration changes are going to be applied, please inform us in advance so we can handle the impact of them.

Comment 22

[Wayne Thayer]

(In reply to Francesc Ferrer from comment #21)

(In reply to Wayne Thayer [:wayne] from comment #20)

Francesc: can you confirm that Consorci has stopped issuing TLS certificates from all subCAs signed by the EC-ACC root as of 2019-12-27?

Yes, I can confirm that TLS certificates issuance finished by 2019-12-27 from all subCAs signed by the EC-ACC.

Thank you. This means that we can distrust certificates issued after that date. Adding N-I for Kathleen to process this distrust, after which I believe this bug may be resolved.

Having confirmed that, Consorci AOC assumes that CCADB trust in the already issued and still valid TLS certificates would last until its expiration in 2021-12-27 at latest.

Correct, all TLS certificates issued on or before 2019-12-27 will remain valid at this time and until they expire, assuming that Consorci remains in compliance with the Mozilla Root Store Policy.

And, in particular, that e-mail trust bit trust for the EC-ACC root is going to remain active.

The Mozilla email trust bit is not enabled for the EC-ACC root (https://searchfox.org/mozilla-central/source/security/nss/lib/ckfw/builtins/certdata.txt#11629)

Of course, by complying at all times with the Mozilla Root Store policy. If we are wrong in this assumption and trust configuration changes are going to be applied, please inform us in advance so we can handle the impact of them.

Comment 23

[Kathleen Wilson]

(In reply to Wayne Thayer from comment #22)

(In reply to Francesc Ferrer from comment #21)

(In reply to Wayne Thayer [:wayne] from comment #20)

Francesc: can you confirm that Consorci has stopped issuing TLS certificates from all subCAs signed by the EC-ACC root as of 2019-12-27?

Yes, I can confirm that TLS certificates issuance finished by 2019-12-27 from all subCAs signed by the EC-ACC.

Thank you. This means that we can distrust certificates issued after that date. Adding N-I for Kathleen to process this distrust, after which I believe this bug may be resolved.

I have filed Bug #1621159 to set CKA_NSS_SERVER_DISTRUST_AFTER to 12/28/2019 for this root:
Subject: CN=EC-ACC; OU=Serveis Publics de Certificacio, Vegeu https://www.catcert.net/verarrel (c)03, Jerarquia Entitats de Certificacio Catalanes; O=Agencia Catalana de Certificacio (NIF Q-0801176-I); C=ES
Certificate Serial Number: EE2B3DEBD421DE14A862AC04F3DDC401
SHA-1 Fingerprint: 28903A635B5280FAE6774C0B6DA7D6BAA64AF2E8
SHA-256 Fingerprint: 88497F01602F3154246AE28C4D5AEF10F1D87EBB76626F4AE0B7F95BA7968799

Comment 24

[Ben Wilson]

In comment #20, Wayne asked "Francesc: can you confirm that Consorci has stopped issuing TLS certificates from all subCAs signed by the EC-ACC root as of 2019-12-27?" Which subCAs were being used to issue TLS certificates? I would like to verify that no new TLS certificates have been issued.

Comment 25

[Francesc Ferrer]

(In reply to Ben Wilson from comment #24)

In comment #20, Wayne asked "Francesc: can you confirm that Consorci has stopped issuing TLS certificates from all subCAs signed by the EC-ACC root as of 2019-12-27?" Which subCAs were being used to issue TLS certificates? I would like to verify that no new TLS certificates have been issued.

Dear Mr. Wilson, the one and only SSL issuing CA in EC-ACC hierachy was EC-SECTORPUBLIC : https://crt.sh/?caid=8050. Thank you,