Closed Bug 1502957 Opened 2 years ago Closed 5 months ago

Camerfirma: MULTICERT Misissuance and missing audits

Categories

(NSS :: CA Certificate Compliance, task)

task
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: wthayer, Assigned: martin_ja)

Details

(Whiteboard: [ca-compliance])

Attachments

(9 files, 2 obsolete files)

Ryan Sleevi reported the following MULTICERT certificate as misissued because the syntax of the qcStatements extension is violated by inclusion of the UTF8String message "Certificate for website authentication as defined in Regulation (EU) No 910/2014"

https://crt.sh/?q=5bada9c841242c13c035496d5668d4a59cb91bb839e9e625a9c63c0e687269ab

In addition, Ryan noticed that the issuer of this certificate "MULTICERT SSL Certification Authority 001" is marked in CCADB as falling under the same audit as its parent, the AC Camerfirma Global Chambersign Root - 2008. However, the audit for the root does not include the MULTICERT intermediate in-scope, so either the information is incorrect and the proper MULTICERT audits have not been supplied, or the MULTICERT intermediate has not been properly audited by Camerfirma.

Please provide an incident report for these problems, as described here:
https://wiki.mozilla.org/CA/Responding_To_A_Misissuance#Incident_Report
The incident report should be posted to the mozilla.dev.security.policy forum and added to this bug.
Attached file MULTICERT's audit attestation letter (obsolete) —
The uploaded file is just a text file that contains the URL http://docs.camerfirma.com/publico/Ficheros/I1002_v2_EN_Audit_letter_eIDAS_SSL.PDF
Attachment #9022651 - Attachment is obsolete: true
Attachment #9022652 - Attachment is obsolete: true
Incident report:
1) How we became aware of the problem:
Through a discussion in mozilla.dev.security.policy https://groups.google.com/forum/#!topic/mozilla.dev.security.policy/x3s3CSKdIlY

2) Actions and Timeline to resolve:
CAMERFIRMA.2.1) Send a comunication asking for information about the misissued certificates - duedate: 29/10/2018; 
MULTICERT.2.1) Stop the qualified web authentication certificates issuance until the problem is resolved.
MULTICERT.2.2) Fix the code and test it in DEV environment – duedate 29/10/2018.
MULTICERT.2.3) Apply and test the patch in CERT environment – duedate 29/10/2018.
MULTICERT.2.4) Apply correction in PROD environment to remove the string message in the QCstatement (after testing in development environment) - duedate: 30/10/2018;
MULTICERT.2.5) Search all the certificates affected for this issue – duedate 29/10/2018.
MULTICERT.2.6) Revoke all 11 test certificates - duedate: 30/10/2018;
MULTICERT.2.7) Change the certificate profile in DEV and CERT environment (disable the QC Statement, remove the QWAC`s OID (0.4.0.194112.1.4), and add the OVCP OID (0.4.0.2042.1.7) – duedate 07/11/2018.
CAMERFIRMA.2.2) Incorporate in CCADB MULTICERT's audit information - duedate: 05/11/2018;
CAMERFIRMA.2.3) Incorporate a manual control over the QcStatements compliance of at least 6% of the certificates issued with QcStatements - duedate: 07/11/2018; 
MULTICERT.2.8) Search all the certificates affected for this issue – duedate 07/11/2018.
MULTICERT.2.9) Reissue all 174 digital certificates with the correction in the QCstatement field (same key pairs and same notafter date) - duedate: 07/11/2018;
MULTICERT.2.10) Revoke all 174 digital certificates identified in the attachment - duedate: 12/12/2018.

3) Stop issuing certificates: 
From the moment MULTICERT receive notification of this problem (29/10/2018), MULTICERT stopped the issuance of qualified web authentication certificates until MULTICERT apply the correction in PROD environment.
                
4) Summary of the problematic certificates:
This problem only exists in certificates issued after 2 July 2018. All the certificates listed below were affected with this problem and all of them will be submited to the actions defined above.

5) Complete certificate data for the problematic certificates:
All certificates listed below were affected for this problem. All of them have the UTF8 string after the OID in the QCstatement configuration and all of them have the QCStatement enabled and the QWAC OID configured in the Certificate Policies.
                
6) Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now:
CAMERFIRMA) In the first instance, CAMERFIRMA research confirms that the audits of the intermediate CAs operated externally were provided by the holders. CAMERFIRMA have failed in its publication in CCADB.
MULTICERT) MULTICERT looked for the root cause and detected a wrong implementation of the QCType encoding: a UTF8String containing the comment on the ASN.1 definition in ETSI EN 319 412-5 v2.2.1 was incorrectly added as a value of that particular QCStatement. MULTICERT found no lint tool capable of detecting the wrong encoding of the QCStatement.
MULTICERT) MULTICERT also detected a wrong interpretation in the ETSI requirements aplicability. It was understood that it was applied the requirements of EU qualified certificates for website plus NCP requirements, and not the EV requirements.

7) List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things:
CAMERFIRMA) In spite of CAMERFIRMA had the right audit report needed to issue the subCA certificate, obviously fails to disclose it instead of file this subCA in CCADB with a "same as parent" value. Reviewing this process CAMERFIRMA are realized that this failure was produced in a high work overload and not enough resources was involved in the process at that time. CCADB management is realized by one of our PKI expert. From November 2018 three persons will be involved in this management.
CAMERFIRMA) Regarding the control over MULTICERT, CAMERFIRMA is aware of any certificate issued by any of our subCAs (MULTICERT included) by means of crt.sh. Daily CAMERFIRMA check any SSL certificate issued. Qcstatement issue was not detected by lint tools, that’s why CAMERFIRMA was not aware of this specific problem. CAMERFIRMA incorporates a control over the QcStatements compliance of at least 6% of the certificates issued with it as it's claimed at CAMERFIRMA.2.3).
MULTICERT) Listed in steps MULTICERT.2.1 to MULTICERT.2.10.
The attestation letter that Ryan attached has the following problems:
* the sha-256 fingerprint for the MULTICERT SSL Certification Authority 001 is wrong
* The period of time the audit covers is not clearly stated (unless the two sets of dates in second paragraph refer to the audit period, but in that case the period is not continuous and is less than one year
* Page 4 says that the certificate is valid until September 14, 2019. Since eIDAS certificates are valid for two years, I think this means that another audit is required by Mozilla for the period ending on September 14, 2018. If so, when will that attestation letter be delivered to Mozilla?
Flags: needinfo?(martin_ja)
* The sha-256 fingerprint for the MULTICERT SSL Certification Authority 001 is wrong
(MULTICERT) The sha-256 fingerprint for the MULTICERT SSL Certification Authority 001 is related to the certificate signed by the MULTICERT Root Certification Authority 01. When the audit was performed, CA MULTICERT SSL Certificate Authority 001 was only signed by our Root CA.

* The period of time the audit covers is not clearly stated (unless the two sets of dates in second paragraph refer to the audit period, but in that case the period is not continuous and is less than one year
(MULTICERT) We have performed an audit that detected non conformities, and a follow up audit to validate the corrective actions to resolve this non conformities. In accordance with ETSI EN 319 403, point 7.6 b) "a TSP audit may be passed with pending nonconformities provided that these do not impact the ability of the TSP to meet the intended service. This certification decision is conditional upon to the implementation of corrective actions within 3 months after conclusion of the audit (depending on the type and criticality of the correction(s)". So, we performed the audit from 03/08/2017 to 24/08/2017, and 3 months later (from 03/11/2017 to 19/12/2017) we performed a new audit to validate the corrective actions.

* Page 4 says that the certificate is valid until September 14, 2019. Since eIDAS certificates are valid for two years, I think this means that another audit is required by Mozilla for the period ending on September 14, 2018. If so, when will that attestation letter be delivered to Mozilla?
(MULTICERT) We have a new audit scheduled to start on 3 December this year. 
(CAMERFIRMA) We'll disclose into CCADB the new CAR.
Flags: needinfo?(martin_ja)
(In reply to Juan Angel Martin from comment #8)
> * The sha-256 fingerprint for the MULTICERT SSL Certification Authority 001
> is wrong
> (MULTICERT) The sha-256 fingerprint for the MULTICERT SSL Certification
> Authority 001 is related to the certificate signed by the MULTICERT Root
> Certification Authority 01. When the audit was performed, CA MULTICERT SSL
> Certificate Authority 001 was only signed by our Root CA.
> 
Thank you - that makes sense.

> * The period of time the audit covers is not clearly stated (unless the two
> sets of dates in second paragraph refer to the audit period, but in that
> case the period is not continuous and is less than one year
> (MULTICERT) We have performed an audit that detected non conformities, and a
> follow up audit to validate the corrective actions to resolve this non
> conformities. In accordance with ETSI EN 319 403, point 7.6 b) "a TSP audit
> may be passed with pending nonconformities provided that these do not impact
> the ability of the TSP to meet the intended service. This certification
> decision is conditional upon to the implementation of corrective actions
> within 3 months after conclusion of the audit (depending on the type and
> criticality of the correction(s)". So, we performed the audit from
> 03/08/2017 to 24/08/2017, and 3 months later (from 03/11/2017 to 19/12/2017)
> we performed a new audit to validate the corrective actions.
> 
Please explain how this meets the requirements of Mozilla policy section 3.1.3:

Full-surveillance period-of-time audits MUST be conducted and updated audit information provided no less frequently than annually. Successive audits MUST be contiguous (no gaps).

In other words, if the attached attestation letter covers the period from 03/08/2017 to 19/12/2017, then can you provide the attestation statement that covers the period ending on 02/08/2017 for MULTICERT? Based on the information in bug #1040072, there is a gap from 01/04/2017 to 02/08/2017 if the attestation latter attached to this bug does not cover that period.

> * Page 4 says that the certificate is valid until September 14, 2019. Since
> eIDAS certificates are valid for two years, I think this means that another
> audit is required by Mozilla for the period ending on September 14, 2018. If
> so, when will that attestation letter be delivered to Mozilla?
> (MULTICERT) We have a new audit scheduled to start on 3 December this year. 
> (CAMERFIRMA) We'll disclose into CCADB the new CAR.
Flags: needinfo?(martin_ja)
(In reply to Wayne Thayer from comment #9)
"Please explain how this meets the requirements of Mozilla policy section 3.1.3:

Full-surveillance period-of-time audits MUST be conducted and updated audit information provided no less frequently than annually. Successive audits MUST be contiguous (no gaps).

In other words, if the attached attestation letter covers the period from 03/08/2017 to 19/12/2017, then can you provide the attestation statement that covers the period ending on 02/08/2017 for MULTICERT? Based on the information in bug #1040072, there is a gap from 01/04/2017 to 02/08/2017 if the attestation latter attached to this bug does not cover that period."


CAMERFIRMA have spoken with MULTICERT to correct this anomalous situation. 

MULTICERT informs us that the audit scheduled to start on December 3 will also include the period from 01/04/2017 to 02/08/2017.

Could it be considered that this would correct the problem detected?
Flags: needinfo?(martin_ja)
(In reply to Juan Angel Martin from comment #10)
> CAMERFIRMA have spoken with MULTICERT to correct this anomalous situation. 
> 
> MULTICERT informs us that the audit scheduled to start on December 3 will
> also include the period from 01/04/2017 to 02/08/2017.
> 
> Could it be considered that this would correct the problem detected?

Providing an audit covering the period from 01/04/2017 to 02/08/2017 will help to remediate the problem. Because the audit for that period will be very late, it will not fix the problem. There is no way to correct the problem at this point. The best remediation would be revoking the MULTICERT SSL Certification Authority 001 and creating a new one with proper audit coverage.

Should MULTICERT decide to add the period from 01/04/2018 to 02/08/2017 to the upcoming audit, then I would suggest that they also add the period from 25/08/2017 to 02/11/2017 that also does not appear to be covered by the previous audit.
A similar audit issue applies to the following two CA certificates:
* InfoCert Organization Validation CA 3, 247A6D807FF164031E0EB22CA85DE329A3A4E6603DBC6203F0C6E282A9C9EA84 issued 06/07/2017
    - First known publicly-trusted certificate issued 28/09/2017 - https://crt.sh/?id=221198192
* Intesa Sanpaolo Organization Validation CA, 27CDD699DE15EE88A05BB10ED9DF2FC5E4CA25B5FDD42988963A38EC8940D55A issued 06/07/2017
    - First known publicly-trusted certificate issued 14/09/2017 - https://crt.sh/?id=307667156

Point-in-time audit reports dated December-2017 were provided for both of these certificates well after issuing their first Publicly-Trusted Certificates. Neither have received a full period-of-time report since then. These are both appear to be violations of BR section 8.1.

Please explain how these certificates meet the requirements of BR section 8.1, and if they do not, explain why not and how the problem will be remediated.
(In reply to Wayne Thayer from comment #12)
Since July 2018 ETSI TS 119 403-2 audit attestation state the start and end of the period that was audited.
Audits with a scope ETSI EN 319 401, 319-411-1 and 411-2 covering the period from the issuance of ‘InfoCert Organization Validation CA 3’ and ‘Intesa Sanpaolo Organization Validation CA’ to December 2018 is scheduled.
(In reply to Juan Angel Martin from comment #13)
> (In reply to Wayne Thayer from comment #12)
> Since July 2018 ETSI TS 119 403-2 audit attestation state the start and end
> of the period that was audited.

Are you stating that the December 2017 audits cover the period from 06/07/2017 through December 2017, but that is just not stated in the audit reports? If so, will the auditors confirm that?

> Audits with a scope ETSI EN 319 401, 319-411-1 and 411-2 covering the period
> from the issuance of ‘InfoCert Organization Validation CA 3’ and ‘Intesa
> Sanpaolo Organization Validation CA’ to December 2018 is scheduled.

This is not satisfactory because the audit period will be greater than one year.
(In reply to Wayne Thayer from comment #14)

> (In reply to Juan Angel Martin from comment #13)
>> (In reply to Wayne Thayer from comment #12)
>> Since July 2018 ETSI TS 119 403-2 audit attestation state the start and end
>> of the period that was audited.
>
> Are you stating that the December 2017 audits cover the period from 06/07/2017 through December 2017, but that is just not stated in the audit reports? If so, will the auditors confirm that?
 
I'm stating that since July 2018 in the ETSI audits that follow the TS 119 403-2 and EN 319 411-1 the period covered by the audit must be indicated. 
In the audit reports disclosed by Infocert and Intesa Sanpaolo, only the dates in which the audit was performed are stated. They're made in December 2017.
 
The auditor has confirmed that the audit performed last year covered the period from the subCAs certificates generation to the 6th of December. 

I understand that the problem with this issue comes from the declaration in CCADB of the audit start periods. If I am right, I propose to incorporate in CCADB the Audit Period Start Date on 06/07/2017 as the auditors confirms. 

>> Audits with a scope ETSI EN 319 401, 319-411-1 and 411-2 covering the period
>> from the issuance of ‘InfoCert Organization Validation CA 3’ and ‘Intesa
>> Sanpaolo Organization Validation CA’ to December 2018 is scheduled.
>
> This is not satisfactory because the audit period will be greater than one year.
 
It's just a proposal. This year audit is covering the period from the 7th of December 2017 to the 2nd of December 2018.
(In reply to Juan Angel Martin from comment #15)
> (In reply to Wayne Thayer from comment #14)
> 
> > (In reply to Juan Angel Martin from comment #13)
> >> (In reply to Wayne Thayer from comment #12)
> >> Since July 2018 ETSI TS 119 403-2 audit attestation state the start and end
> >> of the period that was audited.
> >
> The auditor has confirmed that the audit performed last year covered the
> period from the subCAs certificates generation to the 6th of December. 
> 
> I understand that the problem with this issue comes from the declaration in
> CCADB of the audit start periods. If I am right, I propose to incorporate in
> CCADB the Audit Period Start Date on 06/07/2017 as the auditors confirms. 
> 
Yes, we could update the dates in CCADB with the correct audit period, but we will need a confirmation directly from the auditor. The preferred way for the auditor to confirm this information would be for them to issue new attestation statements listing the audit period.
Blocks: 1040072
The MULTICERT SSL Certification Authority 001 cross-certificate was issued by Camerfirma in June 2018. Because of this, I no longer think that the audit statement provided in this bug is insufficient. However:
* MULTICERT will need to provide the corrected audit statement as part of the inclusion request in bug 1040072
* Camerfirma will still need to resolve the other two issues identified in comment #12
No longer blocks: 1040072
Awaiting new period-of-time audit statements.
Whiteboard: [ca-compliance] → [ca-compliance] - Next Update - 07-March 2019
Status: NEW → ASSIGNED

It's been 3 months without updates. Did I miss a status update?

Flags: needinfo?(martin_ja)

We are updating now the CCADB audits for GLOBAL CORPORATE SERVER -the issuer of InfoCert Organization Validation CA 3 and Intesa Sanpaolo Organization Validation CA - comment #12 - according to WebTrust Principles and Criteria for Certification Authorities v2.1

Flags: needinfo?(wthayer)
QA Contact: kwilson → wthayer
Flags: needinfo?(martin_ja)

I'm very confused by the response in comments 20-22. Those audits do not include the certificates from comment #12 in scope, and they do not cover the full audit period back to Dec 2017. However, it appears that audit reports dating back to Dec 2017 were provided for these two CA certificates in bug #1549861, so this problem has been remediated.

Eusebio: is my explanation correct? What is the purpose of comments 20-22?

Flags: needinfo?(wthayer) → needinfo?(eusebio.herrera)
Whiteboard: [ca-compliance] - Next Update - 07-March 2019 → [ca-compliance]

Hi Wayne,

I posted the wrong documents. Now I post the rigth ones.

The Infocert & Intesa auditor considered in the audit held in december 2018 all the certificates issued since the CA has started its activities in September 2017. Indeed in clause 3.3 of the attached report the misissuances occurred in 2017 have been listed too, plus the few occurred in 2018.

We had already opened a bug with the certificates misissued by Infocert
https://bugzilla.mozilla.org/show_bug.cgi?id=1556806
And in the same way with the certificates misissued by Intesa
https://bugzilla.mozilla.org/show_bug.cgi?id=1557085.

So these audit reports considered not only the certificates issued from the stardate of the audit report 7-December-2017 till the enddate 4-December-2018, but the certificates issues since the issuance of the subCA certificates (6-July-2017).
The first certificates issued with these subCAs was:

  • 28-September-2017 (InfoCert Organization Validation CA 3)
  • 14- September-2017 (Intesa Sanpaolo Organization Validation CA)

Best regards

Flags: needinfo?(eusebio.herrera)

Eusebio: These audits cover the period beginning in December 2017. Please explain how this answers the question I asked in comment #12:

Please explain how these certificates meet the requirements of BR section 8.1

Specifically, did these two subCAs have audits covering the period from September to December 2017?

Hi Wayne,

As you requested in Comment #16, https://bugzilla.mozilla.org/show_bug.cgi?id=1502957#c16, now I will attach the auditor's new attestation statements.

So we have for both subCAs ( InfoCert Organization Validation CA 3 & Intesa Sanpaolo Organization Validation CA):

  • This new attestation statements from the auditors assuring that they checked from the date of issuance of the subCAs (6 - September-2017) to 6-December-2017.
  • The standard audits from 7-December-2017.

More specifically,

InfoCert Organization Validation CA 3:

  • Audit memo: from 6-July-2017 to 6-December-2017
  • Standard audit: from 7-December-2017 to 2-December-2018

Intesa Sanpaolo Organization Validation CA:

  • Audit memo: from 6-July-2017 to 6-December-2017
  • Standard audit: from 7-December-2017 to 5-December-2018

Eusebio: thank you for providing these new documents. Both documents state:

During the audit, minor non-conformities were identified and, as a result:
An assessment was conducted between the issue of the subCA certificates (6-July-2017) until 6-December-2017.

What does this mean? Is the auditor stating that an audit was conducted but no public report or seal was generated due to the minor non-conformities that were identified? What were these non-conformities?

Flags: needinfo?(eusebio.herrera)

There existed a gap in the period covered by the reports that we published in the begining (between 6-July-2017 and 6-December-2017), however that period had been audited.

We asked the auditor too an audit memo that states that they’ve audited the period not covered into the reports. You can find the corresponding memos on the links below:

https://bug1502957.bmoattachments.org/attachment.cgi?id=9100429
https://bug1502957.bmoattachments.org/attachment.cgi?id=9100428

Is the auditor stating that an audit was conducted but no public report or seal was generated?

Yes

due to the minor non-conformities that were identified?

No, it was due to the fact that they should issued an attestation statement in July-2018 but they issued it in December-2018.

Related to the non-confomities and problems detected in previous reports, all of them have been solved. You can can find all the information about them on the bugs below:

https://bugzilla.mozilla.org/show_bug.cgi?id=1556806
https://bugzilla.mozilla.org/show_bug.cgi?id=1557085

Best regards

Flags: needinfo?(eusebio.herrera)

Wayne, Kathleen: I'm still a bit confused on the attachments Comment #32, and I'm not sure what the most productive next steps would be. One path is to say that if we're not able to make sense of it, we're not able to accept it, since the audits exist to benefit "us". Another option would be to give the CA the benefit of the doubt, and try to work out with the auditor what happened, but I'm fundamentally worried that does not scale and would not be something we reasonably could or should extend to all CAs.

I'd love to get your perspective here on next steps.

Flags: needinfo?(wthayer)
Flags: needinfo?(kwilson)

I am also confused by this statement:

No, it was due to the fact that they should issued an attestation statement in July-2018 but they issued it in December-2018.

It does seem to be clear that the InfoCert Organization Validation CA 3 & Intesa Sanpaolo Organization Validation CA have audit gaps that violate the BRs and Mozilla policy. I believe the only way to remediate the problem is to revoke these two CA certificates. Therefore the next steps I would propose are:

  1. Ask Camerfirma again if they can clearly explain why these certificates are compliant, and if they cannot, ask Camerfirma if they plan to revoke them (and if so, by when).
  2. If Camerfirma chooses not to revoke them, consider adding them to OneCRL.
Flags: needinfo?(wthayer)

Hi all,

We do not think the revocation or addition to OneCRL are needed in this case because the period between July 7 and December 6 was covered by additional memos wherein the auditors assured that they checked from the date of issuance of the subCAs (6 - September-2017) to 6-December-2017 for both CAs so we think they did not violate the BRs and Mozilla policy

As you can see, we provided information about it in comment 28: https://bugzilla.mozilla.org/show_bug.cgi?id=1502957#c28

Maybe we were not clear enough with the explanation we gave in the past, so we want to clarify the situation.

In that comment we asumed that there existed a gap between the date of creation and December 2017 for the intermediate CAs InfoCert Organization Validation CA 3 and Intesa Sanpaolo Organization Validation CA.

To solve the situation we asked the auditor for an Audit memo to cover that period and we uploaded the memos in the comments 29 and 30
https://bug1502957.bmoattachments.org/attachment.cgi?id=9100428
https://bug1502957.bmoattachments.org/attachment.cgi?id=9100429

In comment 32 we also provided the explanations about those reports in response to your questions https://bugzilla.mozilla.org/show_bug.cgi?id=1502957#c32

Related to the non-conformities that appear in the memos that cover the period between July 7 and December 6, the results of the reviews were the following:

  • Infocert: Some minor non-conformities were detected in the period between July 7 and December 6. They were solved during the audit period. All the details about the non-conformities and their remediation appear in section 1.3. of the report.
  • Intesa San Paolo: Some minor non-conformities were detected in the period between July 7 and December 6. They were solved during the audit period. All the details about the non-conformities and their remediation appear in section 3.3. of the report.

As you can see, no non-conformities continued open after the end of the audit period.

Please let us know if you need more information.

Clearing my needinfo, so I will stop receiving daily email about this bug. Wayne is looking into this further.

Flags: needinfo?(kwilson)
QA Contact: wthayer → bwilson

In summary:

  • Camerfirma/MULTICERT misissued 174 certificates with invalid QCStatements
  • In researching that issue, it was found that the "MULTICERT SSL Certification Authority 001" was not included in the scope of the MULTICERT audit.
  • It was later found that 2 other subCAs were not included in audit statements (comment #12). There was a 6-month gap in 2017 that is only covered by a memo provided by the auditor. No other remediation is planned.

I'm disappointed by the response in comment #34, but at this point I don't expect to make any further progress. I think it's best to resolve this bug and consider it in the context of all the other Camerfirma issues that have been documented.

Status: ASSIGNED → RESOLVED
Closed: 5 months ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.