Closed Bug 1481862 Opened 6 years ago Closed 6 years ago

Camerfirma: MULTICERT organizationName Too Long

Categories

(CA Program :: CA Certificate Compliance, task)

Product:
CA Program
Component:
CA Certificate Compliance
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: wthayer, Assigned: martin_ja)

Details

(Whiteboard: [ca-compliance] [ov-misissuance])

Juan posted the following incident report to the mozilla.dev.security.policy forum today: We detected 5 certificates issued with ERROR: organizationName too long (X.509 lint) 1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date. We detected these certificates checking the CA issued certificates into crt.sh on August 3, 2018. 2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done. 2018-08-03 09:58 UTC --> We detected these 5 certificates and asked the team that manages them to revoke them. 2018-08-03 15:35 UTC --> All the certificates are revoked. 3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation. The issuance of certificates from this CA was suspended until the operational control was deployed. 4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued. https://crt.sh/?id=617995390 https://crt.sh/?id=606954201 https://crt.sh/?id=606953975 https://crt.sh/?id=606953727 https://crt.sh/?id=604874282 5. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem. https://crt.sh/?id=617995390 https://crt.sh/?id=606954201 https://crt.sh/?id=606953975 https://crt.sh/?id=606953727 https://crt.sh/?id=604874282 6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now. There was no effective control into Multicert's PKI platform about DN's O lenth and this CA wasn't included into Camerfirma's quality controls until 2018-08-03. 7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things. - Multicert's team have added an operational control and they'll delploy the techinical control on August 9 - Multicert's team will check crt.sh for misissued certificates (from today forward). - Camerfirma will check for certificates issued by new intermedite CAs into crt.sh no more than 24 hours after the CA certificate issuance (from today forward). Your comments and suggestions will be appreciated.
'- Multicert's team have added an operational control and they'll delploy the techinical control on August 9' I've confirmed that the technical control was not deployed on August 9. It was deployed on August 14 and it's working since then.
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
Reopening this because I just noticed that Camerfirma has not answered the questions asked by Ryan Sleevi on 8-August on the mozilla.dev.security.policy list: Can you speak more to what the technical controls being deployed are? The DNs O length comes from X.509v3 and RFC 5280, so it's a bit baffling to understand how there wasn't an effective control there. It does call into question the potential for a lack of other effective controls, which could be concerning. With respect to Camerfirma checking post-issuance what Multicert is doing, that's certainly the minimum expected, as you've cross-certified them. Can you help explain why this wasn't already part of your process and controls when working with third-party CAs? Can you explain why Multicert's PKI platform is allowed independent issuance, rather than having Camerfirma manage and maintain that for them, and how the community can rely on Camerfirma appropriately supervising and mitigating that risk going forward? I think it is good that you detected this, but note that your incident report doesn't actually examine when Multicert first had this issue. Looking at https://crt.sh/?id=617995390 for example, I see it being July 24. Can you explain why it took so long to detect? Can you discuss how far back you've examined Multicerts issuance?
Status: RESOLVED → REOPENED
Flags: needinfo?(martin_ja)
Resolution: FIXED → ---
Juan: are you planning to respond to the question in comment 2?

Hello,

Due to a mistake of mine the answer to the questions has not been made previously. I can only apologize to the community and especially to Ryan Sleevi. I'm sorry and I'll work to prevent a similar situation from happening again.

About the first point:

Can you speak more to what the technical controls being deployed are?

The DNs O length comes from X.509v3 and RFC 5280, so it's a bit baffling to
understand how there wasn't an effective control >there. It does call into
question the potential for a lack of other effective controls, which could
be concerning.

We are retrieving information (from Multicert) to provide an answer as soon as possible. We will offer it before Friday of this week.

About the second and third points:

With respect to Camerfirma checking post-issuance what Multicert is doing,
that's certainly the minimum expected, as you've cross-certified them. Can
you help explain why this wasn't already part of your process and controls
when working with third-party CAs? Can you explain why Multicert's PKI
platform is allowed independent issuance, rather than having Camerfirma
manage and maintain that for them, and how the community can rely on
Camerfirma appropriately supervising and mitigating that risk going forward?

I think it is good that you detected this, but note that your incident
report doesn't actually examine when Multicert first had this issue.
Looking at https://crt.sh/?id=617995390 for example, I see it being July
24. Can you explain why it took so long to detect? Can you discuss how far
back you've examined Multicerts issuance?

CAMERFIRMA before issuing the cross-certificate requested a Conformity Assessment Report about a Webtrust or ETSI 319 411 to MULTICERT issued by a qualified auditor on it and on the basis of it we issued the certificate. The CAR was disclosed at: https://bugzilla.mozilla.org/attachment.cgi?id=9022788

MULTICERT requires an independent issuance and on the basis of this CAR we allow it to them.

There was a delay from the issuance of the intermediate CA certificate until we deployed a manual control of the issued certificates as there was a delay in the incorporation of the intermediate CA in CCADB (https://bugzilla.mozilla.org/show_bug.cgi?id=1455147#c5).

The intermediate CA was incorporated into CCADB on 2018-07-31 and later, in the control of 2018-08-03, this problem was detected.

Since 2018-08-03 every certificate issued by this intermediate CA are controlled daily through crt.sh.

This control is currently incorporated when the new intermediate CA certificate is delivered to the holder organization.

Flags: needinfo?(martin_ja)

(In reply to Juan Angel Martin from comment #4)

There was a delay from the issuance of the intermediate CA certificate until we deployed a manual control of the issued certificates as there was a delay in the incorporation of the intermediate CA in CCADB (https://bugzilla.mozilla.org/show_bug.cgi?id=1455147#c5).

The intermediate CA was incorporated into CCADB on 2018-07-31 and later, in the control of 2018-08-03, this problem was detected.

Since 2018-08-03 every certificate issued by this intermediate CA are controlled daily through crt.sh.

This control is currently incorporated when the new intermediate CA certificate is delivered to the holder organization.

Thanks for providing these details.

I would express concern about this approach. I think it represents something that might be temporary, rather than complete.

In particular, it sounds like you don't operate your own linting / examination of certificates, and instead rely on crt.sh. If there was a bug in crt.sh, or it was unavailable - both of which have happened in the past - it sounds like your organization would not be suited to oversee these sub-CAs.

In the spirit of ensuring there are robust controls and mitigations, could you highlight if any changes or improvements are planned?

Similarly, is the understanding correct that the examination of past issuance is based purely on crt.sh?

Flags: needinfo?(martin_ja)

I would express concern about this approach. I think it represents something that might be temporary, rather than complete.
In particular, it sounds like you don't operate your own linting / examination of certificates, and instead rely on crt.sh. If there was a bug in crt.sh, or it was unavailable - both of which have happened in the past - it sounds like your organization would not be suited to oversee these sub-CAs.
In the spirit of ensuring there are robust controls and mitigations, could you highlight if any changes or improvements are planned?

We use crt.sh until we deploy our lint tool. It is scheduled to be deployed in mid-February.

Similarly, is the understanding correct that the examination of past issuance is based purely on crt.sh?

No, validation specialists examines 3% of the certificates issued by intermediate CA operated by Camerfirma and 6% of the certificates issued by intermediate CA not operated by Camerfirma.
They check about the QcStatements of the certificates eIDAS QCP-w.

We are retrieving information (from Multicert) to provide an answer as soon as possible. We will offer it before Friday of this week.

I hope to be able to provide news of this topic soon.

Flags: needinfo?(martin_ja)
Whiteboard: [ca-compliance] → [ca-compliance] Next Update - 14-February 2019

We are retrieving information (from Multicert) to provide an answer as soon as possible. We will offer it before Friday of this week.

Multicert has the following controls in place:

  • Server side validation of all web form fields against regular expressions, including pattern and length
  • Database column length constraints

Given that any such requirements can be derived from the technical documents, and thus should not represent anything 'sensitive', could you please share the regular expressions being used to validate the fields?

The goal is to ensure robust and comprehensive remediation, and the use of regular expressions for validation can and often has lead to oversight of the various fields. This is particularly concerning for string fields in certificates, which do have a number of restrictions to them.

Flags: needinfo?(martin_ja)

(In reply to Ryan Sleevi from comment #8)

Given that any such requirements can be derived from the technical documents, and thus should not represent anything 'sensitive', could you please share the regular expressions being used to validate the fields?

The goal is to ensure robust and comprehensive remediation, and the use of regular expressions for validation can and often has lead to oversight of the various fields. This is particularly concerning for string fields in certificates, which do have a number of restrictions to them.

We are using the following Spring annotations for sanitizing the Organization form field:

@Pattern =([|"\@#&/{()}=\+\-_\[\]?\,:.'\u00AA\u00BA0-9a-zA-Z\u00C0-\u00FF]+(?:\.)?(?:(?: )[|"\@#&/{()}=\+\-\[\]_?\,:.'\u00AA\u00BA0-9a-zA-Z\u00C0-\u00FF]+(?:\.)?)*)$

@Size(max = 64)

(In reply to Ryan Sleevi from comment #8)

Do you consider that the answer given by Multicert (comment #9) is a satisfactory answer to the question raised?

Flags: needinfo?(martin_ja)
Summary: Camerfirma: organizationName Too Long → Camerfirma: MULTICERT organizationName Too Long

Trying to summarize the issue so far, could you confirm this is accurate?

  • Incident #1 (Multicert): Issuing certificates with invalid organizationName (violating ASN.1 size constraint)

    • Timeline:
      • (On or before 2018-07-07) - Bug introduced
      • 2018-07-17 14:12:36 - First misissued certificate issued (Comment #0)
      • 2018-07-24 11:14:54 - Last misissued certificate issued (Comment #0)
      • 2018-08-03 09:58 - Camerfirma detects misissued certificates
      • 2018-08-03 15:18 - 3 replacement certificates were issued for INTIC (Comment #12)
      • 2018-08-03 16:36:04 – Multicert completes revocation of these 3 misissued certificates.
    • Cause: Multicert lacked technical controls for the DN's O field (Comment #0)
    • Existing mitigations:
      • None
    • Changes made:
      • Implemented - 2018-08-08 Multicert begins monitoring crt.sh for misissued certificates post-issuance
      • Implemented - 2018-08-14 Multicert corrects platform to include limits on length
      • Not yet implemented - 2019-03-21 Multicert to start linting of pre-certificates

  • Incident #2 (Camerfirma): Sub-CA violating the Baseline Requirements

    • Timeline:
      • 2018-07-07 14:12:36 - First misissued certificate issued (Comment #0)
      • 2018-07-24 11:14:54 - Last misissued certificate issued (Comment #0)
      • 2018-08-03 09:58 - Camerfirma detects misissued certificates
    • Cause: Camerfirma relied on crt.sh for detecting misissuance. This intermediate was not disclosed to CCADB until 2018-07-31 (Comment #4) and Camerfirma did not begin monitoring crt.sh until 2018-08-03
    • Existing mitigations:
      • Camerfirma required a successful audit report from an appropriate auditor prior to issuing the sub-CA
      • Camerfirma monitors crt.sh for misissued certificates under their hierarchy (Comment #4)
      • Camerfirma evaluates 6% of the certificates issued by third-party subCAs (for QCStatements - Comment #6, Comment #13)
    • Changes made:
      • Implemented - 2018-08-03 Camerfirma will not release the issued subCA to third-party organizations until it has configured its post-issuance monitoring (Comment #4, Comment #13)
      • Not yet implemented - 2019-02-14 Camerfirma will deploy its own post-issuance linting tool to ensure redundancy and resiliency in the event of crt.sh outages or delays
Flags: needinfo?(martin_ja)

Few corrections below inline to Incident #1.

(In reply to Ryan Sleevi from comment #11)

Trying to summarize the issue so far, could you confirm this is accurate?

  • Incident #1 (Multicert): Issuing certificates with invalid organizationName (violating ASN.1 size constraint)
    • Timeline:
      • (On or before 2018-07-07) - Bug introduced
      • 2018-07-07 14:12:36 - First misissued certificate issued (Comment #0)

Correct date is 2018-07-17.

* 2018-07-24 11:14:54 - Last misissued certificate issued (Comment #0)
* 2018-08-03 09:58 - Camerfirma detects misissued certificates
* 2018-08-03 15:35 - Multicert completes revocation

For full disclosure, we also want to report the following additional events related to this incident:

Since then, Multicert had no more certificates issued with organizationName field longer than 64 characters.

  • Cause: Multicert lacked technical controls for the DN's O field (Comment #0)
  • Existing mitigations:
    • None
  • Changes made:
    • Implemented - 2018-08-08 Multicert begins monitoring crt.sh for misissued certificates post-issuance
    • Implemented - 2018-08-14 Multicert corrects platform to include limits on length
  • Not yet implemented - 2019-03-21 Multicert to start linting of pre-certificates
  • Incident #2 (Camerfirma): Sub-CA violating the Baseline Requirements
    • Timeline:
      • 2018-07-07 14:12:36 - First misissued certificate issued (Comment #0)
      • 2018-07-24 11:14:54 - Last misissued certificate issued (Comment #0)
      • 2018-08-03 09:58 - Camerfirma detects misissued certificates
    • Cause: Camerfirma relied on crt.sh for detecting misissuance. This intermediate was not disclosed to CCADB until 2018-07-31 (Comment #4) and Camerfirma did not begin monitoring crt.sh until 2018-08-03
    • Existing mitigations:
      • Camerfirma required a successful audit report from an appropriate auditor prior to issuing the sub-CA
      • Camerfirma monitors crt.sh for misissued certificates under their hierarchy (Comment #4)
      • Camerfirma evaluates 6% of the certificates issued by third-party subCAs
    • Changes made:
      • Implemented - 2019-01-08 Camerfirma will not release the issued subCA to third-party organizations until it has configured its post-issuance monitoring
      • Not yet implemented - 2019-02-14 Camerfirma will deploy its own post-issuance linting tool to ensure redundancy and resiliency in the event of crt.sh outages or delays

Few corrections below inline to Incident #2.

(In reply to Ryan Sleevi from comment #11)

Incident #2 (Camerfirma): Sub-CA violating the Baseline Requirements

Timeline:
2018-07-07 14:12:36 - First misissued certificate issued (Comment #0)

Correct date is 2018-07-17.

   2018-07-24 11:14:54 - Last misissued certificate issued (Comment #0)
   2018-08-03 09:58 - Camerfirma detects misissued certificates

Cause: Camerfirma relied on crt.sh for detecting misissuance. This intermediate was not disclosed to CCADB until 2018-07-31 (Comment #4) and Camerfirma did not begin monitoring crt.sh until 2018-08-03
Existing mitigations:
Camerfirma required a successful audit report from an appropriate auditor prior to issuing the sub-CA
Camerfirma monitors crt.sh for misissued certificates under their hierarchy (Comment #4)
Camerfirma evaluates 6% of the certificates issued by third-party subCAs

This evaluation is about the QcStatements of the certificates eIDAS QCP-w.

Changes made:
Implemented - 2019-01-08 Camerfirma will not release the issued subCA to third-party organizations until it has configured its post-issuance monitoring

Correct date is 2018-08-18.

   Not yet implemented - 2019-02-14 Camerfirma will deploy its own post-issuance linting tool to ensure redundancy and resiliency in the event of crt.sh outages or delays
Flags: needinfo?(martin_ja)

Thanks. I've updated Comment #11 with these edits, and passing over to Wayne for review.

Flags: needinfo?(wthayer)

Juan: please update this bug when each remaining remediation actions (post- and pre-issunace linting) have been completed.

Flags: needinfo?(wthayer)

Implemented - 2019-02-14 Camerfirma deployed our own post-issuance linting tool to ensure redundancy and resiliency in the event of crt.sh outages or delays

MULTICERT 2019-03-22 -> Pre-issuance linting has been implemented. (https://bugzilla.mozilla.org/show_bug.cgi?id=1509002#c23)

It appears that all remediation has been completed.

Status: REOPENED → RESOLVED
Closed: 6 years ago → 6 years ago
Resolution: --- → FIXED
Whiteboard: [ca-compliance] Next Update - 14-February 2019 → [ca-compliance]
Product: NSS → CA Program
Whiteboard: [ca-compliance] → [ca-compliance] [ov-misissuance]
You need to log in before you can comment on or make changes to this bug.