Closed Bug 1635096 Opened 4 years ago Closed 4 years ago

Entrust: Printable String Constraint Failure

Categories

(CA Program :: CA Certificate Compliance, task)

Product:
CA Program
Component:
CA Certificate Compliance
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: bruce.morton, Assigned: bruce.morton)

Details

(Whiteboard: [ca-compliance] [ov-misissuance])

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36

Actual results:

Entrust Datacard compliance team discovered that an SSL certificate was issued with quotation marks in the printable string format in the Organization field of the Subject DN.

  1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

On 30 April March 2020, Entrust Datacard compliance team discovered that an SSL certificate was issued with quotation marks in the printable string format in the Organization field of the Subject DN.

  1. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

30 April 2020, 04:13 UTC - An SSL Certificate was issued with printable string in the Organization field.
30 April 2020, 04:13 UTC - Certificate failed with OCSP and was blocked by OCSP responder.
30 April 2020, 05:31 UTC - Posting linting check provided notice, " Constraint failure in X520OrganizationName: ASN.1 constraint check failed: PrintableString: constraint failed"
1 May 2020, 18:27 UTC - Certificate was revoked on CRL

  1. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

The CA software has a bug which encodes quotation marks in the organization field using PrintableString instead of UTF8String. This software has not been fixed at this time.

  1. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

One SSL Signing certificate was issued with PrintableString in the Organization field. There were no other certificates issued with this error.

  1. The complete certificate data for the problematic certificates.

https://crt.sh/?id=2746128448

  1. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

This error was unknown as quotation marks are not a character that is typically used in the Organization field. The certificate information did pass through pre-issuance linting using zlint without error. The certificate was post-issuance linted using cablint where the error was found.

  1. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

The CA will be migrated to a more modern software which does not have the bug. The more modern CA software performs the same checks as the updated OCSP software. In this configuration, the OCSP responder will be put into blocking mode to prevent a miss-issued certificate from being delivered to the Subscriber.

The pre-linting software will be updated to perform the missing check.

The certificate has been revoked per CRL, but has not been revoked per OCSP. The certificate is not in our OCSP responder, so the response is unknown. We are working on a solution to address the OCSP issue.

Thank you for reporting this issue Bruce. I have a few questions:

  • It looks like zlint currently detects this error: https://crt.sh/?id=2746128448&opt=zlint How did zlint not detect it when run pre-issuance?
  • What does "Certificate failed with OCSP and was blocked by OCSP responder" mean? It sounds like your system wasn't able to produce an OCSP response for this cert?
  • What is being done to prevent another misissuance prior to updating Entrust's systems to prevent such an occurrence?
  • Was Entrust aware of the encoding bug in the CA software?
  • What is the timeline for migrating the CA software, updating the pre-linting software, and correcting the OCSP status returned for this cert?
Assignee: wthayer → bruce.morton
Status: UNCONFIRMED → ASSIGNED
Type: defect → task
Ever confirmed: true
Flags: needinfo?(bruce.morton)
Whiteboard: [ca-compliance]

(In reply to Wayne Thayer from comment #2)

Thank you for reporting this issue Bruce. I have a few questions:

  • It looks like zlint currently detects this error: https://crt.sh/?id=2746128448&opt=zlint How did zlint not detect it when run pre-issuance?
    We cannot reproduce this error through a direct call to zlint. We have used more than one version to try to reproduce. We can only conclude that the error “FATAL: asn1: syntax error: PrintableString contains invalid character” is not actually a zlint error, and is returned from some sort of pre-processor in crt.sh before calling zlint.
  • What does "Certificate failed with OCSP and was blocked by OCSP responder" mean? It sounds like your system wasn't able to produce an OCSP response for this cert?
    The certificate issuance system uses older legacy libraries. The OCSP system has been replaced this year and is using more modern libraries. While the old libraries are capable of creating a non-compliant structure, the OCSP system will not accept these certificates. We are migrating our issuance system to the same libraries over the coming year.
  • What is being done to prevent another misissuance prior to updating Entrust's systems to prevent such an occurrence?
    The Entrust verification team now knows that the double quotation mark does not work correctly and they will not approve it in any verified organization name.
  • Was Entrust aware of the encoding bug in the CA software?
    No, this is the first time an organization name containing a double quotation mark has been encountered and approved for use in an SSL certificate.
  • What is the timeline for migrating the CA software, updating the pre-linting software, and correcting the OCSP status returned for this cert?
  • Migration of the CA is scheduled to be completed in July 2020.
  • Pre-linting software will be updated to catch invalid characters in ASN.1 PrintableString, in the August 2020 release.
  • We are still working on the OCSP issue. The work around proposed will make the system either more risky to attacks or harder to keep up-to-date with the open source code we are using. Are there any alternatives? Could the certificate be revoked using OneCRL?
Flags: needinfo?(bruce.morton)

(In reply to Bruce Morton from comment #3)

  • We are still working on the OCSP issue. The work around proposed will make the system either more risky to attacks or harder to keep up-to-date with the open source code we are using. Are there any alternatives? Could the certificate be revoked using OneCRL?

We are working on a patch to our OCSP system. The patch will allow a serial number to be uploaded to OCSP, where the serial number will be revoked by default. The patch and certificate revocation should be in place by 29 May 2020.

I filed Bug 1636339 to track the specific delay in revocation. Supporting the ability to revoke by serial is one that's come up in several CA incidents in the past, and is fairly standard practice as a capability as I understand it, so I'm a bit surprised it's not there already.

(In reply to Bruce Morton from comment #4)

We are working on a patch to our OCSP system. The patch will allow a serial number to be uploaded to OCSP, where the serial number will be revoked by default. The patch and certificate revocation should be in place by 29 May 2020.

The system has been patched and the certificate has been revoked on OCSP.

When is your August 2020 release? I am going to put this down on the Whiteboard for a Next Update date of 15-Aug 2020.

Whiteboard: [ca-compliance] → [ca-compliance] Next Update 15-Aug 2020

(In reply to Ben Wilson from comment #7)

When is your August 2020 release? I am going to put this down on the Whiteboard for a Next Update date of 15-Aug 2020.

Hi Ben, the update to the linting has been delayed. Let's keep the next update for 15 Aug 2020. Thanks, Bruce.

Ben, we are planning to have zlint updated on post-issuance linting in a release which is targeted for 5 August 2020.

We are planning to have pre-issuance linting updated with the latest version of zlint in a patch targeted for 31 August 2020.

We will update post each release.

(In reply to Bruce Morton from comment #3)

(In reply to Wayne Thayer from comment #2)

Thank you for reporting this issue Bruce. I have a few questions:

We cannot reproduce this error through a direct call to zlint. We have used more than one version to try to reproduce. We can only conclude that the error “FATAL: asn1: syntax error: PrintableString contains invalid character” is not actually a zlint error, and is returned from some sort of pre-processor in crt.sh before calling zlint.

Could you describe more how your system invokes ZLint?

This isn't a ZLint error, it's a Golang error, from the ASN.1 module ( https://golang.org/src/encoding/asn1/asn1.go ), which causes a syntax error when parsing the certificate. That is, the certificate should fundamentally fail to decode into an x509.Certificate.

Understanding more about how you're invoking this might help further identify if an error is being suppressed by Entrust's implementation (e.g. treating a failure to parse as a 'success')

Flags: needinfo?(bruce.morton)

(In reply to Ryan Sleevi from comment #10)

Could you describe more how your system invokes ZLint?

Thanks for the comments. In investigating this issue while working on the upgrade to the latest zlint version, we finally realized why this error got past our inline linter but not our post-issuance checker. The ASN.1 parse error causes zlint to abort very early on, and this error return from zlint doesn’t get processed in our inline linting code the same way as other linter alarms. We are fixing our inline integration with zlint to recognize this scenario and report an error rather than reporting that nothing’s wrong and leave it to the post-issuance checker to catch.

Flags: needinfo?(bruce.morton)

Zlint has been updated on post-issuance linting on 5 August 2020.

Thanks Bruce.

This incident report helped highlight an issue in the ZLint Readme that was able to be improved as a result of Comment #11, so this will hopefully prevent future mistakes from CAs (and hopefully all CAs with ZLint integration are following this CA incident bug and aware of this change).

Zlint has been updated on pre-issuance linting on 28 August 2020.

I believe this can be closed. I'll close this on or about 21-Sept-2020 unless there are additional questions or issues raised.

Flags: needinfo?(bwilson)
Flags: needinfo?(bwilson)
Whiteboard: [ca-compliance] Next Update 15-Aug 2020 → [ca-compliance] Next Update 21-Sept-2020
Status: ASSIGNED → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
Product: NSS → CA Program
Whiteboard: [ca-compliance] Next Update 21-Sept-2020 → [ca-compliance] [ov-misissuance]
You need to log in before you can comment on or make changes to this bug.