1759854 - GlobalSign: Certificate issued to FQDN with malformed CAA

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Christophe Bonjean]

Steps to reproduce:

GlobalSign was made aware it has issued a DV TLS certificate to a FQDN with a malformed CAA record. We are currently investigating and will post a full incident report by March 21 2022.

Comment 1

[Christophe Bonjean]

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

We became aware following a certificate problem report, which was filed at 16/03/2022 09:45 UTC (All times in this report are in UTC). It reported a certificate issued for domain unither.com, which contained a malformed, ABNF-incompliant CAA record "globalsign.com; digicert.com" which breaches RFC 8659.

The report was escalated to the Compliance team, who started the investigation on 16/03/2022 11:51. We confirmed that the reported certificate that was misissued required revocation on 16/03/2022 13:23 and responded to the reporter on 16/03/2022 14:08.

In parallel, at 16/03/2022 - 13:43, we started reviewing our historically issued certificates and associated CAA validation logs for the same issue. On 16/03/2022 16:22 we confirmed 28 additional domains were misconfigured with malformed CAA records. All of the additional domains were included in the same single certificate, for which we initiated the process of revocation and replacement.

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

DD/MM/YYYY - Time in UTC	Description
14/03/2022 - 17:32	First certificate issued against a malformed CAA record (unither.com).
16/03/2022 - 09:29	Second certificate issued against malformed CAA records for 28 other domains.
16/03/2022 - 09:45	Certificate problem report received
16/03/2022 - 11:51	Investigation started
16/03/2022 - 13:23	CAA log for certificate reviewed, confirmed the CAA DNS record value was malformed at the time of validation and confirmed the certificate in report requires revocation.
16/03/2022 - 13:43	Started review of historically issued certificates and corresponding CAA records.
16/03/2022 - 14:08	Initial response to reporter.
16/03/2022 - 16:22	Completed review of historically issued certificates. One additional certificate identified. Started revocation and replacement process.
16/03/2022 - 16:37	Responded to reporter with complete list of domains and certificates affected.
17/03/2022 - 12:53	System changes with CAA logic fix deployed in production.
17/03/2022 - 07:36	All affected certificates revoked.

3. Whether your CA has stopped, or has not yet stopped, certificate issuance or the process giving rise to the problem or incident. A statement that you have stopped will be considered a pledge to the community; a statement that you have not stopped requires an explanation.

Monitoring reports were set up on 16/03/2022 15:32 and shared with Compliance for review, with no pending requests affected by this issue. These measures remained in place until all remedial actions were completed.

4. In a case involving certificates, a summary of the problematic certificates. For each problem: the number of certificates, and the date the first and last certificates with that problem were issued. In other incidents that do not involve enumerating the affected certificates (e.g. OCSP failures, audit findings, delayed responses, etc.), please provide other similar statistics, aggregates, and a summary for each type of problem identified. This will help us measure the severity of each problem.

Certificates:

1: https://crt.sh/?id=6352061257
2: https://crt.sh/?id=6340989073

Domains:

adcirca.com
basicbreathingawareness.com
fightingpah.com
impactpahnow.com
lungbio.com
lungbioengineering.com
lungbiotechnology.com
neuroblastoma-info.com
neuroblastomainfo.com
northernther.com
orenitram.com
pahinitiative.com
perfectclinicaltrial.com
rehtinu.com
remodulin.com
remodulinpro.com
revivicor.com
steadymed.com
thebiowall.com
tyvaso.com
unitedpahsupport.com
unither.aero
unither.com
unithermedaffairs.com
unituxin.com
utassist.com
utcopay.com
utunisphere.com
william-harvey.org

5. In a case involving certificates, the complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem. In other cases not involving a review of affected certificates, please provide other similar, relevant specifics, if any.

See #4.

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

The initial logic in the CAA validation component stripped any characters occurring after a semicolon ";". This was based on the interpretation that the CA may optionally specify parameters to further constrain the issuance of certificates. This interpretation is accurate, however the structure of a CAA issuer record must follow the ABNF grammar (rfc8659 section 4.2). If the structure does not match the ABNF grammar, the record must be treated the same as one specifying an empty issuer-domain-name which forbids issuance.

The CAA records of the affected domains were of the format "globalsign.com; digicert.com" and were stripped by the CAA logic to "globalsign.com" which appeared to be a valid issue record to allow certificates to be issued by GlobalSign to this domain.

At the time of development of the CAA validation component (3 years ago), the code was not reviewed by the compliance team for compliance with the RFC specification. We are working to increase capacity to have a compliance RFC analysis of all current and historically released code.

7. List of steps your CA is taking to resolve the situation and ensure that such situation or incident will not be repeated in the future, accompanied with a binding timeline of when your CA expects to accomplish each of these remediation steps.

We have updated the CAA verification logic to reject CAA records with any character specified after the semicolon. The update was deployed in production on 17/03/2022 12:53.

Comment 2

[Christophe Bonjean]

All remedial activities have been completed - unless there are any further questions we believe this issue can be closed.

Comment 3

[Ryan Sleevi]

We are working to increase capacity to have a compliance RFC analysis of all current and historically released code.

There doesn’t seem to be a binding timeline here?

This is excellent in its analysis in digging deeper, and looking past the symptom and into the systemic causes. This sort of thing - not syntactically validating CAA - is potentially something to imagine other CAs may be affected by, and it’s great to see y’all identify that.

However, it’s unclear from Comment #2 if y’all are only considering remediation as being of the symptom, rather than that systemic issue. Can you clarify?

Comment 4

[Christophe Bonjean]

We consider the changes made to the CAA verification logic as the remediation of the very specific issue raised in this ticket.

In terms of RFC, features that are internally developed have always been and will continue to be developed with specifications to meet the applicable RFC requirements. However, we are looking into expanding team capacity to provide an additional resource that would allow us to increase the holistic review of developed code against RFC requirements, with the aim to further reduce potential issues such as the one raised in this ticket.

Initial steps have been taken towards this expansion, however due to the scope and expertise required for this type of analysis and review and given to the scarcity of profiles with this expertise, we are careful in committing to a precise timeline.

Comment 5

[Christophe Bonjean]

If there are no further questions, we believe this issue can be closed.

Comment 6

[Ben Wilson]

I'll close this on Friday, 6-May-2022, unless there are additional issues or questions to discuss.