1393557 - GlobalSign: Non-BR-Compliant Certificate Issuance -- RSA key smaller than 2048 bits

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Kathleen Wilson]

This bug is in regards to the problem: SA key smaller than 2048 bits
https://misissued.com/batch/17/

~~
To continue inclusion of your CA’s root certificates in Mozilla’s Root Store, you must respond in this bug to provide the following information:
1) How your CA first became aware of the problems listed below (e.g. via a Problem Report, via the discussion in mozilla.dev.security.policy, or via this Bugzilla Bug), and the date.
2) Prompt confirmation that your CA has stopped issuing TLS/SSL certificates with the problems listed below.
3) Complete list of certificates that your CA finds with each of the listed issues during the remediation process. The recommended way to handle this is to ensure each certificate is logged to CT and then attach a CSV file/spreadsheet of the fingerprints or crt.sh IDs, with one list per distinct problem.
4) Summary of the problematic certificates. For each problem listed below: number of certs, date first and last certs with that problem were issued.
5) Explanation about how and why the mistakes were made, and not caught and fixed earlier.
6) List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.
7) Regular updates to confirm when those steps have been completed.

Note Section 4.9.1.1 of the CA/Browser Forum’s Baseline Requirements, which states:
“The CA SHALL revoke a Certificate within 24 hours if one or more of the following occurs: …
9. The CA is made aware that the Certificate was not issued in accordance with these Requirements or the CA’s Certificate Policy or Certification Practice Statement; 
10. The CA determines that any of the information appearing in the Certificate is inaccurate or misleading; …
14. Revocation is required by the CA’s Certificate Policy and/or Certification Practice Statement; or 
15. The technical content or format of the Certificate presents an unacceptable risk to Application Software Suppliers or Relying Parties (e.g. the CA/Browser Forum might determine that a deprecated cryptographic/signature algorithm or key size presents an unacceptable risk and that such Certificates should be revoked and replaced by CAs within a given period of time).

However, it is not our intent to introduce additional problems by forcing the immediate revocation of certificates that are not BR compliant when they do not pose an urgent security concern. Therefore, we request that your CA perform careful analysis of the situation. If there is justification to not revoke the problematic certificates, then explain those reasons and provide a timeline for when the bulks of the certificates will expire or be revoked/replaced. 

We expect that your forthcoming audit statements will indicate the findings of these problems. If your CA will not be revoking the certificates within 24 hours in accordance with the BRs, then that will also need to be listed as a finding in your CA’s BR audit statement.

We expect that your CA will work with your auditor (and supervisory body, as appropriate) and the Root Store(s) that your CA participates in to ensure your analysis of the risk and plan of remediation is acceptable. If your CA will not be revoking the problematic certificates as required by the BRs, then we recommend that you also contact the other root programs that your CA participates in to acknowledge this non-compliance and discuss what expectations their Root Programs have with respect to these certificates.
~~

Comment 1

[douglas.beattie]

We're requesting the full list of SHA-256 certificates issued by AT&T, a Trusted Root customer that runs their own CA using a Name Constrained CA issued by GlobalSign.  This will allow us to report on the full scope of the problem.  We'll run these certificates through our certlint checker and provide the results.  Since there are several million certificates to dump, transfer and process, this will take us another day or maybe 2 to finalize.

Comment 2

[Ryan Sleevi]

Setting NeedsInfo so that the system will remind you to provide an update :)

Comment 3

[douglas.beattie]

We received the certificates late today and will be running them through our tools Monday.

Comment 4

[douglas.beattie]

We've had several exchanges with AT&T and here is the latest status:
- 3.1M certificates issued since mid 2013 have been received and processed by our certlint tool
- 18 OCSP signing certificates had AIA, which is not permitted.  The active OCSP signing certificates are being replaced with compliant certificates this week
- For those certificates that had an issue, many had multiple issues (the number of issues are listed below, not the number of certificates)
- 453 individual errors
  * One certificate with no AIA, CDP, SAN, Locality, State/prov - being revoked
  * Approximately 89 with 1024 bit keys, although the report could be inaccurate as we thought there were roughly 199 (we're actively checking this)
  * 3 with CN not in the SAN
  * All remaining (about 356) relate to certificates with internal server names.  Initial indications show that they were not aware of the internal server name requirement.  We're working to understand how this requirement was missed.

In all, about 291 certificates with issues out of 3.1M so the percentage is low.  The majority relate to internal server names and we're working with them to get those replaced and revoked.

We're planning to move them over to a hosted CA solution so this can be avoided in the future.

Comment 5

[Jonathan Rudenberg]

(In reply to douglas.beattie from comment #4)
>   * One certificate with no AIA, CDP, SAN, Locality, State/prov - being
> revoked

Give the lack of AIA OCSP URL and CRL Distribution Point, the only way to revoke this certificate is to revoke the intermediate that issued it. Is this what you plan to do? If so, when? If not, what do you mean by 'being revoked' and how do you square this with your responsibilities under Baseline Requirements section 4.9.1.1?

Additionally, when will these certificates be logged to CT and listed here?

Comment 6

[Tim Smith (inactive) 👨‍🔬 [:tdsmith]]

Perhaps this is a naive question but, considering that the AT&T CA was name-constrained, is it correct that none of the misissued certificates with internal server names were publicly trusted for those names? That doesn't appear to have much utility so I'm curious why it was repeated hundreds of times.

Comment 7

[Gervase Markham [:gerv]]

It seems like there are two versions of the issuing intermediate used by the 3 certs in the batch on misissued.com ("ATT Wi-Fi Services Corporate Certificate Authority G3"), with slightly different notBefore and notAfter dates:
https://crt.sh/?id=11351488
https://crt.sh/?id=11957247

Then, there are two versions of the "root" cert that chains up to, "ATT Wi-Fi Services Root Certificate Authority G3":
https://crt.sh/?caid=10154
https://crt.sh/?id=11957246
https://crt.sh/?id=11351487

Both versions appear to be properly name-constrained to four domains (for both email and TLS), including an IP address exclusion for both IPv4 and IPv6.

That root has a couple of other subCAs:
https://crt.sh/?caid=12658 ("ATT Wi-Fi Services Managed Device Certificate Authority G3")
https://crt.sh/?caid=10157 ("ATT Wi-Fi Services Root Certificate Authority G3")

Doug: can you confirm that you've looked at all the certs issued in every part of this hierarchy, i.e. everything issued by "ATT Wi-Fi Services Root Certificate Authority G3"?

Gerv

Comment 8

[douglas.beattie]

(In reply to Tim Smith from comment #6)
> Perhaps this is a naive question but, considering that the AT&T CA was
> name-constrained, is it correct that none of the misissued certificates with
> internal server names were publicly trusted for those names? That doesn't
> appear to have much utility so I'm curious why it was repeated hundreds of
> times.

Tim - I'm not sure what you mean by "were publicly trusted for those names".  Can you explain?

Comment 9

[Gervase Markham [:gerv]]

I guess perhaps what Tim is asking is, if AT&T issued a certificate for "mailserver", given the name constraints on the root and the fact that the internal name doesn't match them, how would that ever work anywhere, whether on AT&T's internal network or elsewhere?

Gerv

Comment 10

[douglas.beattie]

If a browser was the client accessing the internal server names, then it wouldn't work, agree.

Comment 11

[douglas.beattie]

(In reply to Gervase Markham [:gerv] from comment #7)
> 
> Doug: can you confirm that you've looked at all the certs issued in every
> part of this hierarchy, i.e. everything issued by "ATT Wi-Fi Services Root
> Certificate Authority G3"?
> 
> Gerv

Here's the hierarchy we're reviewing:

This is the GlobalSign CA we use to sign customer CAs:
- GlobalSign Trusted Root, SHA-256: https://crt.sh/?caid=1423 

We used that Trusted Root CA to sign this for AT&T (2 certificates):
ATT Wi-Fi Services Root Certificate Authority G3  https://crt.sh/?caid=10154 
 - https://crt.sh/?id=11957246
 - https://crt.sh/?id=11351487

There are 3 CAs issued under this "Root":

1) ATT Wi-Fi Services Corporate Certificate Authority G3 https://crt.sh/?caid=10155 
- https://crt.sh/?id=11957247
- https://crt.sh/?id=11351488

2) ATT Wi-Fi Services Managed Device Certificate Authority G3 https://crt.sh/?caid=12658 
- https://crt.sh/?id=12721667
- https://crt.sh/?id=12625559

3) ATT Wi-Fi Services Partner Certificate Authority G3 https://crt.sh/?caid=10157
- https://crt.sh/?id=11366031

Yes, we have requested and received certificates for all of these CAs; however we do not feel we're received the entire list, perhaps revoked certs were not provided.  We've asked for the report again.

Comment 12

[douglas.beattie]

(In reply to Jonathan Rudenberg from comment #5)
> (In reply to douglas.beattie from comment #4)
> >   * One certificate with no AIA, CDP, SAN, Locality, State/prov - being
> > revoked
> 
> Give the lack of AIA OCSP URL and CRL Distribution Point, the only way to
> revoke this certificate is to revoke the intermediate that issued it. Is
> this what you plan to do? If so, when? If not, what do you mean by 'being
> revoked' and how do you square this with your responsibilities under
> Baseline Requirements section 4.9.1.1?

This certificate just expired on 9/1.

> Additionally, when will these certificates be logged to CT and listed here?

Yes, we're working to get them posted and we will provide the fingerprints here.

Comment 13

[Ryan Sleevi]

Any updates on Comment #12?

Comment 14

[douglas.beattie]

AT&T has previously supplied a combination of certificates issued under the GlobalSign root and private rooted certificates, so the final report is a bit different than first reported.  For all SSL certificates issued under the GlobalSign Root we have the following:

- 5 with CN not listed in SAN
- 172 with 1028 bit keys
- 168 with internal server names
- The certificate without AIA, CDP, SAN, Locality, State/prov was issued under a private root.

A total of 340 unique certificates (some had multiple errors).  The list of fingerprints is attached.

Comment 15

[douglas.beattie]

Attachment: ATT Certificate fingerprints

This is the list of fingerprints.  We'll let you know when we've posted these 340 certificates to CT logs

Comment 16

[douglas.beattie]

All certificates have been posted to CT logs.

Comment 17

[Ryan Sleevi]

Below is my summary of the issues and remediation plan:

1) CN not listed in SAN
  - See Comment #4, Comment #14
  - AT&T Remediation:
    - Unclear
  - GlobalSign Remediation:
    - XXXX-XX-XX - Move AT&T to a Hosted/Managed CA

2) 1028-bit (1024-bit?) keys
  - See Comment #4, Comment #14
  - AT&T Remediation:
    - Unclear
  - GlobalSign Remediation:
    - XXXX-XX-XX - Move AT&T to a Hosted/Managed CA

3) 168 internal server names
  - See Comment #4, Comment #14
  - AT&T Remediation:
    - Unclear
  - GlobalSign Remediation:
    - XXXX-XX-XX - Move AT&T to a Hosted/Managed CA

With respect to both root cause analysis and mitigations, it doesn't appear that there is sufficient detail to indicate that either AT&T or GlobalSign have performed this step yet.

For AT&T, it's useful to understand how this situation occurred, and what steps are being taken to prevent further such situations while that certificate remains trusted.

For GlobalSign, it's useful to understand the extent to which it supervises its sub-CAs, how it reviews its sub-CAs, and what steps are being taken if it will continue to allow independently operated subordinate CAs.

Comment 18

[douglas.beattie]

We have reinforced several times the importance of compliance to AT&T and now after this incident they fully understand the possible consequences of future misissuances.  They are keenly aware of the importance of compliance and we are in regular communications with them about important BR changes to assure they don't miss anything.

GlobalSign Remediation plan:
- Starting this month, we are doing monthly audits of 100% of the certificates they issue (previously we did a quarterly audit of 3% of the issued certificates)
- We have an ongoing dialog with AT&T to move them to a hosted solution (we had our most recent technical discussion today).  We plan to move them to a hosted model by the end of 2018.

Comment 19

[douglas.beattie]

I wanted to pass along the mitigation from AT&T for the 3 items listed above:

1) CN not listed in SAN
* These were due to human error. New checks and awareness are now in place.

2) 1024-bit keys
* These certificates were generated using the ejbca.sh batch script due to them having multiple SANs. At that time the 1024 key option in the profiles was still not specifically disallowed and the ejbca.sh uses 1024 keys by default, so unfortunately these were generated without any alarms going off.   Since then, the 1024 key options have been removed from the profiles, and the ejbca.sh script has been modified to use 2048 bit keys as its default.

3) 168 internal server names
* These hosts were built without FQDN in account management, which allowed our Certificate Management (certman) process to pull and generate certificates against these incorrectly named hosts from Oracle.  To mitigate human error, a domain check is being added to our registration authority (certman) process.

Comment 20

[Gervase Markham [:gerv]]

Given that AT&T is being transitioned to a hosted model, we can call this closed.

Gerv