GlobalSign: AT&T SSL certificates without the AIA extension

ASSIGNED
Assigned to

Status

task
ASSIGNED
4 months ago
11 days ago

People

(Reporter: douglas.beattie, Assigned: douglas.beattie)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [ca-compliance] - Next Update - 22-August 2019)

Attachments

(3 attachments)

2.05 KB, application/x-x509-ca-cert
Details
26.67 KB, application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
Details
16.48 KB, application/vnd.openxmlformats-officedocument.wordprocessingml.document
Details

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36

Steps to reproduce:

In the course of normal communications with AT&T, we came across an SSL certificate that did not have the required AIA extension in it on Friday April 16th (attached). We had a conference call shortly thereafter and they verified that one of their current EJBCA certificate profiles is missing this extension.

They think that the certificate profile was not maintained when they performed a recent EJBCA upgrade. They believe the upgrade was done in March and that most of the certificates that were replaced due to the 63 bit serial numbers have been replaced with certificates that do not contain the AIA extension.

GlobalSign would have been detected this during our 100% audit of their March certificates; however due to AT&T staff vacation schedules, the March upload of issued certificates was delayed.

We're working with them to obtain the timeline for the change, the dates during which they misissued certificates, the list of affected certificates, and the replacement and revocation schedule.

It should be noted that these certificates are not posted to CT logs nor are they accessed via browsers as they are used within closed networks, but we'll get more details on their exact usage shortly.

Doug: thanks for reporting this. Please provide a full incident report as soon as the information is available.

Assignee: wthayer → douglas.beattie
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true
Flags: needinfo?(douglas.beattie)
Summary: AT&T SSL certificates without the AIA extension → GlobalSign: AT&T SSL certificates without the AIA extension
Whiteboard: [ca-compliance]

Correcting bug type to task.

Type: defect → task

Status update:

MISISSUANCE REPORT

ATT is working on creating the incident report with the specified details.

BACKGROUND INFORMATION

GlobalSign issued ATT this CA:

  • ATT Wi-Fi Services Root Certificate Authority G3

ATT is running the following 3 CAs under the above CA:

  • ATT Wi-Fi Services Managed Device Certificate Authority G3
  • ATT Wi-Fi Services Corporate Certificate Authority G3
  • ATT Wi-Fi Services Partner Certificate Authority G3

You can find these CAs here: https://crt.sh/?caid=10154

ABOUT THIS INCIDENT

The Managed Device CA:

  • This CA was misconfigured in March and the profile was updated Friday (4/26) afternoon immediately following our conference call on the topic.
  • This CA issues the majority of their SSL certificates, approximately 17,000, and issues only to the wayport.net domain.
  • The validity of these certificates is 30 days or less
  • These are issued for machine to machine purposes and are not accessed via browsers
  • Issuance is fully automated and they plan to have the majority certificates replaced by COB today. All certificates will be replaced and revoked no later than Friday, 4/29.
  • We'll report on the status of this on Friday.

The Managed Service CA

  • This CA was also misconfigured on the same date and the profile issue resolved Friday (4/16) afternoon.
  • This CA has approximately 3000 misissued certificates and these are issued only to the wayport.net and attwifi.com domains.
  • These certificates are also for machine to machine communications and no browsers access these servers.
  • This CA is used for securing incoming and outgoing communications with ATT devices. Many of these are on their customers' sites so they require manual replacement and operational downtime (in some cases). The replacement of all of these certificates will follow a timeline similar to the recent 63 bit misissuance incident of approximately 3 weeks because of the manual work and scheduling required with their customers

The Partner CA

  • This CA is no longer being used.
  • We will be asking them to revoke this CA

Status update

ATT is working on the details of the Incident report, but here is what we know so far:

The Managed Service CA

  • 99% of the certificates have been replaced
  • Most certificates are 10 day validity and they will expire no later than Monday, 5/6, so they will not be revoked
  • There are some 30 day certificates and they will be revoked.

The Managed Device CA

  • There are 1320 certificates. The list of CNs is posted as an attachment.
  • These will all be replaced and revoked within 3 weeks of the incident, or 5/17
  • I have requested the PEMs for these certificates

List of certificate CNs issued from the CorpCA

This is the incident report provided by ATT.

The answer to question #6 ("This issue was not caught during certificate reviews following the upgrade.") in the incident report is completely unsatisfactory. My conclusion from that response, combined with the findings in bug #1535873 is that AT&T is not suited to operate a publicly-trusted CA, and the remediation for this incident will not be complete until these intermediates have been revoked.

Doug: is August the time frame in which CAs controlled by AT&T will be revoked, or is that just when they plan to transition to the managed service?

Also, please update this bug when all misissued certificates have been revoked.

Wayne,

Since these certificates do not contain AIA or CDP, I believe we are better suited to focus all efforts on shortening the timeline for moving to the GlobalSign hosted service. The majority of their certificates have expired and the remainder are not accessible via browsers. If you believe that we should focus on the replacement (without revocation) task anyway, we can do that, but we'd rather start replacing them all in June (for the final time) with certificates issued from the GlobalSign hosted CA. At the moment we're collectively 100% focused on closing down their CAs and setting up a hosted service.

We just had a management call and they have committed to us that we will be able to revoke their CA on August 1st.

I plan to post the PEM files for all misissued certificates this week.

(In reply to douglas.beattie from comment #8)

Since these certificates do not contain AIA or CDP, I believe we are better suited to focus all efforts on shortening the timeline for moving to the GlobalSign hosted service. The majority of their certificates have expired and the remainder are not accessible via browsers. If you believe that we should focus on the replacement (without revocation) task anyway, we can do that, but we'd rather start replacing them all in June (for the final time) with certificates issued from the GlobalSign hosted CA. At the moment we're collectively 100% focused on closing down their CAs and setting up a hosted service.

I agree that focusing on the move to the managed CA makes sense.

We just had a management call and they have committed to us that we will be able to revoke their CA on August 1st.

I've changed the 'next update' on this bug to August 1st...

I plan to post the PEM files for all misissued certificates this week.

...but please do still post this information.

Whiteboard: [ca-compliance] → [ca-compliance] - Next Update - 01-August 2019

I've received a file with 220303 misissued certificates from ATT, but it's too large to include as an attachment (max is 10Mb and this is 120Mb). They will be available at this link for a couple of weeks and after that you can request them from me directly.

https://drive.google.com/open?id=1l3g6FLGXAhzgseC-ZbG0GcTbtNrjfUrC

Doug: I realize the update is not scheduled until August 1, but I want to make sure that things are progressing as scheduled.

Yes, I confirmed today that they have already started replacing some of the Manual certificates (the ones they need to work with their customers on) and they have actively been testing their automated provisioning system against a test account.

You probably know this already, but they are issuing certificates under this CA.
https://crt.sh/?caid=126736&opt=cablint
Some will be compliant with the Chrome CT policy, but the majority are for machine to machine use and won't be posted to CT logs.

Flags: needinfo?(douglas.beattie)

AT&T is on track to complete their certificate replacement this week and will be ready to revoke the 2 active CAs on Monday, August 5th. I'll report back when that has been completed.

Whiteboard: [ca-compliance] - Next Update - 01-August 2019 → [ca-compliance] - Next Update - 05-August 2019

AT&T revoked their 2 CAs yesterday:
https://crt.sh/?id=11351488
https://crt.sh/?id=12625559

On August 21st, GlobalSign will be revoking the CAs we issued to AT&T to completely close the loop on this:
https://crt.sh/?id=11957246
https://crt.sh/?id=11351487

Whiteboard: [ca-compliance] - Next Update - 05-August 2019 → [ca-compliance] - Next Update - 22-August 2019
You need to log in before you can comment on or make changes to this bug.