1547691 - GlobalSign: AT&T SSL certificates without the AIA extension

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[douglas.beattie]

Attachment: ATT SSL cert without AIA.txt.cer

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36

Steps to reproduce:

In the course of normal communications with AT&T, we came across an SSL certificate that did not have the required AIA extension in it on Friday April 16th (attached). We had a conference call shortly thereafter and they verified that one of their current EJBCA certificate profiles is missing this extension.

They think that the certificate profile was not maintained when they performed a recent EJBCA upgrade. They believe the upgrade was done in March and that most of the certificates that were replaced due to the 63 bit serial numbers have been replaced with certificates that do not contain the AIA extension.

GlobalSign would have been detected this during our 100% audit of their March certificates; however due to AT&T staff vacation schedules, the March upload of issued certificates was delayed.

We're working with them to obtain the timeline for the change, the dates during which they misissued certificates, the list of affected certificates, and the replacement and revocation schedule.

It should be noted that these certificates are not posted to CT logs nor are they accessed via browsers as they are used within closed networks, but we'll get more details on their exact usage shortly.

Comment 1

[Wayne Thayer]

Doug: thanks for reporting this. Please provide a full incident report as soon as the information is available.

Comment 2

[Firefox Bug Husbandry Bot]

Correcting bug type to task.

Comment 3

[douglas.beattie]

Status update:

MISISSUANCE REPORT

ATT is working on creating the incident report with the specified details.

BACKGROUND INFORMATION

GlobalSign issued ATT this CA:

• ATT Wi-Fi Services Root Certificate Authority G3

ATT is running the following 3 CAs under the above CA:

• ATT Wi-Fi Services Managed Device Certificate Authority G3
• ATT Wi-Fi Services Corporate Certificate Authority G3
• ATT Wi-Fi Services Partner Certificate Authority G3

You can find these CAs here: https://crt.sh/?caid=10154

ABOUT THIS INCIDENT

The Managed Device CA:

• This CA was misconfigured in March and the profile was updated Friday (4/26) afternoon immediately following our conference call on the topic.
• This CA issues the majority of their SSL certificates, approximately 17,000, and issues only to the wayport.net domain.
• The validity of these certificates is 30 days or less
• These are issued for machine to machine purposes and are not accessed via browsers
• Issuance is fully automated and they plan to have the majority certificates replaced by COB today. All certificates will be replaced and revoked no later than Friday, 4/29.
• We'll report on the status of this on Friday.

The Managed Service CA

• This CA was also misconfigured on the same date and the profile issue resolved Friday (4/16) afternoon.
• This CA has approximately 3000 misissued certificates and these are issued only to the wayport.net and attwifi.com domains.
• These certificates are also for machine to machine communications and no browsers access these servers.
• This CA is used for securing incoming and outgoing communications with ATT devices. Many of these are on their customers' sites so they require manual replacement and operational downtime (in some cases). The replacement of all of these certificates will follow a timeline similar to the recent 63 bit misissuance incident of approximately 3 weeks because of the manual work and scheduling required with their customers

The Partner CA

• This CA is no longer being used.
• We will be asking them to revoke this CA

Comment 4

[douglas.beattie]

Status update

ATT is working on the details of the Incident report, but here is what we know so far:

The Managed Service CA

• 99% of the certificates have been replaced
• Most certificates are 10 day validity and they will expire no later than Monday, 5/6, so they will not be revoked
• There are some 30 day certificates and they will be revoked.

The Managed Device CA

• There are 1320 certificates. The list of CNs is posted as an attachment.
• These will all be replaced and revoked within 3 weeks of the incident, or 5/17
• I have requested the PEMs for these certificates

Comment 5

[douglas.beattie]

Attachment: CorpCA certificate list 5-1-19.xlsx

List of certificate CNs issued from the CorpCA

Comment 6

[douglas.beattie]

Attachment: Incident report for ATT AIA issue.docx

This is the incident report provided by ATT.

Comment 7

[Wayne Thayer]

The answer to question #6 ("This issue was not caught during certificate reviews following the upgrade.") in the incident report is completely unsatisfactory. My conclusion from that response, combined with the findings in bug #1535873 is that AT&T is not suited to operate a publicly-trusted CA, and the remediation for this incident will not be complete until these intermediates have been revoked.

Doug: is August the time frame in which CAs controlled by AT&T will be revoked, or is that just when they plan to transition to the managed service?

Also, please update this bug when all misissued certificates have been revoked.

Comment 8

[douglas.beattie]

Wayne,

Since these certificates do not contain AIA or CDP, I believe we are better suited to focus all efforts on shortening the timeline for moving to the GlobalSign hosted service. The majority of their certificates have expired and the remainder are not accessible via browsers. If you believe that we should focus on the replacement (without revocation) task anyway, we can do that, but we'd rather start replacing them all in June (for the final time) with certificates issued from the GlobalSign hosted CA. At the moment we're collectively 100% focused on closing down their CAs and setting up a hosted service.

We just had a management call and they have committed to us that we will be able to revoke their CA on August 1st.

I plan to post the PEM files for all misissued certificates this week.

Comment 9

[Wayne Thayer]

(In reply to douglas.beattie from comment #8)

Since these certificates do not contain AIA or CDP, I believe we are better suited to focus all efforts on shortening the timeline for moving to the GlobalSign hosted service. The majority of their certificates have expired and the remainder are not accessible via browsers. If you believe that we should focus on the replacement (without revocation) task anyway, we can do that, but we'd rather start replacing them all in June (for the final time) with certificates issued from the GlobalSign hosted CA. At the moment we're collectively 100% focused on closing down their CAs and setting up a hosted service.

I agree that focusing on the move to the managed CA makes sense.

We just had a management call and they have committed to us that we will be able to revoke their CA on August 1st.

I've changed the 'next update' on this bug to August 1st...

I plan to post the PEM files for all misissued certificates this week.

...but please do still post this information.

Comment 10

[douglas.beattie]

I've received a file with 220303 misissued certificates from ATT, but it's too large to include as an attachment (max is 10Mb and this is 120Mb). They will be available at this link for a couple of weeks and after that you can request them from me directly.

https://drive.google.com/open?id=1l3g6FLGXAhzgseC-ZbG0GcTbtNrjfUrC

Comment 11

[Ryan Sleevi]

Doug: I realize the update is not scheduled until August 1, but I want to make sure that things are progressing as scheduled.

Comment 12

[douglas.beattie]

Yes, I confirmed today that they have already started replacing some of the Manual certificates (the ones they need to work with their customers on) and they have actively been testing their automated provisioning system against a test account.

You probably know this already, but they are issuing certificates under this CA.
https://crt.sh/?caid=126736&opt=cablint
Some will be compliant with the Chrome CT policy, but the majority are for machine to machine use and won't be posted to CT logs.

Comment 13

[douglas.beattie]

AT&T is on track to complete their certificate replacement this week and will be ready to revoke the 2 active CAs on Monday, August 5th. I'll report back when that has been completed.

Comment 14

[douglas.beattie]

AT&T revoked their 2 CAs yesterday:
https://crt.sh/?id=11351488
https://crt.sh/?id=12625559

On August 21st, GlobalSign will be revoking the CAs we issued to AT&T to completely close the loop on this:
https://crt.sh/?id=11957246
https://crt.sh/?id=11351487

Comment 15

[douglas.beattie]

We're going to post the new CRL on 30 August, sorry for the delay.

Comment 16

[douglas.beattie]

These 2 CAs have been revoked as discussed above:
https://crt.sh/?id=11957246
https://crt.sh/?id=11351487

and the CRL has been updated and is now available at both of these locations:
http://crl.globalsign.com/gs/trustrootsha2g2.crl
http://crl.globalsign.com/trustrootsha2g2.crl

Comment 17

[Ryan Sleevi]

Doug: Could you explain the delays between Comment #14 and Comment #16. What caused them, and what GlobalSign is doing to address that cause going forward?

Comment 18

[douglas.beattie]

Ryan, yes here is some background information:

1. AT&T had revoked the issuing CAs, so all AT&T SSL certificates were already successfully revoked, so the timeline was not deemed to be critical to revoke the Intermediate CAs held by AT&T.

2. If this had been a critical revocation to push out we would have created a key ceremony for this one action and it would have been done quickly and efficiently, but given that this was to just wrap things up (revocation at 2 levels in the hierarchy) with AT&T at the next higher level in he hierarchy the priority was not deemed to be as high.

3. This key ceremony was conducted in a new location (first time) and we wanted to have a buffer in the tested/approved scripts such that we could conduct this at our primary location the following week if needed.

Does that answer this question?

Comment 19

[Ryan Sleevi]

I think so. To make sure I parse the narrative correctly:

GlobalSign was going to revoke the certificates it had issued on 08/21 (Comment #14). In Comment #15, on 08/22, GlobalSign determined to delay until 08/30, without providing more detail. Now that the actual revocation has happened, the reason for the delay was that the nature of these certificates required performing a key ceremony in order to produce a CRL that revoked them. This key ceremony was planned to be performed at a new location, and thus the additional time (from 08/22 until 08/30) was used to review the tested/approved scripts. Had this failed, GlobalSign would have delayed a further week from now, and performed the same activities at the primary location.

GlobalSign made a determination that this was not an urgent/critical matter, on the basis that despite AT&T maintaining the technical capability to issue new certificates, GlobalSign felt confident that they would not do so. As a consequence, GlobalSign did not see an urgency to revoke, versus the benefit of being able to perform this at a new location.

Did I get things roughly right?

If so, I think the question is: Why didn't you just say that in Comment #15? The information in Comment #18 is exactly the kind of information that we want CAs to be providing early and often, and it sounds like this information was determined well before the deadline had been exceeded (by Comment #15).

I mean, we similarly saw a deadline slip from Comment #13 (committing to Aug 5) to Comment #14 (actually performed Aug 7).

Do you have a plan in place to keep the community better informed, and to provide clearer timelines and commitments going forward?

Comment 20

[douglas.beattie]

Ryan,
Yes, you have the basics accurate, and to provide additional clarification: The key ceremony was scheduled for 8/21 and I thought that was also the date for the CRL; however, I was incorrect. The Effective Date for the CRL was always planned to be 8/30 and I missed that detail in the plan when I reported on this bug above in comment #14.

Comment 21

[Ryan Sleevi]

Thanks Doug.

I'm a little concerned with Comment #20's answer to Comment #19's question, as it seems like the answer to "Why didn't you just say that?" is "I made a mistake" and the answer to "How will we prevent those mistakes going forward?" is... unanswered.

I think had the details from Comment #18 and Comment #20 been available in Comment #14 / Comment #15, then I wouldn't have needed the follow-ups in Comment #17 / Comment #19. In general, sharing more information, clearly, is highly desirable, and helps understand the nature and causes for delay and thus helps ensure that the community that trusts on the CA is reliably informed.

I'm passing this over to Wayne to see if he has any follow-ups, otherwise it sounds like Wayne's concerns in Comment #7 have been addressed.

Comment 22

[Wayne Thayer]

It appears that all questions have been answered and remediation is complete.