1535873 - GlobalSign: AT&T Insufficient Serial Number Entropy

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Wayne Thayer]

Doug Beattie posted the following incident report to the mozilla.dev.security.policy mailing list on 13-March:

When the serial number issue was first disclosed we reviewed all GlobalSign
certificates issued from our systems and found no issues wrt serial number
length. While all GlobalSign systems are compliant, one of our customers
running an on-premise CA that chains to a GlobalSign root, AT&T, uses EJBCA
and has been using the default configuration. They have been notified to
immediately stop issuance, update their configurations, replace and then
revoke all affected certificates.

Their Intermediate CA is: https://crt.sh/?caid=10154

Under that CA they have 3 CAs, and here is the estimated number of
non-compliant active certificates:
https://crt.sh/?caid=10155 (fewer than 200 active certificates)
https://crt.sh/?caid=12658 (14 active 10-day certificates)
https://crt.sh/?caid=10157 (4 active certificates)

1. How your CA first became aware of the problem (e.g. via a problem report
   submitted to your Problem Reporting Mechanism, a discussion in
   mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and
   the time and date.

When performing self-compliance check on our Trusted Root customers based on
emails to mdsp list with similar issues.

2. A timeline of the actions your CA took in response. A timeline is a
   date-and-time-stamped sequence of all relevant events. This may include
   events before the incident was reported, such as when a particular
   requirement became applicable, or a document changed, or a bug was
   introduced, or an audit was done.

3/1/2019: GlobalSign self-assessment on certificates issued from our data
center. All certificates are compliant as we had set sufficient serial
number lengths prior to the CABF requirement to move to 64 bits of entropy.

3/13/2019: GlobalSign initiated and completed assessment of SSL certificates
issued by our 3 remaining customers that have CAs chaining to GlobalSign
roots. We observed that one of these customers, AT&T, uses EJBCA with the
default serial number settings.

3. Whether your CA has stopped, or has not yet stopped, issuing certificates
   with the problem. A statement that you have will be considered a pledge to
   the community; a statement that you have not requires an explanation.

We have informed AT&T to stop issuance and will confirm that this is the
case tomorrow morning.

4. A summary of the problematic certificates. For each problem: number of
   certs, and the date the first and last certs with that problem were issued.

Initial reporting indicates there are fewer than 200 active certificates.
The links above can be used to identify the detailed list of certificates
and we will compile a complete list based on input from AT&T

5. The complete certificate data for the problematic certificates. The
   recommended way to provide this is to ensure each certificate is logged to
   CT and then list the fingerprints or crt.sh IDs, either in the report or as
   an attached spreadsheet, with one list per distinct problem.

We will compute a report shortly, but currently the scope is limited to the
3 CAs are listed above. Every active certificate under these CAs has serial
numbers that contain fewer than 64 bits of entropy.

6. Explanation about how and why the mistakes were made or bugs introduced,
   and how they avoided detection until now.

We will collect this information from AT&T in the coming days

7. List of steps your CA is taking to resolve the situation and ensure such
   issuance will not be repeated in the future, accompanied with a timeline of
   when your CA expects to accomplish these things.

We are working with AT&T to correct this problem. Our plans to revoke these
CAs and to terminate all Trusted Root SSL CAs is on track for August.

Comment 1

[douglas.beattie]

Status update from ATT:

Per our plan communicated on Friday, we successfully upgraded our EJBCA version in test/dev to a more recent release that supports 128 serial number entropy and are in the process of completing all of our testing by EOD today, Tuesday. We have plans to upgrade our production PKI environment tomorrow (Wednesday 3/20). After the upgrade all new certificates will use the 128 bit setting.

In the meantime we are meeting daily and have folks running queries to identify all certificates affected. I have a meeting with the team later this afternoon to review the progress on identifying all affected certificates.

We have confirmed 2023 certificates. We will continue to run reports and validate along the way. After we upgrade EJCBA tomorrow (Wednesday) our plan is to immediately reissue valid certificates for all ~17K NMD’s to make sure nothing falls through the cracks. We will do this over a 3 day period and perform the revocation on Monday 3/25.

That will take care of the majority of the certificates leaving approximately 171 certificates that we will work on in parallel.

Comment 2

[douglas.beattie]

Status update from ATT:

On Wednesday we successfully upgraded EJBCA and all certificates issued beginning 4:00pm CT will have 128 bit serial numbers. Last night we issued over 7000 certificates for NMD’s during our nightly run, all with 128 bit serial numbers. We continue to make good progress. I will send another update out tomorrow.

Comment 3

[Wayne Thayer]

Doug: my understanding from other affected CAs is that this merely requires a configuration change to EJBCA. Can you explain why AT&T needed to perform an upgrade? How old was the version of EJBCA they were running?

Comment 4

[douglas.beattie]

I've asked them to clarify this. In the mean time, here is the current status:

Over the weekend we reissued all NMD certificates (aprox 17K) and will be revoking all of the previous certificates for NMD’s by EOD today (Monday). We are also revoking many other certificates issued prior to the EJBCA upgrade. We still have some one-offs we are working on manually to replace and revoke throughout the week this week. By eod Friday 3/29 all affected certificates will be reissued and revoked.

Comment 5

[douglas.beattie]

Here is the update from AT&T on the version of EJBCA they were using:

The upgrade was from EJBCA 4.0.16 (r17223) to EJBCA 6.2.0 (r19221). Our previous version only supported a max serial number octet of 8 (64 bit) so the upgrade was required to support greater than 64 bit since the first bit was used to designate a positive number. We are now using 128 bit.

Comment 6

[Wayne Thayer]

EJBCA 4.0.16 was released in 2013 [1]. It appears that it relies on Java 6 which is no longer supported.

This is a technically constrained subCA, which means that it is not subject to disclosure or external audit requirements.

This subCA asserts policy 1.3.6.1.4.1.4146.1.60.1 which GlobalSign's CPS says is "Baseline Requirements Compatible". It also asserts the CABF OV OID (and one other policy that belongs to Wayport, Inc.)

How is it possible for AT&T to operate this certificate on 6-year old software while complying with the NCSSRs?

BR section 8.7 states: During the period in which a Technically Constrained Subordinate CA issues Certificates, the CA which signed the Subordinate CA SHALL monitor adherence to the CA’s Certificate Policy and the Subordinate CA’s Certification Practice Statement. On at least a quarterly basis, against a randomly selected sample of the greater of one certificate or at least three percent of the Certificates issued by the Subordinate CA, during the period commencing immediately after the previous audit sample was taken, the CA shall ensure all applicable CP are met.

How is it possible that "all applicable CP are met" when the GlobalSign CP incorporates the NCSSRs by reference?

[1] http://blog.ejbca.org/2013/06/ejbca-4016-released.html

Comment 7

[douglas.beattie]

We require all of our Trusted Root customers to assert their compliance with each of the Network Security Controls (1 by 1) and we review their responses. There wasn't an obvious requirement in the NCSSRs which triggered us to ask for the version of EJBCA they were using, which is why we didn't ask for it.

Comment 8

[Ryan Sleevi]

I believe the control Wayne was referencing was:

1. Each CA or Delegated Third Party SHALL:
   l. Apply recommended security patches to Certificate Systems within six (6) months of the
   security patch’s availability, unless the CA documents that the security patch would
   introduce additional vulnerabilities or instabilities that outweigh the benefits of applying
   the security patch.

How does GlobalSign otherwise measure adherence with this policy?

Comment 9

[douglas.beattie]

Response from AT&T:

Revocation Status:

Over the last 2 weeks we revoked over 42,000 certificates. We are still working through replacing the last of the 64 bit s/n certificates. Some of our customers did not want to move as quickly as we planned so we are working with them to get the last of the certificates reissued asap, moving through the change control process. Due to this our new date to complete this work is 4/10/19. In addition we are currently auditing our certificates to ensure nothing is missed.

Regarding the patching question:

Our patching cycle includes kernel and other OS package updates (RHEL patching), and if required would include EJBCA application upgrades. We upgrade EJBCA for security reasons and to gain additional features-like the failed Certificate Transparency, or most recently, to enable a higher possible serial number octet.

We also rely on our 3rd party PKI support agreement with StrongKey to advise us of known issues, patches or required upgrades. We are not aware of any security issues with the version we were running. The highest serial number entropy ANY of the EJBCA versions had until very recently was set at 8 octets by default. Only within the last few weeks has EJBCA released a version that set the default to 16. That default of 8 is why everyone now has to update their settings - ours just included an extra step to upgrade to a version that would allow up to 20 octets.

We will continue to monitor EJBCA patches and updates required as well as continue to work with our partner, StrongKey, to identify any updates required to EJBCA.

Comment 10

[douglas.beattie]

All misissued certificates with 63 bit serial numbers have been revoked.

Comment 11

[Wayne Thayer]

Doug: thanks for the update. Regarding the EJBCA version that was in use by AT&T, does GlobalSign believe that AT&T was in compliance with the NCSSRs the entire time they were running EJBCA version 4.0.16?

Comment 12

[douglas.beattie]

No we do not believe they were compliant, however we were unaware that they did not update their software in alignment with their patch management procedures.

Comment 13

[Wayne Thayer]

Just to confirm my understanding: GlobalSign's remediation to this oversight failure is that existing subordinate CAs operated by 3rd parties will be revoked by August 2019 and in the future GlobalSign will not delegate the operation of subordinate CAs to 3rd parties. Is that correct?

Comment 14

[douglas.beattie]

Yes, we are in the final stages of closing down all subordinate CAs hosted by 3rd parties. We expect all to be revoked in the August time frame with 2 exceptions:

1. AT&T is actively working on their replacement service and we expect that to be completed in August at which point they will stop issuance from their CAs. The revocation of their CA will likely be several months later when all of their certificates have been replaced. For the most part, they are highly automated and this should go quickly for the majority of the certificates, but there are some that will take longer.

2. We have a contractual relationship with a major browser to cross sign their root if that option is executed.

Comment 15

[Wayne Thayer]

Doug: thank you for the confirmation. With that I believe that this bug is ready to be resolved.