1456655 - DigiCert / ABB: Issues with DN, country code and keyUsage

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Brenda Bernal]

1.	How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

On 2018-03-16, Ryan Sleevi notified DigiCert regarding two separate CA certificates: one certificate issued by ABB for the ABB Issuing CA 8 with violations of DN, keyUsage, and country code (https://crt.sh/?id=78292767&opt=cablint)  and the other, ABB Intermediate CA 5, issued by DigiCert with invalid dNSName nameConstraints and without a country code (https://crt.sh/?id=78290245&opt=cablint).  

However, in preparing this incident report, we also became aware of a third CA certificate with problems similar to the one mentioned first above (https://crt.sh/?id=202668017&opt=cablint) – ABB Issuing CA 9. 

Ryan’s notice also mentioned a server certificate that was in violation of the 825-day maximum validity period.  This has been reported and marked as resolved in Bugzilla: https://bugzilla.mozilla.org/show_bug.cgi?id=1451446#c2 (that certificate will not be covered in this incident report).

2.	A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

-	2016/08/23: DigiCert created three CA certificates for ABB, including a CA certificate for the ABB Intermediate CA 5 with serial number 0f87aa7bb7fa2e649889df8f613ea020 and a subject distinguished name of CN = ABB Intermediate CA 5, O = ABB, C = CH.  The DNS name constraints of this certificate did not have a leading dot on any of the entries.

Upon review of the CA certificate, ABB responded that they were concerned that the domain constraints didn’t have paired entries per domain, which they argued was customary, as example "abb" and ".abb".   DigiCert responded that when we initially looked at RFC 5280 we didn’t think that the dot was needed as a general matter, but we hadn’t focused on the fact that abb is its own top-level domain.  We were thinking that .abb would restrict all FQDNs to the second level, e.g. example.abb, and that the approach conflicted with language in the RFC that says, “When the constraint does not begin with a period, it specifies a host.”  

-	2016/08/25: DigiCert received a CSR file from ABB that had the following distinguished name components: CN = ABB Intermediate CA 5, O = ABB, DC = ABB, DC = COM.
ABB indicated that there were business needs demanding delivery of a working solution by August 26, 2016, so we held another key ceremony and created a second CA certificate for ABB Intermediate CA 5 with serial number 0dddabcf71e65ef92c5d698929aa7aba.  This second CA certificate still had a DN with CN = ABB Intermediate CA 5, O = ABB, C = CH, and added the dot to the domain names, but also had name constraints marked critical.  

-	2016/08/27: ABB contacted us to say that the name constraints could not be marked critical and that we needed to re-perform the key ceremony. 

-	2016/09/01: DigiCert created a third CA certificate for ABB Intermediate CA 5, serial number 0e0c6af7191c5c96a1de52cf09359b5d, without marking the name constraints critical. This third CA certificate still had a DN of CN = ABB Intermediate CA 5, O = ABB, C = CH.  

-	2016/09/02: ABB reviewed the CA certificates and demanded that we explain why we changed the DN of the certificates from the CSR, discussed above, and informed us that the different DN was preventing issuing CAs below the Intermediates to chain.  ABB reiterated that we use the DN provided in the CSR.  From that point forward, we then used the DN from the CSR.
 
-	2016/09/07: DigiCert created a fourth ABB Intermediate CA certificate with serial number 0d711c842fe9d67e9dcc22a21cb65cdf.  It had a subject DN of CN = ABB Intermediate CA 5, O = ABB, DC = ABB, DC = COM.  

-	2016/11/02: ABB contacted DigiCert to request changes to ABB Intermediate CA 5’s name constraints, and on November 15, 2016, we created the fifth CA certificate, serial number 0704af9b01e020849378ce2e5fc154ca, which is the subject of this incident report.

-	In preparing this incident report, DigiCert became aware that the Valid From dates for the two issuing CAs created by ABB is 3-August-2016.

3.	Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

ABB has stopped issuing certificates as of 2018/03/19.

4.	A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

https://crt.sh/?id=78292767&opt=cablint and https://crt.sh/?id=202668017&opt=cablint have valid from dates of 3-August-2016.

https://crt.sh/?id=78290245&opt=cablint was created on 15-November-2016.
These are the only certificates that exhibited the problems discussed in this incident report.  DigiCert is in the process of determining how many end entity certificates have been issued by the ABB Issuing CAs.  We will update this incident report once that number is determined.

5.	The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

See 4 above.

6.	Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

For the certificates with no country code set, we neglected to refer ourselves to Section 7.1.4.3.1 of the Baseline Requirements, which requires that CA certificates have a country code.

For the certificates with the issues with the domain name constraints, we had a different interpretation of the RFC.   We thought that it would depend mainly on how the software being used would interpret RFC 5280, and so decided that it would be the safer approach to include the dot.

7.	List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

This issues noted are attributed to two factors:
1)	 We missed the requirement to implement the required country code when creating the issuing CAs for ABB.  This was one of the first technically constrained CAs we created when we took over the OmniRoot relationships.  We take making these mistakes very seriously and have implemented stringent quality assurance process, with proper checks and balances, over our key ceremonies when we newly create issuing CAs before releasing them for use. We have very experienced and technically qualified crypto personnel who will continue to perform these ceremonies with the proper controls in place now.  

2)	We had a different interpretation of the RFC related to the domain name constraints versus what is being reported in error via cablint.  We are still seeking clarifications on the contradiction between RFC 5280 and these flagged errors.

Lastly, we previously adopted a policy to move all external TLS issuance into our managed services. ABB is migrating to a hosted solution for TLS certs.  There should be no further issuances off their subCA platform.

Comment 1

[Wayne Thayer]

Brenda: thank you for submitting this report. Can you provide an update on the certificate information referenced in #4 and 5 above? Also, in reference to your response to the cablint DNSName nameConstraint errors, do you still believe that cablint is wrong, and if so, have you brought this to the attention of the cablint maintainers?

Comment 2

[Brenda Bernal]

Hi Wayne, we have a preliminary estimate of the cert count for #4 above and waiting for a final and more precise number back from the CA operator.  We will get that information to you shortly.  As for the dot prefix issue, we will clarify that shortly.

Comment 3

[Brenda Bernal]

Hi Wayne:

Here is an update on the cert count for 4 above) regarding end-entity certificates from the ABB Issuing CAs: 
- 502 active public SSL certs from the CA8
- 165 active public S/MIME certs from the CA9

Commentary on the dot prefix issue will be forthcoming.

Thanks, Brenda

Comment 4

[Tim Hollebeek]

With respect to the dot prefix issue, interpretation of the relevant RFCs is extremely subtle.  Peter Bowen actually made the same mistake we made in his original implementation of cablint.  You have to be reading RFC 5280 extremely carefully to get name constraints right, since some of the text is downright misleading.  It took a fairly long internal thread among people who are very familiar with RFC 5280 in order to figure out exactly what the right answer is here.

RFC 5280 explicitly discusses the use of ".example.com" for name constraints.  But you need to look a few paragraphs backwards to realize that the relevant paragraphs are scoped to URIs only, and do NOT apply to dnsName name constraints.  Then you need to look a few paragraphs forward, to realize that dnsName name constraints have completely different rules, where the constraint applies to names that have zero or more labels to the left, so the text about "example.com" vs ".example.com" about URIs is not relevant.  For a dnsName constraint, unlike a URI constraint, everything in the tree is already included without having to include both the domain and the dot-prefixed domain.

Now that we understand the distinction between the semantics of dnsName and URI name constraints, we won't include dot-prefixed dnsName constraints in future name-constrained ICAs we create, since they are both unnecessary and non-compliant with RFC 5280.

Comment 5

[Brenda Bernal]

Hi Wayne, Is there additional info you need at this point? Cert count posted above on comment#3 and point of view on CABlint / dot prefix issues are described above in comment#4 by Tim. Thanks, Brenda

Comment 6

[Wayne Thayer]

Were any certificates issued from CA5? Can it be revoked?

I assume that some certificates issued from CA8 and CA9 are still active and there are no plans to revoke them? If so, when does the last leaf issued from each of these intermediates expire?

Comment 7

[Brenda Bernal]

CA 8 and CA 9 are signed by CA 5 and are therefore dependent on it remaining valid.  CA 8 and CA 9 last leaf certs expire: October 19, 2020; we are working on accelerated replacement.

Comment 8

[Wayne Thayer]

Brenda: I plan to leave this bug open while these leaf certificates are being replaced, until these intermediates can be revoked. Please provide periodic updates.

Comment 9

[Steve Medin]

ABB have not yet begun proactive replacement. The current counts have held steady at 165 email certificates under CA9 and 502 TLS server auth certificates under CA8. Expirations will begin in May for CA8 and March for CA9.

Comment 10

[Ryan Sleevi]

Checking in on this: Comment 7 proposed an accelerated replacement, and three months later, Comment 9 suggests no replacement has begun.

Where do we stand on things? Comment #8 called for periodic updates.

Comment 11

[Brenda Bernal]

My apologies for the long pause for a response due to the holiday break and the pace of getting information back at this time.

ABB has been making progress towards replacing email and server certificates as per our guidance. They expect to have all of their email certificates and a majority of their server certificates replaced by March 2019. Each server certificate requires coordination with each owner to understand deployment impact, which ABB has been working.

ABB’s delay in replacing all certificates is due to a diverse and geographically dispersed install base that is beyond the norm. For example, we are aware that ABB has certificates on servers that are used for critical civil infrastructure.

There’s another complication with this migration that we are working with ABB to resolve. Two of ABB's subordinate CAs issue code signing certificates that are used in Java. Revoking such a CA causes a problem. Even when code is timestamped prior to the revocation effective date of the CA that issued the end entity certificate that signed it, Java ignores the CA revocation effective date and treats the CA as if it was never valid, none of its certs were ever valid, and all code signatures are invalid. ABB shared this concern and we verified this behavior with our contacts at Oracle. We wish to explore alternative options such as OneCRL and CRLSet for the code signing CAs.

Comment 12

[Ryan Sleevi]

Thanks Brenda. I've re-read this bug to make sure I understand the full set of data and proposed plan, and am attempting to summarize it:

Baltimore CyberTrust Root
           |
           V
ABB Intermediate CA 5 (CA5)
    |                   |
    V                   V
Issuing CA 8 (CA8)   Issuing CA 9 (CA9)

• 2016-08-23: DigiCert creates a certificate with Subject "CN = ABB Intermediate CA 5, O = ABB, C = CH".
  • As best I can tell, this certificate, with serial 0f87aa7bb7fa2e649889df8f613ea020, is not disclosed via CT.
  • This certificate had no leading dots in name constraints, with nameConstraints marked critical.
• 2016-08-25: A second intermediate, with the same name, is created
  • As best I can tell, this certificate, with serial 0dddabcf71e65ef92c5d698929aa7aba, is not disclosed via CT.
  • This certificate had leading dots in name constraints, with nameConstraints marked critical.
• 2016-09-01: A third intermediate, with the same name, is created
  • As best I can tell, this certificate, with serial 0e0c6af7191c5c96a1de52cf09359b5d, is not disclosed via CT.
  • This certificate had leading dots in name constraints, with nameConstraints not marked critical.
• 2016-08-03: ABB creates "ABB Issuing CA 9" and "ABB Issuing CA 8"
• 2016-09-07: A fourth intermediate, with the Subject "CN = ABB Intermediate CA 5, O = ABB, DC = ABB, DC = COM" is created
  • As best I can tell, this certificate, with serial 0d711c842fe9d67e9dcc22a21cb65cdf, is not disclosed via CT.
  • This certificate had leading dots in name constraints, with nameConstraints not marked critical.
• 2016-11-15: A fifth intermediate, with the name, is created
  • This is serial 0704af9b01e020849378ce2e5fc154ca, https://crt.sh/?id=78290245
• 2018-03-16: CA is notified of problem
• 2018-03-19: ABB ceases issuing certificates
• 2019-03-XX: The majority of TLS server certificates under CA 9 are expected to be replaced
• 2019-03-XX: The first of 165 email certificates under CA 8 begin to expire
• 2019-05-XX: The first of 502 TLS certificates under CA 9 begin to expire
• 2020-10-19: The last of the certificates under CA8 and CA9 expire

There are, I think, new questions from this:

1. Where are the copies of CA5, versions 1-4? Have I missed their disclosure/existence?
2. The EKUs for CA5, version 5, is https://crt.sh/?id=78290245, and contains "TLS Web Server Authentication, TLS Web Client Authentication, E-mail Protection, 1.3.6.1.4.1.311.21.5, OCSP Signing". Notably, this does not contain the EKU "1.3.6.1.5.5.7.3.3", aka id-kp-codeSigning, which is relevant for Java platforms.
   a) Could you explain to me the basis and relevance for the concerns about Java Code Signing intermediates beneath CA5?
   b) Have these issuing intermediates (presumably, not CA8 or CA9) been disclosed in some fashion?
3. What are the revised timelines, given the statement that the majority will be replaced by March? Have these 502 TLS certificates been disclosed?
4. The root cause analysis only focuses on DigiCert's issuance of CA5; it does not cover ABB's (mis)issuance of CA8 and CA9. Given that CA8 and CA9 are the result of misuse of CA5, it's important to understand what controls exist, failed, and changed around CA5, CA8, and CA9 that justify the risk of non-revocation.

Comment 13

[Ryan Sleevi]

Brenda: Do you have updates?

Comment 14

[Brenda Bernal]

Attachment: ABBIntermediateCA5-constrained.zip

referenced in Jan 14th response to Ryan Sleevi

Comment 15

[Brenda Bernal]

Attachment: ABB Intermediate Hierarchy.png

Submitted as part of January 14 response (2A) to Ryan Sleevi

Comment 16

[Brenda Bernal]

Hi Ryan, Here's the updates:

1. Please find attached contained in the zip file. These were no disclosed (via CCADB) as they are technically constrained.

2 a) CA5 is not relevant for Java Code Signing. It is relevant for CA4. Please see diagram attached from ABB that visually represents this.

2 b) They have not been disclosed on CCADB as they are technically constrained

3. We are awaiting a response from ABB on this timeline.

4)ABB's misuse of CA5 relates to the renewal in 2016 that carried forward an existing domain component naming scheme predating the BR country code mandate. For CA8 and CA9, ABB didn't know that key usage needs to be critical or that country code was required.

DigiCert failed to correct the naming of CA5 in 2016 and we did not adequately advise ABB on the content requirements of their issuing CA’s. ABB's CA5 adequately informs relying parties of their identity. The same can be said for parties who rely on CA8 and CA9. While country code is required and is useful information and ABB does have a headquarters location, they operate globally and the relevance of country code to a relying party is minimal.

Comment 17

[Ryan Sleevi]

Thanks for clarifying, Brenda.

I'm not sure, but it sounds like ABB CA 4 is out of scope. Among other things, I'm showing that it's already revoked - https://crt.sh/?id=881434380 , and that quick read is that it would also have counted as not capable of causing TLS issuance and thus out of scope. So I'm not sure I understand the concerns from Comment #11 regarding revoking CA5 / CA8 / CA9.

Regarding the ABB root cause analysis, I'm not sure that really helps much. Subordinate CAs, technically constrained or otherwise, are expected to adhere to the BRs, and that both the issuing CA and supervising CA (if distinct) have controls in place to ensure this. Regardless of the views of the importance of naming, I think there is a concern in which BRs weren't followed by ABB, and understanding why that happened and the steps to mitigating that are important. I think this is even more important if the BR-specified revocation is not being followed, in order to understand how the demonstrated risk is being addressed, mitigated, and in the future, prevented.

Comment 18

[Ryan Sleevi]

Any updates, particularly on how Java factors in to these delays?

Comment 19

[Brenda Bernal]

Thanks, Ryan, it’s correct that CA4 and the ECC CA1 intermediate and issuer are not related to CA5, CA8, and CA9. There is no Java influence on these three CAs. We brought these into this discussion because ABB raised concerns that we shared with Oracle and that affected the speed of our responses here.

ABB will be ready to revoke CA9 by replacing all email certificates by the end of March. Revoking CA5 and CA8 requires that all clients trust the new server chain. The clients involve a mix of easily adaptable browsers and field-deployed non-browsers installed in equipment that is owned by ABB’s customers. The latter requires customer/vendor coordination, testing, and often physical deployment with tedious logistics after devices are fielded. ABB provides power distribution and other civil infrastructure services. This is what revocation breaks.

We understand our RCA is lacking risk mitigation and prevention. We agree that the primary risk is failed adherence to the BRs. We currently follow a documented CA review and creation process that examines the request before it is signed and examines the result before it is released. When the version of CA5 created before the BRs came up for renewal, we failed to fix the domain component naming in part because we were not subjecting CA signing requests to the current compliance team review process established in December 2017. Less people were examining requests than now, even though one person should be enough to have caught this problem.

Because we didn’t change CA5’s DC naming (or dot-prefix constraints), which would have required a conversation with ABB and reconfiguration of their ADCS-based PKI, we didn’t communicate the need to use country code rather than domain components in CA8 and CA9. ABB would have needed to stand up a duplicate ADCS PKI for the country code naming, manage both concurrently, and migrate users to the newer PKI. This didn’t happen because we didn’t stop the renewal of CA5 with domain component naming.

Like we confessed above, when we put dot prefixes on the dNSName constraints, we thought we were doing name constraints right.

The risk we present at a high level is BR violation, and we took steps to address that before this incident was reported. At a low level, we have a misleading representation of ABB’s identity that doesn’t include their country, and we have unneeded name constraints. We feel that relying parties are not confused by the DC naming in these three CAs. We feel that including the dot-prefix constraints does not alter the base domains and FQDNs that are enabled by CA5. But those aren’t the risks. The risk is tolerance of BR violation, a mantra of brushing these matters aside.

We feel this is a managed risk given the openness of our disclosures and willingness to engage. We ask Mozilla to allow ABB to solve the TLS server issuing CA problem in coordination with their customers. We acknowledge that this is a BR violation. We know that the limited browser-using relying party impact doesn’t matter. That isn’t the only relying party for this PKI, though, it’s also the people who want their streetlights on.

We propose to follow up in early April with the CA9 revocation and a status on remaining certificates relying on CA8.

Comment 20

[Steve Medin]

ABB revoked CA9 today and they report 192 of 492 certificates under CA8 are replaced. We have requested a monthly update until the remaining 300 are resolved and CA8 and CA5 are revoked.

Comment 21

[Ryan Sleevi]

Brenda / Steve: It's been two months with no updates. Where do things stand?

Comment 22

[Steve Medin]

Thanks for the reminder. We asked ABB to provide monthly updates, we'll contact them now.

Comment 23

[Steve Medin]

248 remaining.

Comment 24

[Ryan Sleevi]

Note https://wiki.mozilla.org/CA/Responding_To_An_Incident#Keeping_Us_Informed notes weekly updates.

This is concerning on the slow progress on CA8, which as noted in Comment #3, refers to TLS issuance. From Comment #12, the original timeline was 2019-03, and Comment #13 modified this to 2019-04. In the intervening time between Comment #20 and Comment #23, a period of 2.5 months, a total of 52 certificates were revoked.

Comment #19 attempts to explain the challenges, but this doesn't really clarify the lack of progress here, or whether the expectation is that these will continue until they naturally expire.

I'm pushing to Wayne for a further analysis of this, to make sure I've not missed any explanations. I haven't really seen any discussion about how issues like this, systemically, will be prevented in the future. Comment #19 explains several challenges, but no mitigations seem in place to prevent further delays in revocation.

Comment 25

[Steve Medin]

ABB reports a troubling 31 certificates revoked, leaving 217 remaining. They were impacted by parental leave of a lead person in the PKI team. In their comments, they stated that they are frustrated with the lack of progress as we are and they have now set an aggressive goal to reach 80% completion (based on the initial 492 certs) by 9 August 2019 which would revoke 119 certificates within the coming month, leaving 98 valid at the milestone.

We ask that ABB are allowed to demonstrate this commitment before other actions are considered.

Comment 26

[Ryan Sleevi]

Wayne: I recommend this CA be moved to OneCRL. A lack of meaningful progress since 2018-04, and no concrete deadline for revocation, combined with the overall handling of this incident, have lead me to a vote of No Confidence in ABB. Every opportunity has been given here.

I want to draw special attention to Comment #10, as this also concerns DigiCert's management of Sub-CAs.

Comment 27

[Wayne Thayer]

I created bug 1566162 to document the subordinate CA oversight failure and remediation as a separate issue.

Steve: will ABB commit to, and/or will DigiCert specify a date by which this CA will be revoked?

Comment 28

[Steve Medin]

If you recommend a target, I am certain that the new management of the PKI would appreciate the opportunity to comply. We are prepared to take action as required. We would like to see what is accomplished as of 9 August given their commitment to step up the pace.

Comment 29

[Jeremy Rowley]

We will come up with a date today and post it.

Comment 30

[Jeremy Rowley]

We are currently proposing Aug 30th as the deadline, but are waiting to hear back from ABB. The original proposal was Oct 2020 so this will be a pretty significant acceleration. Will post a firm accelerated timeline when they respond in the morning. We were currently expecting end of they year but will convey that needs acceleration.

Comment 31

[Jeremy Rowley]

We are still on track to revoke on August 30th and are tracking progress.

Comment 32

[Brenda Bernal]

Update: We are on track with revoking on 30-August.

Comment 33

[Jeremy Rowley]

Still just waiting for Aug 30. That should be our next update.

Comment 34

[Brenda Bernal]

ABB CA 5 has been revoked. https://crt.sh/?id=78290245. I've also posted an update also on this bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1566162

Comment 35

[Tom Smith]

(In reply to Ryan Sleevi from comment #17)
I do not see CA4 as being revoked.

https://crt.sh/?id=155179795&opt=ocsp,zlint,x509lint

Comment 36

[Jeremy Rowley]

That CA4 cert is constrained to code signing only and out of scope of the Mozilla policy. It's shut down is being worked out with Microsoft.

Comment 37

[Jeremy Rowley]

I think this one is ready to close too?

Comment 38

[Wayne Thayer]

It appears that all questions have been answered and remediation is complete.