Closed Bug 1401407 Opened 8 years ago Closed 7 years ago

DigiCert: Mis-Issuance Rekey certificates

Categories

(CA Program :: CA Certificate Compliance, task)

Product:
CA Program
Component:
CA Certificate Compliance
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: jeremy.rowley, Assigned: jeremy.rowley)

References

Details

(Whiteboard: [ca-compliance] [ov-misissuance] [ev-misissuance])

Attachments

(1 file)

Expired DCV.csv
90.96 KB, application/vnd.ms-excel
Details
User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.91 Safari/537.36 Steps to reproduce: On Friday, Sep 15, we discovered that 1090 certificates were rekeyed using expired domain validation documents. In each case, the original certificate’s domain was properly verified at time of issuance using an approved method. Organization verification properly completed for each rekey prior to issuance. This means, to get a rekeyed certificate with expired validation data the following had to be true: A. Unexpired organization validation information, B. A previously issued, unexpired, and unrevoked certificate for the same domain as the rekeyed certificate, C. Valid domain documentation for the original certificate at time of issuance, and D. All necessary access and permissions on the account issuing the original certificate. In parallel to the on-going investigation, we are reverifying the affected domains using one of the approved domain methods. So far, we have reverified 1021 of the 1090 affected certificates, leaving 69 domains outstanding. We are working with the domain owners to re-establish domain approval, but plan to revoke any certs that cannot be revalidated by end of this week. 1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, via a discussion in mozilla.dev.security.policy, or via a Bugzilla bug), and the date. [DC] We first became aware that there was a potential issue on Sep 14th during part of a routine internal review. We noticed some of the domain control validation documents were older than permitted under both the Baseline Requirements and EV Guidelines. On Sep 14th, we confirmed that rekeys in a validation system did not follow the proper path for verification of the timestamp associated with domain documents, instead using the timestamp of the domain documents on the original certificate. We patched our system early Sep 15th, preventing the system from further re-using expired domain documentation to rekey a certificate. Over the weekend, we worked on this report while revalidating domains associated with the misissued rekeys. 2. A timeline of the actions your CA took in response. [DC] The same day the staff reported the potential issue, we began investigating the root cause. On Sep 15th, we confirmed that older validation code improperly reused domain validation when rekeying a certificate. We thought we patched this after the CAB Forum’s previous discussion on what constitutes a new certificate. If you recall those discussions, DigiCert mistakenly believed rekeyed certificates were considered the same certificate, not requiring domain reverification. The original system believed duplicate certificates and rekeyed certificates were identical to the original cert (despite a new serial number) and relied on a previously completed validation outside of the permitted reuse period. Ultimately, we decided to amend our issuance process because of the CAB Forum discussions but failed to properly communicate this requirement internally. Although most workflows were patched to follow our “new issuance” process, one workflow path was not. Validation documents properly expired in the system, preventing issuance of “new” certificate orders using expired documentation. All organization documents expired in the system according to the proper guidelines, blocking both new issuance and issuance of rekeyed certificates. 3. Confirmation that your CA has stopped issuing TLS/SSL certificates with the problem. [DC] Confirmed. The system was patched on Sep 15th to require valid domain validation prior to rekey. This was within 24 hours of receiving an internal message about the issue and within two hours of confirming the issue’s existence. 4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued. [DC] 1090 certificate rekeys so far. Impacted certificates will be provided in the bug list and will also be available through CT (we’re still working on this part). 5. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem. [DC] Provided as an excel file on the bug list (once compiled). 6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now. [DC] The issue was intentionally introduced during DigiCert’s earlier days because of a mis-understanding by the original DigiCert team about whether rekeys constituted new issuance rather than duplication of a previous certificate. We thought we fixed the issue back when the CAB Forum discussed this topic, but apparently missed a work flow path. One mitigating factor is that all rekeys were initiated at the request of a previously verified and currently active account holder associated with the domains and a currently active certificate holder. Although the certificates had improper documentation, there was an active cert with the same name in each case. Also note that all rekeys have the same expiration date as the original certificate, meaning the rekeyed certificate would expire the same time as the currently active, non-revoked certificate. Over the weekend, we revalidated 1021 domains, limited the non-verified pool to 69 domains. The error avoided detection because there was a misunderstanding by former DigiCert staff that all certificates, regardless of whether the certificate is a rekey, duplicate, modification, etc, requires domain verification completed within the established BR timeframe. The new team discovered this exception for rekeys during their routine code review. 7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things. [DC] Everyone is now fully aware that all certificates are “new” certificates, meaning they follow the same process regardless of whether the certificate is a duplicate, rekey, or modification. We’ve patched the system to remove this alternate work flow and prevent additional issuance without proper validation. We are currently revalidating the 1090 certificates. We are also deprecating this validation service in favor of a new, robust system. Although the system is still heavily used, a complete switch to DigiCert’s v3 validation services will occur early next year.
Assignee: kwilson → jeremy.rowley
Blocks: BR-Compliance
Whiteboard: [ca-compliance]
> On Sep 14th, we confirmed that rekeys in a validation system did not follow the proper path for verification of the timestamp associated with domain documents, instead using the timestamp of the domain documents on the original certificate. I'm not sure I fully understand this - could you explain a bit more about this? I'm wondering if maybe you forgot a few words or phrased it incorrectly. > but failed to properly communicate this requirement internally This seems like an important part of the root cause. That is, the reuse of documents explains the what, but this begins to explore the why. It would be useful to continue unpacking this - you can get a sense of what I mean through a process like https://www.linkedin.com/pulse/use-5-whys-find-root-causes-peter-abilla > but apparently missed a work flow path. Similar/identical to the above, it's good to unpack this through several layers. > will occur early next year. It sounds like monthly updates on the progress would be a reasonable expectation.
Did any of the domain revalidations fail? Gerv
No. None have.
Here's another go to try and answer Ryan's questions: On Sep 14th, a routine review of our code revealed that we rekeyed 1090 certificates without updating the validation document. In each case, the domain validation documentation was outside the time frame permitted under the EV Guidelines and BRs. Basically, the system ignored the expiration date of the validation document if the original certificate was valid and the organization documentation was not expired. The system copied the validation documents from the original certificate, reusing it for the rekeyed certificate with an updated timestamp so everything looked perfect from the audit perspective. Note, that the expiration date of the documentation was not updated by the process so no new certificates could be issued except for additional rekeys of the original certificate. Although we internally discussed the need to ensure all certificates followed the same validation path, regardless of how DigiCert originally classified them, this change was never fully implemented. Part of the reason was the people originally designing the code no longer worked at DigiCert. There was one workflow that was missed. DigiCert originally categorized certificates into three types: 1. New issuance (changed domain, changed org, etc) 2. Duplicates (same cert, could be a new key) 3. Rekeys (new key but a revoked certificate) 4. Modifications (new org info) After the CAB Forum discussion on the issue, we modified our systems to treat each cert as a new cert with respect to all validation. However, not everyone was on-board with the reclassification of "new". The rationale was that if the original certificate is still valid and unrevoked, why would it matter if a new cert was issued with the same expiration date with the same key (or a different key even) duplicate? This meant duplicates (under our definition) followed a different path from the other three. In this case, the duplicate path was not properly patched because of a confusion on which of the four counted as a new certificate. Everyone was clear that the DigCert definition of Modification, New, and Rekey counted as new. However, duplicates was ambiguous. What happened is one path on one platform wasn't modified to require a proper domain verification refresh. This is where the certificate was rekeyed but classified as "Duplicate" because the cert wasn't revoked, modified, or expired. The communication from me was that, "All new certificates, including rekeys and modifications, should follow the same validation path." I thought this was industry-wide vernacular, meaning all certificates would use the correct process to expire and revoke certificates. As you can imagine, this left a gray area on duplicates, which is why one path remained unchanged. Two of the reasons this was not properly communicated to the dev team is DigiCert didn't have a product team back then and multiple dev teams could touch the CA. Each of our handful of dev worked on the CA as necessary. Over the last coupled years, we restructured things to create a product team (lead by me) to ensure all products are properly defined, scoped, and built in compliance with the BRs. We also created a CA dev team that is the sole set of trusted developers who can touch and maintain the CA operations. This gives us tighter control over CA changes and assurance over the BR compliance. I realize sometimes I'm not as precise as I should be (this thread is case in point), so we have multiple individuals involved in making sure we are correctly distilling BR requirements into dev projects. I think you know Clint Wilson, who is one of these primary people. He's awesome. Happy to give monthly updates on our new validation system roll out.
As an update - Of the 1090, we revoked five certs where we couldn't reverify or reach the original requester. I'm still working on getting them all submitted to CT logs. We've also finished a complete scan of all certs to see if any other followed this same path. Should have that data posted soon.
The five revoked certs were: *.kit.nl; 0723B1BD0770FF40C5D6C2DA8C5C1B74 *.txunited.com; 0DB4EE06CB0ADBCD9C55677B740BDAFB mh.bcpss.org; 0DA7E6A98FEF93BDD2EB38BEE615C10F www.bcpss.org; 077AE974560C75041A26B5030C39110A *.lindquist.com; 08893A184CECCF928C615E6C0690B358 Of these five, three were replaced after the domain owner completed the appropriate verification. Two never replied post revocation.
Our finished scan showed a total of 2739 certs issued with expired domain validation, including the 1090 previously reported. We are currently working through the remaining certs to re-verify the domain with the contact owner. All the certs should be uploaded to CT this next week. Anything that doesn't complete re-verification will be revoked by end of week.
I should clarify that last number. There were actually 4263 certs issued with expired domain validation docs. However, only 2739 were not reconfirmed post-issuance through an approved domain validation process.
Attached file Expired DCV.csv —
All certs have been added to the CT database. We're still working through the remaining cert validations.
Hi Jeremy: any update? Gerv
We're still working through them. Down to 187 certs left. If it's okay with Mozilla, we plan to revoke any certs not re-validated and publish a list here of the revoked certs.
Getting a final list today. I've been out for a few days. Should have an updated before tomorrow morning.
There were a total of 112 certs revoked in batch 2. These were: 0B16173E70A77C813D31ACB3D8A3A758 011DA9E0DB948D26B7149A1E286386B9 0FE07A1FAF30382668AB178C349F7A11 083FF3CD2CA49899C5B8C1030D6D1DAC 08B7BB24AE4791588E42E5E208570D2D 0EF25D5BD659A57E0423B810D70D4FD1 0CE817F028571741BE47E91D405630FE 0792B0E7A03F3867E6C87B4003FFA7C8 0BC2C8341F3B3E3E604850B1D682168E 06264731F6D8C646ECCD56CEBD17331E 07A7171B50D04DB104330354ACD2575B 0DEAFEAF60D2D6BEDE235E977BE948B6 053DB7E60AF762655EEE4FBBC072ED63 014CF2DC4D2DDEC0DAA9EEDC7F864F42 03E6186B8F0A625C1D045079BDFEF87A 04A4EB0D4A4E461F95EC2136C36AFF9A 082C5D15FBD271866FBFEA3B3366C772 08F530AC337B02F78890BB1A3DD8F409 06E6C0811963A6FAA4379F46793C6D50 0A2E88E1D6629FFEDCD8FB058A8FA34F 0C87BF037D10554C1FDA366A696608B8 02877CB3B4F134492E0282E7DD88D855 051F2BF1BEFF65B4C6431227E5B6846E 08F7F0A3A1A9A4E7808ED0E93EF2477C 07A08FE896A9318174AA75F4E3E8BA15 07BFA228273BDA9DA30C96972766CBD5 071E58BA0ACD27B4FE457F80F8326B77 0F70B65591CC3D117B6F125365436C2D 0147C96BFC47ACF0B47DBB366F500935 0F5B546E2F298BAC942E920E4BE060E6 06B7F2A131108365CBE9F907DA487464 0764339FFFBC112E6A11A3EA6951A0B8 0BC831A9DD805823F23A73F3448BDD50 01E55C5545427F1B74082391065B63D1 088C4C2426D3E266E42213BCC9859C6D 01EFE817490585A145A7C3FB0D2830E4 0EAB6AA1428D813BF4B7B17B33872D69 0AE93E37840AD02D760E12C644CAEEC8 0E7760CDE9CD8748A31FFA61FF36DD6C 0E804AAA888ED5E6BD1FCF785F4DD478 08E633A25941C43A71CF5E5F8FE12544 08EBA31B4366832B887AFC6A8A552A4E 08506E6A7ABD57D9FD525DE0C8CC0C07 0EF76DD337DF21CE5FD290D06550488E 05105BA127B7A38B78F68AC0DE3BDDCE 069B0F9BBB2C3280BECC88B6BDF99DCE 0DDC135384657DAF62100C5768D4A005 021821DE8D1CDA26C4E689A60DA99AE1 0A69296189531434050FDAE8A6370EEA 068EF1B10A80A05B13E66F3E072F6EA7 09AF2946931CC82080DF9B453E987C4A 067D00875D52855DCF4570DB183E129B 09E17843F89A33B6AF66677C59C861F1 034FE5578BEFC9F38AADE7E283B68D73 0ADDE0DB2D1996006B5AD28E3734F69F 0D1239374198DFC838D87D48FB6B8C12 0540DE52340042EF417DAD7AFE461A65 0CC740D25F1166E72F387250406332B3 0F14BDD2BE6255FDCF8469FFDB41031E 036D0B115A333782654C5BB611955A64 0160AFC5A87D6EF2BD8CF4587A6FC3FF 0E163A808E49F75EB57FBC760D800822 0B504E6903E0AE625149AA09A4F577FD 06C02FF6AA890EE985377E2D77B9EF8F 0B95245A2AF082FFCC411C82A1F4206F 0FEA84BC54D5ADD63B04649B2202C474 0A9EE30403B5B27EECE1AA276C062435 04A283D6343DB2B210F916485B1EFEE1 0A04A889589E7441AD8149FF74A7F3B7 049267659953BE82C238A1753F47259A 05D0336A1C9EBD34CA2C3ADA85278501 07C09D6BEDBC35881B76E62CE589ABF8 08045AB95FCE4AE150F9308D0C1AFF88 05AEE06CD57CAB3EF6F384C5E478D65D 0B8B95B00F89C1FE809D790931EC84F1 0E147A71ABB11647D6907B0ADF5023FB 0F93A3A5BE4C1845A200F766CD27B266 0D33DB724006FE0FC5705678A8155754 03B0A05803C01B69734370400E8DD535 020C15650531CEB365A5C59A6D142983 081667C64E6B9EC73E2A08A856058840 09B9B6AE8DE9F2FDD275F6D9EEA8CA93 09679CE2B71AC50DD8860E14B1536B90 0F65941F8CA0A3C1541C3DD91A16B733 0D45C69B92444769D576B09EEA7B524A 0FF93AB347B592C1DF120AA8BA11EACB 0B20027304CF1AAE53E46FA9D060C64B 0AC2487E90C6B801BF4D93071E3BE2E5 0716460C11DBBC2F6AEF58654B7BD1BD 0ACFFA538CE299C99F3B9E208DAD501E 0AD7A3043EF8338219B861411943438D 0D470E6CAE991B0C9B9575397A908A1D 0F8E24863BDE865AB8ABE91AD600BD98 05E6C6E360450EE95E0122ACDD6BED77 0EBC8F87AFB816BD6BCB819395F70258 01C9F74C0B9DB09AA847AE88B6B0DAEC 03D9FDB5900240AC0796F850D7709C39 0C8F0FFB9C38A9BCE391878A2DB1F45A 06A250C383E24F6E518EBFA5FBDC8CF4 07D8EEFEFB6B01221B991306A078D83C 07A178D74D822AC1A30E660B1550E72D 01A99AEF9863F5C7CC8974CE05266E7B 0FEDD796AC819F3C8A88F3D113E7043D 07E411383AECE2C935981250332F6CB7 07F7AB085C359850DFA64959FE511C6C 0EEC33C7C7DFD0FB525870453605776B 04502B83A53C9E9F542D45EE36FAA3B3 098443FFEF060C4CF09CE388553DAD5C 0A7E0C4ABFC6A816E05FD924EC0B2977 02A1D7B69D1EEB4C08224824E9DF120A 054D780912A3CAFC04B3701DB7EA6CBE 011A732559FA97400441E4C17A937FEF This issue is complete minus the updates to our validation system. The status on that is that v3 is still being rolled out. The Symantec acquisition has temporarily delayed customer migration while we finish integration with various APIs.
Jeremy: please provide an update on the validation system improvements.
Flags: needinfo?(jeremy.rowley)
Whiteboard: [ca-compliance] → [ca-incident] - Need remediation status
They are pretty much done. We've got migration to it planned for everyone by the end of this year with all of the former Sym customers transferred over already. The transfer of remaining is going to happen in phases in Q1, Q2, and Q3. At that point, there will be a single point for all validation and issuance.
Flags: needinfo?(jeremy.rowley)
All certificates have been remediated and system updates are in place, so I'm closing this out.
Status: UNCONFIRMED → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
Product: NSS → CA Program
Summary: DigiCert Mis-Issuance: Rekey certificates → DigiCert: Mis-Issuance Rekey certificates
Whiteboard: [ca-incident] - Need remediation status → [ca-compliance] [ov-misissuance] [ev-misissuance]
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: