Entrust: Subscriber provides private key with CSR
Categories
(CA Program :: CA Certificate Compliance, task)
Tracking
(Not tracked)
People
(Reporter: bruce.morton, Assigned: bruce.morton)
Details
(Whiteboard: [ca-compliance] [dv-misissuance] [ov-misissuance] [ev-misissuance])
Attachments
(3 files)
User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36
Steps to reproduce:
- How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.
On 19 October 2020, Entrust compliance team was advised through a support escalation that a customer CSR was rejected as a private key was included with the CSR. Upon investigation, it was determined that there were previous requests with when the private key was provided that the certificate was issued. This was determined to be a miss-issuance as the private key was compromised.
- A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.
19 October 2020, 14:00 UTC: Entrust discovers that a private key was provided with a CSR. The certificate was rejected by the new CA software.
19 October 2020, 16:30 UTC: It was also discovered that the same private key was used with the previous certificate (https://crt.sh/?id=1954272817), as such, the certificate (Trigger Cert) must be revoked as the key was compromised.
19-22 October 2020: Investigation started to search for certificates which were issued where private keys had been provided. Process was also looking for application which created the issue and to eliminate false positives. During this process, there were 6 certificates which were issued to Entrust demo accounts. These certificates are listed in the were revoked during the investigation.
20 October 2020, 16:00 UTC: Trigger Cert was revoked.
22 October 2020, 14:55 UTC: Certificate list with compromised private keys for Entrust product line confirmed.
22 October 2020, 21:25 UTC: Certificate list with compromised private keys for AffirmTrust product line confirmed.
23 October 2020: 14:20 UTC: Confirmed all Entrust certificates were revoked.
23 October 2020, 20:10 UTC: Confirmed all AffirmTrust certificates were revoked.
23 October 2020, 20:22 UTC: Patch installed to reject CSRs which also include extra data.
23 October 2020, 20:40 UTC: Searched and found that 1 additional certificate was issued before the patch was installed.
23 October 2020, 21:24 UTC: Revoked additional certificate.
- Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.
The CA has stopped issuing certificates where the private key is provided with the CSR based.
- A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.
Entrust - 121 certificates
AffirmTrust - 12 certificates
- The complete certificate data for the problematic certificates.
Trigger Cert
https://crt.sh/?id=1954272817
Entrust Product Line
https://crt.sh/?q=C8641AB62BA90AB0B9A60A4438B75F9D4863CC64
https://crt.sh/?q=2774A944BB8879058EABABB097224B2AFA198DCF
https://crt.sh/?q=9A3301594A5A1F891A455658E0955204B8A923F8
https://crt.sh/?q=7A5893AA12438DD1CB1AF257E08E037498800926
https://crt.sh/?q=CD84A083F8CC5B495AC32DFE8B34E8D7C525B677
https://crt.sh/?q=49C2F917F1C79A9ADBD0C2B64E0F619369DCE97F
https://crt.sh/?q=76C91CA3DA155619DC01191635B9CBC735F6A359
https://crt.sh/?q=87CA9932DC1D3D054A7D9566FF9782D6910D5859
https://crt.sh/?q=8C7610FD89FD456E4984E00418DB3631A11778A7
https://crt.sh/?q=4C56CD77144B9D35D83EEED6904DA3AC781FDD2E
https://crt.sh/?q=5A303CED7C3E756F2C90F8083AC54BC72F44E11B
https://crt.sh/?q=1BD37E75EBACBDCB4DB3D5507F96E9EA321DB336
https://crt.sh/?q=7DFC70455DD33086C8DA47E19C4A7BA256A5C5BC
https://crt.sh/?q=7F83161486DA7D571F3D9249BEA0E5401C042FE8
https://crt.sh/?q=97844532B448DBA230E2125D46934A29E66ED16E
https://crt.sh/?id=1815647978
https://crt.sh/?q=6A6B781B9CB9C172A6D7F432B8EBEA170C11D654
https://crt.sh/?q=F6C7B895766B11DB490AA4CB10CA5C12F84215D9
https://crt.sh/?q=C96A824644D0D5100D5C7AE8F331B4EBCD2A3D11
https://crt.sh/?q=057C09FE32E36DD546EB02DC12FF48D00FBCE657
https://crt.sh/?q=0F7D001E7F6D850F1B54FD3ABA4627AA455C954D
https://crt.sh/?q=B5406D87E98105BB4BA96910CD00B68A76172481
https://crt.sh/?q=F195F6963AB489C2696B5A0B94D9E6547EEFAF97
https://crt.sh/?q=749CDBEBFE3D3CE550B73D277E312EEA18B6EA53
https://crt.sh/?q=DA57AD132C4C130E61B2C780F230F380D7856339
https://crt.sh/?q=FA0B793CB01F91CA4A7A9FB7F7A7213347ED66C7
https://crt.sh/?q=A9E1F7A780E54780084172FF95F68486B052370B
https://crt.sh/?q=E334FBAAA87399EEA0961CBF8BF1F56A2B4FC69F
https://crt.sh/?q=9DBDB33F72136AA4533DC2B1AA8097634A1B1D77
https://crt.sh/?q=559E2B273494BF4AD21403AADCBF7C0F0079EF53
https://crt.sh/?q=CFF85BF6BDFE639EC58035F9297BCD9B953F6967
https://crt.sh/?q=9FFF1B16CF57CB56D07408E892AFE2E479DCFA4D
https://crt.sh/?q=E7E18639FEB93A6B52D1F5DD24FFC6617E9B2A0C
https://crt.sh/?q=1694D641B177E76B830819B0971B1B7E40842423
https://crt.sh/?q=A9AE9CD27CA364D6C854C79539338A991F044038
https://crt.sh/?q=8C44ACB8AE9022A99472CD0C9411882DD46E63B9
https://crt.sh/?q=52A39232EA863F61EADA6CAA84DA4A399C5900AA
https://crt.sh/?q=8610171476F86CB89EC40817DFC967A85DC2C6DE
https://crt.sh/?q=3A0A845FDC0900E1141CF08E72585A5F3DDE432C
https://crt.sh/?q=4457B736F8B95A034067A9BA3F099729E4984027
https://crt.sh/?q=BAC83E638B0830A4B78662164FCC167E20719E52
https://crt.sh/?q=E26CC7048219A9234C37DC9A317856B8B089A0D4
https://crt.sh/?q=A2C975A665350C068BA12D2A6FF5918D6A95ED72
https://crt.sh/?q=E909681368CCAA662CFC18FFEB17627AB233BCB0
https://crt.sh/?q=E6B94584E588291F594A4A95BD651F24653E5ADB
https://crt.sh/?q=1BC73BA2CEBFF7906E2435192998EE5B20DE78AA
https://crt.sh/?q=BCBE6E3644DED2D1931C2EC952809A11E69807F5
https://crt.sh/?q=1B488A43032991312943811BFBA752D30341999C
https://crt.sh/?q=7FC475F1F0295A3E2EE2F6CD8D86F8826D24EF10
https://crt.sh/?q=DAD0C84EAF0431F3B648F4C9C6E581DF0937FE1F
https://crt.sh/?q=9E28A7C81977649641B03BC291FAB3C19B53C85C
https://crt.sh/?q=4C81D7E4363D8F79F50E9D30A46214E60B7F5EDC
https://crt.sh/?q=A4FA30F6587C9061F583C35AB11EFFDCA27590D4
https://crt.sh/?q=EFAB46E0F36CCD5E3A91474C5E11E007D848FE06
https://crt.sh/?q=C7A867C0D94E21D21827A4EE085E080C6D94648C
https://crt.sh/?q=D04E44663AA63812B9796150AFB32B421DACDBF4
https://crt.sh/?q=E333BA89FA4303266C0C7A23E7A1FC40227D27E0
https://crt.sh/?q=41CB9ADA9AEEA22B9072E19433A4FCF4C26A1F3C
https://crt.sh/?q=4B927C3280048ECB7304DE0CEBD9D63BCA894787
https://crt.sh/?q=69896C775DAD4C645ADF724B3E8D2A72B2CE7B49
https://crt.sh/?q=2F61C7D7FE56B44590890B8E01EEFE728E45AE9C
https://crt.sh/?q=5A868150F7FD6D1DD098FCF6DE7044B3D66ABBE0
https://crt.sh/?q=2F2C28A0A4803F74F1E80C3E546C7AFB5BB5BD0F
https://crt.sh/?q=6403E2F95AF980E6BC409BFC2EE4C269F855ED38
https://crt.sh/?q=D5A2CEFD69190856F3B13BA0C9EC4A04B8B14AAA
https://crt.sh/?q=362B3905FD95EB3FF681A4EDBE38098BB2C1B356
https://crt.sh/?q=C22B99A0AF31F4DDC050916E8BF557D2E0865B8A
https://crt.sh/?q=CF21151735A3B448886D9A3DDD8BB75E41D97466
https://crt.sh/?q=66559C94AA27E212E302D178A3242683E0F9985D
https://crt.sh/?q=BDA76A37F7ABC2484B7B6041FB4D3EF7B19B84B4
https://crt.sh/?q=9F17A5628DE6376D6C9D368368823341910EE773
https://crt.sh/?q=58D047481F104EC4B3CF5A4C945125983C22CC21
https://crt.sh/?q=12B9AF88F3867EE72D2FA307C95390B8A1C09E83
https://crt.sh/?q=E5225314A1E4AA4076E2FA1F35FF8E8ECC06D431
https://crt.sh/?q=594B12C147CE422D0B2B4A034C0D99AE3A40D09B
https://crt.sh/?q=13F1D4C61B482CA0F39016CF1D5D2AF593B104C2
https://crt.sh/?q=6B2D354183CAEF225C32EE903EEF12707C551260
https://crt.sh/?q=1BEAD7C55D84E1700AE56ED8E0A1E22501AE8B56
https://crt.sh/?q=70C51D9A753AA09DAA1F06885AA88C183DAE0E2F
https://crt.sh/?q=CA434CD76A215E208450777BE536EEB94C8488F0
https://crt.sh/?q=3FAA45AEB7E2485F34BD34AC016B35519312DB70
https://crt.sh/?q=D8B9BCF3630C8176B8C1B76CB7BE06C0B6D1304A
https://crt.sh/?q=10CBF2746CA8EFA983D45FCBFEDC1CEBD6397EB2
https://crt.sh/?q=A25062AA0F73E6294068672CBFD827DCC7E4DB1E
https://crt.sh/?q=446B7786CE3F2414B64A11125684F75B178B933B
https://crt.sh/?q=11E6BF94078CCA2C4ECF0E0C09CC9AC9D4E741DA
https://crt.sh/?q=B004A66A25CE0D79B6BC7532BD0AA114053DDE1C
https://crt.sh/?q=40A62D7F3ECF27D86F9D8607DA5905FCEF222B00
https://crt.sh/?q=A8468F84D23E87A5ADBDD808A949800033036622
https://crt.sh/?q=C2B45DCEFE7B813CC1F49BD39BA9A14DAC27D6E5
https://crt.sh/?q=4761CDAA61193E4CAA773ABE5A7FBBB767085BCE
https://crt.sh/?q=57193685BA683A8145B1CFF88C746D649B239B04
https://crt.sh/?q=237AAE41328A2EB8F6194B0845B215E088B192D4
https://crt.sh/?q=ADEAF3A18D2FF9CB5CD88C6A8BFBDE304EB65E12
https://crt.sh/?q=C3A5E2A0E662FF3E585F3114BD96DDE1CA1C20F5
https://crt.sh/?q=728FCF32B4D95A0A0EF0D87250304BC8C4A5645D
https://crt.sh/?q=A50FB51F1174901923EC864A1BCEB5F275B4D798
https://crt.sh/?q=5506F5E49BEAEAD28A182A78B0C117F812055A0D
https://crt.sh/?q=EF370C0D1489A4730AF004BAD91898DEFF39AE8C
https://crt.sh/?q=49746262F2BCD24E62AA9D2E3E5A8E0FFD246DA7
https://crt.sh/?q=41E937CD59FD29FA88CDEC74B5B38E48E6C1916F
https://crt.sh/?q=1CB7A552161921173AFAB0C495476374F109EC40
https://crt.sh/?q=AF0B319BE293B35ACDFFF14E64C2C30CF72E59C9
https://crt.sh/?q=C67C04F1E7430991EECF49B215F0F52A8EAF002D
https://crt.sh/?q=3849D95060F166CBBDB8CC4844F164FE081CF843
https://crt.sh/?q=ED535799D2277017071ED796E80E4B23B0B8F76A
https://crt.sh/?q=9ACD82C9D27A6A5B6951D9607055F40749D5DA78
https://crt.sh/?q=0A45B711803B35FF8BAE6601AEC1A2A7496D635B
https://crt.sh/?q=4D340E50E77946EB7E8914BFD3BC56B29935045E
https://crt.sh/?q=98AE0BC61127CE56C3E92D263261151C9621C057
https://crt.sh/?q=16430FE249442AAACE757A091B0CCD25B77F6781
https://crt.sh/?q=B7310B84AEB3ADD86BAC9826F639D24C92C6BE05
https://crt.sh/?q=1B20E23AB28C1FF87DC1F57B7A91F2848EE47C42
https://crt.sh/?q=819ED8BEFBAE47E106B022ED065BC62B4E9425C6
https://crt.sh/?q=A1EEFDB4ACFDEEB1C660AF17E322D71D351F5285
https://crt.sh/?q=1B7F12C2E65FAF1F025A3BCF16C666DCC3E26ED5
3 certificates without CT logs issued from this CA https://crt.sh/?caid=1586 with these serial numbers: 29C76CDD40C71DE10000000050DF3292, FECAF3547EA7F1F0000000050EEF055, and A5F4A163C158E7ADC33F585EBA348FE
AffirmTrust product line
https://crt.sh/?id=904297338
https://crt.sh/?id=1026787394
https://crt.sh/?id=1273737575
https://crt.sh/?id=1080994048
https://crt.sh/?id=1228457795
https://crt.sh/?id=1300896325
https://crt.sh/?id=1235654030
https://crt.sh/?id=1395619057
https://crt.sh/?id=1351904437
https://crt.sh/?id=1822398312
https://crt.sh/?id=1821884371
https://crt.sh/?id=1984845822
Additional certificate issued before patch installed
https://crt.sh/?id=3548310006
A postmortem is under way and Entrust will provide a further update on (6) how the mistakes were introduced, how they avoided detection, and (7) steps that have been taken.
Updated•5 years ago
|
Comment 1•5 years ago
|
||
Bruce:
Thanks for reporting this. This is hugely valuable for all CAs to examine. Is my understanding correct that all of the identified certificates had attached private keys to the CSRs? It seems important to clarify, to examine if these keys had been used to request certificates from any other CAs, if Entrust has the private keys.
Assignee | ||
Comment 2•5 years ago
|
||
Hi Ryan:
In most case this appears to be a cut/paste error where the private key was pasted in with the CSR. In some cases the private key was encrypted and in some cases part of the private key was pasted. How do you suggest that we allow other CAs to examine the private keys to see if they were used by other CAs?
I also think that other CAs should investigate the issue of receiving a private key with the CSR. I did not find another incident report on this issue, but perhaps this vulnerability is not known.
Thanks, Bruce.
Updated•5 years ago
|
Comment 3•5 years ago
|
||
(In reply to Bruce Morton from comment #2)
In most case this appears to be a cut/paste error where the private key was pasted in with the CSR. In some cases the private key was encrypted and in some cases part of the private key was pasted. How do you suggest that we allow other CAs to examine the private keys to see if they were used by other CAs?
One approach, which has been used by at least one other CA, is to examine the keys you have and use crt.sh to scan for other certificates using those keys ( https://crt.sh/?a=1 - SHA-256 SubjectPublicKeyInfo). You can work with those CAs for appropriate proof of possession checks (e.g. signing a CSR with a CN="Entrust has the private key").
I also think that other CAs should investigate the issue of receiving a private key with the CSR. I did not find another incident report on this issue, but perhaps this vulnerability is not known.
Wholeheartedly agree.
Comment 4•5 years ago
|
||
Thanks for posting this Bruce. We are reviewing our systems. After reviewing this weekend, we know at least one system allows extra data beyond the end of the CSR so we will be reviewing all of the contents submitted through that system for issues. I'll file a separate incident report when I get the report back.
Assignee | ||
Comment 5•5 years ago
|
||
We updated the search criteria and found more certificates, so add the following to the timeline:
26 October 2020: Search criteria updated and search started on all certificates issued in 2020.
27 October 2020, 14:06 UTC: Found the following certificates with compromised private keys:
https://crt.sh/?q=6E3EA2399F1BD3624F5C8D7FD84B90646F7B7066
https://crt.sh/?q=F4D4FE61A641A6480F471A52597F46F8420C5F5A
https://crt.sh/?q=E2CA869E11FDF480E312E0D510DE6923531D09CE
https://crt.sh/?q=5077A1438FD1C7B676A69938718C32D3D4669072
28 October 2020, 13:53 UTC: The certificates found on the October 27th were revoked.
Assignee | ||
Comment 6•5 years ago
|
||
- Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
The application layer CSR parser did not reject otherwise valid CSRs that had trailing data. The provided CSR was parsed and stored in its entirety. After parsing, the application layer cleans the CSR before submitting it to the CA software, so in some cases the CA layer was not even aware of the other data. However, in certain cases the cleanse did not strip the extra data, and extra data was provided to the CA layer. The latest CA software (as of June 2020) rejected this CSR. This resulted in an investigation, which determined that other certificates were issued, when the private key was provided.
The bug avoided detection until the CA software was upgraded with new code that rejects CSRs with extra data. Prior to the CA software upgrade in June 2020, the application CSR parser logic was shared with the CA parser logic, and no errors were reported. This combined with the infrequent occurrence of private key submission led to an October 2020 discovery date.
The bug passed testing because it did not occur to us that a private key would be submitted with the CSR, so this was not tested.
- List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.
All detected certificates have been revoked.
The application layer parser was updated to reject CSRs that have unexpected trailing data. Previous permissiveness and tolerance to poorly-formatted CSRs has been tightened considerably.
A list of public keys will be added to this Incident Report to allow other CAs search their data for certificates which were issued with compromised keys.
If CAs are searching their data for CSRs with private keys, the attached file has some samples.
Assignee | ||
Comment 7•5 years ago
|
||
Samples of bad CSRs with private key.
The bug avoided detection until the CA software was upgraded with new code that rejects CSRs with extra data. Prior to the CA software upgrade in June 2020, the application CSR parser logic was shared with the CA parser logic, and no errors were reported. This combined with the infrequent occurrence of private key submission led to an October 2020 discovery date.
[...]
The application layer parser was updated to reject CSRs that have unexpected trailing data.
Do you currently detect and act on private keys in submissions, or is this new implementation only a filter that only allow correct CSRs?
Assignee | ||
Comment 9•5 years ago
|
||
(In reply to Matthias from comment #8)
Do you currently detect and act on private keys in submissions, or is this new implementation only a filter that only allow correct CSRs?
The current implementation to avoid the issue of capturing private keys is a filter to allow correct CSRs. We are looking into how we would detect private keys and what actions we would take if a private key was detected.
Assignee | ||
Comment 10•5 years ago
|
||
This is the SHA1 hash of the hex representation of the key modulus for the certificates with compromised keys. This will help other CAs assess if they issued any certificates with a compromised key as indicated in this bug.
Assignee | ||
Comment 11•5 years ago
|
||
For clarification, we posted the Debian Weak Key list style of hash as we assume that the CAs would already be used to ingesting it. This representation is only the last 10 bytes of the hash. I believe that it is discussed here, https://wiki.debian.org/SSLkeys.
We have no other updates. We believe all actions are completed.
Assignee | ||
Comment 12•5 years ago
|
||
Here is an updated file which has the subjectKeyIdentifier, full hash and Debian hash of each certificate with a compromised key.
Comment 13•5 years ago
|
||
After reading this bug we checked our systems and we discovered that we had about the same input tolerance: even in our certificate request web form it was possible to paste a private key after the CSR. However the CSR is automatically "cleaned" of any extra stuff right after submission, so that only the actual CSR arrives at the CA and is kept on records. Therefore, any private key (or other stuff) possibly appended to the CSR is not stored (nor logged) anywhere, so we have no way of determining if that has ever happened. But in case it did happen, for the reasons I explained above the private key was not compromised (it was just discarded "on the fly").
Nonetheless we have decided to modify our certificate request web form so that doing that (adding extra stuff after the CSR) , although it's harmless, not be possible anymore. We plan to release this patch into production by EOY.
We also checked the other two cases of anomalous input described here by Bruce, but found that they don't apply to us (our certificate request web form does not accept them).
In the meantime we have also thoroughly scanned our database for any certificates matching the SKIs in the list attached by Bruce, and found nothing.
Comment 14•5 years ago
|
||
Thanks for providing this information. I believe this bug can be closed and will do so on or about this Friday, 11-Dec, unless anyone knows of issues to still address.
Updated•5 years ago
|
Updated•3 years ago
|
Updated•2 years ago
|
Description
•