Closed Bug 1516453 Opened 6 years ago Closed 6 years ago

DigiCert: Underscores - Discover

Categories

(CA Program :: CA Certificate Compliance, task)

Product:
CA Program
Component:
CA Certificate Compliance
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: jeremy.rowley, Assigned: brenda.bernal)

Details

(Whiteboard: [ca-compliance] [ov-misissuance])

User Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:64.0) Gecko/20100101 Firefox/64.0 Steps to reproduce: This is the incident report for some certificates that will potentially not be revoked on Jan 15th that contain underscore characters. They request to be identified as "Large Financial Institution". 049B67F0E3C45BB07C881700AF0C4F3E https://crt.sh/?id=628906124 05636298FC5B1642A784023FF2E0B4D3 https://crt.sh/?id=628912394 0E831E65110614B8F9BC294E5F291F17 https://crt.sh/?id=628920909 01E532A602E6F11E208E33F86F07D55D https://crt.sh/?id=1045077704 07896F80F8F33640F7C2D64EB416314F https://crt.sh/?id=1045077705 0109EB4AD4B8909C2FDBE6790E44CD55 https://crt.sh/?id=1045077706 0D3E28F8D62D7E4D24C0DCF6D11D2F87 https://crt.sh/?id=1045077709 05A885B7DA75337B716752211CAD0D2F https://crt.sh/?id=1045077703 07F907423EBD4E45C28A2FE12EE3657A https://crt.sh/?id=1045077707 0A17B933BE45D387EC42DE26AACFF6A3 https://crt.sh/?id=350254382 0C0504ED362DCB9D53061DAB00AB9E70 https://crt.sh/?id=350266037 0B1325FD302B33F340CB9ECD40E12AF4 https://crt.sh/?id=350261306 09A173971935496B7C0F22E9D17093EF https://crt.sh/?id=483157170 0166A67ACCFF5FD813681A550DCB125E https://crt.sh/?id=483155735 0314C43C3C36B079361BA1F68A5C057A https://crt.sh/?id=483154636 098F51500239A0702004AEB710BDB009 https://crt.sh/?id=328734985 0391E231A3F86D70DA56E942AF399C94 https://crt.sh/?id=328616952 0AE2FE1F5707241513275D4376473014 https://crt.sh/?id=328734617 04E4140620511DF42403BE8BBED4C7B2 https://crt.sh/?id=621157153 0F3C43F9DCECFB9D1D7FDD7035805E3D https://crt.sh/?id=620948877 05C00A1D6129461818820226879AD91A https://crt.sh/?id=615925574 05CEE9A451B0C33F804AB24F74A01B01 https://crt.sh/?id=621263390 0F3E6B7FE044159412591FCAB0767E24 https://crt.sh/?id=1045077708 09F1096DC6251D12114CF54A3C7AE07A https://crt.sh/?id=1045077710 068DA5648358BD8CDFBEEC3ECC657E68 https://crt.sh/?id=425233159 094E7A54557E1E9A60E68A72E9F801CC https://crt.sh/?id=425243513 0480BCB83664742D1A95D6B5ECE13985 https://crt.sh/?id=425247444 03855B787CF7E6575B665047E2980122 https://crt.sh/?id=1045077711 03ECA48C3BAF5C380D5EE3603FCF2F05 https://crt.sh/?id=1045077712 0E29799AFC9B5B8120573B8FA5C27C48 https://crt.sh/?id=1045077713 0AB8FE62479D0EFD935FDD440BD90980 https://crt.sh/?id=1045077718 08313EFDB2F6EEE79FB1088DCEBA47B3 https://crt.sh/?id=1045077719 075CF71BE294C8AF8AB4B653AF9B1CC1 https://crt.sh/?id=1045077716 01C7E0272B7C1234E243F7D0F5C4F2E6 https://crt.sh/?id=1045077714 0C91E74EFFE36AF086354C14CB912284 https://crt.sh/?id=360589511 05A8F4B721E85081F2573A5CD8126EBB https://crt.sh/?id=361333124 052CE5CEC1459E3BBA0FE4515673309E https://crt.sh/?id=360569739 07DDE996546DA66798FE5D29E6F5C52C https://crt.sh/?id=647014049 0850679F8005929A015469478C5747A7 https://crt.sh/?id=647012723 09953688DDEDA9885BD4D90400933FBD https://crt.sh/?id=638682945 224aa0fad41b843007b09d8d75c520f6 https://crt.sh/?id=674699114 224aa0fad41b843007b09d8d75c520f6 https://crt.sh/?id=674699114 224aa0fad41b843007b09d8d75c520f6 https://crt.sh/?id=674699114 0FD0DFFCDCE82CA59C085A388E5F525D https://crt.sh/?id=638304282 0FD0DFFCDCE82CA59C085A388E5F525D https://crt.sh/?id=638304282 0a2f137acf001c993ebfbc2e4f57153b https://crt.sh/?id=674699163 0a2f137acf001c993ebfbc2e4f57153b https://crt.sh/?id=674699163 0a2f137acf001c993ebfbc2e4f57153b https://crt.sh/?id=674699163 3120be576853bda2eb5ee5b098865015 https://crt.sh/?id=674699142 3120be576853bda2eb5ee5b098865015 https://crt.sh/?id=674699142 3120be576853bda2eb5ee5b098865015 https://crt.sh/?id=674699142 07749F9C7447922830588EA16FAE4048 https://crt.sh/?id=425334061 0265A54A08DC16874CA6AC6D57C17A1D https://crt.sh/?id=497406099 01AC6E331587F5FFF134F721D2E08869 https://crt.sh/?id=425363968 0F4D58A16E41301962575845EBCBD5CC https://crt.sh/?id=497354390 06ACC2A3E7FF3B92133901764C1F52FA https://crt.sh/?id=425441160 0DE17FC56447F91966AB1471D581F34D https://crt.sh/?id=497378526 Their black out dates are November 17th, 2018 to January 5th, 2019. The change freeze dates start just before, and end just after, the highest consumer financial transaction time of the year when customers will be purchasing during the holidays, black Friday, and cyber Monday. In order to mitigate as much risk as possible to those financial services being unavailable or interrupted during that time, a company wide change freeze is established. The only approved changes during that period are pre-approved prior to September 30th, 2018. These certificates are utilized for business to business transactions processes which support customer facing services. In order to replace these certificates, new certificates are generated and staged on both our side and the 3rd party business side. Coordinated cutover times must be established with all parties. An implementation date is scheduled and developer and support teams are organized to validate service functionality after the change. If the certificates are not loaded correctly or cutover times are not coordinated, the service may be unavailable or interrupted, causing direct customer impact. The challenge with the timeline established by Ballot SC12 stems from 3 components: 1) Timing – Explained above. 2) Scope – Large number of certificates which span multiple 3rd party businesses. Large coordination effort in short period of time. 3) Resources – Both internal and external resources are reduced during the holiday season. This includes resources for planning, coordination, and execution. The 30 day option does not help because a) the company is migrating hierarchies in some cases and b) the domain names are easier to change out than the certificates.
Jeremy, this doesn't seem to match the template of https://wiki.mozilla.org/CA/Responding_To_An_Incident at all. I think it's especially concerning the absence of #7 from that list. The response that "the domains are easier to change out than the certificates" is very troubling, given the expectations of revocation (without remarks about replacement) as quickly as 24 hours. Please ensure the template is followed, and in doing so, please help explain what steps your CA is taking to resolve the situation and prevent future issues.
Flags: needinfo?(jeremy.rowley)
Assignee: wthayer → jeremy.rowley
Summary: DigiCert Underscore - Large Financial Institution → DigiCert: Underscores - Large Financial Institution
Whiteboard: [ca-compliance]
Wanting to make sure this is flagged, as it's now been over a week. It'd be quite concerning if this was the incident report filed post-an-incident.
Flags: needinfo?(brenda.bernal)
Summary: DigiCert: Underscores - Large Financial Institution → DigiCert: Underscores - Discover

Based on the conversation on the forum, the post from Wayne, and instruction from Google, our understanding is there is no exception or extension possible and the expectation is that all CAs will revoke the certificates on the date required by the BRs. We hope that the same rules/penalties/expectations will be applied to those CAs who fail to revoke on the required date. Thank you for the discussion. Although we were hoping for more compassionate results, we do appreciate the feedback and clarification on expectations.

Flags: needinfo?(jeremy.rowley)

Seems there was a mis-communication on the intent of the discussions. We will post an update answering Ryan's questions tomorrow. Please ignore my previous post. Apologies for the confusion.

The following certs from the above list posted by Jeremy has been replaced and will be revoked (crt.sh links followed by serial no):

https://crt.sh/?id=1045077703 05A885B7DA75337B716752211CAD0D2F
https://crt.sh/?id=1045077707 07F907423EBD4E45C28A2FE12EE3657A
https://crt.sh/?id=483157170 09A173971935496B7C0F22E9D17093EF
https://crt.sh/?id=483155735 0166A67ACCFF5FD813681A550DCB125E
https://crt.sh/?id=483154636 0314C43C3C36B079361BA1F68A5C057A
https://crt.sh/?id=328734985 098F51500239A0702004AEB710BDB009
https://crt.sh/?id=328616952 0391E231A3F86D70DA56E942AF399C94
https://crt.sh/?id=328734617 0AE2FE1F5707241513275D4376473014
https://crt.sh/?id=1045077708 0F3E6B7FE044159412591FCAB0767E24
https://crt.sh/?id=1045077719 08313EFDB2F6EEE79FB1088DCEBA47B3
https://crt.sh/?id=1045077716 075CF71BE294C8AF8AB4B653AF9B1CC1
https://crt.sh/?id=1045077714 01C7E0272B7C1234E243F7D0F5C4F2E6

All other certs originally listed above require an extension of time, by no later than February 14, 2019 to complete revocations.

As requested, here is our response to 7) List of steps CA is taking to resolve the situation and ensure it will not be repeated.

Digicert will improve flow and pace of communication, and ensure all customers are aware that the CPS and other documents specify that timely revocation is possible once ballots take effect. It is our contractual right to revoke. Because of the timing of this revocation (during when most of our customers have their IT Standard Code Freeze policy is in effect), we have had to weigh the end user risks and impact, and request an extension of time before revocation. We will ensure that our end users are clear that it is our responsibility to execute revocations based on policy changes as specified in our agreements.

In this case, the complexity of change in the customer's environment (3rd party dependencies), the black out period and number of certs to coordinate replacements contributed to the need for an extension of time to replace / revoke.

Flags: needinfo?(brenda.bernal)

Jeremy: Can you confirm whether an incident occurred and ensure that all of the details of affected certs are accurate?

Flags: needinfo?(jeremy.rowley)

Hi Ryan, I will be responding to provide updates on the underscore incidents. I can confirm that an incident has occurred and the details provided are accurate to the best of our knowledge. Our planned extension to revoke the remaining certificates (listed above) is 14-February-2019. We will provide periodic updates as progress is made.

Assignee: jeremy.rowley → brenda.bernal
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true
QA Contact: kwilson → wthayer

Update: All remaining underscore certs for this customer has been revoked as of today (14-Feb-2019).

Thanks for the update, Brenda.

I'm marking this matter as Resolved.

Status: ASSIGNED → RESOLVED
Closed: 6 years ago
Flags: needinfo?(jeremy.rowley)
Resolution: --- → FIXED
Product: NSS → CA Program
Whiteboard: [ca-compliance] → [ca-compliance] [ov-misissuance]
You need to log in before you can comment on or make changes to this bug.