Closed Bug 1519572 Opened 8 months ago Closed 5 months ago

DigiCert: Underscores - Intuit

Categories

(NSS :: CA Certificate Compliance, task)

task
Not set

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: brenda.bernal, Assigned: brenda.bernal)

Details

(Whiteboard: [ca-compliance])

  1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

All essential notice dates:

  1. September 5, 2018 - the issue is raised by the browsers on the CA/Browser Forum.
  2. October 10, 2018 – The CAB Forum discussions on the validation working group indicated that the browsers believed this was mis-issuance
  3. October 16, 201 – Tim reports back on the status of the Shanghai meeting. This is when we first know of the proposal
  4. October 19, 2018 – Ballot was proposed by Wayne to the validation working group. This is the first we aware that the certs may require revocation. Note the revocation date was still being debated.
  5. October 26, 2018 – Final ballot was proposed.
  6. November 2, 2018 – Voting period starts.
  7. November 9, 2018 – Voting period ends. This is when we first know there is a requirement in the CAB Forum to revoke the certs.
  8. November 19, 2018 – We first hear of customers not being able to meet the revocation timeline.
  9. January 15, 2019 – First time we will be in non-compliance for certs we don't revoke.

Customer was given a list of all their impacted certificates on Dec 5, 2018.

  1. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

  2. September 5, 2018 - the issue is raised by the browsers on the CA/Browser Forum.

  3. October 1, 2018 – We cease issuance of underscore characters in case the discussion move towards removal of underscores

  4. October 2, 2018 – We notify customers that the browsers are raising an issue with underscores. Incomplete data leads to only some customers being notified.

  5. October 10, 2018 – The CAB Forum discussions on the validation working group indicated that the browsers believed this was a mis-issuance

  6. October 10, 2018 – Internal advisory sent that this is picking up speed and external comms provided in KB article (support page)

  7. October 11, 2018 – Discussion with customers about potential impact. Turns out they are required for certain IBM systems.

  8. October 16, 201 – Tim reports back on the status of the Shanghai meeting. This is when we first know of the proposal

  9. October 17, 2018 – Internal discussion about whether we allow underscore character renewals and whether the ballot is likely to pass. We decide it is but are hoping existing certs will be allowed to expire.

  10. October 19, 2018 – Ballot was proposed by Wayne to the validation working group. This is the first we aware that the certs may require revocation. Note the revocation date was still being debated.

  11. October 19, 2018 – Internal discussion to start communications about CAB Forum plan.

  12. October 20, 2018 – Second emergency meeting to start customer outreach.

  13. October 24, 2018 – Gather data on all impacted certs across the different systems.

  14. October 26, 2018 – Final ballot was proposed.

  15. November 1, 2018 – We had to re-pull data due to issues with the information.

  16. November 2, 2018 – Voting period starts.

  17. November 9, 2018 – Voting period ends. This is when we first know there is a requirement in the CAB Forum to revoke the certs.

  18. November 29, 2018 – Posted to Mozilla about concerns with ballot.

  19. November 29, 2018 – Final communications is dropped about the ballot and its impact.

  20. November 30, 2018 – Final internal advisory on issue.

  21. November 20, 2018 – Customer given list of certificates and advised to participate in the Mozilla discussion. All exceptions to the revocation date are denied. People to start to escalate to demand that there is an exception process, we just don't know about it yet.

  22. December 7, 2018 – Customers engage with Mozilla community.

  23. December 5, 2018 – Daily calls start to try and identify why people can’t migrate by the required timeline.

  24. December 12, 2018 – Question about scope asked of Mozilla. Does legacy Symantec really need to be replaced? They aren’t trusted by Mozilla anymore.

  25. December 19, 2018 – Post of future incident report to start discussion on what will happen if we don’t revoke the certs. The goal is to provide better information on the scope of impact.

  26. January 15, 2019 – First time we will be in non-compliance (assuming we don’t revoke all the certs of course)

  27. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

We stopped issuing certs with underscore characters on Oct 1. We re-enabled 30 day certificates per the ballot for any customers that can use that option. We found that exactly no customers can use that option. We will shut down the 30 day certs per the ballot requirements. All certs for this particular entity will be revoked on January 26.

  1. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

A list of the certs can be found further below. From the customer, "These public certificates are utilized for the communication with our external partners. As part of this replacement, we need to coordinate the revoke and replace with our external partners, and in many cases, this requires a configuration change on the partner side. The notice period is too short to coordinate with all 3rd parties and get their commitment to replace them."

  1. The complete certificate data for the problematic certificates.
    Listed below.

  2. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

We have a list of 18 certs that can not be revoked by the January 15th deadline.
From our customer's input, "Due to upcoming tax season the operational risk associated with alternative methods (e.g. 30 days certs, rename vs replace) is much higher. Alternative methods do not reduce the effort or coordination requirements that are associated with these types of changes. Additionally, there is a change restriction for tax peak period, which will prevent us from making any changes every 30 days during the tax season."

  1. List of steps CA is taking to resolve the situation and ensure it will not be repeated.
    Digicert will improve flow and pace of communication, and ensure all customers are aware that the CPS and other documents specify that timely revocation is possible once ballots take effect. It is our contractual right to revoke. Because of the timing of this revocation (during when most of our customers have their IT Standard Code Freeze policy is in effect), we have had to weigh the end user risks and impact, and request an extension of time before revocation. We will ensure that our end users are clear that it is our responsibility to execute revocations based on policy changes as specified in our agreements.

List of certs in scope:
https://crt.sh/?id=17010204
https://crt.sh/?id=17010641
https://crt.sh/?id=276157353
https://crt.sh/?id=276169547
https://crt.sh/?id=276169549
https://crt.sh/?id=276169543
https://crt.sh/?id=276169548
https://crt.sh/?id=276169554
https://crt.sh/?id=353647080
https://crt.sh/?id=353647409
https://crt.sh/?id=354483095
https://crt.sh/?id=354582394
https://crt.sh/?id=496713129
https://crt.sh/?id=499862291
https://crt.sh/?id=596629217
https://crt.sh/?id=596629319
https://crt.sh/?id=596629503
https://crt.sh/?id=771865132

Thanks Brenda.

Please confirm: these 18 certificates will be revoked on 26-January 2019?

Assignee: wthayer → brenda.bernal
Summary: DigiCert - Underscores - Software and Services Company → DigiCert: Underscores - Intuit
Whiteboard: [ca-compliance]

Hi Wayne,

Due to the tax season impact, the request is to extend the revocation to April 30, 2019 for these 18 certificates. Thank you.

(In reply to Brenda Bernal from comment #2)

Hi Wayne,

Due to the tax season impact, the request is to extend the revocation to April 30, 2019 for these 18 certificates. Thank you.

That makes more sense in the context of the answer to question #6, but it leaves a few other questions:

  • why couldn't these certificates be replaced before tax season?
  • what does the 26-Jan date referenced at the end of question #1 mean?

Thanks!

Hi Wayne, Please disregard the 26-Jan referenced. That was the original date we were hoping all certificates would require an extension of time. With that said, the company referenced in this incident, has made significant progress in replacing the certs with underscores that they were able to fit in with the time they had. Their code freeze period started on December 1st, and ends on February 20th. There is also change management policy in effect during the tax season period that is described above (item 6).

Brenda: Can you confirm whether an incident occurred and ensure that all of the details of affected certs are accurate?

Flags: needinfo?(brenda.bernal)

I can confirm that an incident has occurred and the details provided are accurate to the best of our knowledge. Our planned extension to revoke the remaining certificates (listed above) is 30-April-2019. We will provide periodic updates as progress is made.

Flags: needinfo?(brenda.bernal)
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true
Whiteboard: [ca-compliance] → [ca-compliance] Next Update - 30-April 2019

Update: All remaining underscore certs for this customer has been revoked as of today (30-April-2019).

Status: ASSIGNED → RESOLVED
Closed: 5 months ago
Resolution: --- → FIXED
Whiteboard: [ca-compliance] Next Update - 30-April 2019 → [ca-compliance]
You need to log in before you can comment on or make changes to this bug.