Closed Bug 1524815 Opened 4 years ago Closed 3 years ago

GoDaddy: failure to revoke underscores

Categories

(CA Program :: CA Certificate Compliance, task)

Product:
CA Program
Component:
CA Certificate Compliance
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: jonathan, Assigned: jfox)

Details

(Whiteboard: [ca-compliance] [leaf-revocation-delay])

GoDaddy failed to revoke the following certificates containing dnsNames with underscores that were required to be revoked by 2019-01-15 (CABF ballot SC12). I notified them via an email to their problem reporting email address at 2019-01-29 18:18 UTC and they were revoked the next day.

https://crt.sh/?opt=zlint&id=626981823
https://crt.sh/?opt=zlint&id=637047181

Daymion: Please provide an incident report, as per https://wiki.mozilla.org/CA/Responding_To_An_Incident

If GoDaddy intentionally delayed revocation, please include an explanation for why this wasn't disclosed prior to Jan 15 as discussed on the mozilla.dev.security.policy list.

Assignee: wthayer → dreynolds
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true
Flags: needinfo?(dreynolds)
QA Contact: kwilson → wthayer
Whiteboard: [ca-compliance]

GoDaddy Underscore Revocation Disclosure

GoDaddy received a certificate problem report on 1/29/2019 for 2 unrevoked unexpired certificates have underscores in the DNS name that did not meet the January 15th deadline for revocation. The certificates reported are as follows:
https://crt.sh/?opt=zlint&id=626981823
https://crt.sh/?opt=zlint&id=637047181

  1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.
    Certificate Problem Reporting via email address supplied in CP/CPS on Tuesday, January 29, 2019 11:18:42 AM

  2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

1/29/2019 11:18:42 AM AZ time Problem Report received by GoDaddy
1/29/2019 3:55 PM AZ time Problem report was viewed by an agent and escalated appropriately.
1/29/2019 to 1/30/2019 Researched as to why these certificates were not caught in the initial data set for underscore revocation.
1/30/2019 Identified root cause as order of an operation problem where certificates could be CT logged but never be delivered to the certificate requestor.
1/30/2019 23:36:32 UTC and 1/30/2019 23:35:55 UTC Certificates that were reported are revoked.

  1. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.
    We have prevented new certificates with underscores from being provisioned since November 8, 2018.

  2. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.
    https://crt.sh/?opt=zlint&id=626981823
    https://crt.sh/?opt=zlint&id=637047181

  3. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
    Our initial effort to bring underscore certificates into compliance was dependent upon an ‘active’ flag in our database. In rare cases, due to an order of operation problem, non-active certificates were being logged to CT. These non-active certificates were fully vetted meeting the BR standards but failed to be delivered to the certificate requester due to a software defect.

  4. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.
    We have prevented new certificates with underscores from being provisioned since November 8, 2018. These non-active certificates were revoked and we are taking steps to prevent non-active certificates from being logged to CT within the next 2 weeks.

Flags: needinfo?(dreynolds)

we are taking steps to prevent non-active certificates from being logged to CT within the next 2 weeks.

This doesn't address the root cause of the noncompliance because even certificates not logged to CT need to comply with the Baseline Requirements.

What are you changing to ensure that in the future even "non-active" certificates remain compliant - not just with the underscore rule, but with other potential BR changes?

Flags: needinfo?(dreynolds)

Prior issue with a similar cause: https://bugzilla.mozilla.org/show_bug.cgi?id=1462844#c7

(In reply to Andrew Ayer from comment #3)

we are taking steps to prevent non-active certificates from being logged to CT within the next 2 weeks.

This doesn't address the root cause of the noncompliance because even certificates not logged to CT need to comply with the Baseline Requirements.

What are you changing to ensure that in the future even "non-active" certificates remain compliant - not just with the underscore rule, but with other potential BR changes?

Andrew, Not certain I understand your concern. We closed the underscore issue on November 8th 2018, well ahead of the January deadline. The identified certs should never have been in CT since we never fully provisioned them. For future issues we make changes to the system to meet compliance as the BR changes are approved. Please let me know if this doesn't answer your question.

Flags: needinfo?(dreynolds)

(In reply to Wayne Thayer [:wayne] from comment #4)

Prior issue with a similar cause: https://bugzilla.mozilla.org/show_bug.cgi?id=1462844#c7

Wayne, As stated in the Mozilla group they are actually not the same issue. The defect 1462844 certs missed in first pass were fully provisioned, live certificates missed due to a missing live status in the query. This issue's certificates were never issued to the requester, but were logging to CT. It was missed as the certs should never have been logged, and they were never live. We are changing the order of operation to prevent future occurrences.

(In reply to Daymion Reynolds from comment #5)

(In reply to Andrew Ayer from comment #3)

we are taking steps to prevent non-active certificates from being logged to CT within the next 2 weeks.

This doesn't address the root cause of the noncompliance because even certificates not logged to CT need to comply with the Baseline Requirements.

What are you changing to ensure that in the future even "non-active" certificates remain compliant - not just with the underscore rule, but with other potential BR changes?

Andrew, Not certain I understand your concern. We closed the underscore issue on November 8th 2018, well ahead of the January deadline. The identified certs should never have been in CT since we never fully provisioned them. For future issues we make changes to the system to meet compliance as the BR changes are approved. Please let me know if this doesn't answer your question.

Your mitigation seems to be addressing the wrong problem. The problem is not that these were logged to CT, it's that they should have been revoked, whether logged to CT or not.

(In reply to Julien Cristau [:jcristau] from comment #7)

(In reply to Daymion Reynolds from comment #5)

(In reply to Andrew Ayer from comment #3)

we are taking steps to prevent non-active certificates from being logged to CT within the next 2 weeks.

This doesn't address the root cause of the noncompliance because even certificates not logged to CT need to comply with the Baseline Requirements.

What are you changing to ensure that in the future even "non-active" certificates remain compliant - not just with the underscore rule, but with other potential BR changes?

Andrew, Not certain I understand your concern. We closed the underscore issue on November 8th 2018, well ahead of the January deadline. The identified certs should never have been in CT since we never fully provisioned them. For future issues we make changes to the system to meet compliance as the BR changes are approved. Please let me know if this doesn't answer your question.

Your mitigation seems to be addressing the wrong problem. The problem is not that these were logged to CT, it's that they should have been revoked, whether logged to CT or not.

We stated in the Mozilla policy group we will be our enhancing our search capabilities. This shall alleviate the issue of finding certs regardless of state.

(In reply to Daymion Reynolds from comment #8)

We stated in the Mozilla policy group we will be our enhancing our search capabilities. This shall alleviate the issue of finding certs regardless of state.

Just to make sure I've got the right messages:
Was this the Feb 8 response from Joanna

For the underscore certificates, these were non-active, not even considered as provisioned since they were not delivered to a customer and not publicly used for any encryption. These certificates were effectively abandoned by our system.

Or was there a difference response here?

I ask, because there are still a number of unanswered questions from folks waiting answers on that thread, so I wanted to make sure things were getting delivered appropriately. I don't think the above response quite explains the search issue as you summarized, so I'm guessing I'm missing a message?

Assignee: dreynolds → jfox
Flags: needinfo?(jfox)

Joanna: Could you make sure to respond to Comment #9?

Emailed POCs on 2019-07-15 regarding this issue, highlighting https://wiki.mozilla.org/CA/Responding_To_An_Incident#Keeping_Us_Informed

Apologies for the delayed response. GoDaddy acknowledges the inquiry. We will work to have a response to the community by EOD, July 19th.

Flags: needinfo?(jfox)

In regards to Ryan's question asking about unanswered questions from folks that are waiting. What specific questions do we feel have not been answered so that we may respond accordingly?

I consider the following unresolved:

Flags: needinfo?(jfox)

(In reply to Ryan Sleevi from comment #14)

I consider the following unresolved:

To clarify, the certificates linked in this issue are indeed pre-certificates. We agreed with revoking these 2 certificates and had acknowledged this fact by completing the revocations on January 30, 2019.
Yes, the M.D.S.P. post is the same as this bug 1524815.
We updated our query to appropriately search all certificates in our database and introduced a peer review process and going forward we will cross-check our results with CT logs. We have also updated linters and updated our code to ensure compliance with the BR’s.

Flags: needinfo?(jfox)

Comment #3 as amended by Comment #7

Joanna: please clarify how your response in comment 15 answers this question?

Flags: needinfo?(jfox)

(In reply to Wayne Thayer [:wayne] from comment #16)

Comment #3 as amended by Comment #7

Joanna: please clarify how your response in comment 15 answers this question?

We are interpreting Comment 3 to be asking ‘how do we assure all certificates issued by our CA server will be BR compliant from now on’ and Comment 7 to be stating that ‘the certificates should have been revoked.’ To us, these comments are clearly answered by the statement given. Perhaps we are interpreting the comments differently and if so would like more clarification as to what specifically is being missed so that we may address accordingly.

Flags: needinfo?(jfox)

(In reply to Joanna from comment #17)

(In reply to Wayne Thayer [:wayne] from comment #16)

Comment #3 as amended by Comment #7

Joanna: please clarify how your response in comment 15 answers this question?

We are interpreting Comment 3 to be asking ‘how do we assure all certificates issued by our CA server will be BR compliant from now on’ and Comment 7 to be stating that ‘the certificates should have been revoked.’ To us, these comments are clearly answered by the statement given. Perhaps we are interpreting the comments differently and if so would like more clarification as to what specifically is being missed so that we may address accordingly.

Joanna: I'm interpreting comment #3 to mean "there is no such thing as a non-active certificate" and comment 7 to mean that "the fact that these certificates were only known because they were accidentally logged is (to be generous) irrelevant". So the question is "how is GoDaddy ensuring that ALL [pre-]certificates that are misissued in any future incident will be identified and revoked?" From what I can gather the answer is:

  • logging all "non-active" certificates
  • peer review of revocation queries
  • a check of CT logs to verify that no certificates were missed by the query

Is that correct? Is anything else being done to remediate this incident?

Flags: needinfo?(jfox)

Wayne: Thank you for the clarification. You are correct in all bullets listed, we believe that concludes remediation of this incident.

Flags: needinfo?(jfox)

In the spirit of transparency, and to circle back on this thread with respect to non-active certificates, as we continue to improve our systems and processes via rigorous audits and testing, we recently discovered an anomaly in our API reseller space. In two instances, we received a revocation request and revoked the newest certificate issued as part of the associated CSR. However, there were older certificates associated with the two CSRs (non-active) and we were unable to verify whether those certificates also needed to be revoked. We subsequently revoked the non-active certificates. To ensure these rare cases do not again occur in the future, we have implemented a process that will in all cases also identify and revoke any existing non-active certificate as part of a revocation request. There was no reported problem with revocation, but we wanted to raise to keep the community aware of our continued diligence efforts.

Joanna: thank you for the update. It appears that all questions have been answered and remediation is complete.

Status: ASSIGNED → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
Product: NSS → CA Program
Whiteboard: [ca-compliance] → [ca-compliance] [leaf-revocation-delay]
You need to log in before you can comment on or make changes to this bug.