Closed Bug 1723121 Opened 4 months ago Closed 1 month ago

Web.com: Failure to respond in time to revocation requests

Categories

(NSS :: CA Certificate Compliance, task)

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: gaurav.makkar, Assigned: keith.mckenney)

References

Details

(Whiteboard: [ca-compliance])

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Safari/537.36

Steps to reproduce:

Our auditors looked at a sample of our certificate revocation communications and found compliance issues as per the standards laid out in NetSol CPS

Actual results:

We were in breach of WTBR 2-5.2 & WTEV SSL 2-5.2 criteria, BRs 4.9.1.1

For 1 out of 10 Certificate Problem Reports selected for testing, evidence couldn't be provided to show notification was accepted, acknowledged, and the investigation started within 24 hours.

For 3 out of 10 Certificate Problem Reports selected for testing, evidence couldn't be provided to show preliminary report was sent to the entity who filed the report within 24 hours.

For 2 out of 10 Certificate Problem Reports selected for testing, revocation of all the affected certificates did not occur within the five day timeline required.

Expected results:

The CA maintains controls to provide reasonable assurance that it:

  • has the capability to accept and acknowledge Certificate Problem Reports on a 24x7 basis;
  • identifies high priority Certificate Problem Reports;
  • begin investigation of Certificate Problem Reports within 24 hours and provide a - preliminary report on its findings to both the Subscriber and the entity who filed the Certificate Problem Report: decides whether revocation or other appropriate action is warranted;
  • if revocation is deemed the appropriate action, the elapsed time from receipt of the Certificate Problem Report or revocation request and revocation status information does not exceed the timelines in SSL Baseline Requirements 4.9.1.1; and
  • where appropriate, forwards such complaints to law enforcement.

As per NetSol’s CPS:

4.10.5. Time Within which Network Solutions Will Process the Revocation Request

Network Solutions SHALL process revocation requests in accordance with BR sections 4.9.1.1 and 4.9.5. Once a certificate has been revoked the revocation will be reflected in the OCSP responses issued within 1 hour, and in the CRLs within 24 hours.

Type: defect → task
Assignee: bwilson → gaurav.makkar
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true
Summary: failure to respond in time to revocation requests → Web.com: Failure to respond in time to revocation requests
Whiteboard: [ca-compliance]
Assignee: gaurav.makkar → keith.mckenney

We are conducting an investigation of the issues laid out in this thread with the intention of posting a full write-up including our mitigation plan to prevent further instances of these errors.

1. How your CA first became aware of the problem.

During our most recent WebTrust audit, our auditors brought to our attention (note that Network Solutions and Web.com are used interchangeably):

For 1 out of 10 Certificate Problem Reports selected for testing by BDO, Network Solutions couldn't provide evidence to show notification was accepted, acknowledged, and the investigation started within 24 hours of the report being received.

  • 10/10/2020 1:29 a.m. Professional Market Report On Domain Name System Tools For You And Your Management Team

For 3 out of 10 Certificate Problem Reports selected for testing by BDO, Network Solutions couldn't provide evidence to show a preliminary report was sent to the entity who filed the report within 24 hours.

  • 7/14/2020 1:55 p.m. CA certificate must include the Key Usage Digital Signature for CA signed responses - Network Solutions
  • 7/17/2020 1:03 a.m. Certificates with keys not divisible by 8
  • 8/10/2020 2:14 p.m. Network Solutions StateProvince is a Non-Location Value or Not a Valid State

For 2 out of 10 Certificate Problem Reports selected for testing by BDO, revocation of the affected certificates did not occur within the five-day timeline required.

  • 7/17/2020 1:03 a.m. Certificates with keys not divisible by 8
  • 8/10/2020 2:14 p.m. Network Solutions StateProvince is a Non-Location Value or Not a Valid State

On June 15, 2021 the requested audit information wasn’t immediately available. By June 29 we concluded we would not be able to get this information.

2. Timeline.

July 14, 2020
Inbound report received by email Subject: “CA certificate must include the Key Usage Digital Signature for CA signed responses - Network Solutions”

We have no evidence of response to the reporter advising that an investigation was in progress until July 28. We have no evidence that we informed the reporter of the outcome of the investigation.

July 17
Inbound report received by email Subject: “Certificates with keys not divisible by 8”

We have no evidence that we responded to the reporter or the subscriber.

Aug 10
Inbound report received by email Subject: “Network Solutions StateProvince is a Non-Location Value or Not a Valid State”

We have no evidence that we responded to the reporter or the subscriber.

Oct 10
Inbound report received by email with Subject: “Professional Market Report On Domain Name System Tools For You And Your Management Team”

This was deemed to be a junk solicitation message. No evidence is available to show when this message was read/determined to be junk.

December
Web.com implements an internal Ticketing based system to handle inbound abuse reports.

June 15, 2021
As part of this year’s WebTrust audit, BDO publishes its Certificate Problem Reporting sample. Evidence gathering starts.

June 21
As part of gathering evidence for the auditors’ list, we identify that we have a problem finding this information. By June 29 we were confident we do not have it.

June 29
Automated forwarding of all inbound abuse reports to Sectigo's automated response mechanism.

Web.com informs BDO that no further evidence can be found.

July 30
This bug is opened to report errors in responses to inbound problem reports.

3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem.

This is not an issuance problem but concerns over missed responses to inbound abuse reports. Automated systems put in place in June have eliminated this problem going forward.

4 & 5. A summary of the problematic certificates and affected certificates

This is not a case of misissuance but rather failure to respond to abuse reports within guidelines.

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

Prior to December 2020, Web.com had an entirely manual process for accepting and dispositioning reports to our inbound SSL abuse email address, which led to errors. By December 2020 Web.com implemented an internal ticketing-based system to handle inbound abuse reports. Based on the inability to provide full and timely evidence for our WebTrust audit, by June 29, 2021, we decided that this system was not as effective as hoped and the decision was made to forward all inbound abuse reports to Sectigo's automated response mechanism.

7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future.

We have remediated this issue through automated forwarding of all inbound abuse reports to Sectigo's automated response mechanism as described in bug 1648717.

Thanks for the detailed report here. It's difficult to determine whether this was a lack of responsiveness, or a lack of evidence of responsiveness. Could you clarify?

Similarly, can you detail the processes at Web.com that exist to ensure compliance with the BRs? That is, while I'm incredibly happy that BDO detected this issue, and demonstrates a quality and attention to detail, I'm concerned to understand why this appears to have been an "unknown unknown" for Web.com. That is, what sort of self-audits are regularly performed to ensure compliance with the BRs? Any? Section 8.7 of the BRs requires the CA monitor itself for compliance, and it sounds like, beyond the immediate symptom (which this incident report appears to be tackling), there's a deeper root cause issue.

With that mindset, I think it's important to ask yourself: Why didn't you detect that you had a problem until the auditor detected it, and what is changing about that process for detection?

Flags: needinfo?(keith.mckenney)
Duplicate of this bug: 1725047

(In reply to Ryan Sleevi from comment #3)

It's difficult to determine whether this was a lack of responsiveness or a lack of evidence of responsiveness. Could you clarify?

That’s difficult to answer. In this case, the “evidence” is a record of the response and when it occurred. We lack that record for these incidents. One possible reason is that no response occurred. Another is that we failed to record the response. From the auditors’ perspective, the distinction is immaterial. From the reporter’s and subscriber’s perspectives, the distinction is material, but we cannot say which it was. Were that evidence available, we would have offered it during the audit.

As we have announced in other open bugs, we are moving entirely to a managed CA model, with CA services to be managed by Sectigo. Previously to that decision, we already had moved revocation requests and abuse line notification entirely to Sectigo’s service, and that practice will continue. These errors occurred prior to moving these services to Sectigo.

Section 8.7 of the BRs requires the CA monitor itself for compliance, and it sounds like, beyond the immediate symptom (which this incident report appears to be tackling), there's a deeper root cause issue.

We have alluded in our responses to other open incidents that changes in the organization and our CA team led to gaps in Network Solutions’ ability to execute on CABF and root store requirements – furthermore to our ability to detect these gaps and raise a flag. These errors have directly led us to examine how we can best meet the expectations of a public CA while providing a superior customer experience. That includes things referenced in this incident like self-audits, abuse queue response, and monitoring other CAs’ practices. That process led to the conclusion that we should offload those activities to a partner that is more deeply focused on operating a public CA and gain the benefits of that deep focus.

It is worth mentioning that the majority of our certificate volume already is part of our managed CA service from Sectigo and that model has proven effective and compliant for those certificates. We at present are investigating the details for what must happen to enable this model for ALL CA operations, including all certificates. We intend to announce a target date by the end of this month.

Flags: needinfo?(keith.mckenney)

We are working on the details of our full transition to managed CA services. We are aiming to announce a transition schedule by the end of the month.

We have targeted a full transition to the managed service on or before November 8, 2021.

Are there any other questions on this issue?

Ben, is this bug ready to close?

Flags: needinfo?(bwilson)

I'll close this on Wed. 20-Oct-2021.

Status: ASSIGNED → RESOLVED
Closed: 1 month ago
Flags: needinfo?(bwilson)
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.