Closed Bug 1725043 Opened 3 years ago Closed 3 years ago

Network Solutions: 2021 Audit Observation #3

Categories

(CA Program :: CA Certificate Compliance, task)

Product:
CA Program
Component:
CA Certificate Compliance
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: bwilson, Assigned: keith.mckenney)

Details

(Whiteboard: [ca-compliance] [audit-finding])

As noted in https://bugzilla.mozilla.org/attachment.cgi?id=9233219, https://bugzilla.mozilla.org/attachment.cgi?id=9232451, and
https://bugzilla.mozilla.org/attachment.cgi?id=9233220, ten (10) certificates were issued with incorrect state/province information.

Assignee: bwilson → keith.mckenney
Status: NEW → ASSIGNED

Network Solutions is required to file an Incident Report - https://wiki.mozilla.org/CA/Responding_To_An_Incident#Incident_Report

Flags: needinfo?(keith.mckenney)

Do we know which batches these were in, as reported in bug #1645686? And if these certificates were handled in that bug, then why wasn't that bug number referenced in the audit reports?

Do we know which batches these were in, as reported in bug #1645686?

One certificate revoked August 15:
https://crt.sh/?serial=00d6ad43fd5284de89dd5a7e54283205ad

The remaining seven certificates revoked October 26:
https://crt.sh/?serial=00ed668507bfe4431b281e0aa8224384f5
https://crt.sh/?serial=524f7650b21cc5f2287e34192ca470f3
https://crt.sh/?serial=4247ca86502743cc4efda3f1cf2ca53a
https://crt.sh/?serial=7679ce1c66c60fc9cafeb5af10a3b9ff
https://crt.sh/?serial=009bb57e80d7a8d9527f9b78237c8b53cc
https://crt.sh/?serial=7061b255f4b8fa022ef7d1cfb2c229e4
https://crt.sh/?serial=5fc6582dbc8b148d93b5d92b00233b0b

And if these certificates were handled in that bug, then why wasn't that bug number referenced in the audit reports?

The Subordinate CAs that issued the leaf certificates fall out of scope of the Network Solutions audit and were disclosed in Sectigo’s audit report for the same period, for which they were in scope. The reason for this decision is that BDO elected to treat NetSol-branded Sub-CAs that chain only to Sectigo Roots as "managed CAs" that are only in scope for Sectigo's audit. BDO considers Network Solutions to be the CA Owner of Sub-CAs that chain to our own roots (irrespective of whether or not a cross-certified chain up to a Sectigo Root also exists).

My question was not for the revocation date for these certificates. I am trying to verify the statement that bug 1645686 identified and reported all of the certificates in question. The crt.sh URLs with certificate serial numbers are not as helpful as the references used in bug 1645686, which were crt.sh SHA256 hashes, crt.sh database cert IDs, and batch numbers from misissued.com (which then reference crt.sh database cert IDs). Even a pointer to a particular comment # in bug 1645686 would help. I'm just having a hard time correlating your list of certificates with those certificates reported in bug 1645686.

Flags: needinfo?(keith.mckenney)

(In reply to Ben Wilson from comment #5)
Seven of these certificates were included in misissued.com batch 185, which was mentioned in bug 1645686, comment 56:
https://crt.sh/?serial=5fc6582dbc8b148d93b5d92b00233b0b
https://crt.sh/?serial=7061b255f4b8fa022ef7d1cfb2c229e4
https://crt.sh/?serial=009bb57e80d7a8d9527f9b78237c8b53cc
https://crt.sh/?serial=7679ce1c66c60fc9cafeb5af10a3b9ff
https://crt.sh/?serial=4247ca86502743cc4efda3f1cf2ca53a
https://crt.sh/?serial=524f7650b21cc5f2287e34192ca470f3
https://crt.sh/?serial=00ed668507bfe4431b281e0aa8224384f5

The remaining three were included in multiple places in our records of the certificate list to be uploaded. However, they are not visible in any batch we can find on misissued.com. Our best surmise is that we experienced an error when adding these certificates to misissued.com, which were already revoked at that time. As our current practice is to include affected certificates directly in the relevant bug (as text, the way you see here, for short lists and in attached files for longer lists), we do not expect repetition of this error.

The certificates missing from that batch are:
https://crt.sh/?serial=7d0e4af7e0e576f0c0e0613a51ba8b45
https://crt.sh/?serial=00bd02a789e7e4bb9ce26145334398c98d
https://crt.sh/?serial=00d6ad43fd5284de89dd5a7e54283205ad

Thanks!

Flags: needinfo?(keith.mckenney)

As a matter that is directly relevant to this conversion, Network Solutions and Sectigo have decided to transition the Network Solutions and Web.com branded digital certificate businesses entirely to a managed CA model, managed by Sectigo.

We intend to announce a target transition date and other relevant details by the end of this month.

We are working on a full writeup for this incident.

Note that Sectigo provided a full writeup of the original issue in bug 1645686 comment 61, with a great deal of additional discussion among the community on that thread. Please see that thread and that comment for discussion of the original issue.

This writeup will cover the three missing certificates from the original writeup.

1. How your CA first became aware of the problem

In its role providing information to us in response to Ben’s question in comment 5, Sectigo discovered that three affected Network Solutions certificates were not visible at misissued.com, despite internal records indicating that they had been added. Sectigo informed us of that discovery.

2. Timeline

August 10, 2021
Ben Wilson writes up this bug based on our recent WebTrust reports.

August 13
After investigation, we conclude that this incident is a duplicate for bug 1645686 and express that belief in comment 2.

August 16
Ben asks for specifics of where these certificates were reported in comment 3.

August 26
Ben clarifies in comment 5 that he’s looking for specifics of when they were reported, not when they were revoked.

August 26 to September 13
We quickly identify seven certificates by where and when they were reported. Unable to find the other three, we recruit Sectigo’s help. Sectigo eventually concludes that, although it possesses multiple internal records including these certificates for uploading to misissued.com, for an unknown reason those certificates are not visible in the expected batch. This failure to report was unknown until this investigation.

September 14
In comment 6 Sectigo explains its conclusion and reports the missing certificates.

3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem.

This writeup is regarding failure to report certificates, not the original misissuance. The original misissuance was resolved in 2020 and covered in bug 1645686.

4. Summary of the problematic certificates

Three certificates were not reported in bug 1645686, issued between December 18, 2019 and April 10, 2020.

5. Affected certificates

The certificates that were not reported successfully are:
https://crt.sh/?serial=7d0e4af7e0e576f0c0e0613a51ba8b45
https://crt.sh/?serial=00bd02a789e7e4bb9ce26145334398c98d
https://crt.sh/?serial=00d6ad43fd5284de89dd5a7e54283205ad

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now

During its response to bug 1645686 Sectigo chose to load identified certificates to misissued.com in what eventually was batch 185. Internal records for this batch included these three certificates, in addition to the seven reported. However, these certificates do not currently appear in that batch, or any other misissued.com batch that we can identify.

We have been unable to determine the process or software error that led to these certificates failing to appear.

7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future

As described in comment 6, Sectigo has ceased using misissued.com for reporting certificates on Bugzilla, in favor of direct reporting. The direct reporting model will not result in this same error again.

Network Solutions is moving all our remaining CA operations to a pure managed service from Sectigo. Once our existing certificates have expired out, we will cease to behave as an active CA on Bugzilla. We have targeted a full transition to the managed service on or before November 8, 2021.

Are there any other questions on this issue?

Ben, this discussion appears to have run its course. Should we close this bug?

Flags: needinfo?(bwilson)

I'll close this on Wed. 20-Oct-2021.

Status: ASSIGNED → RESOLVED
Closed: 3 years ago
Flags: needinfo?(bwilson)
Resolution: --- → FIXED
Product: NSS → CA Program
Whiteboard: [ca-compliance] → [ca-compliance] [audit-finding]
You need to log in before you can comment on or make changes to this bug.