Closed Bug 1648593 Opened 4 years ago Closed 4 years ago

Sectigo: Potential audit report delay

Categories

(CA Program :: CA Certificate Compliance, task)

Product:
CA Program
Component:
CA Certificate Compliance
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: nick, Assigned: nick)

Details

(Whiteboard: [ca-compliance] [audit-failure] [audit-delay])

Attachments

(1 file)

Sectigo-ExplanatoryLetter.pdf
159.64 KB, application/pdf
Details
  1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

Sectigo were informed by our WebTrust auditors of a possible delay to our audit report on June 25th, 2020. The report was due to be submitted to Mozilla by way of the CCADB on or before Monday 29th of June 2020.

  1. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

As soon as our Compliance team were made aware of the possible delay by the auditors, the message was communicated internally to a wider distribution group including C-level staff.
This incident reports was drafted, approved, and filed as a result.

Additionally, representatives at our auditor were contacted to stress the urgency of completion of the report by the the deadline.

  1. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

Current CA operations are continuing as normal. We are still aiming to deliver the report on-time by Monday June 29th, but with possible delays no later than Monday July 6th.

  1. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

N/A

  1. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

N/A

  1. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

The current restrictions on work and travel throughout the world has impacted the usual timelines for audit procedures. Though we had hoped to have the final report before the due date, we have found some last-minute delays are likely.
As soon as we were aware of this possible delay, even of a few days, we wanted to ensure the delay was documented.

  1. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

As per item 3, we intend to have the audit report with Mozilla by no later than July 6th.

An update will be posted on Monday June 29th, should the report not be completed then.

Assignee: bwilson → nick
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true

Further communications with the auditors today and over the weekend suggest the report is not likely to be available today (Monday, June 29th), but we are still targeting before a week from today (Monday, July 6th).

I'll update daily until the report is delivered.

Whiteboard: [ca-compliance] [audit-delay]

We have received the final reports from our auditors and will be updating them to CCADB once we have processed the final signed versions.

Robin has added the reports: https://bugzilla.mozilla.org/show_bug.cgi?id=1472993

It looks like they passed ALV for everything except file location. I'm trying to find out why.

I am going to wait until I have the cpacanada URLs to re-run the ALV process.

If I may ask:

If your CA will not be revoking the certificates within the time period required by the BRs, our expectations are that:

  • ...
  • The issue will need to be listed as a finding in your CA’s next BR audit statement.
  • Your CA will work with your auditor (and supervisory body, as appropriate) and the Root Store(s) that your CA participates in to ensure your analysis of the risk and plan of remediation is acceptable.
  • ...

- Mozilla wiki - CA/Responding To An Incident: Revocation (https://wiki.mozilla.org/CA/Responding_To_An_Incident#Revocation)

The audit reports added to Bug 1472993 do not have 'findings', only a list of 'comments', and it is my understanding that these two terms have a substantially different in meaning.

If my understanding is correct, then I am missing Bug 1620561 -
Non-revocation of certificates with subject:organizationalUnitName in DV certificates - as a 'finding' in the audit statement, as per the information provided in the quoted wiki page.

If I may ask:
...

Hmm, seems like I didn't phrase it as a question, so I'll try again:

Could someone from Sectigo or Mozilla verify that my understanding (regarding the distinction between 'comments' and 'findings' in audit reports) is correct, and subsequently, that the audit report as uploaded in Bug 1472993 is not conforming to expectations as documented on the wiki?

Matthias: I see it listed as Appendix C, Item 9 in https://bug1472993.bmoattachments.org/attachment.cgi?id=9161258

The relevant distinction here is whether or not these incidents caused the auditor to issue a qualified opinion. EY is asserting that, despite the unprecedented number of systemic failures by Sectigo, their opinion about Sectigo's compliance to WebTrust is not modified in any way, and they believe it's fairly stated that Sectigo fully complied with the BRs.

The intent of the language you mentioned in Comment #6 is to ensure the CA and their management disclose to the auditor each of these incidents. At present, because the CA contracts the auditor, the set of controls examined to allow the auditor to determine compliance is determined by the auditor and CA, and not disclosed. While the proposed Detailed Control Report would provide greater transparency, our options are either to accept the auditor's opinion or, based on understanding of the evidence, choose to no longer accept reports from this auditor over concerns about the degree of expertise.

Am I looking at the wrong attachment here?

Flags: needinfo?(matthias.vandemeent)

I see it listed as Appendix C, Item 9 in https://bug1472993.bmoattachments.org/attachment.cgi?id=9161258
[...]
Am I looking at the wrong attachment here?

We're most definately looking at the same attachment; all of the added reports indeed have the bug listed in Appendix C, but my perceived problem is that it is not listed as a 'finding' in these audit statements.

Further explanation as to why I expect Bug 1620561 reported as a finding:

The MRSP and corresponding documents are not quite consistent on what specific kind of document must be supplied to Mozilla as a report on the audit:

  • 3.1.3 (Audit Parameters) mentions "point-in-time audit statements", "qualified audit statement" and "audit reports"
  • 5.3.2 (Publicly Disclosed and Audited) mentions "audit report"
  • 8.2 (Change in Operational Personnel) mentions "audit statement (or opinion letter)"
  • 8.3 (Change in Secure Location) mentions "audit statements, opinion letter, or point-in-time audit statement", closing off with "The regular annual audit statements MUST still happen in a timely manner"

Of the above, the wiki CA/Responding To An Incident - Revocation only mentions "audit statement", and the wiki CA/Audit Statements - Intermediate Certificates states the following:

Remediation may include one of the following when a non-technically-constrained intermediate certificate is missing from an audit statement. Note that Mozilla's Root Store Policy says: "If the CA has a currently valid audit report at the time of creation of the certificate, then the new certificate MUST appear on the CA's next periodic audit reports."

  • Have your auditor issue a revised report that includes the intermediate certificate.

All of the above mentioned sections of Mozilla documentation make me believe that an audit report as mentioned in the documentation is indistinguishable from an audit statement, and that these two terms describe the same document. It is also of note that the ALV documentation on the wiki references the CCADB, which also intermingles the audit report and audit statement terminology, further solidifying this conclusion.

With my above understanding that the uploaded documents are mentioned in the documentation as both audit statements and audit reports, and that there exists a material difference between 'comment' and 'finding', the supplied audit statement fails my expectations as aroused in CA/Responding To An Incident - Revocation, namely that Bug 1620561 would need to be listed as a finding in the CA's next BR audit statement, not as an externally supplied comment.

Extending from this failed expectation, the current wording of the supplied audit statements and the information that is available in Bug 1620561, I am concerned that Sectigo has not communicated about Bug 1620561 with the auditor in a manner as referred to in the wiki, and that the audit statement consequently would not be a fully qualified audit statement.

Flags: needinfo?(matthias.vandemeent)

(In reply to Matthias from comment #9)

With my above understanding that the uploaded documents are mentioned in the documentation as both audit statements and audit reports, and that there exists a material difference between 'comment' and 'finding', the supplied audit statement fails my expectations as aroused in CA/Responding To An Incident - Revocation, namely that Bug 1620561 would need to be listed as a finding in the CA's next BR audit statement, not as an externally supplied comment.

Extending from this failed expectation, the current wording of the supplied audit statements and the information that is available in Bug 1620561, I am concerned that Sectigo has not communicated about Bug 1620561 with the auditor in a manner as referred to in the wiki, and that the audit statement consequently would not be a fully qualified audit statement.

I'm still having trouble understanding your understanding/interpretation, or your expectation. I'm hoping this is just communication difficulties, and not that I'm missing an important detail you're trying to highlight.

The extent to which Mozilla, or any browser, can require anything without directly initiating the audit report itself, is to ensure that the information is disclosed to the auditor. Seeing it incorporated as part of Management's Assertion is the expectation, in that it makes it unambiguous that the auditor had this information available to them. However, whether or not the auditor qualifies their opinion or not is subjective, based on their auditor.

You're correct that, at present, we largely don't make the distinction between audit statements and reports (realizing that ETSI seems to make a distinction, and in a way that's contrary to browser expectations, which is its own whole thing). However, we (the collective browser community) do not currently demand an auditor qualify their opinion, because, well, "no one" can do that: it's the auditor and their reputation as to whether they examine the comments and believe whether or not the CA filled their obligations.

As best I can tell, the reports meet the expectation, and having been involved in drafting that language, I'm not sure I understand what you're expecting. Are you expecting a guaranteed qualified audit?

Are you expecting a guaranteed qualified audit?

I agree that my terminology was not great, 'would not be a fully qualified audit statement' were not the right words to use. The concern I was trying to convey was as such:

  • The auditor may have had incomplete or incorrect information due to incorrect labeling of some published issues (specifically Bug 1620561), and as a consequence that the auditor may not have audited or investigated the issue as thoroughly as they would otherwise have investigated that issue.

This concern is based on the following:

  • Assumption: There is indeed a distinction between 'comment' and 'finding' in the auditing framework that is being used by the auditor.
  • Observation: There are 13 other issues listed with arguably the same or higher priority in the list of 'comments', as they are higher on the published list of seemingly arbitrarily ordered bugzilla issues.
  • Observation: There is no indication in the communications from the CA on Bug 1620561, nor its parent Bug 1593776, that the CA has worked with the auditor on the delayed revocation, as all opinions and statements seem to be the CA's and no indication is given regarding communications with the auditor.
    Although reporting on this work with the auditor is not required, I had at least expected some indication that the auditor has been involved in the statements supplied by Sectigo.

Following from the above assumptions and observations, combined with the wiki section on Revocation, I had expected that the following expectation would be held:

  • I expected that the Management's Assertion or the "Independent Assurance Report" had included Bug 1620561 not as a comment but as a finding.

If the distinction between 'comment' and 'finding' does not exist in this auditing framework, then this is indeed not an issue. Otherwise, my concerns regarding the completeness and correctness of the information supplied to the auditor remain.

Although reporting on this work with the auditor is not required, I had at least expected some indication that the auditor has been involved in the statements supplied by Sectigo.

Right, but we have that (or, at least as best we get) from the auditor in:

Sectigo’s management has disclosed to us the attached comments (Appendix C) that have been posted publicly in the online forums of the Bugzilla site, as well as the online forums of individual internet browsers that comprise the CA/Browser Forum. We have considered the nature of these comments in determining the nature, timing and extent of our procedures.

I do agree with in principle, which I'm concerned that none of these issues led the auditor to believe Sectigo had trouble with the principles or controls in a way that resulted in a qualification of the opinion. That Sectigo would be able to obtain an unqualified report, for the time period in question, absolutely raises questions about what the auditor performed, did, and their judgement. The auditor is free to have their opinion, and there are inherent limitations, but you would think some of this evidence would materially affect things. I think that's more of an EY quality issue, and more broadly fits into the question about Detailed Control Reporting going forward to understand what exactly were the tests the auditor performed to help them reach their opinion that the principles and controls were met, despite the evidence.

There are ways for an auditor to both report on facts that didn't modify their opinion, along with the relevant control and remediation, and facts that did cause their opinion to be modified, and so you're right for questioning whether this is sufficient. However, those methods aren't (yet) mandatory, so we're in a gray area now, and instead should look askance at EY more than Sectigo. I think it sounds like your expectation was something of a qualified opinion, in line with the ISAE 3000 Illustrative Reports for WebTrust , such as IN 1.4 or IN 1.6

It's worth noting that this is a pattern: Sectigo received a clean report from EY (New York office) in 2018 despite numerous misissuances. Bug 1525446 was filed against EY, but the focus at that time was on disclosure rather than the auditor's opinion.

The 2019 BR report from EY was also clean despite listing 3 BR compliance issues in appendix C.

Wayne: Huge thanks for highlighting this deeply troubling pattern.

Ben: I believe you're looking into the overall set of issues with Sectigo, and you mentioned in Comment #5 waiting to rerun these reports, so setting N-I for you. When we have a system of issues that make the audits unreliable, where a CA has such poor incident response that they ignore issues for weeks on end, I cannot help but see a more systemic set of issues at the CA that are not easily resolved and may bear further discussion on appropriate next steps.

Flags: needinfo?(bwilson)

Ben - (c5): We're still pushing the auditors for updates on the WebTrust seals, we're told they're still waiting on CPA Canada and other internal processes to get them ready. I'll update again next week.

Still waiting on a further update from our auditors - I have chased them again this today.

We have no further update at this time. We continue to communicate with our auditors on this issue.

We have had a good number of further useful exchanges with our auditors.
Our ability to get resolution of this has been impaired by holiday schedules beyond our control.
We now feel we are on the home straight and expect a September release.

In the last week we have made good progress. We continue to expect a September release.

I think it sounds like your expectation was something of a qualified opinion, in line with the ISAE 3000 Illustrative Reports for WebTrust , such as IN 1.4 or IN 1.6

My understanding of what a qualified audit means was incomplete. Apparently, I was indeed looking for what amounts to a qualified opinion.

Ryan, thanks for your careful explanations.

  1. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

The current restrictions on work and travel throughout the world has impacted the usual timelines for audit procedures. Though we had hoped to have the final report before the due date, we have found some last-minute delays are likely.
As soon as we were aware of this possible delay, even of a few days, we wanted to ensure the delay was documented.

  1. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.
    As per item 3, we intend to have the audit report with Mozilla by no later than July 6th.

An update will be posted on Monday June 29th, should the report not be completed then.

Did you further investigate why you only got notice of a potential delay 4 days before the deadline, and how you will be preventing this from occuring again?

Flags: needinfo?(nick)

Did you further investigate why you only got notice of a potential delay 4 days before the deadline, and how you will be preventing this from occuring again?

We had clear expectations from our auditors that we were on track for the deadline until shortly before we informed this community otherwise.

We have been investigating and continue to investigate how to prevent such problems in the future. We will seek to understand how this process could have been better. Our ability to do so may be limited by the auditors’ ability and willingness to share detailed information about their processes.

Flags: needinfo?(nick)

(In reply to Nick France from comment #21)

We had clear expectations from our auditors that we were on track for the deadline until shortly before we informed this community otherwise.

As Sectigo is ultimately in control of selecting the auditor(s) you will use, how has this informed Sectigo's choices going forward? This seems an extremely unique scenario, with the exception of course of the remarkable and concerning lack of qualifications, given the available information.

We have been investigating and continue to investigate how to prevent such problems in the future. We will seek to understand how this process could have been better. Our ability to do so may be limited by the auditors’ ability and willingness to share detailed information about their processes.

To the extent it's useful, both to this community and, apparently, to Sectigo, it seems important that per BRs Section 8.6, an explanatory letter be provided by the auditor as to the delays. Can you ensure this is provided within the next 7 days?

Flags: needinfo?(nick)

Looking back to Comment #21, we made a comment which may have made it look as though our auditors were not cooperating with us, although that was not our intent, and so we must address that.
When we said “We will seek to understand how this process could have been better. Our ability to do so may be limited by the auditors’ ability and willingness to share detailed information about their processes”, that was with a mindset that we were addressing only the 3 day delay in the initial report and even then it was a one-sided comment and we had a number of deliverables for the auditors that slipped somewhat during that period.
The client review meeting after every audit engagement always identifies points of improvement on both sides, and I am sure that when we come to it this year we will find that we had at least a hand in almost every one of the slips to the delivery of these reports.

(In reply to Ryan Sleevi from comment #22)
An issue here is that we were concentrating on the 3 day delay in obtaining the initial versions of the WebTrust reports at the end of June / start of July, but we feel we must address the elephant in the room which is that we are now almost another three months down the line and we don’t have new WebTrust audit reports and the associated seals.

We have experienced two significant setbacks during the audit work that have resulted in this long delay in getting audit reports out.

1) CAs in scope

In our previous year’s management assertions we had included the list of all of our publicly trusted issuing CAs as being in scope for all of the sets of WebTrust Criteria on which we receive reports, i.e. we previously had the same list of CAs in each of our WebTrust reports. This year we provided just such a single list shortly after the end of the audit period. We were asked for distinct CA lists for each report.
We produced the lists from the understanding given to us by the audit team which was that they should be broken down by our future Intent to issue.
We appreciate and understand that these management assertions are signed by Sectigo and that we take the responsibility for them, but with the benefit of hindsight we can see that we did not apply an appropriate level of rigour to our examination of that understanding (that they should be broken down by our future Intent) before we applied it to provide the initial lists, and we should have been able to do that as we had in-house expertise that we could and should have consulted, specifically Rob Stradling and his work up to that time on https://crt.sh/mozilla-disclosures.
Of course, the appearance of the arguably somewhat related bug (https://bugzilla.mozilla.org/show_bug.cgi?id=1650910 on intent vs. capability) around the same time increased the significance of this mistake substantially and that is something that we were unaware of when we produced those initial CA lists.

We received the initial signed audit reports from our auditors on July 3rd, and by July 13th we realized that our lists of issuing CAs for each management assertion (apart from WebTrust for CAs) were not as required, and we reported this to our auditors. The auditors halted the seal issuance process for all except the WebTrust for CAs report.

The auditors asked for further data, including data to construct the revised sets of issuing CAs for each assertion and also a count of certificates of each type issued from each CA in the audit period.

2) Partial population listing

We provided this revised ‘sets of issuing CAs and counts of certificates of each type’ data by August 19th. The auditors reviewed this data and compared it with the earlier detailed population listings (i.e. lists of certificates issued in the audit period) and found that those earlier population listings were partial, i.e. that they did not contain as many rows as would be expected to match the count of certificates of each type issued from each CA. They informed us of this by August 27th, and we investigated and found the cause of the shortfall in the population listings and provided correct certificate populations listings on September 1, 2020.

By September 17 the auditor asked CPA Canada to initiate the temporary suspension of the earlier issued WebTrust for CA seal (seal ID 10475) as they realized that even this report could not stand because of the original partial population report.

Since then we have repeated and completed the sampling and evidence gathering process from the full population report.

We continue to work with our auditors positively and diligently (both them and us) toward getting new audit reports signed off and seals issued. We still anticipate getting the new signed reports by the end of September and expect to see the seals issued within about 7 days of the reports being issued.

(In reply to Robin Alden from comment #23)

1) CAs in scope

In our previous year’s management assertions we had included the list of all of our publicly trusted issuing CAs as being in scope for all of the sets of WebTrust Criteria on which we receive reports, i.e. we previously had the same list of CAs in each of our WebTrust reports. This year we provided just such a single list shortly after the end of the audit period. We were asked for distinct CA lists for each report.
We produced the lists from the understanding given to us by the audit team which was that they should be broken down by our future Intent to issue.
We appreciate and understand that these management assertions are signed by Sectigo and that we take the responsibility for them, but with the benefit of hindsight we can see that we did not apply an appropriate level of rigour to our examination of that understanding (that they should be broken down by our future Intent) before we applied it to provide the initial lists, and we should have been able to do that as we had in-house expertise that we could and should have consulted, specifically Rob Stradling and his work up to that time on https://crt.sh/mozilla-disclosures.
Of course, the appearance of the arguably somewhat related bug (https://bugzilla.mozilla.org/show_bug.cgi?id=1650910 on intent vs. capability) around the same time increased the significance of this mistake substantially and that is something that we were unaware of when we produced those initial CA lists.

It could be a lack of coffee / too much coffee, but I'm having trouble understanding this. If I'm following correctly, you started off with a complete list, based on capability ("all of our publicly trusted issuing CAs as being in scope"), then reduced that list ("We were asked for distinct CA lists for each report"), then expanded that list ("we realized that our lists... were not as required")

That seems a bit confusing for a sequence, so I suspect I'm probably misunderstanding something.

If I understand the second part, it sounds like the initial population list (i.e. that given to the auditor during the first phase) actually omitted certificates that Sectigo had issued, which the auditors did not detect until Sectigo sent a list on August 19, which the auditors tried to reconcile with the previous data, and discovered on August 27 that Sectigo had omitted certificates from the initial set. Sectigo then sent a corrected set on Sept 1. Is that... roughly correct?

If so, why were Comment #16, Comment #17, and Comment #18 so totally and utterly devoid of substance and details? Did these facts, even if they were preliminary discussions, not merit being disclosed as part of explaining the issues here?

I think it's useful to understand the tragedy of errors, here, but I'd like to reiterate the request in Comment #22 for a proper letter. Comment #23 contains a number of very useful details, so I don't want to overlook that, but it's omitting what was requested, and it's not clear to me why that is. While I certainly want to value transparency, which Comment #23 does seek to provide in better detail than previous comments on this bug, I also want to make sure we're mitigating risk appropriately. In my mind, there's both risk that Sectigo is not being as candid with all of the facts, and the risk of auditor issues, both of which are very relevant for our community.

I'm also wanting to understand more about the root causes about the sequence of audit scoping issues, especially in light of the many conversations regarding CCADB scoping, ALV, and of course, browsers' regular feedback to the Webtrust TF. That there is still confusion in September 2020 suggests deeply problematic issues are still at play here, and we need to identify the factors in order to better resolve them.

In trying to update the CCADB, I reached out to the E&Y auditors to see whether I should use the current WebTrust audits (linked in Bug 1472993) in the CCADB. They recommended that I wait for an updated set of audit letters before I update the CCADB.

Flags: needinfo?(bwilson)

(In reply to Ryan Sleevi from comment #24)

It could be a lack of coffee / too much coffee, but I'm having trouble understanding this. If I'm following correctly, you started off with a complete list, based on capability ("all of our publicly trusted issuing CAs as being in scope"), then reduced that list ("We were asked for distinct CA lists for each report"), then expanded that list ("we realized that our lists... were not as required")

That seems a bit confusing for a sequence, so I suspect I'm probably misunderstanding something.

If I understand the second part, it sounds like the initial population list (i.e. that given to the auditor during the first phase) actually omitted certificates that Sectigo had issued, which the auditors did not detect until Sectigo sent a list on August 19, which the auditors tried to reconcile with the previous data, and discovered on August 27 that Sectigo had omitted certificates from the initial set. Sectigo then sent a corrected set on Sept 1. Is that... roughly correct?

Given that the report and explanatory letter are due imminently, we hope the letter will clarify the changes in CA lists and scope. In the interest of including the auditors' viewpoint in addition to our own, we would like to wait for that letter. In the event that the community needs further explanation in beyond what the letter contains, we are available to provide it.

If so, why were Comment #16, Comment #17, and Comment #18 so totally and utterly devoid of substance and details? Did these facts, even if they were preliminary discussions, not merit being disclosed as part of explaining the issues here?

During this time period we were highly focused on the successful resolution of our audit. We had been presented with a new approach than we had seen in the past and needed to start with fundamentals like determining the proper methodology to proceed. We didn’t want to make statements that might later turn out to be wrong and discerned that the immediate value of revealing bits and pieces of our ongoing challenges was less than that of getting the audit and its report right. Comment #23 was intended to begin to share information once we were feeling confident in our read on the facts, including agreement from the auditors. It was never intended as a substitute for the specified letter.

I think it's useful to understand the tragedy of errors, here, but I'd like to reiterate the request in Comment #22 for a proper letter. Comment #23 contains a number of very useful details, so I don't want to overlook that, but it's omitting what was requested, and it's not clear to me why that is. While I certainly want to value transparency, which Comment #23 does seek to provide in better detail than previous comments on this bug, I also want to make sure we're mitigating risk appropriately. In my mind, there's both risk that Sectigo is not being as candid with all of the facts, and the risk of auditor issues, both of which are very relevant for our community.

I'm also wanting to understand more about the root causes about the sequence of audit scoping issues, especially in light of the many conversations regarding CCADB scoping, ALV, and of course, browsers' regular feedback to the Webtrust TF. That there is still confusion in September 2020 suggests deeply problematic issues are still at play here, and we need to identify the factors in order to better resolve them.

(In reply to Ben Wilson from comment #25)

In trying to update the CCADB, I reached out to the E&Y auditors to see whether I should use the current WebTrust audits (linked in Bug 1472993) in the CCADB. They recommended that I wait for an updated set of audit letters before I update the CCADB.

Hi Ben. We've received the updated set of audit letters from E&Y. I've attached them to bug #1472993 and updated the links in our Audit Case accordingly. If you want to process our Audit Case now, please go ahead. Or, if we receive the Seal URLs before you process our Audit Case, then I'll update the links in our Audit Case again.

(In reply to Ryan Sleevi from comment #22)

(In reply to Nick France from comment #21)
<snip>
To the extent it's useful, both to this community and, apparently, to Sectigo, it seems important that per BRs Section 8.6, an explanatory letter be provided by the auditor as to the delays. Can you ensure this is provided within the next 7 days?

Hi Ryan. An explanatory letter is attached.

Flags: needinfo?(nick)

(In reply to Rob Stradling from comment #28)

Hi Ryan. An explanatory letter is attached.

The auditor enquired whether certificates were issued by the Certification Authorities amended in the list on July 23, 2020. On August 19, 2020, Sectigo provided the auditor with the number of certificates issued by each Certification Authority.

This seems to point to a delayed communication from Sectigo to the auditor, coinciding with the lack of substantive updates regarding the state of this issue (Comment #15, Comment #16, Comment #17) and contradicts Comment #15 (on 2020-08-02): "We're still pushing the auditors for updates on the WebTrust seals, we're told they're still waiting on CPA Canada and other internal processes to get them ready."

Could you further elaborate on what happened in that time frame, including why it took almost 4 weeks to provide the data to the auditor? This seems to have been a significant factor in what is currently a 3 month delay and counting.

Then, I noticed that the WebTrust EVSSL Independent Assurance Report referenced the WebTrust [..] Version 1.6.8, which is valid for audit periods commencing on or after 1 June 2019. As the audit period starts at 1 April 2019, that does seem like a discrepancy.

Similar issues can be found for the SSL Baseline report and the WebTrust CA report, as the referenced WebTrust criteria are for audits commencing on or after 1 June 2019, while the audit periods that are being reported on start at 1 April 2019, so that would technically leave the period 1 April 2019 through 31 May 2019 uncovered for EVSSL, SSL Baseline and WebTrust CA.

This issue already existed for the previous audit reports of the same period, but I only noticed it just now. It really makes me question the accuracy of EY's audit reports, I've noticed this issue in multiple assurance reports that have been added before 2020-06-01 (the theoretical first date that audit periods of 1 year which use WebTrust criteria 1.6.8 can be reported on).

Flags: needinfo?(nick)

Rob, Tim, Nick,

Could you further elaborate on what happened in the time frame between July 23, 2020 and August 19, 2020, as the explanatory letter does not explain that 4 week delay in communication?

Also, regarding the audit criteria used, could you verify with your auditor that the criteria used for the audit were correct, valid, and available for the full audit period?

Flags: needinfo?(tim.callan)
Flags: needinfo?(rob)

(In reply to Matthias from comment #30)

Could you further elaborate on what happened in the time frame between July 23, 2020 and August 19, 2020, as the explanatory letter does not explain that 4 week delay in communication?

That portion of the delay hinged primarily on disagreement between ourselves and our auditors about the correct scope of the CAs to be included in these audit reports.

To understand, you need to know why the original reports we received on July 3 were not usable. On June 22 our auditors asked us for the list of CAs that will be used for specific types of certs, with a separate list for each audit report. This was in addition to the full list of public-trusted root and intermediate certificates for the audit period that we had provided on April 3, 2020.

We put together these “future intent” lists and delivered them to our auditors on June 23, not realizing that these lists would limit the CAs considered in each report. It was when we read the delivered reports on July 3 that we understood how the future intent lists had been applied. This application was, based on our experience, contrary to WebTrust audit requirements.

We then went into a lengthy period of back-and-forth discussion with our auditors about the correct scope of CAs to be audited. We were advised that CPA Canada had told EY to follow its current course, which Sectigo didn’t believe would be correct advice. Eventually we received authoritative guidance from Kathleen Wilson on this issue and could proceed. Shortly after, on August 20, Kathleen clarified this requirement publicly in the Mozilla PKI policy’s 2.7.1 label (https://github.com/mozilla/pkipolicy/issues/147), stating in part, “EV-capable intermediate certs must be specifically listed in an EV audit.”

Flags: needinfo?(tim.callan)

We have no update at this time.

(In reply to Rob Stradling from comment #27)

(In reply to Ben Wilson from comment #25)

In trying to update the CCADB, I reached out to the E&Y auditors to see whether I should use the current WebTrust audits (linked in Bug 1472993) in the CCADB. They recommended that I wait for an updated set of audit letters before I update the CCADB.

Hi Ben. We've received the updated set of audit letters from E&Y. I've attached them to bug #1472993 and updated the links in our Audit Case accordingly. If you want to process our Audit Case now, please go ahead. Or, if we receive the Seal URLs before you process our Audit Case, then I'll update the links in our Audit Case again.

Ben, we have now received the Seal URLs, and so I've just updated our CCADB Audit Case accordingly. Please would you continue to process our Audit Case now?

Flags: needinfo?(rob)
Flags: needinfo?(nick)
Flags: needinfo?(bwilson)

I've updated the CCADB. Please let me know if anything needs to be corrected. I'll close this bug unless there are any other issues to be discussed.

Thanks Ben. We have nothing further to discuss on this bug.

Status: ASSIGNED → RESOLVED
Closed: 4 years ago
Flags: needinfo?(bwilson)
Resolution: --- → FIXED
Product: NSS → CA Program
Whiteboard: [ca-compliance] [audit-delay] → [ca-compliance] [audit-failure]
Whiteboard: [ca-compliance] [audit-failure] → [ca-compliance] [audit-failure] [audit-delay]
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: