1724530 - Microsoft PKI Services: Overdue Audit Reports 2021

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[mohanr]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.131 Safari/537.36 Edg/92.0.902.67

Steps to reproduce:

Microsoft PKI Services did not upload audit reports to CCADB within 3 months of the audit period end date.

Timeline:
2021-04-30 End of audit period
2021-07-26 Preliminary reports posted
2021-08-03 Notified by Cross-Signed CA of missing ICAs in report
2021-08-03 Analysis and work with auditor
2021-08-06 Corrected reports provided by auditor
2021-08-06 Final audit reports posted to Bugzilla
2021-08-06 Final audit report links added to CCADB

Actual results:

We are working on understanding the root cause for why this incident occurred. We have posted the final reports to Bugzilla (1722411) and links to them within CCADB. We are currently troubleshooting an issue in the CCADB submission.
This is a preliminary report due to some key personnel being out of communication until next week. We wanted to inform the community as soon as we realized the issue even though we do not have full information yet.
We will provide a full report by EOD 2021-08-11.

Expected results:

Microsoft PKI Services should have uploaded audit reports to CCADB within 3 months of the audit period end date.

Comment 1

[John Mason]

Incident Report

1. How your CA first became aware of the problem.

There are two problems that we have run into with regard to posting our WebTrust Audit Letters. Specifically, for our WTBR and WTCA reports. Our previous audit concluded on 2020-04-30 and our most recent audits concluded on 2021-04-30.

The first is that we were late in posting our final audit reports to CCADB. We were aware that we were in the three-month grace period allowed by the Mozilla Trusted Root Program, as we had been getting automated notifications of the impending deadline from Mozilla/CCADB. We were able to get the final assertions/audit letters completed and signed off by our management and our auditors on 2021-07-30, but we were not able to post the final audit letters to a CCADB audit case until 2021-08-06.

The second issue we had is that our 2021-07-30 reports failed to include 16 ICA’s that were included in the scope of the audit, but did not appear in our assertion or audit letters. These were Cross Signed ICAs issued by a partner. This partner notified us on 2021-08-03 that our preliminary audit reports were missing these ICA’s and that they needed an updated report that included them to post their audit letters successfully. Our management and our auditors were able to issue updated reports with the ICA’s in the assertions and the audit letters on 2021-08-05.

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

• 2021-04-30 End of audit period
• 2021-07-26 Preliminary reports posted to CCADB using the “Test Preliminary Audit Statements” process published here (https://www.ccadb.org/cas/updates)
• 2021-08-03 Notified by Cross-Signed CA of missing ICAs in report
• 2021-08-03 Analysis and work with auditor
• 2021-08-05 Corrected reports provided by auditor
• 2021-08-06 Final audit reports posted to Bugzilla (https://bugzilla.mozilla.org/show_bug.cgi?id=1722411)
• 2021-08-06 Final audit report links added to CCADB (https://ccadb.force.com/s/case/5004o00000LOuRUAA1/apr-2021-audit-microsoft-corporation-public-ssl)
• 2021-08-10 Final Seals issued from CPA Canada

3. Whether your CA has stopped, or has not yet stopped, certificate issuance or the process giving rise to the problem or incident. A statement that you have stopped will be considered a pledge to the community; a statement that you have not stopped requires an explanation.

Issuance was not stopped during this incident as there were no technical issues related to certificate issuance because of the late audit updates.

4. In a case involving certificates, a summary of the problematic certificates. For each problem: the number of certificates, and the date the first and last certificates with that problem were issued. In other incidents that do not involve enumerating the affected certificates (e.g. OCSP failures, audit findings, delayed responses, etc.), please provide other similar statistics, aggregates, and a summary for each type of problem identified. This will help us measure the severity of each problem.

No certificates were mis-issued during this case. Here is a list of the certificates that were omitted from the final Audit Letters/Assertions from 2021-07-30:

https://crt.sh/?id=2841732943
https://crt.sh/?id=3232541596
https://crt.sh/?id=2841732835
https://crt.sh/?id=3232541597
https://crt.sh/?id=2841732847
https://crt.sh/?id=3232541594
https://crt.sh/?id=2841732837
https://crt.sh/?id=3232541595
https://crt.sh/?id=2841732843
https://crt.sh/?id=3163654574
https://crt.sh/?id=2841732828
https://crt.sh/?id=3163546037
https://crt.sh/?id=2841732842
https://crt.sh/?id=3163600408
https://crt.sh/?id=2841732827
https://crt.sh/?id=3163654575

5. In a case involving certificates, the complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem. In other cases not involving a review of affected certificates, please provide other similar, relevant specifics, if any.

See list above.

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

As discussed above there are two problems that we need to address with this bug. The first being that the reports were late in being posted to CCADB. The second being that we missed 16 ICAs that were in scope of the audit in our initial audit report of 2021-07-30.

With regard to the late audit reports. The root cause of this problem came down to deficient project management. We were running late with closing out the audit and we ran out of time to complete all of the required tasks. In 2020 our audit reports were final and complete on 2020-06-29 and this year it was 2021-07-30 (both audit periods ended on April 30). We were a full month later than last year and we ran out of time to receive the final reports and get them uploaded to CCADB before the three-month deadline. We had hoped that we would be able to complete all tasks before the end of the three-month period and were driving to finish on time. We ran into issues with key personnel being out of the office for planned vacation and we ran out of time to post. We should have managed the schedule to conclude the final audit reports within two-months of the audit end and began appropriate escalations when those milestones slipped. Additionally, we need to have more cross training of personnel and improved documentation of the full Audit End process and timelines.

With regard to the 16 missed ICAs in the assertions and audit reports. The root cause of this problem comes down to the processes we use at the end of audit cycle to check the scope of each of our assertions/audits. This year is the first year that we needed to include CAs that were Cross Signed into our audit reports, and we missed the fact that we needed to include certificates that do not chain to Microsoft roots into our reports. Our existing processes start and end with the master list of certificates generated from Microsoft CA databases and did not also include a check of any Cross Signed agreements. Of course, we understood that we had a Cross Sign relationship that needed audit support, but we failed to integrate that into these processes. Fortunately, our partners pointed out our misstep and the reports were amended quickly, as the 16 ICA’s were in scope of the audit (but not listed in the initial reports). Here we need to update our processes to include Cross Signed certificates that are managed by Microsoft PKI Services.

7. List of steps your CA is taking to resolve the situation and ensure that such a situation or incident will not be repeated in the future, accompanied with a binding timeline of when your CA expects to accomplish each of these remediation steps.

Completed Remediation's:
• Audit reports were amended, and final reports updated on 2021-08-05.
• Audit Case was opened in CCADB with updated reports on 2021-08-06. (https://ccadb.force.com/s/case/5004o00000LOuRUAA1/apr-2021-audit-microsoft-corporation-public-ssl).

Open Remediation's:
• We will update our documentation and timelines around the audit end processes to include clear timelines and metrics for reporting during the process (2021-08-31).
• We will ensure that all personnel with access to CCADB understand the clearer timelines and requirements and have the training needed to execute the audit end processes (2021-08-31)
• We will update our documentation around scoping audits to include a check for all Cross Signed agreements (or other external agreements) that we have in place during the audit period and ensure we include all necessary CAs in our reports (2021-09-17).

Comment 2

[John Mason]

We are still on track with the Open Remediation's.

Open Remediation's:
• We will update our documentation and timelines around the audit end processes to include clear timelines and metrics for reporting during the process (2021-08-31).
• We will ensure that all personnel with access to CCADB understand the clearer timelines and requirements and have the training needed to execute the audit end processes (2021-08-31)
• We will update our documentation around scoping audits to include a check for all Cross Signed agreements (or other external agreements) that we have in place during the audit period and ensure we include all necessary CAs in our reports (2021-09-17).

Comment 3

[John Mason]

We are still on track with the Open Remediation's.

Open Remediation's:
• We will update our documentation and timelines around the audit end processes to include clear timelines and metrics for reporting during the process (2021-08-31).
• We will ensure that all personnel with access to CCADB understand the clearer timelines and requirements and have the training needed to execute the audit end processes (2021-08-31)
• We will update our documentation around scoping audits to include a check for all Cross Signed agreements (or other external agreements) that we have in place during the audit period and ensure we include all necessary CAs in our reports (2021-09-17).

Comment 4

[John Mason]

We closed two of the Open Remediation's on time this week.

The additional training and documentation is already having a positive impact on the planning for the current audit cycles and should ensure that we do not have a repeat of this incident.

Closed Remediation's:
• We will update our documentation and timelines around the audit end processes to include clear timelines and metrics for reporting during the process (2021-08-31).
• We will ensure that all personnel with access to CCADB understand the clearer timelines and requirements and have the training needed to execute the audit end processes (2021-08-31)

Open Remediation's:
• We will update our documentation around scoping audits to include a check for all Cross Signed agreements (or other external agreements) that we have in place during the audit period and ensure we include all necessary CAs in our reports (2021-09-17).

Comment 5

[John Mason]

We are still on track with the Open Remediation

Open Remediation:
• We will update our documentation around scoping audits to include a check for all Cross Signed agreements (or other external agreements) that we have in place during the audit period and ensure we include all necessary CAs in our reports (2021-09-17).

Comment 6

[John Mason]

We are still on track with the last Open Remediation and will update this bug either Friday 2021-09-17 or Monday 2021-09-20 to confirm.

Open Remediation:
• We will update our documentation around scoping audits to include a check for all Cross Signed agreements (or other external agreements) that we have in place during the audit period and ensure we include all necessary CAs in our reports (2021-09-17).

Comment 7

[John Mason]

Confirming that all remediation's identified in this bug have now been completed. We formally request for this bug to please be marked as Resolved.

Comment 8

[John Mason]

If there are no more comments, can we please have this bug marked as Resolved?

Comment 9

[Ben Wilson]

Unless there are any objections, I'll close this on Friday 1-Oct-2021.