Microsoft PKI Services: Trusted Role Control Failure
Categories
(CA Program :: CA Certificate Compliance, task)
Tracking
(Not tracked)
People
(Reporter: u654666, Assigned: u654666)
Details
(Whiteboard: [ca-compliance] [policy-failure] Next update 2023-10-23)
Preliminary report
A Microsoft PKI Services engineer identified that a user account had been provisioned for an employee that is not in a Trusted Role. The user is a Microsoft employee who works on other internal CAs and has completed a background check, but has not received Trusted Role training or been assigned within a Trusted Role. This failed to meet section 2.c. of the Network Security Requirements: “Ensure that only personnel assigned to Trusted Roles have access to Secure Zones and High Security Zones”.
The initial investigation also identified a separate problem related to 3-month access reviews which will be opened as a separate Bugzilla bug.
How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in the MDSP mailing list, a Bugzilla bug, or internal self-audit), and the time and date.
This problem was identified by an internal self-audit and an internal incident was opened on 2023-08-09 at 13:15 Pacific time.
A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.
Note: All times in Pacific Time (PT).
2022-11-15 18:17: Non-Trusted Role user requested a user account within a High Security Zone
2022-11-15 19:01: Service owner approved request outside of process
2022-11-17 18:05: Security Engineer assigned ticket to perform the account creation
2022-11-22 13:54: User account created for Non-Trusted Role user outside of process for a High Security Zone (Note that account is unusable without associated smart card)
2023-08-02: Smart card issued to Non-Trusted Role user
2023-08-03 16:49: Non-Trusted Role successful logon to file server within a High Security Zone
2023-08-04 14:38: Non-Trusted Role successful logon to file server within a High Security Zone
2023-08-09 13:09: Trusted Role Engineer performed random audit and discovered Non-Trusted Role user account within a High Security Zone and opened internal Incident
2023-08-09 13:15: User account for Non-Trusted Role user was deleted
Whether your CA has stopped, or has not yet stopped, certificate issuance or the process giving rise to the problem or incident. A statement that you have stopped will be considered a pledge to the community; a statement that you have not stopped requires an explanation.
Microsoft PKI Services has not stopped certificate issuance. While we consider this a serious process failure by approving a user who has not been formally assigned to a Trusted Role, we are confident that the environment was secure. The user who was granted access is a Microsoft employee who regularly works on internal CA systems, but was not assigned to a Trusted Role and restricted from our High Security Zone only for the purposes of limiting access.
In a case involving certificates, a summary of the problematic certificates. For each problem: the number of certificates, and the date the first and last certificates with that problem were issued. In other incidents that do not involve enumerating the affected certificates (e.g. OCSP failures, audit findings, delayed responses, etc.), please provide other similar statistics, aggregates, and a summary for each type of problem identified. This will help us measure the severity of each problem.
Certificates were not impacted by this process failure. We identified a single problem user account that was deleted quickly after being identified.
In a case involving TLS server certificates, the complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem. It is also recommended that you use this form in your list "https://crt.sh/?sha256=[sha256-hash]", unless circumstances dictate otherwise. When the incident being reported involves an SMIME certificate, if disclosure of personally identifiable information in the certificate may be contrary to applicable law, please provide at least the certificate serial number and SHA256 hash of the certificate. In other cases not involving a review of affected certificates, please provide other similar, relevant specifics, if any.
Certificates were not impacted by this process failure.
Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
We are still investigating and expect to have a complete report within one week.
List of steps your CA is taking to resolve the situation and ensure that such situation or incident will not be repeated in the future, accompanied with a binding timeline of when your CA expects to accomplish each of these remediation steps.
We deleted the problem user account and are still investigating the root causes and remediation steps to prevent repeating this issue in the future.
|
||
Updated•3 years ago
|
|
||
Comment 1•3 years ago
|
||
Incident Report
A Microsoft PKI Services engineer identified that a user account had been provisioned for an employee that is not in a Trusted Role. The user is a Microsoft employee who works on other internal CAs and has completed a background check, but has not received Trusted Role training or been assigned within a Trusted Role. This failed to meet section 2.c. of the Network Security Requirements: “Ensure that only personnel assigned to Trusted Roles have access to Secure Zone and High Security Zones”.
The initial investigation also identified a separate problem related to 3-month access reviews which was opened as a separate Bugzilla bug (https://bugzilla.mozilla.org/show_bug.cgi?id=1848280).
How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in the MDSP mailing list, a Bugzilla bug, or internal self-audit), and the time and date.
This problem was identified by an internal self-audit and an internal incident was opened on 2023-08-09 at 13:15 Pacific time.
A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.
Note: All times in Pacific Time (PT).
2022-11-15 18:17: Non-Trusted Role user requested a user account within a Secure Zone
2022-11-15 19:01: Service owner approved request outside of process
2022-11-17 18:05: Security Engineer assigned ticket to perform the account creation
2022-11-22 13:54: User account created for Non-Trusted Role user outside of process for a Secure Zone (Note that account is unusable without associated smart card)
2023-08-02: Smart card issued to Non-Trusted Role user
2023-08-03 16:49: Non-Trusted Role successful logon to file server within a Secure Zone
2023-08-04 14:38: Non-Trusted Role successful logon to file server within a Secure Zone
2023-08-09 13:09: Trusted Role Engineer performed random audit and discovered Non-Trusted Role user account within a Secure Zone and opened internal Incident
2023-08-09 13:15: User account for Non-Trusted Role user was deleted
2023-08-17 13:17: Updated manual provisioning process for access to Secure Zone and High Security Zone to include an independent check for Trusted Role group membership
Whether your CA has stopped, or has not yet stopped, certificate issuance or the process giving rise to the problem or incident. A statement that you have stopped will be considered a pledge to the community; a statement that you have not stopped requires an explanation.
Microsoft PKI Services has not stopped certificate issuance. While we consider this a serious process failure by approving a user who has not been formally assigned to a Trusted Role, we are confident that the environment was secure. The user who was granted access is a Microsoft employee who regularly works on internal CA systems but was not formally assigned to a Trusted Role and restricted from our Secure Zone.
In a case involving certificates, a summary of the problematic certificates. For each problem: the number of certificates, and the date the first and last certificates with that problem were issued. In other incidents that do not involve enumerating the affected certificates (e.g. OCSP failures, audit findings, delayed responses, etc.), please provide other similar statistics, aggregates, and a summary for each type of problem identified. This will help us measure the severity of each problem.
Certificates were not impacted by this process failure. We identified a single problem user account that was deleted quickly after being identified.
In a case involving TLS server certificates, the complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem. It is also recommended that you use this form in your list "https://crt.sh/?sha256=[sha256-hash]", unless circumstances dictate otherwise. When the incident being reported involves an SMIME certificate, if disclosure of personally identifiable information in the certificate may be contrary to applicable law, please provide at least the certificate serial number and SHA256 hash of the certificate. In other cases not involving a review of affected certificates, please provide other similar, relevant specifics, if any.
Certificates were not impacted by this process failure.
Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
Our provisioning processes for access to the Secure Zone and High Security Zone are a manual human process. The process relies on the manager approving the provisioning request to ensure that the user is a member that is listed in our Trusted Role group. The manager in this case approved a user that was not in the Trusted Role group for access to the Secure Zone.
Today, there is de-centralized tracking of the Trusted Role group list, that depends on each manager to be responsible for ensuring that their team members have been onboarded and offboarded correctly to the Trusted Role group.
In this case, the mistake could have been detected from 2022-11-22 but was not detected until 2023-08-09. The reasons for this are discussed in the other Bugzilla Bug that we opened specific to the 3-Month Access Review Process Failure (https://bugzilla.mozilla.org/show_bug.cgi?id=1848280).
List of steps your CA is taking to resolve the situation and ensure that such situation or incident will not be repeated in the future, accompanied with a binding timeline of when your CA expects to accomplish each of these remediation steps.
On 2023-08-09 we deleted the problem user account to mitigate the immediate Trusted Role control failure.
On 2023-08-17 we updated the manual provisioning processes for our Secure Zone and High Security Zone to include a check of the Trusted Role group list before the manager is asked to approve the inclusion of the user. This should ensure that the user is a member of the Trusted Role group before the manager is asked to approve the request in the first place.
We will centralize management of Trusted Role group list and improve the process and availability of the list within CA Systems to allow for further automation. This will be implemented by Sept 1, 2023. This will allow us to automate the verification of all user accounts that have access to the Secure Zone.
We will automate verification of all user accounts that have access to the Secure Zone to ensure they are a valid user in the Trusted Role group and take appropriate action in the case of a failure. The date for completion of this automation will be provided by Aug 25, 2023.
|
||
Updated•3 years ago
|
We are still making progress towards the centralized management and process improvements related to the Trusted Role list. Due to several staff on vacation, it has slowed down some of the decision making and need to remove the 2023-09-01 commitment date. We should be able to provide a more accurate commitment date by next week.
Also, due to the staffing issues that are blocking decisions, we have not been able to lock on the final scope of work for the Secure Zone member verification automation and will provide a better estimate next week with a date we can commit to completing the work.
Since our last update, we were able to finalize the commitment dates for open work to ensure this issue does not recur.
We will centralize management of Trusted Role group list and improve the process and availability of the list within CA Systems to allow for further automation. This will move the list from a static list to a list that is more visible and able to be queried via automation. This will be implemented by 2023-09-29.
We will automate verification of all user accounts that have access to the Secure Zone to ensure they are a valid user in the Trusted Role group and take appropriate action in the case of a failure. This will be implemented by 2023-10-13.
|
|
||
Updated•3 years ago
|
MS PKI Services completed the centralized management of the Trusted Role group list which has allowed automation to replace several manual processes for comparing with that list. This included adding automation to verify all users with access to the Secure Zone are listed within an appropriate Trusted Role.
With that work completed, we respectfully request for this bug to be closed.
|
|
||
Updated•2 years ago
|
Description
•