Closed Bug 1604124 Opened 4 years ago Closed 4 years ago

Microsoft DSRE PKI: problem reporting e-mail in CPS does not work

Categories

(CA Program :: CA Certificate Compliance, task)

task
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: oscarkarlsson72, Assigned: Dustin.Hollenback)

Details

(Whiteboard: [ca-compliance] [policy-failure])

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.79 Safari/537.36

Steps to reproduce:

I sent an e-mail to pkiwg@microsoft.com listed in their CPS to report a possible compliance issue and received a bounce.

Actual results:

Your message to pkiwg@microsoft.com couldn't be delivered.
The group pkiwg only accepts messages from people in its organization or on its allowed senders list, and your email address isn't on the list.

550 5.7.133 RESOLVER.RST.SenderNotAuthenticatedForGroup; authentication required; Delivery restriction check failed because the sender was not authenticated when sending to this group

Expected results:

E-mail should have been delivered to the Microsoft CA compliance team.

This e-mail address is also discussed in Bug 1424305 as the correct address for compliance issues but does not appear to accept outside e-mails.

Assignee: wthayer → jcooper
Status: UNCONFIRMED → ASSIGNED
Type: defect → task
Ever confirmed: true
Summary: Microsoft: e-mail in CPS does not work → Microsoft: problem reporting e-mail in CPS does not work
Whiteboard: [ca-compliance]

Hi all,

The email contact in the Microsoft PKI Services CPS is certificateauthority@microsoft.com, also listed on our repository page: https://www.microsoft.com/pkiops/docs/repository.htm. I contacted the team that uses pkiwg@microsoft.com and notified them of this bug and they tell me they have fixed the issue reported above.

Oscar, if you are trying to reach Microsoft PKI Services team please contact us at certificateauthority@microsoft.com. If you are trying to reach pkiwg@microsoft.com their inbox should accept external mail now. Please let me know if you have questions or encounter further problems.

Thanks,
Jason

Flags: needinfo?(oscarkarlsson72)

Oscar: What version of the CP/CPS did you examine to find that e-mail?

CP v3.1.2 notes the email Jason mentioned, in Comment #1, as does CPS v3.1.3

Jason: I believe the issue/confusion is related not to the Microsoft roots (those listed at https://www.microsoft.com/pkiops/docs/repository.htm ) but to the MSIT roots. The CCADB All Problem Reporting Mechanisms Report is what presently lists that contact information, but only for the MS IT SSL sub-CAs under the Baltimore CyberTrust Root.

Those CAs (example 1) appear to be under a different set of audits (WebTrust, BR), and as sub-CAs of DigiCert, fall under DigiCert's responsibility.

It looks like I made a mistake, and this should be flagged as a DigiCert sub-CA compliance issue rather than as a Microsoft Root CA compliance issue? I think this will be an interesting case for Wayne/Kathleen, about how to handle Roots and sub-CAs potentially operated by different teams of the same company.

Flags: needinfo?(jeremy.rowley)

I agree that this technically appears to be DigiCert's issue, but I'm also concerned by the implication that Microsoft, as a Mozilla Root Store Program member, bears no direct responsibility to Mozilla for their intermediates that chain to DigiCert roots.

Jason: what would happen if a problem report for the roots that you manage is sent to pkiwg@microsoft.com (assuming that address is now accepting external email)? If the burden is on relying parties to understand the structure of Microsoft's PKIs, I think we have a problem. As best I can tell, Microsoft is unique in having two distinct problem reporting addresses for their root and externally operated subordinate hierarchies.

Flags: needinfo?(jcooper)

I confirmed that the issue was with the email in the CPS chained to us. I also confirmed the email is currently operational, so the outage is resolved. Microsoft said they are working on an incident report about the cause of the outage and will post it here.

Flags: needinfo?(jeremy.rowley)

Thank you, I did not realize there were two different teams for PKI at Microsoft. To answer Ryan, it was Microsoft DSRE PKI CP-CPS for TLS Ver 2.3 June 2019. I have reached out to the proper team as per Jason's update.

Flags: needinfo?(oscarkarlsson72)

Here is a report for the email issue reported via this specific bug. Please let me know if there are any questions.

  1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

We were first made aware of this email issue via Bugzilla 1604124 that was then assigned to Jason Cooper (Microsoft). The email from the Bugzilla bug was forwarded to the other CA team on Monday, 16 December 2019 at 09:43 AM.

  1. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

Tuesday, 03 December 2019 at 04:03 PM: A modification was made to the configuration for the pkiwg@microsoft.com email address, which is a distribution list. A flag was set for Microsoft internal use only, preventing external email from being delivered.

Monday, 16 December 2019 at 01:05 AM: Oscar Karlsson (Reporter) sent original email to pkiwg@microsoft.com that was returned as undeliverable.

Monday, 16 December 2019 at 02:08 AM: Bugzilla 1604124 (https://bugzilla.mozilla.org/show_bug.cgi?id=1604124) was opened to notify of the issue with undeliverable mail sent to pkiwg@microsoft.com.

Monday, 16 December 2019 at 05:23 AM: Jason Cooper (Microsoft) was added to Bugzilla 1604124 and notified via email.

Monday, 16 December 2019 at 09:43 AM: Jason Cooper (Microsoft) forwarded this to the some members on the pkiwg@microsoft.com distribution list.

Monday, 16 December 2019 at 09:57 AM: Change was made to pkiwg@microsoft.com email address to re-allow external email.

Monday, 16 December 2019 at 04:52 PM: A test email was received from an external (non-Microsoft corporate) email account and was successfully received.

  1. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

No non-compliant certificates were issued.

  1. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

No non-compliant certificates were issued.

  1. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

No non-compliant certificates were issued.

  1. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

While editing membership of the pkiwg@microsoft.com distribution list, a setting was mistakenly configured that blocked email from external email addresses. The CA team had not received an indication that anyone was attempting to send to this address between the time of the change (03 December 2019) and when Oscar Karlsson (Reporter) reported the issue (16 December 2019). Once reported, the root cause was quickly identified and remediated.

The CA team is investigating how to monitor for changes to this setting in the future so that we will know immediately if it is changed.

There are currently two groups in Microsoft that manage publicly trusted CAs. Each team has their own email address. Both teams are investigating whether it is possible to have one email address for communication to both teams to avoid any potential confusion related to who to contact.

  1. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

There are currently two groups in Microsoft that manage publicly trusted CAs. Each team has their own email address. Both teams are investigating whether it is possible to have one email address for communication to both teams to avoid any potential confusion related to who to contact. In the mean time both groups are adding personnel from the other group to each mailbox.

Flags: needinfo?(jcooper)

Jason: Thank you for the incident report. My understanding is that from now on an email sent to either address can be assumed to have been received by the correct group within Microsoft - please correct me if I'm wrong. Also, please update this bug with the findings of the investigation into consolidating into a single problem reporting address.

Flags: needinfo?(mohanr)
Flags: needinfo?(julio.montano)

First, my apologies for the delays in responding to this bug. I am still in the process of transitioning into Jason's role. A new email address, centralpki@microsoft.com has been created and both teams are included as recipients for the address. The Microsoft DSRE PKI team will update their documents and repository site by June of this year. The Microsoft PKI team will update our repository site in the next week and our CP and CPS documents by August of this year. In the interim, we have been added to each other's email alias so that either team can route requests or inquiries appropriately.

Flags: needinfo?(julio.montano)
Assignee: jcooper → julio.montano

Thanks for the update, Julio!

It looks like there wasn't a smooth transition from Jason's open bugs to these, and so just wanted to make sure to draw attention to Keeping Us Informed

I think one thing that would be extremely valuable to understand, and I'm hoping you can share more context about, is the significant timeframes involved for what would nominally seem like a simple change. It stands out, for example, that the DSRE team will take until June to update both, while the Microsoft PKI team is able to do the site update quickly, but the CP/CPS update much longer.

Could you share a bit more about the challenges involved? It seems extremely useful to know, especially when policy updates (such as the recently released Policy 2.7) may require CAs to update their CP/CPSes. Naively, it seems like the sort of change that can be done within O(days), between drafting, reviews, approvals, and publications, but it sounds like that's not a realistic expectation?

Flags: needinfo?(mohanr) → needinfo?(julio.montano)

The reason for the prolonged timelines provided are that the updates to the sites and the CP/CPS updates were appended to planned updates each team had in progress to their respective CP/CPS and repository sites. If circumstances arose that mandated edits to the sites or CP/CPS documents those changes could be made in a more expedited fashion. That was not deemed necessary in this instance given the mitigation I mentioned in my last post (we added members of each team to each other’s email alias). Please let us know if there is any further concerns with these timelines.

Flags: needinfo?(julio.montano)

Julio: It has been well over a week and I'm not seeing the new address on https://www.microsoft.com/pkiops/docs/repository.htm as you indicated in comment #8. Am I missing something?

Thank you for the explanation in comment #10. While it helps to ease my concern, it doesn't eliminate it. Waiting 3+ months for a simple update does still give me the impression that Microsoft can't or won't make policy changes promptly.

Flags: needinfo?(julio.montano)

Wayne, you're correct and I apologize for the delay. The team had a couple of other updates that needed to be done to the site and these were bundled with those changes. As mentioned above, the issue was mitigated by adding members of the DSRE PKI team to the microsoftpki email address so there was no concern that messages to either team would be missed. The updated has been completed.

Flags: needinfo?(julio.montano)

It appears that all questions have been answered and remediation is complete.

Status: ASSIGNED → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
Assignee: julio.montano → Dustin.Hollenback
Summary: Microsoft: problem reporting e-mail in CPS does not work → Microsoft DSRE PKI: problem reporting e-mail in CPS does not work
Product: NSS → CA Program
Whiteboard: [ca-compliance] → [ca-compliance] [policy-failure]
You need to log in before you can comment on or make changes to this bug.