Closed Bug 1730291 Opened 3 months ago Closed 2 months ago

Apple: Test web page certificates expired

Categories

(NSS :: CA Certificate Compliance, task)

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: certification_authority, Assigned: certification_authority)

Details

(Whiteboard: [ca-compliance])

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.1.2 Safari/605.1.15

On September 09, 2021 the Apple CA Compliance team received notification that certificates for Apple CA’s test Web pages (https://www.apple.com/certificateauthority/public/) expired. The issue has since been resolved.

A full report will be provided no later than September 17, 2021.

Assignee: bwilson → certification_authority
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true
Whiteboard: [ca-compliance]
Type: defect → task

Incident Report

1 ) How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

On September 09, 2021 the Apple CA compliance team received notification from an internal security team that certificates for some of Apple CA’s test Web pages (https://www.apple.com/certificateauthority/public/) had expired.

2) A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

  • 2020-04-03 at 14:45 PT: The certificates for Apple CA’s test Web pages were deployed.
  • 2021-01-21 at 17:47 PT: The operations team was inadvertently removed from the email group used for expiration notifications.
  • 2021-09-08 at 21:26 PT: The operations team was notified by an internal security team that certificates for some of Apple CA’s test Web pages had expired and began investigating the issue.
  • 2021-09-08 at 22:36 PT: The operations team confirmed that some of the certificates for Apple CA’s test Web pages had expired, determined the issue was not business impacting, and deferred further action until the following morning. They also discovered that expiration notifications were not received because the notification group was empty. **
  • 2021-09-09 at 08:46 PT: The operations team resumed investigating the issue.
  • 2021-09-09 at 09:15 PT: The operations team notified the compliance team.
  • 2021-09-09 at 10:00 PT: The compliance team met to review the situation.
  • 2021-09-09 at 14:00 PT: The operations team was added back to the email group used for expiration notifications.
  • 2021-09-09 at 17:51 PT: New certificates were issued.
  • 2021-09-09 at 21:16 PT: The certificates were deployed to Apple CA’s test Web pages.
  • 2021-09-10 at 11:00 PT: We held a post-mortem and determined that the email group used for the certificates in question was leveraged for additional purposes after the certificates were issued, which lead to changes in the group membership that removed the operations team responsible for these certificates. Therefore, the expiration notifications, although sent, were not received by the operations team.
  • 2021-09-14 at 19:39 PT: The test website endpoints used in Apple CA’s test Web pages were added to Apple CA’s monitoring application that proactively monitors endpoints directly for certificate expiration.

3) Whether your CA has stopped, or has not yet stopped, certificate issuance or the process giving rise to the problem or incident. A statement that you have stopped will be considered a pledge to the community; a statement that you have not stopped requires an explanation.

This did not affect certificate issuance.

4) In a case involving certificates, a summary of the problematic certificates. For each problem: the number of certificates, and the date the first and last certificates with that problem were issued. In other incidents that do not involve enumerating the affected certificates (e.g. OCSP failures, audit findings, delayed responses, etc.), please provide other similar statistics, aggregates, and a summary for each type of problem identified. This will help us measure the severity of each problem.

There were no problematic certificates.

5) In a case involving certificates, the complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem. In other cases not involving a review of affected certificates, please provide other similar, relevant specifics, if any.

The following certificates for Apple CA’s test Web pages expired without being renewed:

6) Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

For renewal notifications, the operations team managing the test websites utilizes email groups. The email group used for the certificates in question was leveraged for additional purposes after the certificates were issued, which lead to changes in the group membership that removed the operations team responsible for these certificates. Therefore, the expiration notifications, although sent, were not received by the operations team.

7) List of steps your CA is taking to resolve the situation and ensure that such situation or incident will not be repeated in the future, accompanied with a binding timeline of when your CA expects to accomplish each of these remediation steps.

  • We contemplated whether we should automate the management of these certificates. We determined that since the management of the certificates used for the test sites is unique among all the certificates we manage, automation efforts would have broader impact and value in other places. For these certificates, we will rely on expiration monitoring, as discussed below.
  • We have added the test website endpoints to our monitoring solution that proactively monitors endpoints directly for certificate expiration. This will be our primary expiration monitoring mechanism moving forward. While this monitoring system did exist when Apple CA’s test Web pages were deployed, it was not capable of monitoring endpoints outside of Apple’s internal network at that time.
  • The membership of the notification group was remediated to include the operations team and the group will not be used for additional purposes moving forward.

We have added the test website endpoints to our monitoring solution that proactively monitors endpoints directly for certificate expiration.

Does that monitoring notify you some time in advance, or only after the certificates that endpoint is serving expire?

(In reply to Matthias from comment #3)

Does that monitoring notify you some time in advance, or only after the certificates that endpoint is serving expire?

Our monitoring system checks the status of certificates for identified endpoints on a daily basis and starts alerting 30 days before a certificate expires.

There are no outstanding tasks for this incident report. Since no other questions or concerns have been raised, can this incident be closed?

Unless there are any objections, I'll close this tomorrow.

Flags: needinfo?(bwilson)
Status: ASSIGNED → RESOLVED
Closed: 2 months ago
Flags: needinfo?(bwilson)
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.