1750631 - SSL.com: Issuance of TLS certificates with domain validation methods prohibited by SC-45

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[SSL.com]

Attachment: affected_certificates_1.3.csv

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36

Steps to reproduce:

This is a preliminary incident report. Our investigation into this matter is ongoing.

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

On 2021-12-29, our internal Quarterly Certificate Review identified three (3) DV TLS certificates which were issued with validation methods prohibited by SC-45. Per our Incident Management Policy, this triggered an investigation to discover the entire population of affected certificates (see sections 4 and 5 below).

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

2021-06-03T00:00+00:00 Ballot SC-45 passes; effective 2021-12-01 the validation methods 3.2.2.4.6, 3.2.2.4.18, and 3.2.2.4.19 of the Baseline Requirements MUST NOT be used to issue Wildcard Certificates and MUST NOT be used as Authorization Domain Names for subordinate FQDNs of the validated FQDN.

2021-06-04T07:56+00:00 Ballot is analyzed internally by the Compliance Department and results are documented in our “Incoming Ballot” ticket; the need to update the CP/CPS and the validation systems (RA Portal, ACME component) is identified.

2021-06-30T17:00+00:00 PMA approves CP/CPS 1.13, setting deadline of 2021-09-01 for adoption of SC-45 changes (earlier than the one required by SC-45)

2021-09-03T09:25+00:00 CA Administrators deploy SC-45 update to production.

2021-09-09T21:29+00:00 RA Administrators deploy SC-45 update to production.

2021-12-29T00:21+00:00 Security Auditors discover three potentially problematic certificates (including one wildcard certificate). The initial evidence shows that method 3.2.2.4.18 was used to validate the entire domain namespace for these certificates. Further evidence is sought in collaboration with RA Administrators.

2021-12-30T22:06+00:00 Security Auditors inform the team about the issue, along with a timeline of actions to be performed by each role, should no further evidence to support issuance is found. The actions include creation of an event ticket on the same day, the notification of the affected customers before 2022-01-03, the revocation of these three problematic certificates before 2022-01-04T18:00 and further actions regarding the escalation of the event to an incident and the investigation for other problematic certificates.

2021-12-30T22:56+00:00 Security event ticket created.

2022-01-02T01:03+00:00 Validation Specialists report that two out of the three affected certificates were already expired.

2022-01-03T21:09+00:00 RA Administrators report that no additional evidence was found to support the issuance of the three certificates in question, thus the three potentially mis-issued certificates found during QCR, are confirmed. Upon further investigation, a fourth certificate, issued 2021-09-10 reusing validation evidence based on 3.2.2.4.18, is found to be problematic.

2022-01-04T16:03+00:00 Validation Specialists report revocation of the remaining two affected non-expired certificates (as identified at the time).

2022-01-04T18:40+00:00 In accordance with our Incident Management Policy, the security event is formally escalated to an incident.

2022-01-04T22:29+00:00 The investigation is expanded to cover the entire period since 2022-09-01.

2022-01-07T18:45+00:00 The criteria for identifying the targeted population of other problematic certificates are finalized and the RA Administrators begin to research the entire corpus of TLS certificates. The problem which caused the incident is investigated by our engineers and a hotfix is prepared.

2022-01-09T12:55+00:00 The hotfix is deployed to the RA system to ensure no similar issuance can be performed.

2021-01-10T20:21+00:00 Research of the entire corpus of TLS certificates identifies 706 potentially affected certificates. A QA phase is initiated to exclude any false positives and identify any outliers.

2022-01-11T15:35+00:00 Follow-up checks conducted to re-confirm that the hotfix is effective, per request by Security Auditors.

2022-01-11T17:00+00:00 Security Auditors start compiling a preliminary incident report.

2022-01-13T19:50+00:00 Preparations to provide notifications to owners of all affected non-expired non-revoked certificates for upcoming revocation.

2022-01-10 to 2022-01-17 The QA phase of the investigation continues. This includes a manual review of each issuance, enhanced by automated checks using QCR tools.

2022-01-17T08:29+00:00 Investigation is completed and 657 certificates confirmed to be affected. Revocation deadline set to 2022-01-21T18:00+00:00.

2022-01-17T15:00+00:00 Preliminary incident report updated based on the outcome of the abovementioned investigation.

2022-01-17: Filed initial Bugzilla report (this document).

3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

A hotfix has already been deployed (see timeline above); no similar mis-issuances can be performed currently.

4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

The initial tranche included three (3) TLS certificates, issued between 2021-09-01 and 2021-09-08. A fourth TLS certificate, issued 2021-09-10, was found during initial research.

Our investigation has identified a total of six hundred fifty-seven (657) affected TLS certificates, issued between 2021-09-01 and 2022-01-08, of which four hundred nine (409) were identified as non-expired non-revoked certificates.

5. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

Serial numbers of the affected certificates and their crt.sh IDs (pre-certificates confirmed to be logged to CT) are included in the attached file.

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

A delayed implementation of changes to meet the criteria of SC-45 in the timeline specified by our CP/CPS (i.e. before 2021-09-01) caused the issuance of the three (3) problematic certificates identified during our quarterly review. During that period (2021-09-01 to 2021-09-09) the system continued to consider methods 3.2.2.4.18 and 3.2.2.4.19 of the Baseline Requirements as eligible to validate the entire domain namespace and thus issue Wildcard Certificates or be used as Authorization Domain Names for subordinate FQDNs of the validated FQDN.

Upon further investigation, the SC-45 update which was deployed on our production systems on 2021-09-09 proved to be incomplete. Specifically, while it effectively handled issuance based on newly presented validation evidence, it failed to address issuance based on evidence re-use.

Our initial assessment is that the requirement to manage evidence re-use was not captured in the design requirements and acceptance criteria, and thus this use case was neither addressed correctly nor detected during testing. The problematic certificates were thus not identified until examples were captured and examined in our quarterly review process.

This section shall be updated based on the upcoming analysis of the incident, as required by our Incident Management Policy.

7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

Immediate actions are described in steps 2-5 of this report and include the identification of the population of affected certificates, the deployment of the code fix to prevent further problematic issuances, the revocation of the initial tranche of certificates, the scheduled revocation for the newly identified problematic certificates within 5 days and notification of the affected customers.

In particular, a significant amount of time was invested to extend our research to the entire population of TLS certificates issued since 2021-09-01, and to complete a thorough review of each item, to ensure the accuracy of the affected population.

Our investigation is ongoing to reveal more details on this issue and its source. Per our Incident Management Policy, an analysis shall also take place to identify any underlying weaknesses and according to the results, decide the proper measures and improvements in our systems and processes, so that such occurrences are not repeated in the future.

A full incident report shall be filed here when our investigation is complete. In the meantime, we will post regular updates.

Comment 1

[SSL.com]

This is an update to note our progress.

Regarding the revocation of the affected active (non-expired non-revoked) certificates:

• Revocation of the affected active certificates was initiated according to plan on Friday 2022-01-21Τ16:00+00:00 and completed on the same day at 17:36+00:00.
• A follow-up check which was conducted on Monday 2022-01-24 revealed that due to a technical issue, fifty-three (53) certificates were not revoked during the bulk revocation process. Revocation of these certificates took place on the next day.
• As of 2022-01-25T+17:52+00:00 all affected active certificates have been revoked.

We acknowledge that the above constitutes a violation of section “4.9.1.1 Reasons for Revoking a Subscriber Certificate” of our CP/CPS (regarding the revocation timeline of the remaining 53 certificates). A separate bug shall be registered for this issue.

In parallel, an analysis of the current incident shall take place in accordance with our Incident Management Policy to reveal any underlying weaknesses and identify possible remediation measures. We shall follow up with more information in our next update (to follow no later than February 2, 2022).

Comment 2

[SSL.com]

We have now opened Bug 1752636 to document the delayed revocation issue mentioned above.

Comment 3

[SSL.com]

This is an update to note our progress.

Since the previous update, we have focused on our postmortem analysis. Two primary contributing issues have been identified:

• Delayed adoption of SC-45 changes after 2021-09-01. This was not a violation of the BRs (since the effective date for SC-45 changes was 2021-12-01), but it was a violation of our CP/CPS (in which we specified an earlier effective date of 2021-09-01); and
• Insufficient implementation of the SC-45 update in our RA Portal. This caused violation of both the BRs and our CP/CPS, since it allowed issuance of problematic certificates to continue when re-using evidence, including after 2021-12-01.

Our investigation is ongoing, and we intend to present a fuller account of our findings from this analysis when complete. We shall continue to report our progress here, with our next update to follow no later than February 9, 2022.

Comment 4

[SSL.com]

This is an update to note our progress.

Since the previous update, the internal auditing department focused on gathering all the facts that led to this issue, identifying possible failures in that timeline and conducting an analysis of the underlying weaknesses. The results of this analysis were presented and discussed with the engineering teams; apart from confirming the conclusions of the internal audit, the discussion also included possible remediation measures which would prevent this issue and any similar future cases.

As mentioned in comment #3 above, two primary contributing issues have been identified:

• Delayed adoption of SC-45 changes after 2021-09-01, and
• Insufficient implementation of the SC-45 update in our RA Portal.

Regarding the former issue, our analysis has identified the following failures, shortcomings or mistakes:

1. Our ballot monitoring system identified and started work on SC-45 changes immediately after it was approved. However, there was insufficient involvement of our engineering teams to analyze the implications at the RA Portal level and plan ahead.

2. The PMA decision for an earlier adoption of SC-45 policy changes (2022-09-01 rather than 2022-12-01) was not communicated effectively to our engineering teams. In particular, this communication relied on a) the participation of engineering team members as involved participants in the PMA (where the decision was taken) and b) our standard process for internal communication of our updated CP/CPS reflecting this decision (via a redline version and internal channels) to all Trusted Role (TR) members. Thus, no further notifications or updates for this earlier deadline for SC-45 changes were made.

3. The above resulted in failure by the engineering departments involved to recognize, plan for and adhere to the new deadline (2021-09-01) for implementation of the required changes.

Regarding the latter issue, our analysis has identified the following failures, shortcomings or mistakes:

4. The analysis and design of SC-45 changes in the RA Portal were affected by the delayed acknowledgment of the new deadline, and proved to be insufficient.
5. The update was reviewed and tested by the engineering teams; however, the lack of detailed acceptance criteria and poor coordination with the compliance department did not allow detection of the fact that the RA Portal update addressed SC-45 related issuances, but not re-issuances based on re-using validation evidence.

Based on our analysis, issues no. 4-5 are addressed in our new Software Development Lifecycle (SDLC) policy (see also bugs #1722089 and #1724520) and associated procedures. These require software engineers to work with the end user (the compliance officer, in case of incoming ballots) to establish clear and detailed acceptance criteria before any work begins.

For issues no. 1-3, the following remediation measures have been decided:

a. Update of our Monitoring Policy to formally adopt our “Procedure to monitor updates in external requirements” and strengthen the obligation of all TRs to take actions promptly and collaborate with the compliance department to ensure continuous compliance in their respective area of responsibility.
b. Improvement in the above-mentioned procedure, with clear responsibilities for initial assessment, implementation and ballot management (including follow-up responsibilities).
c. Improvement of our methods to communicate and act upon changes due to incoming ballots and their timelines. For example, incoming ballots and other external requirements shall be a standard agenda item in our weekly coordination meetings.
d. Awareness training for all TRs regarding the updated Monitoring Policy (particularly regarding incoming ballot/external requirements) and the new SDLC.

The timeline for the above action plan shall be part of the final report for this issue, which we intend to file within the next week.

Comment 5

[SSL.com]

Our investigation into this matter has been completed, and this is our final report regarding this incident.

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

On 2021-12-29, our internal Quarterly Certificate Review identified three (3) DV TLS certificates which were issued with validation methods prohibited by SC-45. Per our Incident Management Policy, this triggered an investigation to discover the entire population of affected certificates (see sections 4 and 5 below).

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

2021-06-03T00:00+00:00 Ballot SC-45 passes; effective 2021-12-01 the validation methods 3.2.2.4.6, 3.2.2.4.18, and 3.2.2.4.19 of the Baseline Requirements MUST NOT be used to issue Wildcard Certificates and MUST NOT be used as Authorization Domain Names for subordinate FQDNs of the validated FQDN.

2021-06-04T07:56+00:00 Ballot is analyzed internally by the Compliance Department and results are documented in our “Incoming Ballot” ticket; the need to update the CP/CPS and the validation systems (RA Portal, ACME component) is identified.

2021-06-30T17:00+00:00 Policy Management Authority (“PMA”) approves CP/CPS 1.13, setting deadline of 2021-09-01 for adoption of SC-45 changes (earlier than the one required by SC-45)

2021-09-03T09:25+00:00 CA Administrators deploy SC-45 update to production.

2021-09-09T21:29+00:00 RA Administrators deploy SC-45 update to production.

2021-12-29T00:21+00:00 Security Auditors discover three potentially problematic certificates (including one wildcard certificate). The initial evidence shows that method 3.2.2.4.18 was used to validate the entire domain namespace for these certificates. Further evidence is sought in collaboration with RA Administrators.

2021-12-30T22:06+00:00 Security Auditors inform the team about the issue, along with a timeline of actions to be performed by each role, should no further evidence to support issuance is found. The actions include creation of an event ticket on the same day, the notification of the affected customers before 2022-01-03, the revocation of these three problematic certificates before 2022-01-04T18:00 and further actions regarding the escalation of the event to an incident and the investigation for other problematic certificates.

2021-12-30T22:56+00:00 Security event ticket created.

2022-01-02T01:03+00:00 Validation Specialists report that two out of the three affected certificates were already expired.

2022-01-03T21:09+00:00 RA Administrators report that no additional evidence was found to support the issuance of the three certificates in question, thus the three potentially mis-issued certificates found during QCR, are confirmed. Upon further investigation, a fourth certificate, issued 2021-09-10 reusing validation evidence based on 3.2.2.4.18, is found to be problematic.

2022-01-04T16:03+00:00 Validation Specialists report revocation of the remaining two affected non-expired certificates (as identified at the time).

2022-01-04T18:40+00:00 In accordance with our Incident Management Policy, the security event is formally escalated to an incident.

2022-01-04T22:29+00:00 The investigation is expanded to cover the entire period since 2022-09-01.

2022-01-07T18:45+00:00 The criteria for identifying the targeted population of other problematic certificates are finalized and the RA Administrators begin to research the entire corpus of TLS certificates. The problem which caused the incident is investigated by our engineers and a hotfix is prepared.

2022-01-09T12:55+00:00 The hotfix is deployed to the RA system to ensure no similar issuance can be performed.

2021-01-10T20:21+00:00 Research of the entire corpus of TLS certificates identifies 706 potentially affected certificates. A QA phase is initiated to exclude any false positives and identify any outliers.

2022-01-11T15:35+00:00 Follow-up checks conducted to re-confirm that the hotfix is effective, per request by Security Auditors.

2022-01-11T17:00+00:00 Security Auditors start compiling a preliminary incident report.

2022-01-13T19:50+00:00 Preparations to provide notifications to owners of all affected non-expired non-revoked certificates for upcoming revocation.

2022-01-10 to 2022-01-17 The QA phase of the investigation continues. This includes a manual review of each issuance, enhanced by automated checks using QCR tools.

2022-01-17T08:29+00:00 Investigation is completed and 657 certificates confirmed to be affected. Revocation deadline set to 2022-01-21T18:00+00:00.

2022-01-17T15:00+00:00 Preliminary incident report updated based on the outcome of the abovementioned investigation.

2022-01-17T20:21+00:00 Filed initial Bugzilla report.

2022-01-21Τ16:00+00:00 to 2022-01-25T+17:52+00:00 Revocation of all affected certificates. Note that due to a problem in the process, a new delayed revocation incident is caused (see bug #1752636).

2022-01-26T21:13+00:00 Update of bug #1750631, informing about the delayed revocation of the 53 certificates and our intention to file a separate bug.

2022-01-30 to 2022-02-08 Ongoing postmortem investigation by the internal auditing department.

2021-02-02T23:12+00:00 Update of the public bug to report progress with our postmortem analysis.

2021-02-03 to 2022-02-07 Ongoing investigation, focused on gathering all the facts that led to this issue, identifying possible failures in that timeline and conducting an analysis of the underlying weaknesses and the possible remediation measures.

2021-02-08T18:00+00:00 Internal meeting to discuss the results of the investigation of our internal auditors and possible remediation measures. Collaborative analysis of the facts which led to this incident and discussions between the engineering and compliance departments to analyze any underlying weaknesses and, according to the results, decide any additional measures and improvements in our systems and processes, so that such occurrences are not repeated in the future.

2021-02-09T21:51+00:00 Update of the public bug to report progress with our postmortem analysis.

2022-02-14 to 2022-02-15 Finalization of the postmortem analysis, the remediation measures and the timeline.

2021-02-16 Start drafting the final Bugzilla report (this document)

2021-02-18 Filed final Bugzilla report (this document).

3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

A hotfix has already been deployed (see timeline above); no similar mis-issuances can be performed currently.

4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

The initial tranche included three (3) TLS certificates, issued between 2021-09-01 and 2021-09-08. A fourth TLS certificate, issued 2021-09-10, was found during initial research.

Our investigation has identified a total of six hundred and fifty-seven (657) affected TLS certificates, issued between 2021-09-01 and 2022-01-08, of which four hundred and nine (409) were identified as non-expired non-revoked certificates.

5. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

Serial numbers of the affected certificates and their crt.sh IDs (pre-certificates confirmed to be logged to CT) are included in the attached file.

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

A delayed implementation of changes to meet the criteria of SC-45 in the timeline specified by our CP/CPS (i.e. before 2021-09-01) caused the issuance of the three (3) problematic certificates identified during our quarterly review. During that period (2021-09-01 to 2021-09-09) the system continued to consider methods 3.2.2.4.18 and 3.2.2.4.19 of the Baseline Requirements as eligible to validate the entire domain namespace and thus issue Wildcard Certificates or be used as Authorization Domain Names for subordinate FQDNs of the validated FQDN.

Upon further investigation, the SC-45 update which was deployed on our production systems on 2021-09-09 proved to be incomplete. Specifically, while it effectively handled issuance based on newly presented validation evidence, it failed to address issuance based on evidence re-use.

Two primary contributing issues have been identified per our postmortem analysis:

• Delayed adoption of SC-45 changes after 2021-09-01, and

• Insufficient implementation of the SC-45 update in our RA Portal.

Regarding the former issue, our analysis has identified the following issues:

1. Our ballot monitoring system identified and started work on SC-45 changes immediately after it was approved. However, there was insufficient involvement of our engineering teams to analyze the implications at the RA Portal level and plan ahead.

2. The PMA decision for an earlier adoption of SC-45 policy changes (2022-09-01 rather than 2022-12-01) was not communicated effectively to our engineering teams. In particular, this communication relied on a) the participation of engineering team members as involved participants in the PMA (where the decision was taken) and b) our standard process for internal communication of our updated CP/CPS reflecting this decision (via a redline version and internal channels) to all Trusted Role (TR) members. Thus, no further notifications or updates for this earlier deadline for SC-45 changes were made.

3. The above resulted in failure by the engineering departments involved to recognize, plan for and adhere to the new deadline (2021-09-01) for implementation of the required changes.

Regarding the latter issue, our analysis has identified the following issues:

4. The analysis and design of SC-45 changes in the RA Portal were affected by the delayed acknowledgment of the new deadline, and proved to be insufficient.

5. The update was reviewed and tested by the engineering teams; however, the lack of detailed acceptance criteria and poor coordination with the compliance department did not allow detection of the fact that the RA Portal update addressed SC-45 related issuances, but not re-issuances based on re-using validation evidence. The problematic certificates were thus not identified until examples were captured and examined in our quarterly review process.

Corrective actions for the above issues are described in section 7 of this report.

7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

Immediate actions are described in steps 2-5 of this report and include the identification of the population of affected certificates, the deployment of the code fix to prevent further problematic issuances, the revocation of the initial tranche of certificates, the scheduled revocation for the newly identified problematic certificates within 5 days and notification of the affected customers.

Additionally, a significant amount of time was invested to extend our research to the entire population of TLS certificates issued since 2021-09-01, and to complete a thorough review of each item, to ensure the accuracy of the affected population.

Based on our analysis, issues no. 4-5 of section 6 of this report are addressed in our new Software Development Lifecycle (SDLC) policy (see also bugs #1722089 and #1724520) and associated procedures. These require software engineers to work with the end user (the compliance officer, in case of incoming ballots) to establish clear and detailed acceptance criteria before any work begins.

For issues no. 1-3 of section 6 of this report, the following remediation measures have been decided:

a. Update of our Monitoring Policy to formally adopt our “Procedure to monitor updates in external requirements” and strengthen the obligation of all personnel in Trusted Roles (“TRs”) to take actions promptly and collaborate with the compliance department to ensure continuous compliance in their respective area of responsibility.

b. Improvement in the above-mentioned procedure, establishing clear responsibilities for initial assessment, implementation and ballot management (including follow-up responsibilities).

c. Improvement of our methods to communicate and act upon changes due to incoming ballots and their timelines. In particular, incoming ballots and other external requirements are now a standard agenda item in our weekly coordination meetings. Also, a dedicated communication channel is now available to inform all teams, discuss implications in systems and processes, monitor and report completion.

d. Awareness training for all TRs regarding the updated Monitoring Policy (particularly regarding incoming ballot/external requirements) and the new SDLC.

Action (c) has already been completed. Our plan is to complete all other actions before the end of March 2022.

Comment 6

[SSL.com]

This is an update to report our progress on outstanding items for this issue.

All remediation actions for this bug have been completed:

• The Monitoring Policy (MP) has been updated to cover the continuous monitoring of updates in external requirements (e.g. CA/B Forum requirements, updates in Root Store Programs, legislation), strengthen the obligation of all personnel in Trusted Roles and adopt the “Procedure to monitor updates in external requirements” (ERP). The updated MP and the supporting ERP have been officially adopted by the Policy Management Authority and are already operating as part of our standard processes.
• The ERP has been designed to clearly define the workflow and the obligations of Trusted Role members with regards to monitoring, reporting, analyzing, implementing and verifying normative updates. Furthermore, this procedure has been incorporated in our ticketing system to ensure smooth flow of the process.
• Internal training was successfully conducted for all Trusted Role members to ensure awareness / understanding of their obligations related to the updated MP and the ERP. An additional internal training session on SDLC took place for the involved personnel (software engineers and developers).

Comment 7

[Ben Wilson]

Closing this bug as complete.