Closed Bug 1864204 Opened 2 years ago Closed 1 year ago

Buypass: TLS certificates with incorrect Subject attribute order

Categories

(CA Program :: CA Certificate Compliance, task)

Product:
CA Program
Component:
CA Certificate Compliance
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: mads.henriksveen, Assigned: mads.henriksveen)

Details

(Whiteboard: [ca-compliance] [ov-misissuance] [ev-misissuance])

Attachments

(1 file)

Affected certificates
46.75 KB, text/plain
Details
No description provided.

Incident Report

This is a preliminary report.

Summary

Buypass has issued a number of TLS certficates since September 15th 2023 with an incorrect relative order of Subject attributes as defined in BR section 7.1.4.2.

Buypass stopped issuance immediately after discovery. The error has been fixed and certificates issued after this fix have a correct relative order of attributes.

This incident is still under investigation and a full incident report will be provided no later than Tuesday November 14th.

Assignee: nobody → mads.henriksveen
Status: UNCONFIRMED → ASSIGNED
Type: defect → task
Ever confirmed: true
Whiteboard: [ca-compliance] [ov-misissuance]
Attached file Affected certificates —
## Incident Report This is the incident report. ### Summary Buypass has issued 591 TLS certficates since September 15 2023 with an incorrect relative order of Subject attributes as defined in BR section 7.1.4.2. ### Impact After September 15 2023, Buypass have issued 479 OV, 101 EV and 11 QWAC certificates with incorrect relative order of Subject attributes. After discovering the incorrect Subject attribute order in a self-audit of issued certificates on November 10 2023, Buypass stopped issuance of OV, EV and QWAC certificates immediately. The Subject attribute order was then corrected and issuance restarted. Buypass will revoke all affected certificates. ### Timeline All times are CET (UTC+1) 2023-04-11: - BR 2.0.0 was published with changes in certificate profiles effective from September 15 2023 2023-07/08: - Buypass did a thorough analysis of the changes to identify changes to be made in our systems and in the configuration of Certificate Profiles 2023-09-13: - Certificate Profiles for OV, EV and QWAC certificates were updated in the CA-system to be compliant with BR 2.0.0 - Certificates issued were verified against certificate profiles using Windows Certificate Viewer 2023-09-15: - The certificate profile changes in BR 2.0.0 came into effect 2023-11-10: - 09:25 We discovered that the Subject attributes where reversed in the certificate file compared to the Certificate Profile and the order displayed in Windows Certificate Viewer - 10:09 We stopped issuance of OV, EV and QWAC certificates - 10:30 Subject attributes in affected Certificate Profiles were reversed and verified to be correct using an ASN.1 Certificate Viewer - 10:40 We restarted certificate issuance - 12:30 We started investigation to identify affected certificates and Subscribers - 13:00 We informed our Conformity Assessment Body about the incident - 17:21 A preliminary incident report was registered in Bugzilla 2023-11-11: - All affected certificates and Subscribers were identified, and we started notifying Subscribers 2023-11-13: - All affected Subscribers were informed about the incident and instructed to apply for replacement certificates as the affected certificates must be revoked ### Root Cause Analysis Buypass have used inhouse developed CA-systems for many years and always configured certificate content by defining the structure in a Certificate Profile. The Subject attributes in the Certificate Profiles were configured according to BR v2.0.0. We verified that the order of the Subject attributes in the certificate was the same as in the Certificate Profiles. This verification was done by using Windows Certificate Viewer. Our certificate issuance system reversed the order of Subject attributes from the Certificate Profile before writing them to the certificate file. Windows Certificate Viewer also reverses the order of the Subject attributes after reading the certificate file before presentation. In combination, we were confident that we had proper control of the certificate configuration. The linters used do not focus on the order of Subject attributes as defined in BR 2.0.0, and did not report any issue. ### Lessons Learned #### What went well * We had a strong focus on the changes in certificate profiles effective September 15 * All changes were implemented before September 15 * Immediate actions were taken when the error was identified, the error was fixed fast and effective #### What didn't go well * The parsing of Subject attributes from the configured Certificate Profile was reversed before being written to the certificate file * Using a Certificate Viewer who reversed the Subject attributes before presenting for verification * The combination of reversing Subject attribute order both a) during parsing of Certificate Profile and b) during presentation was misinterpreted as the changes having expected effect The verification should have been done using an ASN.1 certificate viewer closer to the actual certificate file * The linters currently used by Buypass did not check the order of the Subject attributes * A strong focus on identifying system changes required to comply with BR 2.0.0 made us underestimate the task of configuring the correct Subject attribute order in our CA-systems #### Where we got lucky * The error was identified by Buypass when performing the quarterly self-audit for issued TLS certificates in Q3 ### Action Items | Action Item | Kind | Due Date | | ----------- | ---- | -------- | | Improve the change process for such configuration changes, i.e. to include a verification at file level | Prevent | 2023-12-01 | | Include digicert/pkilint in the certificate issuance process | Prevent | 2024-01-15 | ### Appendix #### Details of affected certificates

Incident Report

This is the incident report.

Summary

Buypass has issued 591 TLS certficates since September 15 2023 with an incorrect relative order of Subject attributes as defined in BR section 7.1.4.2.

Impact

After September 15 2023, Buypass have issued 479 OV, 101 EV and 11 QWAC certificates with incorrect relative order of Subject attributes.

After discovering the incorrect Subject attribute order in a self-audit of issued certificates on November 10 2023, Buypass stopped issuance of OV, EV and QWAC certificates immediately.

The Subject attribute order was then corrected and issuance restarted.

Buypass will revoke all affected certificates.

Timeline

All times are CET (UTC+1)

2023-04-11:

  • BR 2.0.0 was published with changes in certificate profiles effective from September 15 2023

2023-07/08:

  • Buypass did a thorough analysis of the changes to identify changes to be made in our systems and in the configuration of Certificate Profiles

2023-09-13:

  • Certificate Profiles for OV, EV and QWAC certificates were updated in the CA-system to be compliant with BR 2.0.0
  • Certificates issued were verified against certificate profiles using Windows Certificate Viewer

2023-09-15:

  • The certificate profile changes in BR 2.0.0 came into effect

2023-11-10:

  • 09:25 We discovered that the Subject attributes where reversed in the certificate file compared to the Certificate Profile and the order displayed in Windows Certificate Viewer

  • 10:09 We stopped issuance of OV, EV and QWAC certificates

  • 10:30 Subject attributes in affected Certificate Profiles were reversed and verified to be correct using an ASN.1 Certificate Viewer

  • 10:40 We restarted certificate issuance

  • 12:30 We started investigation to identify affected certificates and Subscribers

  • 13:00 We informed our Conformity Assessment Body about the incident

  • 17:21 A preliminary incident report was registered in Bugzilla

2023-11-11:

  • All affected certificates and Subscribers were identified, and we started notifying Subscribers

2023-11-13:

  • All affected Subscribers were informed about the incident and instructed to apply for replacement certificates as the affected certificates must be revoked

Root Cause Analysis

Buypass have used inhouse developed CA-systems for many years and always configured certificate content by defining the structure in a Certificate Profile.

The Subject attributes in the Certificate Profiles were configured according to BR v2.0.0. We verified that the order of the Subject attributes in the certificate was the same as in the Certificate Profiles. This verification was done by using Windows Certificate Viewer.

Our certificate issuance system reversed the order of Subject attributes from the Certificate Profile before writing them to the certificate file. Windows Certificate Viewer also reverses the order of the Subject attributes after reading the certificate file before presentation.

In combination, we were confident that we had proper control of the certificate configuration.

The linters used do not focus on the order of Subject attributes as defined in BR 2.0.0, and did not report any issue.

Lessons Learned

What went well

  • We had a strong focus on the changes in certificate profiles effective September 15
  • All changes were implemented before September 15
  • Immediate actions were taken when the error was identified, the error was fixed fast and effective

What didn't go well

  • The parsing of Subject attributes from the configured Certificate Profile was reversed before being written to the certificate file
  • Using a Certificate Viewer who reversed the Subject attributes before presenting for verification
  • The combination of reversing Subject attribute order both a) during parsing of Certificate Profile and b) during presentation was misinterpreted as the changes having expected effect
    The verification should have been done using an ASN.1 certificate viewer closer to the actual certificate file
  • The linters currently used by Buypass did not check the order of the Subject attributes
  • A strong focus on identifying system changes required to comply with BR 2.0.0 made us underestimate the task of configuring the correct Subject attribute order in our CA-systems

Where we got lucky

  • The error was identified by Buypass when performing the quarterly self-audit for issued TLS certificates in Q3

Action Items

Action Item Kind Due Date
Improve the change process for such configuration changes, i.e. to include a verification at file level Prevent 2023-12-01
Include digicert/pkilint in the certificate issuance process Prevent 2024-01-15

Appendix

Details of affected certificates

Thanks for the report! You mention that including digicert/pkilint in the issuance process is one of the action items you intend to take. What linter(s) are you currently using (zlint? certlint?), and do you also intend to contribute changes to those linter(s) so that they would catch this same error?

Whiteboard: [ca-compliance] [ov-misissuance] → [ca-compliance] [ov-misissuance] [ev-misissuance]

Buypass use zlint, x509lint and certlint as provided by Sectigo through their linttbscert-service. We currently give priority to include pkilint in our certificate issuance systems and have not yet considered any contribution to update other linters.

We were not able to revoke all affected certificates within 5 days - see https://bugzilla.mozilla.org/show_bug.cgi?id=1865368.

Improve the change process for such configuration changes, i.e. to include a verification at file level was completed by December 1st 2023.

We are aware that this bug should have been updated earlier.

We are delayed with including digicert/pkilint in the certificate issuance process. This is now scheduled to be completed by 2024-02-01.

We have no new information in this bug.

We have included digicert/pkilint in the certificate issuance process today - slightly delayed according to the updated plan.

Here is an update on the Action items:

Action Items

Action Item Kind Due Date
Improve the change process for such configuration changes, i.e. to include a verification at file level Prevent DONE
Include digicert/pkilint in the certificate issuance process Prevent DONE

We have no new information in this bug.

We have no new information in this bug.

We have no new information in this bug.

We kindly request the NextUpdate field be set to 2024-04-03 - the same as for https://bugzilla.mozilla.org/show_bug.cgi?id=1865368 as these bugs are related.

Whiteboard: [ca-compliance] [ov-misissuance] [ev-misissuance] → [ca-compliance] [ov-misissuance] [ev-misissuance] Next update 2024-04-03
Whiteboard: [ca-compliance] [ov-misissuance] [ev-misissuance] Next update 2024-04-03 → [ca-compliance] [ov-misissuance] [ev-misissuance] Next update 2024-05-06

We have no new information in this bug.

We have no more open action items and this bug is related to bug https://bugzilla.mozilla.org/show_bug.cgi?id=1865368 that was closed last week.

If there are no further comments on questions on this bug, I suggest we close it.

We have no new information in this bug.

If there are no further comments or questions on this bug, I suggest we close it.

I'll close this on or about Friday, 10-May-2024.

Flags: needinfo?(bwilson)
Whiteboard: [ca-compliance] [ov-misissuance] [ev-misissuance] Next update 2024-05-06 → [ca-compliance] [ov-misissuance] [ev-misissuance]
Status: ASSIGNED → RESOLVED
Closed: 1 year ago
Flags: needinfo?(bwilson)
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: