1632632 - Buypass: Illegal Business Category in a PSD2 QWAC

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Mads Henriksveen]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36

Comment 1

[Mads Henriksveen]

This is an incident report for one PSD2 QWAC issued by Buypass with an illegal value in the Subject Business Category. The actual value was set to ‘UN’ while it should have been ‘Private Organization’.

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

Buypass became aware of this by verifying the certificate content immediately after issuance.

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

2020-04-21, 16:54 (CEST): The PSD2 QWAC was issued.

The certificate content was verified immediately after issuance and the illegal value was identified.

2020-04-21, 17:34: The certificate was revoked and replaced with a correct certificate.

We stopped issuance of all types of certificates with Business Category (i.e. EV, QWAC and PSD2 QWAC) and started investigating what the cause of the problem was.

2021-04-22, 09:30: We had defined the relevant corrective action including a bug fix in one of our applications.

2021-04-23, 11:29: The bug fix was deployed into production

3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

Buypass stopped issuance of PSD2 QWACs and other types of certificates which includes Business Category immediately when becoming aware of the issue.

4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

We investigated all issued certificates and no other certificate with the same error have been issued.

5. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.spreadsheet, with one list per distinct problem.

The affected certificate is: https://crt.sh/?id=2708443813

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

The root cause for this error was a recent change in our registration systems. We do perform an automatic analysis of different types of organizations in order to categorize them as specific Business Categories according to EVG. In some cases, it is not possible to identify the real category and historically the Business Category has been left empty. The mentioned change was that the category was set to ‘UN’ (meaning unknown) in this case.

The Business Category is only used in high assurance certificates like EV, QWACs and PSD2 QWACs and validation specialists always validate these certificates. In case the Business Category has not been set or set to ‘UN’, the validation specialist will set this to its proper value (from a drop down list comprising only allowed values). So the actual value will be set manually, if it has not been set automatically.

We do also have controls in the issuance system to verify data to be embedded in a certificate, but for the Business Category, the control has only verified that the Business Category was not empty (e.g. based on our historical handling of Business Category). In addition, the actual certificate is always verified by linting before issuance.

In this case, the value of Business Category was still set to ‘UN’ at time of issuance, and our pre-issuance controls failed to identify this (as it was not empty). The linting service we use also did not identify this as an issue, so the certificate was issued.

7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.
   We have added a more strict check in our pre-issuance control for the Business Category value, only values as specified in EVG are accepted.
   The fix for this has already been implemented and deployed to production.

Comment 2

[Ben Wilson]

From your explanation, I understand that the system will now prevent a certificate with a business category of "UN" to proceed to issuance.
Is that a correct understanding?
Are there any other questions or comments about this incident from anyone else?

Comment 3

[Mads Henriksveen]

Yes, that's correct. We will only accept values specified in EVG for the Business Category.

Comment 4

[Ryan Sleevi]

We do also have controls in the issuance system to verify data to be embedded in a certificate, but for the Business Category, the control has only verified that the Business Category was not empty (e.g. based on our historical handling of Business Category). In addition, the actual certificate is always verified by linting before issuance.

In this case, the value of Business Category was still set to ‘UN’ at time of issuance, and our pre-issuance controls failed to identify this (as it was not empty). The linting service we use also did not identify this as an issue, so the certificate was issued.

Mads: Can you speak to any sort of systemic evaluation you're doing?

Looking at businessCategory, for example.

• Version 1 of the EV Guidelines stated an allowlist of the following values:
  • V1.0, Clause 5.(b)
  • V1.0, Clause 5.(c)
  • V1.0, Clause 5.(d)
• Version 1.1 modified that to add
  • V1.0, Clause 5.(e)
• Version 1.3 modified that to be
  • Private Organization
  • Government Entity
  • Business Entity
  • Non-Commercial Entity

from which it's been since.

It's unclear why a restriction wasn't implemented, given that very clear requirement, to make sure those values are the only values. Have you, for example, began a process of evaluating the EV Guidelines and analyzing each requirement, line by line, and ensured there's either a technical control or robust procedural control for those requirements? Is this the only requirement for which controls didn't exist?

Similarly, in calling out linters, are you using an in-house linter or an open-source linter? If an open-source linter, do you plan to contribute back that linter? If you're using ZLint, I filed https://github.com/zmap/zlint/issues/438 to track this

Comment 5

[Mads Henriksveen]

We have had a systematic evaluation of the relevant requirements, including EV Guidelines (EVG), several times since we started issuing EV certificates. However, the responsibility and implementation of the controls have been distributed among several system components and performed at different times during registration, validation and issuance processes. This has made some of these controls less efficient, including the controls of values set for Business Category. One of the actions we took due to this incident were therefore to change the control at time of issuance to also verify that only allowed values as defined in EVG was used – in addition to the control we already had at time of registration.

When we introduced linting as a pre-issuance control in our issuance systems, we found this to be an efficient way of controlling the quality of information to be included in the certificates. Although we have worked to improve other controls as well, we realize that we may have relied too much on the linters and failed to improve our other controls to a satisfactory level.

We have also identified other situations and possible mis-issuance cases that fails to be identified by linters, e.g. correct formatting of postal codes for different countries. We have already identified that a new systematic evaluation of requirements needs to be done.

We use lint services provided by Sectigo made available as a part of their excellent crt.sh. We have identified the need for some changes in some of the linters, but we have only reported this to Sectigo’s representatives and not filed any issue to the relevant lint project. The issues we have identified so far is related to the use of Organization Identifiers in the EV certs (both for Subject and the cabf extension) as both attributes gives a Warning from the linters, i.e. ‘WARNING: Name has unknown attribute 2.5.4.97’ and ‘WARNING: Unknown Extension: 2.23.140.3.1’.

Comment 6

[Ben Wilson]

I have no further questions on this incident and intend to close it on or after 10-Jul-2020 if no other questions or comments are raised.

Comment 7

[Ryan Sleevi]

I don't really see any meaningful fixes for the underlying systemic issues highlighted in Comment #5. Linters are a last line of defense, but it sounds like the complexity at Buypass due to "responsibility and implementation of the controls [being] distributed among several system components and performed at different times during registration, validation, and issuance processes" will continue to result in compliance failures, particularly if those aren't covered by linters, which Buypass doesn't seem to be maintaining.

I don't think answers like "We have already identified that a new systematic evaluation of requirements needs to be done." raises to that level, because it doesn't provide any timeline for when that will be completed, or how it's being performed.

I think, similar to the comments I recently made to Entrust, it would benefit from thinking about this issue as an opportunity to implement more meaningful controls. It's a bit like flipping a coin, and saying if it's all heads, the certificate was misissued, or if any of the flips was tails, it's fine. Flipping the coin only once means you've got a 50/50 chance it will be misissued. The more times you flip the coin, the lower the probability that every flip will be heads, and you lower the risk of misissuance.

Systemic controls are adding extra flips to the coin, and so that's why they're so important. These incident reports are about looking at every chance you had to flip a coin, but decided not to, and understanding what you'll be doing going forward. I realize the analogy is stretched, but I think it's important to understand what opportunities existed to prevent this, and how they're being addressed, and a timeline for addressing them or providing updates.

Comment 8

[Mads Henriksveen]

We have started an evaluation of identified issues and we are discussing internally how to improve our controls. The focus is on how to take down complexity and reduce the risk for further compliance failures. This will affect the way the controls are implemented, including the use of linters. We consider a systematic evaluation of (relevant) requirements as an important element of this work to ensure that controls are meaningful and properly implemented. We will continue this work and will provide an update with a timeline within the next weeks.

Comment 9

[Mads Henriksveen]

We are in process, but due to key personnel on summer holidays we have not been able to complete this work. We will give an update with more details and a timeline by August 25th at the latest.

Comment 10

[Mads Henriksveen]

We have had a systematic walkthrough of our systems and identified all relevant controls and their corresponding responsibilities. We have already identified that some of the existing controls need improvement and that we need to gather distributed controls to reduce the complexity within our systems. Another area for improvement is our use of linters, but we have not yet decided the final approach (e.g. whether to take them inhouse or still use them as a service). We have already started some development activities to implement these identified improvements.

The next step will be a systematic evaluation of all relevant requirements and assign responsibilities for each requirement to specific controls, this may result in new controls in some areas. We have planned for this evaluation to be completed no later than end of September.

We already foresee that we need to spend time on improving and implementing controls after the evaluation and we have given priority to such activities in the next development periods. We plan for new development activities quarterly and the management team has accepted that we spend time on these activities in Q4 2020 and Q1 2021.

We must wait until the systematic evaluation has been performed to be able to give a proper estimate for the development activities required. However, based on our current overview of controls with responsibilities and the identified needs for improvements, we consider it realistic to implement all necessary changes within the two next development periods.

Comment 11

[Mads Henriksveen]

We are continuing the development activities to implement identified improvements.

Comment 12

[Mads Henriksveen]

We have finished the systematic evaluation of all relevant requirements today, this has been a very useful exercise. We have organized this in two workshops with representatives from our product department, the validation team and the development team.

We have focused on the certificate content and how to ensure that any content is properly validated and correctly formatted according to the requirements. During this exercise we have assigned responsibilities for each certificate field to defined components in our system. The controls to be implemented has been defined, but we need some time to summarize the work to be done and to be able to give a brief estimate for the implementation. We will give an update early next week.

Comment 13

[Mads Henriksveen]

Our systems for registration, validation and issuance are composed of several microservices with defined responsibilities. However, these microservices have been updated and changed over years and as a result we have ended up with rather complex systems with not so effective controls distributed among these services.

We decided to focus on the issuance of certificates and collect all controls related to the content of certificate data in existing microservices used for all types of tls certificates, this includes a microservice responsible for domain validation and a microservice responsible for analyzing and approving CSRs including keys and algorithms. All controls related to domains and domain validations are to be collected and managed in the domain validation microservice, while all controls related to keys and algorithms are to be collected and managed by the CSR validation microservice. These controls are to be organized in (controller) components within the microservice responsible for a specific subdomain/area where the controls belong.

At time of issuance all data to be included in the certificate must be complete and properly validated. We intend to add new controls both in existing and new (controller) components. We have identified a certificate pre-issuance data (controller) component which shall ensure that all data used as input to the (TBS) certificate in our certificate generation microservice are properly controlled before creating the TBS certificate. Some content of the certificate is defined by preconfigured certificate templates and other configurations set by the CA team. We have therefore defined a separate (controller) component with a responsibility to ensure that all configurations used in the certificate generation microservice are correct. Finally, we have also identified a TBS certificate (controller) component which shall ensure that the actual TBS certificate content is controlled before the certificate is signed. We will also include linters as (controller) subcomponents here and we have decided to implement the linters inhouse.

All these (controller) components were identified before our systematic evaluation of requirements.

For the systematic evaluation of requirements, we based this on the draft certificate profiles document used in CABF validation subcommittee to identify all fields and attributes to be included in the certificate. The fields and attributes were organized in three sections covering a) general fields and extensions, b) subject data (including SAN) and c) keys and algorithms.

We identified all relevant requirements from different sets of requirements (BR, EVGL, root certificate programs, ETSI-standards etc) and defined the controls for each specific field related to different types of certificates (DV, OV, EV etc). E.g. for the policyIdentifier field, the OIDs shall include an CABF OID according to BR. In addition, according to Buypass policy requirements we include our own OIDs. Both are included in the controls defined for this field.

The responsibility for each control defined for a field were assigned to one or more of the identified (controller) components described above. E.g. the controls for the policyIdentifier field were assigned to the certificate configuration (controller) component and the TBS certificate (controller) component.

The next step will be to implement all controls in all components. The main approach is to give priority to the inner controller components focusing on certificate content, i.e. the pre-issuance certificate data (controller) component, the certificate configuration (controller) component and the TBS certificate (controller) component. Thereafter, we will improve the other controller components, including adding any missing controls.

These changes will affect many of our microservices and we will organize the deliverables per microservice accordingly. We plan to implement all the high prioritized controls for the inner controller components during Q4 2020 and the improvement of existing components during Q1 2021. The actual deliverables will be estimated, scheduled and implemented according to our development process.

Comment 14

[Mads Henriksveen]

We are in progress of implementing the prioritized controls for the inner controller components, but the implementation is slightly delayed compared to the initial plan. According to our current estimates, we will not be able to finalize the controls during Q4 2020 as production ready deliverables. We will give an update early 2021.

Comment 15

[Mads Henriksveen]

We are still working with implementing the prioritized controls for the inner controller components and have decided to split them into several deliverables. The first deliverable is scheduled for next week and we will subsequently work continuously with the next deliverables. The development team is now working with this as their main priority and we plan to complete this activity during Q1, i.e. by the end of the current development period. We will give an update when the work is finalized or no later than end of Q1.

Comment 16

[Mads Henriksveen]

The work is ongoing with good progress. The development team has created a framework for implementation of the controls which includes test automation and documentation. This will ensure that the controls are effective and consistent across all components. We will give an update no later than end of Q1.

Comment 17

[Mads Henriksveen]

There is no new information to this bug.

Comment 18

[Ben Wilson]

Hi Mads,
Just a reminder that another update is due next week.
Thanks,
Ben

Comment 19

[Mads Henriksveen]

We have implemented approximately 50 percent of the planned controls. The process of implementing new controls based on the mentioned framework seems to be effective and gives a consistent implementation across the affected components.

In order to use the controls effectively, some changes has to be made in several of the existing systems. For some of our legacy systems, implementing such changes are more challenging and time consuming than expected so the implementation has been further delayed.

According to our current plan, we expect the work to be completed during the next development period (Q2 2021). We will give an update no later than mid-June.

Comment 20

[Mads Henriksveen]

The new framework and most of the implemented controls are now deployed to our production environment. A few controls are still not 100% effective as they depend on updates in legacy systems. The planned controls are implemented according to priority and some of the lowest prioritized controls are still to be implemented, however we consider these controls to be non-critical as they are covered by other mechanisms. These controls and further improvements will be implemented as standard maintenance activities for the development team. We consider the major part of this activity to be completed.

Comment 21

[Ben Wilson]

Mads,
Do you believe that this bug can now be closed? If not, could you outline and provide a schedule for the low-priority, non-critical controls that you still plan to implement?
Thanks,
Ben

Comment 22

[Mads Henriksveen]

We have completed the major part of changes within our systems as described in this bug, so I believe the bug can be closed now.

Comment 23

[Ben Wilson]

I will close this on or about Friday, 2-July-2021.