1712120 - Sectigo: Inappropriate subject:serialNumber information in EV certificates obtained through ACME

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Tim Callan]

Attachment: sectigo_ev_acme_crtsh_links.txt

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

Internal audit discovered EV TLS certificates containing dates of Incorporation or Registration in the subject:serialNumber field rather than Registration Numbers. For these organizations Registration Numbers were available.

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

May 5, 11:01 am Eastern.
Internal audit discovers Extended Validation TLS certificates containing dates of Incorporation or Registration in the subject:serialNumber field. These certificates are for organizations with available Registration Numbers. Internal audit escalates within the Validation team to confirm that this is a misissuance.

12:16 pm Eastern.
The root cause is determined, which is a coding bug in ACME orders for EV TLS certificates. In this case our software failed to obtain the Registration Number from the appropriate database and therefore wrote in the date of Incorporation or Registration instead. This code appears to have been performing incorrectly since December 2019, when we initially deployed our support for EV with ACME.

4:05 pm Eastern.
Access to our EV ACME server disabled to prevent further instances of this issuance error.

May 6, 10:22 am, Eastern.
Search query yields the attached list of 204 affected certificates.

May 7, 5:58 am, Eastern.
Fix deployed into production. Access to our EV ACME server reenabled.

May 10, 4:06 pm Eastern.
All affected certificates revoked.

3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

We ceased issuance from the affected part of our infrastructure less than four hours after understanding the behavior. We issued no new noncompliant certificates. The misbehavior is now rectified.

4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

204 active certificates

The earliest of these was issued August 27, 2020.

The last was issued May 3, 2021.

5. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

We have attached a list of crt.sh links for the 204 affected certificates.

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

This was a software coding error when we extended our ACME support to OV and EV certificates. Boiled down, because Comodo had maintained Company Registration Numbers in its records prior to the publication of the EV Guidelines, they were stored in a different way than other EV-specific fields such as Jurisdiction, Date of Incorporation/Registration, and Business Category, which Comodo put in place when it developed EV support.

Because the Company Registration Number was treated differently from other fields, our 2019 ACME release failed to pull across this record correctly, and since no Company Registration Number was present, our software added the Date of Incorporation or Registration instead. Fixing this bug involved moving the Company Registration Number into the same table as other EV-specific fields and then updating our code to pull this field as well.

Our QA process should have caught this bug and did not. This QA failure owes itself to human error in the testing process, which in December 2019 was still mostly manual.  It now appears that for this release we did not execute the QA test plan correctly.

Internal audit didn’t find it before now for the simple reason that usage of EV with ACME is small. With a total universe of 204 affected certificates, none of them was in our selection set before now.

7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

This specific bug is fixed, so this problem won’t recur.

As mentioned above, this bug should not have gotten past QA. In 2020 we added a great deal of test automation coverage, which includes full automation of our regression test suite.  We expect that progress on its own to go a long way in mitigating the risk of errors like this one.  Had that automation been in place at the time, QA would have caught this coding error.

Our 2021 roadmap includes full automated checking of the contents of all fields of all certificate types issued through all process flows on our platform. This check will confirm that field content matches expectations set on a field-by-field basis. For some fields that will require that contents match a specific list of acceptable strings (such as country names). For others it will involve the formatting or nature of the field contents (for example, the software might confirm that the contents of a certain field match the allowable character set). This functionality need not come in a single release, and we expect to develop and deploy these checks over the course of the year. Some, such as the countryName field check, are in production today.

Comment 1

[Tim Callan]

We attempted to provide a complete and descriptive report on this incident, including root causes, steps we took to mitigate the problem, and how we are confident this problem will not repeat itself. We also attempted to look not just at the immediate bug and its resolution but also at larger organizational issues that contributed to the bug in the first place.

This is a reminder to the community that we remain available to address any questions or comments that arise from this write up.

Comment 2

[Ryan Sleevi]

(In reply to Tim Callan from comment #0)

As mentioned above, this bug should not have gotten past QA. In 2020 we added a great deal of test automation coverage, which includes full automation of our regression test suite.  We expect that progress on its own to go a long way in mitigating the risk of errors like this one.  Had that automation been in place at the time, QA would have caught this coding error.

Can you share more specific details about how these tests are structured and how they would have caught this? For example, if you have a dozen tests that exercise this, it might be useful to provide "plain English" or pseudo-code descriptions of how/what they test, so that other CAs can consider and learn from Sectigo's approach here to automated testing.

A recurring theme that seems to turn up in CA incidents is the complexity of how data flows through systems, especially legacy systems. Indeed, some CAs ultimately were distrusted over their inability to bring order to this complexity, as demonstrated through continued compliance incidents. I understand Sectigo has taken steps to address this immediate issue, but I'm wondering whether there are further opportunities for systemic improvements, especially following the Comodo/Sectigo separation.

Comment 3

[Tim Callan]

(In reply to Ryan Sleevi from comment #2)

Can you share more specific details about how these tests are structured and how they would have caught this?

FYI, I am putting together a response to this question. QA automation isn't the kind of thing the compliance team is closely paying attention to as a rule, so it involves gathering information from other teams inside the company. I'm leaving the needinfo open until we post that response.

Comment 4

[Tim Callan]

As part of our response to bug #1712188 we have added a senior member of our QA team to the WebPKI Incident Response team. This move has several beneficial effects, including education inside the QA department about compliance needs and a closer marriage of QA activity to the issues driving compliance. Plus it has the side benefit that I now have a QA expert to help me with the request in comment #2.

We are still putting this together, but I feel better now about providing a complete, accurate, and digestible summary of the QA automation we've put in place in the past 18 months or so.

Comment 5

[Tim Callan]

(In reply to Ryan Sleevi from comment #2)
Our QA automation for certificate issuance (public or private) treats each type of certificate as an object with a set of defined attributes. The sum total of these attributes along with relevant input data at order time (by the way of the CSR) define the expected form of the output certificate. Our automation platform is able to order a certificate based on a defined set of values for these attributes and a set of relevant input data and then check the output certificate against expectations based on these inputs.

The available parameters depend on ordering method, such as our partner API, our partner UI portal, or ACME. They are similar but not necessarily identical across ordering methods. A good example set of the available parameters is covered on pages 3 through 11 of the documentation for our AutoApplyOrder API: https://secure.trust-provider.com/api/pdf/latest/AutoApplyOrder.pdf. Pages 16 and 17 detail the parameters required by this API to place an order for SSL certificates at each authentication level.

Input data also affects output expectations. Here are some examples:

• Term: The QA team can order a certificate for anything up to five years. For public-root SSL certificates the system should limit the available term to no longer than 396 days, 23:59:59. A certificate ordered for less time than that (for instance, one year or 90 days) would be expected to match the ordered term.
• companyNumber: If provided among the data inputs, this becomes the default value for the certificate subject:SerialNumber. If no companyNumber is provided, the subject:SerialNumber field should present the date of incorporation. Our automation checks that the appropriate value correctly populates the subject:SerialNumber field. (This check would have caught the error that initiated this bug.)
• Product ID: This is a numerical value that indicates the specific certificate product. For public SSL certs, this includes brand name, authentication level, and domain scope. For example, PositiveSSL single-domain DV and PositiveSSL wildcard DV have distinct product IDs. You can find a list of product IDs on pages 3 to 4 of our AutoApplyOrder API documentation. Our system has defined expectations for the mandatory validation steps, allowed methods, and issuing CA for each product ID.
• stateOrProvince: Our system checks that the value in this field matches our list of acceptable values and that the value is acceptable for the selected countryName. We confirm that in the event of failure to match these expectations, the order will not be accepted and issuance cannot occur.

We consider our issuance workflow in four stages:

1. Order creation. The automated system places an order from one of our available interfaces. We place this order with a set of input parameters and input data as described above and call our issuance system. The expected response is no error codes from the issuance system.
2. Processing of the order. In this stage we check that all appropriate validation steps occur correctly. The steps tested depend on the product type. DV, OV, and EV SSL certificates require a distinct set of steps for each. The expected response to a positive case is “Ready to issue” status for the certificate in the system. We also test negative cases. For example, an order with incomplete DCV should fail this step.
3. Issuance and collection. As soon as all required validation is complete, we trigger issuance, confirm that issuance has occurred, and collect the certificate. The expected result is “Issued” status and successful certificate collection.
4. Certificate content validation. This step checks the content of the collected certificate. We look at the initial parameters used while placing the order to determine a set of expected results in certificate morphology and content.

Comment 6

[Tim Callan]

(In reply to Tim Callan from comment #5)
I hope this general overview is sufficiently detailed to be useful to the community and answer the question. Communication in this forum often requires making a judgement about the right level of detail to provide so that it useful without driving ourselves crazy with details. We're available to respond to questions or comments about what you read above.

Comment 7

[Tim Callan]

We're available for questions or comments on any of the above.

Comment 8

[Ryan Sleevi]

(In reply to Tim Callan from comment #5)

• companyNumber: If provided among the data inputs, this becomes the default value for the certificate subject:SerialNumber. If no companyNumber is provided, the subject:SerialNumber field should present the date of incorporation. Our automation checks that the appropriate value correctly populates the subject:SerialNumber field. (This check would have caught the error that initiated this bug.)

I guess I'm still struggling to better understand how this would have caught things.

Comment #0 stated:

In this case our software failed to obtain the Registration Number from the appropriate database and therefore wrote in the date of Incorporation or Registration instead.

Your description here, of companyNumber, seems to match your description of the root cause. Perhaps its my confusion around what you mean by "the appropriate value". A unit test of the API seems like it would miss this - because the companyNumber would be provided - so perhaps I'm misunderstanding how your tests are actually structured.

We consider our issuance workflow in four stages:

1. Order creation. ....
2. Processing of the order. ...
3. Issuance and collection. ...
4. Certificate content validation. ...

It's unclear to me which of these steps should have failed. It sounds like the issue was in your database storage of validated fields, which if I understand from your description, happens before the "order creation" of the test.

That is, what I'm missing from this is that it appears you've described unit tests for each component, along with an integration test (in step 4) of the API. But it sounds like the issue was in wrong (but valid looking) inputs were provided to the API in 1, which would pass the integration test in 4. So I'm hoping you can provide a bit more description about how this would have been caught.

As someone who has written more than his fair share of tests that "work for me, but miss the bug", I'm trying to better understand how this class of issues can be missed. Figuring out where in the pipeline inputs are validated is important, and for me, the biggest ambiguity is trying to figure out where 1 is describing an "internal API" or an "external API", and where the data validation is performed.

Comment 9

[Ryan Sleevi]

Sectigo replied in Bug 1715929, Comment #9 addressing the questions I mentioned in Comment #8 (just posting on the wrong bug)

Nikola: If you're able to answer for Sectigo, can you share more details about the following?

• In Comment #0, Sectigo stated:

  In 2020 we added a great deal of test automation coverage, which includes full automation of our regression test suite. 

  But this isn't part of his timeline (Question #2 on the incident report). Can you share precisely when this happened?

• In Comment #0, Sectigo stated:

  Had that automation been in place at the time, QA would have caught this coding error.

  However, that same comment stated:

  The last was issued May 3, 2021.

  Could you help me square this away? Should the checks introduced in 2020 have caught this or not? And if not, what's the comment referring to?

• In Comment #0, Sectigo stated:

  Our 2021 roadmap includes full automated checking of the contents of all fields of all certificate types issued through all process flows on our platform.

  What's the concrete timeline for that roadmap?

• In Bug 1715929, Comment #9, Sectigo described three different tests, including the test that would/should have caught this. When were these tests introduced?

Comment 10

[Nikola Maksimovic]

Ryan: my apologies for posting in a wrong bug, beginners mistake :)
I will post a more detailed answer next week.
Have a great weekend!
Nikola

Comment 11

[Ryan Sleevi]

Resetting N-I for next week's update.

Comment 12

[Nikola Maksimovic]

(In reply to Ryan Sleevi from comment #9)

Sectigo replied in Bug 1715929, Comment #9 addressing the questions I mentioned in Comment #8 (just posting on the wrong bug)

Nikola: If you're able to answer for Sectigo, can you share more details about the following?

Before I get into that, let me take a few paragraphs to explain why you’re seeing me on Bugzilla all of the sudden. I’ve become directly involved in this dialog because it’s getting into matters that are the responsibility of the department I manage, and I don’t want the “telephone game” to create accidentally misleading statements. Tim wrote in bug 1712120 comment 3 that,

QA automation isn't the kind of thing the compliance team is closely paying attention to as a rule, so it involves gathering information from other teams inside the company.

This information-gathering is where that risk comes in. Let me give you a couple examples of that telephone game at work.

When he was preparing comment 0, here’s what I sent to Tim in email:

Since December 2019 we have shored up our QA effort by significantly increasing test automation coverage and including all automated tests into our regression test suite.

Here is what appeared in the bug:

In 2020 we added a great deal of test automation coverage, which includes full automation of our regression test suite.

At a quick glance these statements might look equivalent, but they are not. Though I reviewed Tim’s post before it went up, I didn’t notice this difference myself at the time. Obviously we don’t want to be making statements that aren’t completely accurate. Mea culpa - more serious than posting in the wrong bug - and an important lesson learned.

A second example is Tim’s and my differing use of the word “roadmap.” To him, “roadmap” means “Here’s the stuff we intend to do and when we intend to do it.” To me, “roadmap” means we have a specified set of functionality that has been defined and sized with assigned resources and target dates. There are different things.

So I have injected myself into this conversation in hopes of keeping the discussion precise.

• In Comment #0, Sectigo stated:

  In 2020 we added a great deal of test automation coverage, which includes full automation of our regression test suite. 

  But this isn't part of his timeline (Question #2 on the incident report). Can you share precisely when this happened?

As explained above: as we increase automation coverage, we include automated tests into our regression suite. We should not have said that all tests in our regression suite were automated. We should have said that all automated tests are available as part of our regression suite. Some tests used for regression have not yet been automated.

In May 2021 we conducted an internal audit resulting in bug 1712120, after which we added the 3 conditions described in bug 1715929 comment 9. These conditions have been in the regression suite since May 11, 2021. The fact is that those conditions were not part of our manual test suite either, as their first appearance in our test version control happened on that date only.

Additionally, we have not had automated tests for ordering certificates through ACME until June 29, 2021. ACME testing had been manual before that date. Since that date the ACME regression suite is part of automated regression.

• In Comment #0, Sectigo stated:

  Had that automation been in place at the time, QA would have caught this coding error.

  However, that same comment stated:

  The last was issued May 3, 2021.

  Could you help me square this away? Should the checks introduced in 2020 have caught this or not? And if not, what's the comment referring to?

Our original statement turns out to have been incorrect. The specific problem would have been caught through automation only after both automation improvements mentioned above were implemented, which is June 29th, 2021. Furthermore, since the 3 conditions discussed earlier were implemented only on May 11, 2021, we would have not caught the issue before that date (either through automation or manual testing).

• In Comment #0, Sectigo stated:

  Our 2021 roadmap includes full automated checking of the contents of all fields of all certificate types issued through all process flows on our platform.

  What's the concrete timeline for that roadmap?

These improvements are part of Guard Rails initiative. We expect to have 15 – 20 releases for the remainder of the year using them to incrementally close the gaps/improve the checks. We have a clear direction in mind but haven’t yet defined specifically what we will be delivering more than 2-3 releases ahead. Please see bug 1715929 comment 12 for more on this point.

• In Bug 1715929, Comment #9, Sectigo described three different tests, including the test that would/should have caught this. When were these tests introduced?

As explained above they were first introduced on May 11, 2021

Comment 13

[Ryan Sleevi]

Thanks Nikola.

I appreciate the focus on keeping things precise, as we now try to revisit the original incident report in Comment #0, both to make sure we (users and other CAs) understand where things went wrong, and what can be done (systemically) to address this. This is less about assigning blame, so much as working through the process to make sure we're on the same page, as well as that Sectigo has a meaningful path to improvement.

It sounds then, that we have a situation where:

• Prior to 2020, testing was primarily manual (Comment #12, Comment #0)
  • In 2019-12, Sectigo added all of their (existing) automated tests to be part of their regression tests (Comment #12, Comment #0)
• On 2021-05-01, Sectigo identified additional conditions that need testing (Bug 1715929, Comment #9)
  • Date of Incorporation/Registration Provided, Company Registration Number Provided
    • Expected Output: serialNumber == company number
  • Date of Incorporation/Registration Provided, Company Registration Number Absent
    • Expected output: serialNumber == date of incorporation/registration
  • Date of Incorporation/Registration Absent, Company Registration Number Absent
    • Expected output: Failed issuance
• On 2021-06-29, Sectigo added automated tests for ACME-based API issuance (Comment #12)

In terms of thinking root causes:

• A bug was introduced, due to legacy design of Sectigo's database
• The QA process did not have explicit tests to cover this scenario, although it's possible a QA engineer may perform such tests on their own
• There were no automated tests in place for ACME issuance

And in terms of remediation for those root causes:

Date	Reference	Remediation
2021-05-01	Comment #12, Bug 1715929, Comment #9	New automated tests for (some of) these cases
2021-06-29	Comment #12	ACME endpoints now are covered by automated testing
2021 EOY/TBD	Comment #0	Top-to-bottom analysis of testing gaps ("Guard Rails")

While I'm not sure if this is entirely a correct understanding, a few questions stand out:

• The description of ACME endpoint testing can be read several ways.
  • You now perform all automated tests for issuance against the ACME endpoint, in additional to your traditional API flows (e.g. web page, customer-specific APIs, etc)
  • You now test the issuance of certificates with ACME (but not performing positive/negative tests)
• For the two fields (Date of Incorporation/Registration and Company Number), there are a total of four permutations of (present/absent) combinations. Yet only three tests were described. Why isn't the fourth scenario relevant?
• Bug 1715929, Comment #12 highlight challenges that Sectigo perceives to transparency; mostly, y'all are working through these issues, and new requirements or edge cases continue to crop up, making it difficult to provide clear and binding timelines to improvements, since you don't apriori know what the solutions will be, or if they'll work (hopefully, this all reads charitably; as a SWE, I totally understand this challenge). Yet what we look for, with a CA, are ways that we can be confident the CA understands the risks, and ways to measure and understand how a CA resources and prioritizes improvements. For example, a CA that takes 6 months to fix an issue bears a much more thorough explanation than a CA that is able to fix (the same issue) overnight, but it's also useful to understand if that overnight fix runs the risk of causing more harm than good, or failing to actually address the issue vs paper over it (e.g. blocklisting " " from an OU field, versus stopping issuing OUs). Given that Sectigo is constantly leaning on "Guard Rails will solve this", is there a path Sectigo has to propose to provide insight into "Here's what we're prioritizing, here's the overall list of things we see as important (which we will continue to add to), and here's our next expected update"?
  • I want to find a path here to get the transparency, find a path to resolving these bugs, while also finding a way Sectigo has to provide transparency that this class of major misissuance events has been firmly and completely resolved.

While I'd normally want to understand how these issues (like no automated testing in your regression tests, or no automated ACME tests) went on so long, I recognize that Sectigo's efforts on guard rails at least demonstrates they recognize that this is a serious lapse to be corrected ASAP.

Comment 14

[Nikola Maksimovic]

(In reply to Ryan Sleevi from comment #13)

Thanks Nikola.

I appreciate the focus on keeping things precise, as we now try to revisit the original incident report in Comment #0, both to make sure we (users and other CAs) understand where things went wrong, and what can be done (systemically) to address this. This is less about assigning blame, so much as working through the process to make sure we're on the same page, as well as that Sectigo has a meaningful path to improvement.

It sounds then, that we have a situation where:

• Prior to 2020, testing was primarily manual (Comment #12, Comment #0)
  • In 2019-12, Sectigo added all of their (existing) automated tests to be part of their regression tests (Comment #12, Comment #0)
• On 2021-05-01, Sectigo identified additional conditions that need testing (Bug 1715929, Comment #9)

It was May 11 as stated in bug 1712120 comment 12.

• Date of Incorporation/Registration Provided, Company Registration Number Provided
  • Expected Output: serialNumber == company number
• Date of Incorporation/Registration Provided, Company Registration Number Absent
  • Expected output: serialNumber == date of incorporation/registration
• Date of Incorporation/Registration Absent, Company Registration Number Absent
  • Expected output: Failed issuance
• On 2021-06-29, Sectigo added automated tests for ACME-based API issuance (Comment #12)

In terms of thinking root causes:

• A bug was introduced, due to legacy design of Sectigo's database
• The QA process did not have explicit tests to cover this scenario, although it's possible a QA engineer may perform such tests on their own
• There were no automated tests in place for ACME issuance

And in terms of remediation for those root causes:

Date	Reference	Remediation
2021-05-01	Comment #12, Bug 1715929, Comment #9	New automated tests for (some of) these cases

May 11

| 2021-06-29 | Comment #12 | ACME endpoints now are covered by automated testing |
| 2021 EOY/TBD | Comment #0 | Top-to-bottom analysis of testing gaps ("Guard Rails") |

While I'm not sure if this is entirely a correct understanding, a few questions stand out:

We do have partial negative test coverage for ACME and comprehensive negative test coverage for other endpoints. We will continue updating our ACME coverage over time.

• The description of ACME endpoint testing can be read several ways.
  • You now perform all automated tests for issuance against the ACME endpoint, in additional to your traditional API flows (e.g. web page, customer-specific APIs, etc)

We have not yet reached the point of having the same issuance test coverage for ACME compared to other endpoints. We plan to have matched the coverage for other endpoints by the end of September 2021.

• You now test the issuance of certificates with ACME (but not performing positive/negative tests)

Currently we do test certificate issuance with ACME and we do have limited negative test coverage as explained above.

• For the two fields (Date of Incorporation/Registration and Company Number), there are a total of four permutations of (present/absent) combinations. Yet only three tests were described. Why isn't the fourth scenario relevant?

We didn’t mean for those examples to be exhaustive. For ACME we have 30 permutations of this test with different configurations for variable aspects including organization type and the contents of certificate fields.

The test you mention is part of that suite. We didn’t cover that case as our intention for the examples was to be illustrative.

• Bug 1715929, Comment #12 highlight challenges that Sectigo perceives to transparency; mostly, y'all are working through these issues, and new requirements or edge cases continue to crop up, making it difficult to provide clear and binding timelines to improvements, since you don't apriori know what the solutions will be, or if they'll work (hopefully, this all reads charitably; as a SWE, I totally understand this challenge). Yet what we look for, with a CA, are ways that we can be confident the CA understands the risks, and ways to measure and understand how a CA resources and prioritizes improvements. For example, a CA that takes 6 months to fix an issue bears a much more thorough explanation than a CA that is able to fix (the same issue) overnight, but it's also useful to understand if that overnight fix runs the risk of causing more harm than good, or failing to actually address the issue vs paper over it (e.g. blocklisting " " from an OU field, versus stopping issuing OUs). Given that Sectigo is constantly leaning on "Guard Rails will solve this", is there a path Sectigo has to propose to provide insight into "Here's what we're prioritizing, here's the overall list of things we see as important (which we will continue to add to), and here's our next expected update"?

• I want to find a path here to get the transparency, find a path to resolving these bugs, while also finding a way Sectigo has to provide transparency that this class of major misissuance events has been firmly and completely resolved.

While I'd normally want to understand how these issues (like no automated testing in your regression tests, or no automated ACME tests) went on so long, I recognize that Sectigo's efforts on guard rails at least demonstrates they recognize that this is a serious lapse to be corrected ASAP.

We are managing Guard Rails with a window on the next month or so. In addition to the government businessCategory-registrationNumber match we just deployed (see bug 1715929 comment 14) our upcoming release targets are:

August 1: Forced escalation for postalCode match to other fields

• Compare the organization’s postalCode value to the contents of all other variable fields
• In the event of a match, hold issuance for review by escalation team

August 1: Match businessCategory to organization name

• Parse businessCategory value for known markers of organization type such as Inc. or LLC
• Compare to expected businessCategory value
• Bock issuance in case of mismatch

August 15: stateOrProvinceName-localityName exclusivity

• We have described this feature in bug 1715929 comment 12.
  We will add these dates as a new comment to bug 1715929. As features are delivered or expected dates change, we will provide updates on that bug.

Comment 15

[Nikola Maksimovic]

(In reply to Nikola Maksimovic from comment #14)
Oops.

August 1: Match businessCategory to organization name

• Parse businessCategory value for known markers of organization type such as Inc. or LLC
• Compare to expected businessCategory value
• Bock issuance in case of mismatch

The first bullet should read:

• Parse organizationName value for known markers of organization type such as Inc. or LLC

Comment 16

[Tim Callan]

Are there any further questions on this discussion?

Comment 17

[Ryan Sleevi]

So to check where we're at for deliverables, then, based on the updates and corrections to Comment #13

Date	Reference	Remediation
2021-05-01	Comment #12, Bug 1715929, Comment #9	New automated tests for (some of) these cases
2021-06-29	Comment #12	ACME endpoints now are covered by automated testing
2021-09-30	Comment #14	Achieve coverage for ACME endpoints that matches the coverage for other endpoints
2021 EOY/TBD	Comment #0	Top-to-bottom analysis of testing gaps ("Guard Rails")

Comment #14 mentions some GuardRails activity, but the improvements seem mostly unrelated to this specific issue, if I understand correctly.

I'd suggest let's set a NextUpdate for 2021-09-30, and Sectigo can share more details about the coverage they added. I think in light of the disconnect between omitted tests (e.g. Comment #13's 3 tests out of 4 possible, Comment #14's explanation it's actually 30 permutations for this single scenario), it's useful to share as detailed as possible, to the extent it helps other CAs consider what sort of testing scenarios and permutations to consider.

Comment 18

[Tim Callan]

Attachment: ACME SSL_TLS Test Automation Status _October_01_2021.docx

Comment 19

[Tim Callan]

We completed our ACME automation project on September 24. We have attached a detailed overview of the ACME automated test suite as attachment 9244017 [details]. Note that we do not support QWACs through ACME and therefore have not included them in this test suite.

Comment 20

[Tim Callan]

Attachment 9244017 [details] includes the details of our automated test suite for ACME. We hope it will help other members of the community with their own detailed ACME testing. Please share any questions here.

Comment 21

[Tim Callan]

Ben, it’s been two weeks since we posted our detailed ACME test automation architecture. The document will remain there to help other CAs working on ACME. There don’t appear to be any outstanding questions or items on this bug. Is it time to close it?

Comment 22

[Ben Wilson]

I will plan on closing this bug on Friday, 22-Oct-2021, unless there are additional questions to be answered.