1593814 - DigiCert: & character in a printableString in ICA

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Jeremy Rowley]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.87 Safari/537.36

Steps to reproduce:

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

• After the discussion on the Mozilla forum about key ceremonies, we started running queries on our CAs to see if we could detect any issues. Key ceremonies are manual processes and were identified internally as high risk so we decided to investigate them for issues. We discovered three ICAs that had improper encoding.

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

• We reported to Wells Fargo on 10/29 that these ICAs would be revoked because of mis-issuance. These were revoked on 11-01.

3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

• We have a linter for key ceremonies but it's not robust enough. The linter also didn't really exist in 2017 during the Symantec transition. We're working on improving the linter now and then open sourcing it. We stopped issuing certs with incorrect printableStrings as shown by later ICAs.

4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

There are three:
https://crt.sh/?id=250864705
https://crt.sh/?id=250864704
https://crt.sh/?id=254024589

Each one has the same problem. A "&" in the org name that is encoded as a printableString instead of UTF8. This contravenes the requirements of 5280. We've revoked the ICAs of course.

5. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

https://crt.sh/?id=250864705
https://crt.sh/?id=250864704
https://crt.sh/?id=254024589

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

• We didn't have a good enough linter on key ceremonies back in 2017. This was a problem. It also escaped notice because you can't eyeball an encoding issue. We've been running scans on our ICAs and certs to detect problems. This was one we found.

7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

• Since 2017, we've already already developed a linter for key ceremonies. We currently create a non-trusted test cert and run it through the linter. This tells us if the minted ICA will have any problems. However, upon review, we've determined it's not good enough. We also determined that this need to be shared with community so that everyone can benefit from it. Key ceremonies are way too manual right now with a lot of rules. We figure an open-source tool will help CAs across the industry achieve better results with their key ceremonies.

We've found several issues with other non-DigiCert ICAs, which we are sending to the the various issuers as well. Hopefully they will provide an incident report shortly.

Comment 1

[Jeremy Rowley]

We're currently investigating how to use zlint to check ICAs for issues pre-issuance. As stated above, we'd like to better automate the process and the easy place to start is by checking a more official linting source than the internal tool. We currently use zlint for end entity certs so it makes sense to use it for key ceremonies if we can. The review is to see how this could work in key ceremonies and what modifications may be required for ICAs.

Comment 2

[Ryan Sleevi]

Jeremy: Thanks.

I think one thing that would be useful to understand is why DigiCert is still defaulting to PrintableString vs UTF8String. Given PrintableString's many limitations, one might imagine that the only reason to use it is when encoding the Subject in a new certificate that needs to match a previously issued Subject (e.g. when creating a new version of an intermediate), so that you only ever have one canonical encoding. However, for all new certificates, using UTF8String globally, rather than selectively, seems to avoid these issues.

This matches the guidance from RFC 2459 (which said MUST USE UTF8String > 2003-12-31), which was preserved in RFC 3280. It was not until RFC5280 that PrintableString became an option, and only because RFCs setting transition dates tend to get ignored. However, it's something DigiCert could have done ages ago, so it's a bit curious to understand what may have been a factor in still using PrintableString.

I'm trying to be cautious here, to make sure you don't make everything UTF8String only and then end up with a certificate whose subject / issuer don't use the same encoding (despite 5280 permitting this, implementations largely don't support this, and it's a questionable violation of 5280).

Comment 3

[Jeremy Rowley]

We default to minimal encoding necessary. We actually kept it this way because it's how google.com does it - the GO cert building code defaults to PS and only goes to UTF8 if needed. Normally, the process works well for end-entity certs. However, our internal linter for ICAs didn't catch it lacks the robustness of zlint.

I'm going to put a spike in to see about moving to UTF8 instead of printable string though. I don't know a good reason not to get rid of printableStrings completely.

Comment 4

[Rob Stradling]

Sectigo also defaults to PrintableString and only goes to UTF8String if needed. We have no plans to change this. (RFC5280 obsoleted RFC3280, which obsoleted RFC2459, so the historical requirements from 3280/2459 are not relevant).

PrintableString was preferable to UTF8String back when the Windows 9x certificate viewer rendered UTF8String as a hex dump. I'm not aware of any UTF8String compatibility issues with newer clients, but...if it ain't broke, why fix it?

Comment 5

[Ryan Sleevi]

Rob: Thanks for monitoring the other bugs and for chiming in! As I tried to acknowledge, while 5280 allows PrintableString, it’s useful to understand the history, especially where there was a 5-year gap of discouraging anything but UTF8String, and many CAs in the Mozilla program were around then. Reasons like yours are very useful to understand the context and decisions. Perhaps we can/should take it to m.d.s.p.? Understanding if Win9X compatibility is something still very high for CAs decision making processes is useful, for example.

The rationale for the suggestion is in line with the overall set of challenges we see for CAs: When multiple options for doing a thing exist, it can be easy to make a mistake (like this issue) in applying and consistently following the rules. A migration to pure UTF8String over time, since it can’t be done automatically at once (since there are chains to migrate), helps provide a long-term path to reducing risk, especially where PrintableString offers no discernible benefits over UTF8String, but a more restrictive character set.

Comment 6

[Rob Stradling]

Ryan: I think the history lesson is that prohibiting PrintableString turned out to be a bad idea, hence the reversal of that prohibition in RFC5280. If you believe that the world has since moved on and is now ready to prohibit PrintableString (for DirectoryString attribute types), then I think a 5280-bis effort would be the way forward.

I take your point about multiple options, but...
PrintableString is much simpler than UTF8String. Validating whether or not a string contains only characters permitted by the PrintableString character set is not rocket science. Validating the same for UTF8String is harder. If a CA can make a mistake when encoding a PrintableString, what makes you believe that they won't make a mistake when encoding a UTF8String?

Some DN attributes (serialNumber, countryName, joiCountryName) can only be encoded as PrintableString, so many CAs are going to have to deal with this ASN.1 type anyway.

If you were to add new rules (to the BRs, or to the Mozilla and/or Chrome root store policies) to prohibit PrintableString for certain attribute types where RFC5280 permits it, you would further complicate the requirements placed on CAs. The more rules there are, the harder it is to keep them all. The more documents a CA representative has to look at to derive the effective rule set (X.520, RFC3280, RFC5280, BRs, Mozilla/Chrome root store policies) for a particular DN attribute, the more likely it is that their head will explode.

I'm sure nobody cares about Win9x any more. It's the unknown unknowns I'm concerned about though. Many certs we issue use only PrintableStrings. Can you guarantee that all of our customers use their certificates only with software that supports UTF8String?

Comment 7

[Jeremy Rowley]

Looks like the discussion is dead on the Mozilla forum now. To resolve this, we've implemented zlint on the key ceremony. We create a dummy cert, run it through zlint, and then create the real cert. This, of course, creates lots of false positives, but we evaluate each one before issuing. We tested this in the last key ceremony and it worked great, although it didn't identify anything other than false positives. Anything we need to do to resolve this one?

Comment 8

[Daniel]

This, of course, creates lots of false positives, but we evaluate each one before issuing. We tested this in the last key ceremony and it worked great, although it didn't identify anything other than false positives

What sorts of false positives? Would you consider opening an issue with the upstream project with more detailed information so that the linting tool could be improved?

Comment 9

[Jeremy Rowley]

Thanks for asking this Daniel. It turns out I mis-understood. There are actually no false positives when we ran the test with zlint and zlint handles issuing CA requirements quite well. Instead of it failed the test with false positives, what the team was saying is that they ran the tests and it found the error without false positives. Sorry about that confusion.

Comment 10

[Daniel]

Great! Glad to hear it. Thanks for following up.

Comment 11

[Wayne Thayer]

It appears that all questions have been answered and remediation is complete.