1637093 - Multicert: AIA CA Issuer field pointing to PEM encoded cert

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[ca.forum]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:77.0) Gecko/20100101 Firefox/77.0

Steps to reproduce:

1. How your CA first became aware of the problem
2020-05-11 16:05 UTC - A notification was received by email reporting that the CA certificate file available for download on the URL included in the Authority Information Access (AIA) of end entity certificates was encoded in PEM format, while RFC 5280 §4.2.2.1 requires to be in DER format.

CA certificate "MULTICERT SSL Certification Authority 001": https://crt.sh/?caid=84368

AIA URL: http://pki.multicert.com/cert/SSL_CA01.cer

This incident was also reported in m.d.s.p (https://groups.google.com/forum/#!topic/mozilla.dev.security.policy/37aDOIBXN_A)

2. A timeline of the actions your CA took in response.
2020-05-11 16:18 UTC - Investigation on incident started
2020-05-11 16:29 UTC - Incident confirmed. Resolution started.
2020-05-11 19:37 UTC - Finished to replace CA certificate file by DER encoded version in public repository. Problem solved.

3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem.
Certificate issuing was not stopped. Problem could be fixed with a correction on the repository. It was not causing any misissuance and did not required any revocation/replacement of end user certificates.

4. A summary of the problematic certificates.
There were no end entity certificates misissued.

5. The complete certificate data for the problematic certificates.
CA certificate "MULTICERT SSL Certification Authority 001": https://crt.sh/?caid=84368

AIA URL: http://pki.multicert.com/cert/SSL_CA01.cer

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
Following a key ceremony, the distribution of a CA certificate file from the PKI team to the sysadmin team for publishing at the repository, at the AIA URL, is usually sent by email (integrity and authenticity confirmed by with S/MIME or out-of-band confirmation of sha256sum). Because many email UAs block certificate files sent as attachments, it is common practice to send as PEM encoded text files.

The follow-up publishing procedure of CA certificates on the public repository is manual and was not explicitly indicating the DER encoded format.

7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.
2020-05-11 18:15 UTC - Procedure was updated to explicitly indicate CA certificates files MUST be published in DER format in the public repository.

Comment 1

[Ben Wilson]

Have you had the opportunity yet to test your updated procedure to ensure that CA certificates are always published in DER format? If so, please provide a more detailed explanation of that step in the procedure and how it was performed. Thank you.

Comment 2

[ca.forum]

(In reply to Ben Wilson from comment #1)

Have you had the opportunity yet to test your updated procedure to ensure that CA certificates are always published in DER format? If so, please provide a more detailed explanation of that step in the procedure and how it was performed. Thank you.

That will only happen when we publish a new CA certificate.

Meanwhile, I would like to share the monitoring we setup to check continuous compliance with the requirement:

• check HTTP response code is 200 for the URL included in the AIA
• check Content-Type header is application/pkix-cert
• check file is in DER format (implementation detail: check magic number (first two bytes) are 0x30 0x82)

Comment 3

[Ryan Sleevi]

Why only check for 0x30 0x82? Why not check that it's a valid Certificate? And that it's the expected certificate, byte for byte?

There are plenty of ways for things to go wrong here: a file is truncated, a file is invalid DER, a file has trailing data (which may or may not cause client issues, depending on whether or not they permit trailing data)

Comment 4

[ca.forum]

We try to keep it simple in the monitoring system, with the least dependencies and software required to do the checks, balanced to the objectives. Limiting software dependencies reduces the attack surface and facilitates change management over time.

With that in mind, and following your comment, we are replacing the magic number check with a sha256sum comparison of the published file against the reference digest calculated upon initial publishing. It is simple enough and provides high assurance over the objectives.

Comment 5

[Ryan Sleevi]

Thanks, that does sound better.

So now I think the question is "What's the process for producing the sha256sum that you will configure monitoring to monitor for"

That is, if I'm understanding correctly, your process is:

• Certificate signing ceremony is held (since AIA will always necessarily refer back to an intermediate)
• (Something Happens)
• Ongoing monitoring of certain response properties (200 OK, correct Content-Type, correct content via sha256sum)

The (something happens) presumably entails producing the sha256sum, but also verifying those other properties I mentioned as part of an initial configuration step. Could you describe more (and with specific details) about how your new key ceremony works, through to the sha256sum monitoring?

Again, the idea here is sufficient detail that a reasonable third-party (e.g. another CA, a relying party) could implement a similar process, with the idea being that Multicert believes the process it has is the best way to accomplish this, given the constraints Multicert has identified.

Comment 6

[ca.forum]

(In reply to Ryan Sleevi from comment #5)

The (something happens) presumably entails producing the sha256sum, but also verifying those other properties I mentioned as part of an initial configuration step. Could you describe more (and with specific details) about how your new key ceremony works, through to the sha256sum monitoring?

Most of the post-key ceremony verifications are done with openssl verify.

Other checks are done by peer visual inspection of the details using openssl x509 -text. Particularly for the Subject and Issuer DN, we further inspect the order of RDNs, making sure they start from Country as well as binary comparison (openssl asn1parse at hand). This is because textual representation of DNs can be misleading (ordering, spacing, quoting, etc).

Finally, certification chain checking in end-user applications, e.g. NSS-based products, Windows, Apple, Java, etc.

Comment 7

[Ben Wilson]

Unless anyone has any further questions or comments, I intend to close this bug on or after 10-July 2020.

Comment 8

[Ryan Sleevi]

Yeah, I consider the response in Comment #6 disappointing and failing to address the concerns in Comment #5. I think we've tapped the well of reasonable response, and that it's unlikely we'll see improved controls from the CA.

I consider this, overall, a disappointing response from a CA.

Comment 9

[ca.forum]

(In reply to Ryan Sleevi from comment #8)

Yeah, I consider the response in Comment #6 disappointing and failing to address the concerns in Comment #5. I think we've tapped the well of reasonable response, and that it's unlikely we'll see improved controls from the CA.

I consider this, overall, a disappointing response from a CA.

Could you please detail what concerns were not addressed in our response, so that we have the opportunity to further improve our report?

I believe anyone reading this bug report will find interesting details of checks and controls, including some that are not so obvious, and can learn from it. We are also open to learn from others and further improve our own processes.

Comment 10

[Ben Wilson]

I think Comment #6 might have had more detail in the steps you take and could have clearly outlined the order in which the steps are performed. I am still going to close this bug, although I would encourage Multicert to continue efforts to improve its Quality Assurance (QA) processes and automated checking (more processing with computer-based checks and less reliance on visual inspection) when creating CA certificates.

Comment 11

[Ryan Sleevi]

Yes, to clarify, compare how Telia handled this in Bug 1637854 with the handling by Multicert on this issue.