1335132 - DigiCert: Verizon mis-issued test certificates

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Kathleen Wilson]

From
https://groups.google.com/d/msg/mozilla.dev.security.policy/AvmWvPZUFO8/Tcxk5mHwCwAJ

Based on the Symantec disclosure, we ran a test on our own certs (including
cross-signed partners) and found the following certificates that were issued
contrary to the Baseline Requirements. All of these certificates were issued
by Verizon's subordinate certificate. We've requested that the issuer
revoked each of these. So far, two have been revoked. Let me know what
questions you have.

Comment 1

[Kathleen Wilson]

Attachment: VerizonTestCerts.png

Comment 2

[Jeremy Rowley]

Attachment: Verizon Test Certificates.xlsx

Comment 3

[Gervase Markham [:gerv]]

Hi Jeremy,

What news on getting all of these revoked?

Has the source of these certificates been identified and shut down?

Some of the domains in these certs are clearly bogus; how is domain validation done in this system? Was it overridden in this case, or simply bypassed (not done)?

Gerv

Comment 4

[Jeremy Rowley]

1) I've still waiting on a status report from Verizon on whether these are revoked. Our primary contact there has been out. I expect to hear back tomorrow. 

2) Verizon is an intermediate CA with their own issuing system. They are using this intermediate to support services while the transition to DigiCert. I've asked them to identify why these certificates were issued and report back with what steps were taken to address the issue and prevent re-occurrence. 

3) Agreed that some of these are bogus. We're waiting for a report from Verizon on what happened with their domain validation systems.  I've asked them for a report by tomorrow.

Comment 5

[Jeremy Rowley]

26 out of the 28 have now been revoked. The remaining two are expired. Verizon is currently patching its systems to prevent re-occurrence and has conducted training with its team to not circumvent the domain validation for any reason.

Comment 6

[Gervase Markham [:gerv]]

(In reply to Jeremy from comment #4)
> 2) Verizon is an intermediate CA with their own issuing system. They are
> using this intermediate to support services while the transition to
> DigiCert. 

Is there an ETA for when this transition will be completed?

Gerv

Comment 7

[Gervase Markham [:gerv]]

Jeremy: ping?

Gerv

Comment 8

[Jeremy Rowley]

Thanks for the ping. I missed comment 6. Will get info on that right away.

Comment 9

[Jeremy Rowley]

These are all revoked. Still waiting on an update on intermediate usage from Verizon (which will tell us when the transition will be complete).

Comment 10

[Gervase Markham [:gerv]]

Jeremy: ping?

Gerv

Comment 11

[Jeremy Rowley]

Hey Gerv - we've moved a couple of divisions off of external intermediates, but we still don't have a date yet on when the entire transition will complete. Do we need a date to close this particular bug or is revocation of the certs enough? Is there a date Mozilla would like us to set?

Comment 12

[Gervase Markham [:gerv]]

Jeremy: if Verizon's issuing infrastructure has proper up-to-date audits there is no formal concern. But you can imagine I have a general sense of nervousness due to the previous mis-issuance.

You earlier wrote: "They are using this intermediate to support services while the transition to DigiCert", and now you say "we still don't have a date yet on when the entire transition will complete". This surprises me a little; I doubt you still want to be supporting them 5 years from now. Are they really contractually free to take as long as they like?

Gerv

Comment 13

[Jeremy Rowley]

We would like them to transfer as fast as possible, but there are a lot of entities so the transition time is pretty open-ended (we can't easily force everyone over). I'm hoping five years is not in the realm of possibilities. They do have a point in time audit uploaded and will shortly complete their full audit that lists all Sub CAs under Verizon's control.

Comment 14

[Jonathan Rudenberg]

I noticed that there were two additional Verizon test certs discovered in bug 1389172, comment 2. Can you explain why they were not detected during the handling of this issue in January?

Comment 15

[Jeremy Rowley]

It's a good question.I sent a question to Verizon asking why they didn't find all these the first time around.

Comment 16

[Jeremy Rowley]

Sorry for the delay in updating this. Basically the two test certs were CT pre-certs and not TLS certs. The corresponding certs never issued. Because they were pre-certs, the can did not pick them up, and the certs were not revoked.

Comment 17

[Ryan Sleevi]

Jeremy: In Comment #4, DigiCert indicated it was obtaining a report from Verizon as to how this happened. In the spirit of the postmortems regarding BR non-compliance, and in the effort of transparency (given the open-ended nature in Comment #13), it would be good to have a bit more transparency as to how this issue happened (and the root cause(s)), as well as the comprehensive set of steps taken or to be taken.

Further, with respect to the answer in Comment #16, has any further analysis been done, based on the corpus of CT-disclosed certificates, for any other forms of non-compliance with this sub-CA? For example, what steps have been or will be taken to ensure that the issue highlighted in Comment #16 does not occur in the future?

Comment 18

[Jeremy Rowley]

Sure - Verizon hadn't patched their systems.  Similar to the Symantec issue, they issued test certs in their dev environment that were actually live certs.  They patched their systems after receiving notice that this was not okay. They also revoked all of their certificates with internal names.  The scan missed two pre-certificates (as they are stored and handled differently than regular certs). These pre-certs were revoked after being pointed out on the Mozilla list.  This took longer than 24 hours because the Verizon PKi couldn't actually revoke pre-certs.  Verizon modified their systems to permit revocation of pre-certs in addition to standard SSL certs.

In summary, from what I gathered , the root cause was: 
1) lack of a system patch to prevent internal operations from issuing publicly trusted test certs
2) lack of training that even test certs must comply with the BRs

The remediation was:
1) patch systems to prevent any issuance of test certs unless those were verified in accordance with the BRs
2) train the dev team on the BR requirements and that test certs must comply with the BRs.

As for comment #16, Verizon scanned their entire database for additional mis-issued CT proofs with internal names. None were detected. Remediation of 16 was an update to the platform to ensure all CT proofs followed proper validation, certs with the poison extension follow the same path as all other certs.

What else would you like to know?

Comment 19

[Ryan Sleevi]

Regarding holistic analysis and root causes, #1/#2 as both causes and remediations don't seem fully complete. That is, could or should we reasonably expect that Verizon has failed to apply other system patches to comply with the BRs? What sort of holistic evaluation has been done or steps taken to ensure the systems are fully conforming?

Regarding Comment #13, is it correct to understand that the proposed migration is somewhere 5 years out?

Comment 20

[Jeremy Rowley]

I don't think you can assume that they haven't patched their systems completely - there was just the one incident.  The issue was similar to all the other CAs who issued test certs during that same time frame.   

As for a holistic view, Ben's requested that they provide a complete set of certificates issued from their intermediate.  We plan on running all of the certificates through CABLint to determine whether additional issues exist. We'll post the any non-compliance. 

On a perhaps related note, is there a way to require CT on a particular intermediate?  Or can this only be require for CAs at the root level? Asking for a friend.

I know they told me, but I can't recall the exact migration timeline. I'm checking with Ben and the Verizon guys.

Comment 21

[Alex Gaynor [:Alex_Gaynor]]

Jeremy, you probably want to talk to the Chrome team for a policy answer, but my read of their code is that it supports requiring CT for any cert in the chain, by public key.

Comment 22

[Jeremy Rowley]

Right - it would be really cool if we could add individual Sub CAs to Chrome where the Sub CA claims they are no longer issuing to verify that claim.

Comment 24

[Ben Wilson]

Here is a combined incident response report prepared by DigiCert and Verizon

1.	How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, via a discussion in mozilla.dev.security.policy, or via a Bugzilla bug), and the date.
July 31, 2017 email from Alex Gaynor requesting revocation of https://crt.sh/?id=12344381 and https://crt.sh/?id=33626750 

2.	A timeline of the actions your CA took in response. 
Email forwarded to Verizon on 31-July-2017.  https://crt.sh/?id=12344381 was revoked immediately on 2017-07-31, but  https://crt.sh/?id=33626750 only ever existed as a CT pre-certificate, and couldn’t be revoked until 2017-08-08.  (Note that section 7.1.2.5 of the BRs states that a pre-certificate is not considered to be a “certificate” subject to the requirements of RFC 5280 under the BRs.)

3.	Confirmation that your CA has stopped issuing TLS/SSL certificates with the problem.
This is confirmed that “.test”  certificates are no longer being issued by Verizon.

4.	A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued. 
https://crt.sh/?id=12344381 was issued on Dec. 23, 2105 and https://crt.sh/?id=33626750 was issued on Sept. 21, 2016, but see  attachment 8831833 [details].

5.	The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem. 
See the two certificates listed above.

6.	Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now. 
For https://crt.sh/?id=12344381 - A full scan of the database was performed to check for invalid DN’s ( including SAN’s) for all valid certificates.  Also the RA office performed manual verification for all domains allowed for any active user. 
•	All valid certificates were checked for DN compliance.
•	All domains available to issuers were verified and invalid domains removed so no more invalid certificates can be issued.  
Additionally, internal processes for issuance of test certificates have been updated and circulated to all relevant staff.
For  https://crt.sh/?id=33626750 - The pre-certificate has been revoked.  A CA patch was introduced to enable revocation of pre- certificates containing the poison certificate extension.
•	CA patch installed allowing revocation of pre-certificates.
•	Additional monitoring added to be able to detect incomplete/incorrect/failed responses from CT Log servers that prevent a “final” certificate from being issued.

7.	List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things. 
•	Updated internal processes for issuance of test certificates – ALREADY DONE
•	Additional monitoring to detect incomplete/incorrect/failed responses from CT Log Servers – ALREADY DONE
•	Software Change request raised to improve management and error-handling of pre-certificates from CT Log servers – currently planned for Q4/2017.

Comment 25

[Jeremy Rowley]

To hopefully add clarity, the two pre-certificates mentioned were not revoked previously because (1) the certificate underlying the logged pre-cert was never issued and (2) pre-certificates are not within scope of the BRs (as htey are not intended for TLS), meaning there was confusion about whether revocation support was required for pre-certs.  The net effect was the two pre-certificates were discovered and revoked at a much later date than the initial certificates.

Comment 26

[Kathleen Wilson]

Jeremy, What's the state of this bug?

Comment 27

[Jeremy Rowley]

Could this one be closed and start a new one about Sub CA deprecation? The issues here are resolved, but we are still working with Verizon to transition them away from operating an issuing CA. Still shooting for year's end, but I am having a hard time locking down the date.

Comment 28

[Gervase Markham [:gerv]]

I think we can close this out.

Gerv