Closed Bug 1389172 Opened 2 years ago Closed 2 years ago

DigiCert: Certificate Issues Identified on the Mailing List

Categories

(NSS :: CA Certificate Compliance, task)

Product:
NSS
Component:
CA Certificate Compliance
Type:
task
Priority:
Not set

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: jeremy.rowley, Assigned: kwilson)

References

Details

(Whiteboard: [ca-compliance]) 

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36



Actual results:

The following issues were discovered through crt.sh by the Mozilla community and reported on the Mozilla dev-sec mailing list:

1) Serial Numbers less than 64-bit entropy 
a) InfoCert
179801356
47922359
72065818

b) Siemens 
45010775
45000519
45073660
40700407
48827344
40726549
45068282
48827312
48726324
48726344
48726335
45068035
45060262
41273940
179803714
48827347
48827332
48827315
48827348
48726294
45029804
45053316
45061425
43151162
43151152
43151164
45060837
54576144
48881965
48726336
44438214
47906173
45048889
45257938
81268921
45257935
133008905
132946615
48726297
47884474
46110537
47211018
42791689
44377709
91004967
47612195
47646461
47612190
48827338
48715703
48827336
48726376
48773656
47743543
48773648
48773658
48773657
54576085
48773662
47605232
48827352
48773645
49706248
48827353
179801334
48773663
49706251
54576119
54576115
47939190
48827317
54397723
47815889
53205222
54576127
50756547
48773638
50165772
50756565
54576082
95852573
102169703

2) Metadata in OU field
a) CTJ
53176869
57830766
65616693
117197299
93295619
104632894
21853640
22198796
30236343
30547236
32078607
34809638
39655364
40493955
43400295
45247984
47645844
45815760
40840774
48674276
40627018
48832585
57830776
57830777
57830779
57830781
61135216
57785675
57785694
53176900
53176901
53176903
64152048
78648920
53176895
53176898
57830773
102832536
51567050
132148480
110336219
111461024
80230338
81246678
91907174
91907175
110354478
110354479
78766827
79544403
78494050
80022464
15522999
16058171
20501110
20501178
21202825
21202827
21202829
20501158
20501137
20686696
20501187
20501201
20501217
20686693
20835061
21202831
21202833
21202835
21202836
26829282
11658435
11723002
11020877
11758671
12502488
12502489
13247204
13247205
13388567
42698962
16519562
19240755
20479086
20103452
21192792
54658744
21477087
42505252
29807378
57881547
30238735
30238738
30238741
30351648
47300904
54163587
54163620
54163639
51576885
56762416
70713886
73109181
78277867
73090743
77286114
75591249

b) DigiCert
28838043
29719356
41854275
45002523
42416834
42416853
42416863
44421681
45060999
43131232
43131307
43131498
43132441
48279184
48558476
48558484
48908976
58878082
58931131
68837112
71615331
71632071
78328813
49832975
74933357
76418106
76418197
76590177
76639330
76685188
76685195
81975356
82029229
97931808
97931812
106177929
138900619
32267988
41751647
105794465
15326026
47930798
45048314
10859239
19505358
98797497
98797525
17719410
47773488
40937363
48678004
105227697
83837393
27797536
42000357
42281109
42416587
48992066
49529547
64029766
42440045
23016305
42394669
72557347
165031011

c) Terena
32142920
20475084
11984821
23387766
40944750
50979270
66713765
73234645
99764268
134328239

3) Serial Number > 20 Octets
a) TI Systems
176067928
160949550
160813046
166543375
173165333
168397193
168043243
179755654
176673561
179547720
165041960
179791026
173821648
161488914
173821651
165508757
179756125
179790644
Info Cert was revoked Aug 1, 2017.  They shouldn't be a problem.  We're reaching out to the remainder to have the certs revoked and replaced.  

For the CAs:
1) Siemens is no longer issuing certificates under the DigiCert hierarchy. Per our previous discussions, we are maintaining the CA solely to support previously issued certificates.  Once all certs expire, the Sub CA will retire.
2) TI Systems is migrating away from a sub CA. We hope to complete the transition by the end of the year.
3) CTJ is migrating to a new platform. The transition will be lengthy, about a year.

Jeremy
Additional certs
4) Local and test names
a) Verizon
https://crt.sh/?id=33626750&opt=cablint
https://crt.sh/?id=12344381&opt=cablint

5) commonName not in SANS
a) TI Trust Systems
crt.sh/?id=100738527&opt=cablint
crt.sh/?id=113547920&opt=cablint
crt.sh/?id=113197913&opt=cablint
crt.sh/?id=128965579&opt=cablint
crt.sh/?id=139455685&opt=cablint

b) Swiss Government
https://crt.sh/?id=108003741&opt=cablint
https://crt.sh/?id=108003737&opt=cablint
Blocks: BR-Compliance
Summary: Certificate Issues Identified on the Mailing List - DigiCert → DigiCert: Certificate Issues Identified on the Mailing List
Whiteboard: [ca-compliance]
6) Reserved IP Address:
a) CTJ
33034069

b) TI Trust
10620806	crmaffari.telecomitalia.local	crmaffari.telecomitalia.local ,  crmaffari	4 days ago	2017-11-18
10632702	com.telecomitalia.it	com.telecomitalia.it	4 days ago	2017-12-17
10632711	allintranet.telecomitalia.it	allintranet.telecomitalia.it	4 days ago	2018-06-23
13421134	oaforms.telecomitalia.it	oaforms.telecomitalia.it ,  SVOAESEWEBc24	4 days ago	2019-01-14
12819786	oaforms.telecomitalia.it	oaforms.telecomitalia.it ,  oaformsmanager.telecomitalia.it , oaformsapp.telecomitalia.it ,  helpme.telecomitalia.it , attivatore.telecomitalia.it ,  SVOAESEWEBc24	4 days ago	2019-02-03

7) Invalid dNSNnames
a) TI Trust
14568767
10620806
10620776
10620792
9864794
9871186
13421134
12819786

b) Justica
10620923

c) WellsSecure
19560142

8) Bad Characters
a) Wells Fargo
DigiCert	19558707	2015-11-25 11:32:15	2016-11-24 11:32:15	ceomobilefix.wf.com
DigiCert	11382596	2015-11-25 11:31:17	2016-11-24 11:31:17	ceomobile.wf.com

9)Double dot characters
a) Inteso San Paulo
https://crt.sh/?q=2B95B474A2646CA28DC244F1AE829C850EA41CF64C75E11A94FE8D228735977B&opt=cablint,x509lint
Here's the proposed resolution for each of these issues:

1a) InfoCert's subCA was revoked.
1b) We are working towards revocation of the Siemens Sub CA. We are still working out the exact date with them, but we currently plan to add them to OneCRL on Sep 1 and revoke on Sep 15. 
2) Both CTJ and DigiCert have updated their systems to prevent metadata in the OU. Per the previous discussion on the issue, we don't plan to revoke the existing ones. (Source: https://groups.google.com/forum/#!msg/mozilla.dev.security.policy/DgeLqKMzIds/E9--gj5WEAAJ).  
3a) We plan to revoke the TI Trust issuing CA.  We are still working out the exact date with them, but the current plan is to add them to OneCRL on Sep 1 and revoke on Sep 15.
4a) Both certificates are revoked. We've asked Verizon for a more detailed report on why these were missed previously. 
5a) We plan to revoke the TI Trust issuing CA.  We are still working out the exact date with them, but the current plan is to add them to OneCRL on Sep 1 and revoke on Sep 15.
5b) Both are revoked
6a) Revoked
6b) TI Trust systems is being revoked
7a) TI Trust systems is being revoked
7b) Justica revoked the cert
7c) Wells Fargo revoked the cert
8a) These were expired
9a) Revoked
To continue inclusion of your CA’s root certificates in Mozilla’s Root Store, you must respond in this bug to provide the following information:
1) Prompt confirmation that your CA has stopped issuing TLS/SSL certificates with the problems listed below.
- With the exception of the metadata issue, DigiCert is not issuing any of these certs. We've asked each of the Sub CAs to confirm they have updated their systems to prevent further mis-issuance. All confirmed they will fix the problem.  Siemens and TI Trust provided unsatisfactory answers and are being revoked. In each communication, we stressed the need to revoke within 24 hours of receiving a problem report and provide a detailed remediation plan.

2) Explanation about how and why the mistakes were made, and not caught and fixed earlier.
- Most of these certificates are fairly old. With the exception of reserved names, I think most CAs didn't think revoking the existing certs was a big deal as long as they fixed their system. 

1) Serial Numbers less than 64-bit entropy 
a) InfoCert - revoked, issued due to older system
b) Siemens - planned for revocation, no long using the intermediate and are operating in maintenance mode only. 

2) Metadata in OU field - Patched during the previous discussion about metadata, but the decision then was revocation is not necessary. 

3) Serial Number > 20 Octets - TI Trust is being revoked as they are unable to fix this system.
4) Local and test names - Verizon patched their system. We're waiting to hear why it was not caught and revoked earlier.

5) commonName not in SANS
a) TI Trust Systems - Revoking 
b) Swiss Government - Patched earlier.

6) Reserved IP Address:
a) CTJ - This cert was actually properly issued but wasn't revoked as part of the internal name revocation of 2016. They simply forgot to revoke this one.
b) TI Trust - Being revoked

7) Invalid dNSNnames
a) TI Trust - Revoking
b) Justica - Asking for an explanation 
c) WellsSecure - Revoked. Asked for an explanation. 

8) Bad Characters
a) Wells Fargo - Patched and revoked

9)Double dot characters
a) Inteso San Paulo - Revoked. I believed they patched their system, but I'll double check.

3) List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.
a) Revoking TI Trust and Siemens by Sep 15
b) Patching our systems and requiring the Sub CAs to patch their systems - already done
c) Per our previous conversation, we are planning on requiring each CA to use cablint to find and detect errors prior to issuance. We're still working towards that goal by year's end.


4) Regular updates to confirm when those steps have been completed.
a) Will do.  Let me know if there are questions.
Telecom Italia Trust Systems has revoked some of the certificates listed  in this bug.  Siemens has indicated it is working on revoking some too.  Both have indicated that the Sept. 1 (OneCRL) and Sept. 15 (revocation) are too aggressive considering the impact that it will have on their systems. Siemens is launching a new system on Sept. 8 that will CT log all certificates and issue certificates with serial numbers of appropriate length.  We are working with Telecom Italia to create two new CAs (server and client) this week that will be housed inside DigiCert's infrastructure. They currently have approximately 2,200 server certificates and 13,000 client certificates that will need to be transitioned over.
(In reply to Jeremy from comment #6)
> 3) List of steps your CA is taking to resolve the situation and ensure such
> issuance will not be repeated in the future, accompanied with a timeline of
> when your CA expects to accomplish these things.
> a) Revoking TI Trust and Siemens by Sep 15
> b) Patching our systems and requiring the Sub CAs to patch their systems -
> already done
> c) Per our previous conversation, we are planning on requiring each CA to
> use cablint to find and detect errors prior to issuance. We're still working
> towards that goal by year's end.

Can you provide a more descriptive timeline, enumerating which CAs (all sub-CAs or just those identified here), along with expected completion dates for such remediations?

> 4) Regular updates to confirm when those steps have been completed.
> a) Will do.  Let me know if there are questions.

Thanks for proactively identifying and responding to the questions that have been directed at other CAs regarding these misissuances.

I'll note that this reply was over a week ago, and a number of items were listed as "in progress" with limited to no timeline detail.

Can you please review the answers you've provided, so that we can ensure we have a complete and timely understanding?

(In reply to Ben Wilson from comment #7)
> Telecom Italia Trust Systems has revoked some of the certificates listed  in
> this bug. 

Does this mean that it has not revoked others? Please provide further details, such as what certificates have _not_ been revoked.

> Siemens has indicated it is working on revoking some too.

Similarly, can you please indicate which certificates have _not_ been revoked.

> Both have indicated that the Sept. 1 (OneCRL) and Sept. 15 (revocation) are too
> aggressive considering the impact that it will have on their systems.

- Do you believe this is consistent with DigiCert's CP/CPS?
- Does your CP/CPS describe the process that such 'leeway' is granted if something is reported by the customer as 'too aggressive'
- Do you have reason to believe these two CAs comply with all aspects of the Baseline Requirements and Mozilla inclusion policy, such that failing to revoke does not introduce further or additional risk? Please provide details supporting this, if so.

> Siemens is launching a new system on Sept. 8 that will CT log all
> certificates and issue certificates with serial numbers of appropriate
> length. 

Does this mean that, until then, Siemens will continue to violate the BRs and does not have remediation? As previously indicated, these CAs provided "unsatisfactory answers" (which are undisclosed on this bug), and so this does not seem a reasonable mitigation.

> We are working with Telecom Italia to create two new CAs (server
> and client) this week that will be housed inside DigiCert's infrastructure.
> They currently have approximately 2,200 server certificates and 13,000
> client certificates that will need to be transitioned over.

It does not appear, for purposes of OneCRL, that there is a need to transition the 13,000 client certificates in order to ensure a smooth transition. Can you provide a schedule for when Telecom Italia will have servers transitioned (both when it expects to first be able to transition and when it expects to last complete that transition), so that Mozilla can make an informed decision about the compatibility impact of adding to OneCRL.
Hey Ryan, 

Here's a more definitive timeline:
1) Siemens - Revoking the Sub CA. The date is still set to Sep 15, but I'll let Ben manage the conversation. The mis-issued certificates are  still not revoked.
2) InfoCert - already revoked
3) TI Trust - TI Trust systems successfully revoked all of their certificates.  They are migrating away from the Sub CA and asked for more time. Ben is working with them to establish a date. 
4) Verizon - patched.  These were actually mis-issued pre-certs.
5) Well's Fargo - revoked and patched
6) CTJ - patched already
7) Swiss Gov - patched already
8) Justica - patched alreaduu
9) Inteso San Paulo - revoked the cert  

Using cablint for every certificate will be required by the end of the year. We've asked all Sub CAs to provide a complete repository of certificates. These will be scanned through cablint for issues. Some have started delivering. We expect a complete set by Oct.   

(Sorry for the delay in updates - I was gone most of last week)
(In reply to Jeremy from comment #0)
Here is an updated status on revocation:
> 
> 3) Serial Number > 20 Octets
> a) TI Systems
> 176067928
Revoked
> 160949550
Revoked
> 160813046
Revoked
> 166543375
Revoked
> 173165333
Revoked
> 168397193
Revoked
> 168043243
Revoked
> 179755654
Revoked
> 176673561
Revoked
> 179547720
Revoked
> 165041960
Revoked
> 179791026
Revoked
> 173821648
Revoked
> 161488914
Revoked
> 173821651
Revoked
> 165508757
Revoked
> 179756125
Revoked
> 179790644
Revoked
(In reply to Jeremy from comment #2)
Here is a revocation status of the following certificates:

> Additional certs
> 5) commonName not in SANS
> a) TI Trust Systems
> crt.sh/?id=100738527&opt=cablint
Revoked
> crt.sh/?id=113547920&opt=cablint
Revoked
> crt.sh/?id=113197913&opt=cablint
Revoked
> crt.sh/?id=128965579&opt=cablint
Revoked
> crt.sh/?id=139455685&opt=cablint
Revoked
> 
> b) Swiss Government
> https://crt.sh/?id=108003741&opt=cablint
Revoked
> https://crt.sh/?id=108003737&opt=cablint
Revoked
(In reply to Jeremy from comment #3)

Here is the revocation status of the following certificates:
> 
> b) TI Trust

> 10620806	crmaffari.telecomitalia.local	crmaffari.telecomitalia.local , 
> crmaffari	4 days ago	2017-11-18
Revoked

> 10632702	com.telecomitalia.it	com.telecomitalia.it	4 days ago	2017-12-17
Revoked

> 10632711	allintranet.telecomitalia.it	allintranet.telecomitalia.it	4 days
> ago	2018-06-23
Revoked

> 13421134	oaforms.telecomitalia.it	oaforms.telecomitalia.it ,  SVOAESEWEBc24
> 4 days ago	2019-01-14
Revoked

> 12819786	oaforms.telecomitalia.it	oaforms.telecomitalia.it , 
> oaformsmanager.telecomitalia.it , oaformsapp.telecomitalia.it , 
> helpme.telecomitalia.it , attivatore.telecomitalia.it ,  SVOAESEWEBc24	4
> days ago	2019-02-03
Revoked
> 
> 7) Invalid dNSNnames
> a) TI Trust
> 14568767
Revoked

> 10620806
Revoked

> 10620776
Revoked

> 10620792
Revoked

> 9864794
Revoked

> 9871186
Revoked

> 13421134
Revoked

> 12819786
Revoked
(In reply to Ryan Sleevi from comment #8)

> (In reply to Ben Wilson from comment #7)
> > Telecom Italia Trust Systems has revoked some of the certificates listed  in
> > this bug. 
> 
> Does this mean that it has not revoked others? Please provide further
> details, such as what certificates have _not_ been revoked.

I am continuing my review of Telecom Italia Trust Technologies, but I may have been mistaken in saying "some".  So far, all of the certificates issued by them in this bug were revoked a couple of weeks ago.

> 
> > Siemens has indicated it is working on revoking some too.
> 
> Similarly, can you please indicate which certificates have _not_ been
> revoked.

I am continuing my review and will have more information next week.  So far, the list of certificates that I am relying on is in 
https://bugzilla.mozilla.org/attachment.cgi?id=8898848 under the tab "SSL CA 2013 ZZZZZZY9".  I have a call with Siemens on Monday and will have more information then.

> 
> > Both have indicated that the Sept. 1 (OneCRL) and Sept. 15 (revocation) are too
> > aggressive considering the impact that it will have on their systems.
> 
> - Do you believe this is consistent with DigiCert's CP/CPS?

I believe that working with Siemens and Telecom Italia on this issue in the manner currently undertaken is consistent with the DigiCert CP/CPS and good CA practice.

> - Does your CP/CPS describe the process that such 'leeway' is granted if
> something is reported by the customer as 'too aggressive'

As stated above, this is consistent with our CP/CPS and good CA practice.

> - Do you have reason to believe these two CAs comply with all aspects of the
> Baseline Requirements and Mozilla inclusion policy, such that failing to
> revoke does not introduce further or additional risk? Please provide details
> supporting this, if so.

I believe that in some cases a CA could be justified in not revoking certificates within a 24-hour timeframe.  Risks and circumstances should be considered. I do not have knowledge of "all" aspects as they relate to the Baseline Requirements or the Mozilla inclusion policy. However, I have no additional basis, other than these misissuances, to believe that the CA or the failure to revoke all of these certificates within 24 hours poses additional risk.    

> 
> > Siemens is launching a new system on Sept. 8 that will CT log all
> > certificates and issue certificates with serial numbers of appropriate
> > length. 
> 
> Does this mean that, until then, Siemens will continue to violate the BRs
> and does not have remediation? As previously indicated, these CAs provided
> "unsatisfactory answers" (which are undisclosed on this bug), and so this
> does not seem a reasonable mitigation.

Previously and separately elsewhere, Siemens has explained that it has taken its CA offline while it is working to remediate the problem with its serial numbers. I have a call with Siemens on Monday and will have more information then.  As also noted elsewhere, Siemens stopped issuing under the Siemens Issuing CA Class Internet Server 2013 last year.
 
> 
> > We are working with Telecom Italia to create two new CAs (server
> > and client) this week that will be housed inside DigiCert's infrastructure.
> > They currently have approximately 2,200 server certificates and 13,000
> > client certificates that will need to be transitioned over.
> 
> It does not appear, for purposes of OneCRL, that there is a need to
> transition the 13,000 client certificates in order to ensure a smooth
> transition. Can you provide a schedule for when Telecom Italia will have
> servers transitioned (both when it expects to first be able to transition
> and when it expects to last complete that transition), so that Mozilla can
> make an informed decision about the compatibility impact of adding to OneCRL.

Yesterday we created two CAs for Telecom Italia-- https://ccadb.force.com/0011J000019xflQ and https://ccadb.force.com/0011J000019xeiw.  These two CAs are now being tested by Telecom Italia. Until I speak with Telecom Italia next week, I do not have any further information on their SSL/TLS Server Certificate replacement efforts.
This morning we had a call with Siemens. They are re-checking everything on their 2017 CA and are in communication with their auditor before bringing the 2017 CA live on 8-Sept-2017.  Meanwhile they are replacing certificates with ones issued by Symantec. They will continue this work throughout September, when they expect to have all certificates replaced/revoked. This effort requires that they contact their internal customers. Some customers have restrictions on allowed downtime.  Others have implemented certificate pinning, making certificate replacement more difficult. Siemens' process involves identifying and mitigating potential risks that would otherwise be encountered by moving too fast. They are prioritizing the revocation and replacement of the 137 DigiCert-chained certificates with a goal to have those done by September 15.  Measures being implemented to prevent/avoid this situation in the future include: increasing the formal level of review of all changes proposed by browsers/CABF; immediate re-audits based on the adoption of new requirements; and conducting internal lessons-learned workshops. They will be providing DigiCert with a weekly update to their revocation tracking spreadsheet until all certificates are revoked and replaced in September.
The plan is still to add the Siemens intermediate to oneCRL by September 15th.
Attempting to summarize the status below:

Because there's such a large number of issues here, particularly with sub-CAs of DigiCert, this is a bit difficult, particularly as new issues emerged. I've attempted to break this down per-subCA, which may be appropriate to do so in the future.

Note that when I say "Remediation not performed", I am explicitly excluding the notion of simply revoking the certificates. Rather, we want to see a comprehensive root cause analysis as to what caused the issue, and what systems adn controls are in place. Unless DigiCert is planning to revoke these certificates within 7 days, I would further suggest that DigiCert must, for each of these cases, perform their own post-mortem regarding their supervision and oversight of their subordinate CAs, and what steps they have or are taking to resolve these issues.

Kathleen, I suspect that given the number of issues, and the number of sub-CAs for which we are lacking meaningful response or analysis for, that we probably should be looking at creating a bug for each and every one of these issues (e.g. letter and number) to allow DigiCert the opportunity to respond with the 7 questions being asked.

1) InfoCert
  a) Insufficient Serial Number Entropy
    - See Comment #0, Comment #4, Comment #6
    - Remediation
      - 2017-08-01 - CA Certificate was Revoked

2) Siemens
  a) Insufficient Serial Number Entropy
    - See Comment #0, Comment #1, Comment #4, Comment #6, Comment #9, Comment #13, Comment #14, Comment #15
    - Remediation
      - 2017-09-15 - CA certificate will be revoked
3) CTJ
  a) Metadata in OU fields
    - See Comment #0, Comment #1, Comment #4, Comment #9
    - Remediation
      - 2017-08-16 - CTJ systems patched
      - 2018-08-XX - CTJ migrates to a new system (See Comment #1)
  b) Reserved IP Address
    - See Comment #3, Comment #6, Comment #9
    - Remediation
      - None performed; no root cause analysis performed

4) DigiCert
  a) Metadata in OU fields
    - See Comment #0, Comment #4, Comment #6
    - Remediation
      - 2017-08-16 - DigiCert systems patched

5) Terena
  a) Metadata in OU fields
    - See Comment #0
    - Remediation
      - None performed; no root cause analysis performed

6) Telecom Italia Trust Systems (TI Trust Systems / TI Systems)
  a) Serial numbers > 20 octets
    - See Comment #0, Comment #1, Comment #4, Comment #5, Comment #7, Comment #9, Comment #10, Comment #13
    - Remediation
      - None performed.
      - Comment #4 and Comment #5 proposed revoking this sub-CA, but Comment #9 walked this back and indicated DigiCert would not be revoking.
      - Comment #7 suggests standing up a new sub-CA, and expanded upon in Comment #13. Neither of these remedy the problems with the existing CAs.
  b) Common Names not in SANs
    - See Comment #2, Comment #3, Comment #6, Comment #9, Comment #10, Comment #13
    - Remediation
      - None performed. See above.
  c) Reserved/Intranet domain name
    - See Comment #3, Comment #6, Comment #9, Comment #10, Comment #13
    - Remediation
      - None performed. See above.
  d) Invalid DNS names
    - See Comment #3, Comment #6, Comment #9, Comment #10, Comment #13
    - Remediation
      - None performed. See above.

7) Justica
  a) Invalid DNS names
    - See Comment #3, Comment #5, Comment #6, Comment #9
    - Remediation
      - 2017-08-24 - CA patched (See Comment #9)

8) Wells Fargo
  a) Reserved/Intranet domain name
    - See Comment #3, Comment #5, Comment #6, Comment #9
    - Remediation
      - 2017-08-24 - CA patched (See Comment #9)
  b) Invalid DNS name (trailing whitespace)
    - See Comment #3, Comment #5, Comment #6, Comment #9
    - Remediation
      - 2017-08-24 - CA patched (See Comment #9)

9) Swiss Government
  a) CommonName not in SANs
    - See Comment #2, Comment #4, Comment #6, Comment #9
    - Remediation
      - 2017-08-24 - CA patched

10) Verizon
  a) Reserved/Intranet domain name
    - See Comment #2, Comment #4, Comment #6, Comment #9
    - Remediation
      - 2017-08-24 - CA patched

11) Inteso San Paulo
  a) Double dot characters
    - See Comment #3, Comment #5, Comment #6, Comment #9
    - Remediation
      - None performed


Overall, I think the level of communication and detail here is far below the standard of what we'd expect and are expecting of CAs, but the depth and breath of misissuance by DigiCert's subordinate CAs, and the sheer volume of information here, have prevented effectively enforcing this.

For this reason, I suggested the above split-out, to allow each of these issues to be followed up on. On the positive side, a number of these issues were self-reported (See Comment #0, Comment #2, Comment #3) or attempting to be proactive (See Comment #6). This is positive and encouraging, although it's expected of all CAs as a minimum standard. On the negative side, multiple plans for remediation have failed to be met, most notably with both Siemens and TI Trust Systems, which highlights both the importance of managing these as explicit issues and concerns with timelines DigiCert proposes to respond.

Kathleen, Gerv, I'm flagging this for you to review and to determine how to proceed effectively.
Flags: needinfo?(kwilson)
Flags: needinfo?(gerv)
This morning we had a call with Telecom Italia.  They are no longer issuing certificates from the TI Trust Technologies Global CA (the “TI Trust CA”). Instead, their SSL/TLS Server Certificates are being issued from DigiCert’s infrastructure under the Trust Technologies Global CA, which is owned and controlled by DigiCert as a subordinate CA to the Baltimore Cybertrust Root, which is subject to ongoing WebTrust audit and compliance with the Baseline Requirements.  

As to the reason for the misissuances, they were apparently due to use of the Unicert CA platform, which is no longer being supported.  They were not aware of the problem until notified by us.  The misissuances were not detected as part of an annual audit because until recently these certificate criteria were not examined by auditors.

Telecom Italia has already revoked the certificates with problems identified in my following responses:  https://bugzilla.mozilla.org/show_bug.cgi?id=1389172#c10, https://bugzilla.mozilla.org/show_bug.cgi?id=1389172#c11, and https://bugzilla.mozilla.org/show_bug.cgi?id=1389172#c12. 

Telecom Italia now intends to revoke all previously issued certificates with the serial number problem (about 800-900) by 31 December 2017.  Again, these were due to a bug or unsupported feature of the Unicert CA software.  Telecom Italia have represented to us that they will accelerate certificate revocation/replacement as they become more familiar with our systems and API.
  
Stressing the importance that their customers, including the Italian Government, not be unduly disrupted, they would like to avoid having the TI Trust CA being placed on OneCRL or revoked.  They would like the 900 or so remaining SSL/TLS server certificates to be replaced in the normal course of business or when they expire.  This will allow them to replace these otherwise-good certificates over time, as well as the 13,000 client certificates I mentioned in https://bugzilla.mozilla.org/show_bug.cgi?id=1389172#c7.
Kathleen is creating a bug for each Sub CA.  I'll be the owner on each bug and add the Sub CA's to the cc list.  This way we can address each one separately.
(In reply to Ryan Sleevi from comment #16)
> 
> 1) InfoCert
>   a) Insufficient Serial Number Entropy
>     - See Comment #0, Comment #4, Comment #6
>     - Remediation
>       - 2017-08-01 - CA Certificate was Revoked


Bug #1397951

> 
> 2) Siemens
>   a) Insufficient Serial Number Entropy
>     - See Comment #0, Comment #1, Comment #4, Comment #6, Comment #9,
> Comment #13, Comment #14, Comment #15
>     - Remediation
>       - 2017-09-15 - CA certificate will be revoked

Bug #1397954


> 3) CTJ
>   a) Metadata in OU fields
>     - See Comment #0, Comment #1, Comment #4, Comment #9
>     - Remediation
>       - 2017-08-16 - CTJ systems patched
>       - 2018-08-XX - CTJ migrates to a new system (See Comment #1)
>   b) Reserved IP Address
>     - See Comment #3, Comment #6, Comment #9
>     - Remediation
>       - None performed; no root cause analysis performed
> 


Bug #1397957



> 4) DigiCert
>   a) Metadata in OU fields
>     - See Comment #0, Comment #4, Comment #6
>     - Remediation
>       - 2017-08-16 - DigiCert systems patched
> 


Will keep this one in this bug, Bug #1389172
For the DigiCert "Metadata in OU fields" problem listed above, please provide an incident report as described here:
https://wiki.mozilla.org/CA/Responding_To_A_Misissuance#Incident_Report


> 5) Terena
>   a) Metadata in OU fields
>     - See Comment #0
>     - Remediation
>       - None performed; no root cause analysis performed

Bug #1397958


> 
> 6) Telecom Italia Trust Systems (TI Trust Systems / TI Systems)
>   a) Serial numbers > 20 octets
>     - See Comment #0, Comment #1, Comment #4, Comment #5, Comment #7,
> Comment #9, Comment #10, Comment #13
>     - Remediation
>       - None performed.
>       - Comment #4 and Comment #5 proposed revoking this sub-CA, but Comment
> #9 walked this back and indicated DigiCert would not be revoking.
>       - Comment #7 suggests standing up a new sub-CA, and expanded upon in
> Comment #13. Neither of these remedy the problems with the existing CAs.
>   b) Common Names not in SANs
>     - See Comment #2, Comment #3, Comment #6, Comment #9, Comment #10,
> Comment #13
>     - Remediation
>       - None performed. See above.
>   c) Reserved/Intranet domain name
>     - See Comment #3, Comment #6, Comment #9, Comment #10, Comment #13
>     - Remediation
>       - None performed. See above.
>   d) Invalid DNS names
>     - See Comment #3, Comment #6, Comment #9, Comment #10, Comment #13
>     - Remediation
>       - None performed. See above.


Bug #1397960


> 
> 7) Justica
>   a) Invalid DNS names
>     - See Comment #3, Comment #5, Comment #6, Comment #9
>     - Remediation
>       - 2017-08-24 - CA patched (See Comment #9)

Bug #1397961

> 
> 8) Wells Fargo
>   a) Reserved/Intranet domain name
>     - See Comment #3, Comment #5, Comment #6, Comment #9
>     - Remediation
>       - 2017-08-24 - CA patched (See Comment #9)
>   b) Invalid DNS name (trailing whitespace)
>     - See Comment #3, Comment #5, Comment #6, Comment #9
>     - Remediation
>       - 2017-08-24 - CA patched (See Comment #9)

Bug #1397963

> 
> 9) Swiss Government
>   a) CommonName not in SANs
>     - See Comment #2, Comment #4, Comment #6, Comment #9
>     - Remediation
>       - 2017-08-24 - CA patched

Bug #1397965

> 
> 10) Verizon
>   a) Reserved/Intranet domain name
>     - See Comment #2, Comment #4, Comment #6, Comment #9
>     - Remediation
>       - 2017-08-24 - CA patched

Bug #1397968

> 
> 11) Inteso San Paulo
>   a) Double dot characters
>     - See Comment #3, Comment #5, Comment #6, Comment #9
>     - Remediation
>       - None performed

Bug #1397969
Flags: needinfo?(kwilson)
Flags: needinfo?(gerv)
(In reply to Kathleen Wilson from comment #19)
> > 4) DigiCert
> >   a) Metadata in OU fields
> >     - See Comment #0, Comment #4, Comment #6
> >     - Remediation
> >       - 2017-08-16 - DigiCert systems patched
> > 
> 
> 
> Will keep this one in this bug, Bug #1389172
> For the DigiCert "Metadata in OU fields" problem listed above, please
> provide an incident report as described here:
> https://wiki.mozilla.org/CA/Responding_To_A_Misissuance#Incident_Report

This was originally noted as "Issue 2b"

Jeremy, Ben - can one of you make sure there's a consistent reply for the incident report (mentioned with the above template)? I think many of the seeds of the information are here, I think if we can get a consistent report here we can look at closing it out.

Thanks.
Status: UNCONFIRMED → NEW
Ever confirmed: true
Flags: needinfo?(jeremy.rowley)
1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, via a discussion in mozilla.dev.security.policy, or via a Bugzilla bug), and the date.
- The issue was posted to the Mozilla dev list on Aug 9, 2017

2. A timeline of the actions your CA took in response.
- We  investigated to see what was going on that same day, responding to Jonathan. Because this is not a SubCA, we were able to post initial findings to that mailing list. We determined this was not a direct violation of the BRs as it states "All other subject attributes MUST contain information that has been verified by the CA". OU is not an "other subject attribute" and is defined in the section immediately prior to this. As such, we have not revoked these certificates.

3. Confirmation that your CA has stopped issuing TLS/SSL certificates with the problem.
- Confirmed. DigiCert has patched its systems to detect whether the OU field has metadata and prevents issuance if non-validated information is present. Although I don't think this is actually a violation per the above explanation, we do see value in preventing meta-data from all subject fields, not just other subject fields. 

4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.
- Unknown. Customers can enter data they want in the OU Field. These are screened for addresses and names. We continuously patch our systems as new forms of metadata are found.

5. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.
28838043
29719356
41854275
45002523
42416834
42416853
42416863
44421681
45060999
43131232
43131307
43131498
43132441
48279184
48558476
48558484
48908976
58878082
58931131
68837112
71615331
71632071
78328813
49832975
74933357
76418106
76418197
76590177
76639330
76685188
76685195
81975356
82029229
97931808
97931812
106177929
138900619
32267988
41751647
105794465
15326026
47930798
45048314
10859239
19505358
98797497
98797525
17719410
47773488
40937363
48678004
105227697
83837393
27797536
42000357
42281109
42416587
48992066
49529547
64029766
42440045
23016305
42394669
72557347
165031011

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
- OU is traditionally just a dumping ground for non-validated information. Although the BRs adopted the requirement that no other field should contain meta-data, the requirement does not squarely apply to the OU field under the language (as it's not an "Other Subject" field.  However, rather than fight over nuances with respect to the OU, DigiCert patched the issue and put safeguards in place to prevent re-occurrence.

7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.
- We add new metadata to a blacklist as it is discovered.  We are also moving to a platform where all OU information is verified prior to issuance.
Flags: needinfo?(jeremy.rowley)
(In reply to Jeremy from comment #21)
> 2. A timeline of the actions your CA took in response.
> - We  investigated to see what was going on that same day, responding to
> Jonathan. Because this is not a SubCA, we were able to post initial findings
> to that mailing list. We determined this was not a direct violation of the
> BRs as it states "All other subject attributes MUST contain information that
> has been verified by the CA". OU is not an "other subject attribute" and is
> defined in the section immediately prior to this. As such, we have not
> revoked these certificates.

This isn't a timeline :)

For example, this doesn't include the answer below:

(In reply to Jeremy from comment #21)
> 6. Explanation about how and why the mistakes were made or bugs introduced,
> and how they avoided detection until now.
> - OU is traditionally just a dumping ground for non-validated information.
> Although the BRs adopted the requirement that no other field should contain
> meta-data, the requirement does not squarely apply to the OU field under the
> language (as it's not an "Other Subject" field.  However, rather than fight
> over nuances with respect to the OU, DigiCert patched the issue and put
> safeguards in place to prevent re-occurrence.

As captured in https://wiki.mozilla.org/CA/Responding_To_A_Misissuance#Examples_of_Good_Practice , can you provide a timeline (for example, both https://groups.google.com/d/msg/mozilla.dev.security.policy/vl5eq0PoJxY/W1D4oZ__BwAJ and https://groups.google.com/d/msg/mozilla.dev.security.policy/nMxaxhYb_iY/AmjCI3_ZBwAJ provide concrete dates and times for the set of changes made to the sytem)
Flags: needinfo?(jeremy.rowley)
Got it - Here's the timeline:

1. Aug 9, 2017 - Report was made to the mailing list
2. Aug 9, 2017 - DigiCert responded to mailing list and began investigation
3. Aug 10, 2017 - DigiCert decided this is not a violation as posted on the mailing list:

Section 7.1.4.2.2(J) : 
"All other optional attributes, when present within the subject field, MUST 
contain information that has been verified by the CA. Optional attributes 
MUST NOT contain metadata such as ‘.’, ‘‐‘, and ‘ ‘ 
(i.e.        space) characters, and/or any other indication that the value is 
absent, incomplete, or not applicable."  Because OU is not "all other optional attributes", j doesn't apply. 

However, DigiCert decided that preventing metadata in the OU field was a good thing to do so committed to preventing it going forward.

4. Aug 10, 2017 - OU metadata was added to a blacklist on the CA side that prevented issuance 
5. ETA beginning of Nov, 2017 - DigiCert will validate all OU information prior to inclusion. This will eliminate all future OU metadata that is not currently blacklisted.
Flags: needinfo?(jeremy.rowley)
Thanks. I believe we can close this bug out.
Status: NEW → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED
Want me to go add a similar timeline to the TERENA bug and others ones that are ready to close out? TERENA was patched at the same time (it's the same system as ours).
If the other bugs do not have complete details, please do update them :)
You need to log in before you can comment on or make changes to this bug.