Closed Bug 1483715 Opened Last year Closed 10 months ago

DigiCert: improper domain validation

Categories

(NSS :: CA Certificate Compliance, task)

task
Not set

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: jeremy.rowley, Assigned: jeremy.rowley)

Details

(Whiteboard: [ca-compliance])

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36

Steps to reproduce:

1.	How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

On 2018/08/07 at 17:00 UTC, a customer submitted a request for information about our validation process for the verification of four of their domains. Upon investigation, we found that the four domains were not properly validated using a post-Aug 1 domain validation method.  When attempting to revalidated the domains prior to August 1, the random value was sent to an address other than the WHOIS contact. This launched a broader investigation into our overall revalidation efforts. This investigation is ongoing.  
2.	A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.
From approximately February through April 2018, DigiCert permitted some legacy Symantec customers to use Method 1 to validate their domains. Use of the method was subject to manager approval and reserved only for those companies that had urgent replacement deadlines that could not be met with an alternative validation method. Under this process, prior to approval, the validation staff was required to match the WHOIS company information and obtain approval using the WHOIS email address. 

Around April, this process was modified to include a BR-compliant Random Value that the validation staff sent using the WHOIS contact information. Use of the random value indicated acceptance.  Adding the random value effectively transformed the validation from Method 1 to Method 2. The email could include multiple domains with the understanding that the WHOIS contact information had to match each domain listed. 

We believe that in some cases either the validation staff failed to match the WHOIS contact information for each domain listed, approving the certificate solely based on the existing verified registrant info, or the system did not check whether the WHOIS contact information matched the email address used in the original confirmation. 

On Aug 1, 2018, Ballot 218 took effect, deprecating Method 1.

On, August 7, 2018, a customer requested the audit trail of a certificate issued using our new process. Upon review, validation management discovered the validation was improper because the previously verified email contact information did not match the WHOIS contact information.  This discovery created an escalation up to management.

On August 13, 2018, we stopped all issuance based on the process that converted Method 1 validations to Method 2 validations. 

We’re currently investigating and will post an update when we know the number of certificates and more about what went wrong. For now, we know the number of impacted certificates is just under 2,500. We should have a clearer picture shortly, after we have conducted a manual review of all 2,500 certificates.

3.	Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

On August 13, although most of these validations were likely properly completed, we stopped issuance using information converted from Method 1 until completing a more thorough investigation.

4.	A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

Approximately 2,500 certificates are under review for validation issues. We wanted to get the incident report out quickly as an FYI while our investigation continues. We’ll update this section in a final report.

5.	The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

Still under review. We will upload them to the bug once we have a complete list. Because the error was human, we are reviewing each validation to determine whether Method 2 was correctly used.  Once we complete our review, we’ll post a Bugzilla attachment with the links for revoked certificates.

6.	Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

Preliminarily, we believe that the revalidation process that relied on previously verified information failed because  there were not technical controls in place to ensure information integrity and because that the validation staff received insufficient training on the process of uploading WHOIS information., This resulted in validation staff adding some domains to the approval email where the WHOIS contact information didn’t always match all the other domains listed. 
When we performed revalidation of domains in preparation for Ballot 218’s effective date, we considered the existing domain information to be correct and reusable until the normal expiration date. However, this inadvertently blended poor quality validations due to the failure to check WHOIS contact information with properly completed Method 2 validations, allowing certs to issue post Aug 1 without proper domain validation.

7.	List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

We stopped all issuance relying on impacted domains until the domain control is revalidated. We are also investigating each impacted domain’s validation to determine whether the email was sent to the appropriate WHOIS contact. Any certs issued based on an improper validation will be revoked or revalidated using an approved method. 

We will update this issue and post to the Mozilla list when we have a precise number of impacted certs. 

We’re still considering what additional action to take and will post an update when we figure out more about what went wrong.
Assignee: wthayer → jeremy.rowley
Summary: Improper domain validation → DigiCert: improper domain validation
Whiteboard: [ca-compliance]
In all, there were 1233 certs issued using the improper domain val. We're currently working on remediation, which is verifying control with the applicant or revoking.
https://crt.sh/?id=447272346
https://crt.sh/?id=629080809
https://crt.sh/?id=393015397
https://crt.sh/?id=629474593
https://crt.sh/?id=347816816
https://crt.sh/?id=542346159
https://crt.sh/?id=485320643
https://crt.sh/?id=646735319
https://crt.sh/?id=431430133
https://crt.sh/?id=399399474
https://crt.sh/?id=528383167
https://crt.sh/?id=539935167
https://crt.sh/?id=404380456
https://crt.sh/?id=513373795
https://crt.sh/?id=577200852
https://crt.sh/?id=535284330
https://crt.sh/?id=628157748
https://crt.sh/?id=276180690
https://crt.sh/?id=417322937
https://crt.sh/?id=628138831
https://crt.sh/?id=536847873
https://crt.sh/?id=360385306
https://crt.sh/?id=422488795
https://crt.sh/?id=527799442
https://crt.sh/?id=433907520
https://crt.sh/?id=654961282
https://crt.sh/?id=342027911
https://crt.sh/?id=637861914
https://crt.sh/?id=367674763
https://crt.sh/?id=647013578
https://crt.sh/?id=646734933
https://crt.sh/?id=391182110
https://crt.sh/?id=341960784
https://crt.sh/?id=627567643
https://crt.sh/?id=347720808
https://crt.sh/?id=606727334
https://crt.sh/?id=319427196
https://crt.sh/?id=345495945
https://crt.sh/?id=528499670
https://crt.sh/?id=628099698
https://crt.sh/?id=628099711
https://crt.sh/?id=638558126
https://crt.sh/?id=636666745
https://crt.sh/?id=350960062
https://crt.sh/?id=479040545
https://crt.sh/?id=370670783
https://crt.sh/?id=528383154
https://crt.sh/?id=628450241
https://crt.sh/?id=456117285
https://crt.sh/?id=355935667
https://crt.sh/?id=630933290
https://crt.sh/?id=593290954
https://crt.sh/?id=528424018
https://crt.sh/?id=528674716
https://crt.sh/?id=616673932
https://crt.sh/?id=541857017
https://crt.sh/?id=528674723
https://crt.sh/?id=420128378
https://crt.sh/?id=655652554
https://crt.sh/?id=499712081
https://crt.sh/?id=454105054
https://crt.sh/?id=539934733
https://crt.sh/?id=400250852
https://crt.sh/?id=635742221
https://crt.sh/?id=343467688
https://crt.sh/?id=628877506
https://crt.sh/?id=638068807
https://crt.sh/?id=380966213
https://crt.sh/?id=597017911
https://crt.sh/?id=531434920
https://crt.sh/?id=404416912
https://crt.sh/?id=391181448
https://crt.sh/?id=400976389
https://crt.sh/?id=469260704
https://crt.sh/?id=630761424
https://crt.sh/?id=325756999
https://crt.sh/?id=414863543
https://crt.sh/?id=531005664
https://crt.sh/?id=649149334
https://crt.sh/?id=542346219
https://crt.sh/?id=351223684
https://crt.sh/?id=663842748
https://crt.sh/?id=359538436
https://crt.sh/?id=317697810
https://crt.sh/?id=610404819
https://crt.sh/?id=653226796
https://crt.sh/?id=630098197
https://crt.sh/?id=523434986
https://crt.sh/?id=628389801
https://crt.sh/?id=333175265
https://crt.sh/?id=617412416
https://crt.sh/?id=649625121
https://crt.sh/?id=350979922
https://crt.sh/?id=533413436
https://crt.sh/?id=372324385
https://crt.sh/?id=651246456
https://crt.sh/?id=597017904
https://crt.sh/?id=390695988
https://crt.sh/?id=329573041
https://crt.sh/?id=638276120
https://crt.sh/?id=461686953
https://crt.sh/?id=535036689
https://crt.sh/?id=324829788
https://crt.sh/?id=513355695
https://crt.sh/?id=528501482
https://crt.sh/?id=527897742
https://crt.sh/?id=399399649
https://crt.sh/?id=445744072
https://crt.sh/?id=531005661
https://crt.sh/?id=627868278
https://crt.sh/?id=528674725
https://crt.sh/?id=628845877
https://crt.sh/?id=628099728
https://crt.sh/?id=620282902
https://crt.sh/?id=472605073
https://crt.sh/?id=417424064
https://crt.sh/?id=541888665
https://crt.sh/?id=646739127
https://crt.sh/?id=513069032
https://crt.sh/?id=390696231
https://crt.sh/?id=431344129
https://crt.sh/?id=470337877
https://crt.sh/?id=311838004
https://crt.sh/?id=302399700
https://crt.sh/?id=344968953
https://crt.sh/?id=541888192
https://crt.sh/?id=602060547
https://crt.sh/?id=354356092
https://crt.sh/?id=333176665
https://crt.sh/?id=519084723
https://crt.sh/?id=596983793
https://crt.sh/?id=508839567
https://crt.sh/?id=461686974
https://crt.sh/?id=480579777
https://crt.sh/?id=541858089
https://crt.sh/?id=635991037
https://crt.sh/?id=361518723
https://crt.sh/?id=611724616
https://crt.sh/?id=522585874
https://crt.sh/?id=399501716
https://crt.sh/?id=630523627
https://crt.sh/?id=445736473
https://crt.sh/?id=404380861
https://crt.sh/?id=350952589
https://crt.sh/?id=379987174
https://crt.sh/?id=649625175
https://crt.sh/?id=648579614
https://crt.sh/?id=528674821
https://crt.sh/?id=635617618
https://crt.sh/?id=528757816
https://crt.sh/?id=400250989
https://crt.sh/?id=638669433
https://crt.sh/?id=370670385
https://crt.sh/?id=511828405
https://crt.sh/?id=463175591
https://crt.sh/?id=638339233
https://crt.sh/?id=463383299
https://crt.sh/?id=399517639
https://crt.sh/?id=638669428
https://crt.sh/?id=316184425
https://crt.sh/?id=575528105
https://crt.sh/?id=649338435
https://crt.sh/?id=341571268
https://crt.sh/?id=662338175
https://crt.sh/?id=582875498
https://crt.sh/?id=465993410
https://crt.sh/?id=350959131
https://crt.sh/?id=527899237
https://crt.sh/?id=628920074
https://crt.sh/?id=577200847
https://crt.sh/?id=575964996
https://crt.sh/?id=325757061
https://crt.sh/?id=662566050
https://crt.sh/?id=635535035
https://crt.sh/?id=627857792
https://crt.sh/?id=575952016
https://crt.sh/?id=663809902
https://crt.sh/?id=620283920
https://crt.sh/?id=541888447
https://crt.sh/?id=663402252
https://crt.sh/?id=446519094
https://crt.sh/?id=360385125
https://crt.sh/?id=541888518
https://crt.sh/?id=628467676
https://crt.sh/?id=638173798
https://crt.sh/?id=655726471
https://crt.sh/?id=541889050
https://crt.sh/?id=474444834
https://crt.sh/?id=638307794
https://crt.sh/?id=483340960
https://crt.sh/?id=649303368
https://crt.sh/?id=472584080
https://crt.sh/?id=637849621
https://crt.sh/?id=312433410
https://crt.sh/?id=638357167
https://crt.sh/?id=391181105
https://crt.sh/?id=399501672
https://crt.sh/?id=573392094
https://crt.sh/?id=465242315
https://crt.sh/?id=636118953
https://crt.sh/?id=528674707
https://crt.sh/?id=575895347
https://crt.sh/?id=561164464
https://crt.sh/?id=528383048
https://crt.sh/?id=370670719
https://crt.sh/?id=513039307
https://crt.sh/?id=328526952
https://crt.sh/?id=370670625
https://crt.sh/?id=629113154
https://crt.sh/?id=575875566
https://crt.sh/?id=648692728
https://crt.sh/?id=341975218
https://crt.sh/?id=399501629
https://crt.sh/?id=620418624
https://crt.sh/?id=347126450
https://crt.sh/?id=360391352
https://crt.sh/?id=606771919
https://crt.sh/?id=627539995
https://crt.sh/?id=627794777
https://crt.sh/?id=539936539
https://crt.sh/?id=447005696
https://crt.sh/?id=585098234
https://crt.sh/?id=649152701
https://crt.sh/?id=493178437
https://crt.sh/?id=466967548
https://crt.sh/?id=649338353
https://crt.sh/?id=451711281
https://crt.sh/?id=614494356
https://crt.sh/?id=451220232
https://crt.sh/?id=463098049
https://crt.sh/?id=461687052
https://crt.sh/?id=649333709
https://crt.sh/?id=474444831
https://crt.sh/?id=400976346
https://crt.sh/?id=341948028
https://crt.sh/?id=662337455
https://crt.sh/?id=628068923
https://crt.sh/?id=561749349
https://crt.sh/?id=466316608
https://crt.sh/?id=404381320
https://crt.sh/?id=399399441
https://crt.sh/?id=628525139
https://crt.sh/?id=316828831
https://crt.sh/?id=652129446
https://crt.sh/?id=357487320
https://crt.sh/?id=368889859
https://crt.sh/?id=527763985
https://crt.sh/?id=635535019
https://crt.sh/?id=368950130
https://crt.sh/?id=433903741
https://crt.sh/?id=536847504
https://crt.sh/?id=611552776
https://crt.sh/?id=314362744
https://crt.sh/?id=360384959
https://crt.sh/?id=350992139
https://crt.sh/?id=415006762
https://crt.sh/?id=582875500
https://crt.sh/?id=541887948
https://crt.sh/?id=595803809
https://crt.sh/?id=370670425
https://crt.sh/?id=610092745
https://crt.sh/?id=539934564
https://crt.sh/?id=592647669
https://crt.sh/?id=539936910
https://crt.sh/?id=329451018
https://crt.sh/?id=655750282
https://crt.sh/?id=559647923
https://crt.sh/?id=649804550
https://crt.sh/?id=494989810
https://crt.sh/?id=568212526
https://crt.sh/?id=628099739
https://crt.sh/?id=364143888
https://crt.sh/?id=495479962
https://crt.sh/?id=360391439
https://crt.sh/?id=635471145
https://crt.sh/?id=470378861
https://crt.sh/?id=534538158
https://crt.sh/?id=356232960
https://crt.sh/?id=541857966
https://crt.sh/?id=333177669
https://crt.sh/?id=627705964
https://crt.sh/?id=356252566
https://crt.sh/?id=431433121
https://crt.sh/?id=641355258
https://crt.sh/?id=541888876
https://crt.sh/?id=533171463
https://crt.sh/?id=648776508
https://crt.sh/?id=489914984
https://crt.sh/?id=492979136
https://crt.sh/?id=543087939
https://crt.sh/?id=620284368
https://crt.sh/?id=533413439
https://crt.sh/?id=341960528
https://crt.sh/?id=453904231
https://crt.sh/?id=541888989
https://crt.sh/?id=391181682
https://crt.sh/?id=657760481
https://crt.sh/?id=466316557
https://crt.sh/?id=660807483
https://crt.sh/?id=575887041
https://crt.sh/?id=326116423
https://crt.sh/?id=606400503
https://crt.sh/?id=333174432
https://crt.sh/?id=393099918
https://crt.sh/?id=611719918
https://crt.sh/?id=528674728
https://crt.sh/?id=390695760
https://crt.sh/?id=527688307
https://crt.sh/?id=637113931
https://crt.sh/?id=648059113
https://crt.sh/?id=393048617
https://crt.sh/?id=325757059
https://crt.sh/?id=511828411
https://crt.sh/?id=347884041
https://crt.sh/?id=404417540
https://crt.sh/?id=541882855
https://crt.sh/?id=347671628
https://crt.sh/?id=417975630
https://crt.sh/?id=394010431
https://crt.sh/?id=391180956
https://crt.sh/?id=380562277
https://crt.sh/?id=637769180
https://crt.sh/?id=391180768
https://crt.sh/?id=497654710
https://crt.sh/?id=649187984
https://crt.sh/?id=308382967
https://crt.sh/?id=580141302
https://crt.sh/?id=417966140
https://crt.sh/?id=622013966
https://crt.sh/?id=311837901
https://crt.sh/?id=476149646
https://crt.sh/?id=388740460
https://crt.sh/?id=470336733
https://crt.sh/?id=329480432
https://crt.sh/?id=404417615
https://crt.sh/?id=513068214
https://crt.sh/?id=528342031
https://crt.sh/?id=470337309
https://crt.sh/?id=606460786
https://crt.sh/?id=424985408
https://crt.sh/?id=652866569
https://crt.sh/?id=649034335
https://crt.sh/?id=305226395
https://crt.sh/?id=622313628
https://crt.sh/?id=368804637
https://crt.sh/?id=541856859
https://crt.sh/?id=625504691
https://crt.sh/?id=627610049
https://crt.sh/?id=399517188
https://crt.sh/?id=479039430
https://crt.sh/?id=628409277
https://crt.sh/?id=499710575
https://crt.sh/?id=306692846
https://crt.sh/?id=636118962
https://crt.sh/?id=628371029
https://crt.sh/?id=324835753
https://crt.sh/?id=663399161
https://crt.sh/?id=341981135
https://crt.sh/?id=445736235
https://crt.sh/?id=350973065
https://crt.sh/?id=641520602
https://crt.sh/?id=652166292
https://crt.sh/?id=370670692
https://crt.sh/?id=541889391
https://crt.sh/?id=482144437
https://crt.sh/?id=351472839
https://crt.sh/?id=306878540
https://crt.sh/?id=636257521
https://crt.sh/?id=630948235
https://crt.sh/?id=541888034
https://crt.sh/?id=635742299
https://crt.sh/?id=542872086
https://crt.sh/?id=307647017
https://crt.sh/?id=361640155
https://crt.sh/?id=635534983
https://crt.sh/?id=628099747
https://crt.sh/?id=393086420
https://crt.sh/?id=638356363
https://crt.sh/?id=431430291
https://crt.sh/?id=606671057
https://crt.sh/?id=610010240
https://crt.sh/?id=638304662
https://crt.sh/?id=553392822
https://crt.sh/?id=602313333
https://crt.sh/?id=649151610
https://crt.sh/?id=431429985
https://crt.sh/?id=542506572
https://crt.sh/?id=628821681
https://crt.sh/?id=645418449
https://crt.sh/?id=393048572
https://crt.sh/?id=542346179
https://crt.sh/?id=317583734
https://crt.sh/?id=521743586
https://crt.sh/?id=638151775
https://crt.sh/?id=649633126
https://crt.sh/?id=606459107
https://crt.sh/?id=476762895
https://crt.sh/?id=393100067
https://crt.sh/?id=596879689
https://crt.sh/?id=337343909
https://crt.sh/?id=638280487
https://crt.sh/?id=414871288
https://crt.sh/?id=527857777
https://crt.sh/?id=635535007
https://crt.sh/?id=593294814
https://crt.sh/?id=541856748
https://crt.sh/?id=628389875
https://crt.sh/?id=541883612
https://crt.sh/?id=541882203
https://crt.sh/?id=435963328
https://crt.sh/?id=637024158
https://crt.sh/?id=475163185
https://crt.sh/?id=627697584
https://crt.sh/?id=638188397
https://crt.sh/?id=345576703
https://crt.sh/?id=332237752
https://crt.sh/?id=649625083
https://crt.sh/?id=635742082
https://crt.sh/?id=415933437
https://crt.sh/?id=649151130
https://crt.sh/?id=638204041
https://crt.sh/?id=638727024
https://crt.sh/?id=433215070
https://crt.sh/?id=655742320
https://crt.sh/?id=498909766
https://crt.sh/?id=393048459
https://crt.sh/?id=657658301
https://crt.sh/?id=655752520
https://crt.sh/?id=541889559
https://crt.sh/?id=357032718
https://crt.sh/?id=353410902
https://crt.sh/?id=627609398
https://crt.sh/?id=391181794
https://crt.sh/?id=649650838
https://crt.sh/?id=354784696
https://crt.sh/?id=638669443
https://crt.sh/?id=461686760
https://crt.sh/?id=606647874
https://crt.sh/?id=540243634
https://crt.sh/?id=636636673
https://crt.sh/?id=541889110
https://crt.sh/?id=393067878
https://crt.sh/?id=311837960
https://crt.sh/?id=390694981
https://crt.sh/?id=528674706
https://crt.sh/?id=628761954
https://crt.sh/?id=552449135
https://crt.sh/?id=372335059
https://crt.sh/?id=606468924
https://crt.sh/?id=637742698
https://crt.sh/?id=628170201
https://crt.sh/?id=363459804
https://crt.sh/?id=341950839
https://crt.sh/?id=492936803
https://crt.sh/?id=628316821
https://crt.sh/?id=317377094
https://crt.sh/?id=451239558
https://crt.sh/?id=470339217
https://crt.sh/?id=470352282
https://crt.sh/?id=374382683
https://crt.sh/?id=636118969
https://crt.sh/?id=351001693
https://crt.sh/?id=621354852
https://crt.sh/?id=575642190
https://crt.sh/?id=375501759
https://crt.sh/?id=649261817
https://crt.sh/?id=494602802
https://crt.sh/?id=651738147
https://crt.sh/?id=312491357
https://crt.sh/?id=324290031
https://crt.sh/?id=485309036
https://crt.sh/?id=512127430
https://crt.sh/?id=654563525
https://crt.sh/?id=317697787
https://crt.sh/?id=614493018
https://crt.sh/?id=341818964
https://crt.sh/?id=337518028
https://crt.sh/?id=393048655
https://crt.sh/?id=342026286
https://crt.sh/?id=399399497
https://crt.sh/?id=614494235
https://crt.sh/?id=648545234
https://crt.sh/?id=311837771
https://crt.sh/?id=606876673
https://crt.sh/?id=487090495
https://crt.sh/?id=361564577
https://crt.sh/?id=505726176
https://crt.sh/?id=539958619
https://crt.sh/?id=575906524
https://crt.sh/?id=611738690
https://crt.sh/?id=625504689
https://crt.sh/?id=628158036
https://crt.sh/?id=399399686
https://crt.sh/?id=409968798
https://crt.sh/?id=328531849
https://crt.sh/?id=372339174
https://crt.sh/?id=493178429
https://crt.sh/?id=483018569
https://crt.sh/?id=360385532
https://crt.sh/?id=405853491
https://crt.sh/?id=573395251
https://crt.sh/?id=311837422
https://crt.sh/?id=638098648
https://crt.sh/?id=653462443
https://crt.sh/?id=568587350
https://crt.sh/?id=541856594
https://crt.sh/?id=662339114
https://crt.sh/?id=328525901
https://crt.sh/?id=399399397
https://crt.sh/?id=539716353
https://crt.sh/?id=399501590
https://crt.sh/?id=598087170
https://crt.sh/?id=476812342
https://crt.sh/?id=630129949
https://crt.sh/?id=614495433
https://crt.sh/?id=470338041
https://crt.sh/?id=470337413
https://crt.sh/?id=349362509
https://crt.sh/?id=662088101
https://crt.sh/?id=485320641
https://crt.sh/?id=476820243
https://crt.sh/?id=636434780
https://crt.sh/?id=662564517
https://crt.sh/?id=339091781
https://crt.sh/?id=339069258
https://crt.sh/?id=308282779
https://crt.sh/?id=542693831
https://crt.sh/?id=635900681
https://crt.sh/?id=627752227
https://crt.sh/?id=393086670
https://crt.sh/?id=574275562
https://crt.sh/?id=539763494
https://crt.sh/?id=431430234
https://crt.sh/?id=420128017
https://crt.sh/?id=355100231
https://crt.sh/?id=400251143
https://crt.sh/?id=610880752
https://crt.sh/?id=611600808
https://crt.sh/?id=612155273
https://crt.sh/?id=487390909
https://crt.sh/?id=451710469
https://crt.sh/?id=476727264
https://crt.sh/?id=405852487
https://crt.sh/?id=313458149
https://crt.sh/?id=627903814
https://crt.sh/?id=325825146
https://crt.sh/?id=628838183
https://crt.sh/?id=316828002
https://crt.sh/?id=328525646
https://crt.sh/?id=330397308
https://crt.sh/?id=628882127
https://crt.sh/?id=492979120
https://crt.sh/?id=606671947
https://crt.sh/?id=308528766
https://crt.sh/?id=541889240
https://crt.sh/?id=484984774
https://crt.sh/?id=431430193
https://crt.sh/?id=566575595
https://crt.sh/?id=628086637
https://crt.sh/?id=333174356
https://crt.sh/?id=649338254
https://crt.sh/?id=505725399
https://crt.sh/?id=638525513
https://crt.sh/?id=399113860
https://crt.sh/?id=649150989
https://crt.sh/?id=542344582
https://crt.sh/?id=635742104
https://crt.sh/?id=635617503
https://crt.sh/?id=391181271
https://crt.sh/?id=606622136
https://crt.sh/?id=611729030
https://crt.sh/?id=512155426
https://crt.sh/?id=470337966
https://crt.sh/?id=636118983
https://crt.sh/?id=638553961
https://crt.sh/?id=393099963
https://crt.sh/?id=497272118
https://crt.sh/?id=638215957
https://crt.sh/?id=399399248
https://crt.sh/?id=542062547
https://crt.sh/?id=655745575
https://crt.sh/?id=623723465
https://crt.sh/?id=393100138
https://crt.sh/?id=536515968
https://crt.sh/?id=487390913
https://crt.sh/?id=664289923
https://crt.sh/?id=662327657
https://crt.sh/?id=376813915
https://crt.sh/?id=355024373
https://crt.sh/?id=451710955
https://crt.sh/?id=628099757
https://crt.sh/?id=540031989
https://crt.sh/?id=430004271
https://crt.sh/?id=533096025
https://crt.sh/?id=662337579
https://crt.sh/?id=497584764
https://crt.sh/?id=445732184
https://crt.sh/?id=325756336
https://crt.sh/?id=596538521
https://crt.sh/?id=663122051
https://crt.sh/?id=505507724
https://crt.sh/?id=614496101
https://crt.sh/?id=627725792
https://crt.sh/?id=511163512
https://crt.sh/?id=575980882
https://crt.sh/?id=393100105
https://crt.sh/?id=370670579
https://crt.sh/?id=390695301
https://crt.sh/?id=628389852
https://crt.sh/?id=476721755
https://crt.sh/?id=470638248
https://crt.sh/?id=329294688
https://crt.sh/?id=628932695
https://crt.sh/?id=637829763
https://crt.sh/?id=637901051
https://crt.sh/?id=400252630
https://crt.sh/?id=349410155
https://crt.sh/?id=606688000
https://crt.sh/?id=390696505
https://crt.sh/?id=628973982
https://crt.sh/?id=508854187
https://crt.sh/?id=628099763
https://crt.sh/?id=493792562
https://crt.sh/?id=350938471
https://crt.sh/?id=480061472
https://crt.sh/?id=489392786
https://crt.sh/?id=372241304
https://crt.sh/?id=561009481
https://crt.sh/?id=638117632
https://crt.sh/?id=636118988
https://crt.sh/?id=326494312
https://crt.sh/?id=625759336
https://crt.sh/?id=399487468
https://crt.sh/?id=399399724
https://crt.sh/?id=360385015
https://crt.sh/?id=399399261
https://crt.sh/?id=475070863
https://crt.sh/?id=302982096
https://crt.sh/?id=543529143
https://crt.sh/?id=648875383
https://crt.sh/?id=485981384
https://crt.sh/?id=527896584
https://crt.sh/?id=649577570
https://crt.sh/?id=472584083
https://crt.sh/?id=611737580
https://crt.sh/?id=539935063
https://crt.sh/?id=541888127
https://crt.sh/?id=511828408
https://crt.sh/?id=528674718
https://crt.sh/?id=497946781
https://crt.sh/?id=453806353
https://crt.sh/?id=537103450
https://crt.sh/?id=655964351
https://crt.sh/?id=473008145
https://crt.sh/?id=349393297
https://crt.sh/?id=462640910
https://crt.sh/?id=364159728
https://crt.sh/?id=628552553
https://crt.sh/?id=664944220
https://crt.sh/?id=311837842
https://crt.sh/?id=339236242
https://crt.sh/?id=479321347
https://crt.sh/?id=638809027
https://crt.sh/?id=628099778
https://crt.sh/?id=319427090
https://crt.sh/?id=628467470
https://crt.sh/?id=541961935
https://crt.sh/?id=471030189
https://crt.sh/?id=316828374
https://crt.sh/?id=541889507
https://crt.sh/?id=406072800
https://crt.sh/?id=575861272
https://crt.sh/?id=404418043
https://crt.sh/?id=461934085
https://crt.sh/?id=360404401
https://crt.sh/?id=480062945
https://crt.sh/?id=541857339
https://crt.sh/?id=629104217
https://crt.sh/?id=479216194
https://crt.sh/?id=629080504
https://crt.sh/?id=655661543
https://crt.sh/?id=602380330
https://crt.sh/?id=648692825
https://crt.sh/?id=620416027
https://crt.sh/?id=574425245
https://crt.sh/?id=307194659
https://crt.sh/?id=634874914
https://crt.sh/?id=638601400
https://crt.sh/?id=299307274
https://crt.sh/?id=505512406
https://crt.sh/?id=637167367
https://crt.sh/?id=496092251
https://crt.sh/?id=497584686
https://crt.sh/?id=291194975
https://crt.sh/?id=539935746
https://crt.sh/?id=399399609
https://crt.sh/?id=325903986
https://crt.sh/?id=628446495
https://crt.sh/?id=539096974
https://crt.sh/?id=662338030
https://crt.sh/?id=328525385
https://crt.sh/?id=620158103
https://crt.sh/?id=646890111
https://crt.sh/?id=372241109
https://crt.sh/?id=541857903
https://crt.sh/?id=336408093
https://crt.sh/?id=575937053
https://crt.sh/?id=465993415
https://crt.sh/?id=431429824
https://crt.sh/?id=646967819
https://crt.sh/?id=621720729
https://crt.sh/?id=370670462
https://crt.sh/?id=514126843
https://crt.sh/?id=635742204
https://crt.sh/?id=404417092
https://crt.sh/?id=534526844
https://crt.sh/?id=349371090
https://crt.sh/?id=329495911
https://crt.sh/?id=620950535
https://crt.sh/?id=539788961
https://crt.sh/?id=612926725
https://crt.sh/?id=393048680
https://crt.sh/?id=649151485
https://crt.sh/?id=626798033
https://crt.sh/?id=628099680
https://crt.sh/?id=524070249
https://crt.sh/?id=453818067
https://crt.sh/?id=350981233
https://crt.sh/?id=312491362
https://crt.sh/?id=550125177
https://crt.sh/?id=393048527
https://crt.sh/?id=354507226
https://crt.sh/?id=568214038
https://crt.sh/?id=606622128
https://crt.sh/?id=508783831
https://crt.sh/?id=472584092
https://crt.sh/?id=551765737
https://crt.sh/?id=303120467
https://crt.sh/?id=551478996
https://crt.sh/?id=347917529
https://crt.sh/?id=628802467
https://crt.sh/?id=620025155
https://crt.sh/?id=627609990
https://crt.sh/?id=511828410
https://crt.sh/?id=542279296
https://crt.sh/?id=414912297
https://crt.sh/?id=400251014
https://crt.sh/?id=311837541
https://crt.sh/?id=351261190
https://crt.sh/?id=370670816
https://crt.sh/?id=575886894
https://crt.sh/?id=576573389
https://crt.sh/?id=406073004
https://crt.sh/?id=505562851
https://crt.sh/?id=527900427
https://crt.sh/?id=638678315
https://crt.sh/?id=332089321
https://crt.sh/?id=628551663
https://crt.sh/?id=575674650
https://crt.sh/?id=638101920
https://crt.sh/?id=638112740
https://crt.sh/?id=646778552
https://crt.sh/?id=445734290
https://crt.sh/?id=277199550
https://crt.sh/?id=653945200
https://crt.sh/?id=343467292
https://crt.sh/?id=662020028
https://crt.sh/?id=541888379
https://crt.sh/?id=637996299
https://crt.sh/?id=606671441
https://crt.sh/?id=333175038
https://crt.sh/?id=628099700
https://crt.sh/?id=344968945
https://crt.sh/?id=399517614
https://crt.sh/?id=627017639
https://crt.sh/?id=638231552
https://crt.sh/?id=601075945
https://crt.sh/?id=404417446
https://crt.sh/?id=664944130
https://crt.sh/?id=470338504
https://crt.sh/?id=362305160
https://crt.sh/?id=404381520
https://crt.sh/?id=284082513
https://crt.sh/?id=451205002
https://crt.sh/?id=547714198
https://crt.sh/?id=540468109
https://crt.sh/?id=606850694
https://crt.sh/?id=346349346
https://crt.sh/?id=592647878
https://crt.sh/?id=575895630
https://crt.sh/?id=628492711
https://crt.sh/?id=628133185
https://crt.sh/?id=659520124
https://crt.sh/?id=575896458
https://crt.sh/?id=533413434
https://crt.sh/?id=368804699
https://crt.sh/?id=485250040
https://crt.sh/?id=612149440
https://crt.sh/?id=391498055
https://crt.sh/?id=628099721
https://crt.sh/?id=307701122
https://crt.sh/?id=542713232
https://crt.sh/?id=630862195
https://crt.sh/?id=627273319
https://crt.sh/?id=332749215
https://crt.sh/?id=638810956
https://crt.sh/?id=399399594
https://crt.sh/?id=540242228
https://crt.sh/?id=541889627
https://crt.sh/?id=541888079
https://crt.sh/?id=645078057
https://crt.sh/?id=399501754
https://crt.sh/?id=334475772
https://crt.sh/?id=453806348
https://crt.sh/?id=575861935
https://crt.sh/?id=498522495
https://crt.sh/?id=474444836
https://crt.sh/?id=470337770
https://crt.sh/?id=363983242
https://crt.sh/?id=628447520
https://crt.sh/?id=541954059
https://crt.sh/?id=635063464
https://crt.sh/?id=611746451
https://crt.sh/?id=516013380
https://crt.sh/?id=455984014
https://crt.sh/?id=664993473
https://crt.sh/?id=431430042
https://crt.sh/?id=311837595
https://crt.sh/?id=508905529
https://crt.sh/?id=393048820
https://crt.sh/?id=378217794
https://crt.sh/?id=405858535
https://crt.sh/?id=606671598
https://crt.sh/?id=606637593
https://crt.sh/?id=349263797
https://crt.sh/?id=372338938
https://crt.sh/?id=607226455
https://crt.sh/?id=620158023
https://crt.sh/?id=649546398
https://crt.sh/?id=451200673
https://crt.sh/?id=391182348
https://crt.sh/?id=528341847
https://crt.sh/?id=628099731
https://crt.sh/?id=349510302
https://crt.sh/?id=353629306
https://crt.sh/?id=511828413
https://crt.sh/?id=637104902
https://crt.sh/?id=541889472
https://crt.sh/?id=649546543
https://crt.sh/?id=594327056
https://crt.sh/?id=663396026
https://crt.sh/?id=630462695
https://crt.sh/?id=566508716
https://crt.sh/?id=498917243
https://crt.sh/?id=635742166
https://crt.sh/?id=361177847
https://crt.sh/?id=575877451
https://crt.sh/?id=649625106
https://crt.sh/?id=533413442
https://crt.sh/?id=636432902
https://crt.sh/?id=351022095
https://crt.sh/?id=627619560
https://crt.sh/?id=644387351
https://crt.sh/?id=468718794
https://crt.sh/?id=420128014
https://crt.sh/?id=285936900
https://crt.sh/?id=380561269
https://crt.sh/?id=629018578
https://crt.sh/?id=370670666
https://crt.sh/?id=580158112
https://crt.sh/?id=513556906
https://crt.sh/?id=498916894
https://crt.sh/?id=541889534
https://crt.sh/?id=433889043
https://crt.sh/?id=604935920
https://crt.sh/?id=376763328
https://crt.sh/?id=511828414
https://crt.sh/?id=628486320
https://crt.sh/?id=351212479
https://crt.sh/?id=305696133
https://crt.sh/?id=493792877
https://crt.sh/?id=541857169
https://crt.sh/?id=606726399
https://crt.sh/?id=491422904
https://crt.sh/?id=349268115
https://crt.sh/?id=392048104
https://crt.sh/?id=469274631
https://crt.sh/?id=404381798
https://crt.sh/?id=596324986
https://crt.sh/?id=325756997
https://crt.sh/?id=324290073
https://crt.sh/?id=638739364
https://crt.sh/?id=542346244
https://crt.sh/?id=541888835
https://crt.sh/?id=482759717
https://crt.sh/?id=311118443
https://crt.sh/?id=638667135
https://crt.sh/?id=628553375
https://crt.sh/?id=637790510
https://crt.sh/?id=627857797
https://crt.sh/?id=596983646
https://crt.sh/?id=630200489
https://crt.sh/?id=351015554
https://crt.sh/?id=627567531
https://crt.sh/?id=495479966
https://crt.sh/?id=528674712
https://crt.sh/?id=593295859
https://crt.sh/?id=493170517
https://crt.sh/?id=363543255
https://crt.sh/?id=393048676
https://crt.sh/?id=655758566
https://crt.sh/?id=470339176
https://crt.sh/?id=654586932
https://crt.sh/?id=637996327
https://crt.sh/?id=606883375
https://crt.sh/?id=451710039
https://crt.sh/?id=629889553
https://crt.sh/?id=655696656
https://crt.sh/?id=638151669
https://crt.sh/?id=541888292
https://crt.sh/?id=484599565
https://crt.sh/?id=348925513
https://crt.sh/?id=542042470
https://crt.sh/?id=628852187
https://crt.sh/?id=634875830
https://crt.sh/?id=541255365
https://crt.sh/?id=628577234
https://crt.sh/?id=326494329
https://crt.sh/?id=613801741
https://crt.sh/?id=549558317
https://crt.sh/?id=311828233
https://crt.sh/?id=508813129
https://crt.sh/?id=648692914
https://crt.sh/?id=404380690
https://crt.sh/?id=575887730
https://crt.sh/?id=627127261
https://crt.sh/?id=497441413
https://crt.sh/?id=662087606
https://crt.sh/?id=628644511
https://crt.sh/?id=462576876
https://crt.sh/?id=497542565
https://crt.sh/?id=329496567
https://crt.sh/?id=528383185
https://crt.sh/?id=528674731
https://crt.sh/?id=604813792
https://crt.sh/?id=648931087
https://crt.sh/?id=636119000
https://crt.sh/?id=606617214
https://crt.sh/?id=638568816
https://crt.sh/?id=368798495
https://crt.sh/?id=485320645
https://crt.sh/?id=404416791
https://crt.sh/?id=521040876
https://crt.sh/?id=575989928
https://crt.sh/?id=537927538
https://crt.sh/?id=404648445
https://crt.sh/?id=591207122
https://crt.sh/?id=636296852
https://crt.sh/?id=540242234
https://crt.sh/?id=651732649
https://crt.sh/?id=655639078
https://crt.sh/?id=641520607
https://crt.sh/?id=615718396
https://crt.sh/?id=537927953
https://crt.sh/?id=590122515
https://crt.sh/?id=628386877
https://crt.sh/?id=368889611
https://crt.sh/?id=351486707
https://crt.sh/?id=409967924
https://crt.sh/?id=404418157
https://crt.sh/?id=474444827
https://crt.sh/?id=470848515
https://crt.sh/?id=531091702
https://crt.sh/?id=388370125
https://crt.sh/?id=549533499
https://crt.sh/?id=440521534
https://crt.sh/?id=628644353
https://crt.sh/?id=542438808
https://crt.sh/?id=620285111
https://crt.sh/?id=606622095
https://crt.sh/?id=307160586
https://crt.sh/?id=378217777
https://crt.sh/?id=628169380
https://crt.sh/?id=328928890
https://crt.sh/?id=351004855
https://crt.sh/?id=415595210
https://crt.sh/?id=410136512
https://crt.sh/?id=648692763
https://crt.sh/?id=606622113
https://crt.sh/?id=370670756
https://crt.sh/?id=393048936
https://crt.sh/?id=621458141
https://crt.sh/?id=628874592
https://crt.sh/?id=508837501
https://crt.sh/?id=370674908
https://crt.sh/?id=636296439
https://crt.sh/?id=354946848
https://crt.sh/?id=372169668
https://crt.sh/?id=499795327
https://crt.sh/?id=461686927
https://crt.sh/?id=355547138
https://crt.sh/?id=347716260
https://crt.sh/?id=612149280
https://crt.sh/?id=465991590
https://crt.sh/?id=522584002
https://crt.sh/?id=508030527
https://crt.sh/?id=627611862
https://crt.sh/?id=344820986
https://crt.sh/?id=372309514
https://crt.sh/?id=368798521
https://crt.sh/?id=647774865
https://crt.sh/?id=404417857
https://crt.sh/?id=648824099
https://crt.sh/?id=649547504
https://crt.sh/?id=614051959
https://crt.sh/?id=336221298
https://crt.sh/?id=527827771
https://crt.sh/?id=470112850
https://crt.sh/?id=474444838
https://crt.sh/?id=364001624
https://crt.sh/?id=399517267
https://crt.sh/?id=636296308
https://crt.sh/?id=539716362
https://crt.sh/?id=620282552
https://crt.sh/?id=606442633
https://crt.sh/?id=628099743
https://crt.sh/?id=326445651
https://crt.sh/?id=332749333
https://crt.sh/?id=380562493
https://crt.sh/?id=611733214
https://crt.sh/?id=390695118
https://crt.sh/?id=420439772
https://crt.sh/?id=636118944
https://crt.sh/?id=649034864
https://crt.sh/?id=485308069
https://crt.sh/?id=393039114
https://crt.sh/?id=628029268
https://crt.sh/?id=522585917
https://crt.sh/?id=391464413
https://crt.sh/?id=337213254
https://crt.sh/?id=318212268
https://crt.sh/?id=354322991
https://crt.sh/?id=528758908
https://crt.sh/?id=369643415
https://crt.sh/?id=468071323
https://crt.sh/?id=539959843
https://crt.sh/?id=393039119
https://crt.sh/?id=651042327
https://crt.sh/?id=577577495
https://crt.sh/?id=490833013
https://crt.sh/?id=539875008
https://crt.sh/?id=655725080
https://crt.sh/?id=406072907
https://crt.sh/?id=596983796
https://crt.sh/?id=311838133
https://crt.sh/?id=329625668
https://crt.sh/?id=390694754
https://crt.sh/?id=539846725
https://crt.sh/?id=357032704
https://crt.sh/?id=638307620
https://crt.sh/?id=648841933
https://crt.sh/?id=646966393
https://crt.sh/?id=404380968
https://crt.sh/?id=507452649
https://crt.sh/?id=390696322
https://crt.sh/?id=497640586
https://crt.sh/?id=393039122
https://crt.sh/?id=470337500
https://crt.sh/?id=470338170
https://crt.sh/?id=399398493
https://crt.sh/?id=344758471
https://crt.sh/?id=417765469
https://crt.sh/?id=541887900
https://crt.sh/?id=357032757
https://crt.sh/?id=392955947
https://crt.sh/?id=345257447
https://crt.sh/?id=431029919
https://crt.sh/?id=422978747
https://crt.sh/?id=641046640
https://crt.sh/?id=351143364
https://crt.sh/?id=354322070
https://crt.sh/?id=606890064
https://crt.sh/?id=575874018
https://crt.sh/?id=641046212
https://crt.sh/?id=513161960
https://crt.sh/?id=414233232
https://crt.sh/?id=551478990
https://crt.sh/?id=461686818
https://crt.sh/?id=343467837
https://crt.sh/?id=628845917
https://crt.sh/?id=316828266
https://crt.sh/?id=662562156
https://crt.sh/?id=636118959
https://crt.sh/?id=329005049
https://crt.sh/?id=400250930
https://crt.sh/?id=345486407
https://crt.sh/?id=537024729
https://crt.sh/?id=575931599
https://crt.sh/?id=638112761
https://crt.sh/?id=404417258
https://crt.sh/?id=637861916
https://crt.sh/?id=628969251
https://crt.sh/?id=399517134
https://crt.sh/?id=453904156
https://crt.sh/?id=388461988
https://crt.sh/?id=654502939
https://crt.sh/?id=329536329
https://crt.sh/?id=332269821
https://crt.sh/?id=368003423
https://crt.sh/?id=653840963
https://crt.sh/?id=508783822
https://crt.sh/?id=576730909
https://crt.sh/?id=628099750
https://crt.sh/?id=466463317
https://crt.sh/?id=491400303
https://crt.sh/?id=508886703
https://crt.sh/?id=347884310
https://crt.sh/?id=507698261
https://crt.sh/?id=596876862
https://crt.sh/?id=491400326
https://crt.sh/?id=541881051
https://crt.sh/?id=636118964
https://crt.sh/?id=350973249
https://crt.sh/?id=635962442
https://crt.sh/?id=380913957
https://crt.sh/?id=362294575
https://crt.sh/?id=351001503
https://crt.sh/?id=540242243
https://crt.sh/?id=330399941
https://crt.sh/?id=474444843
https://crt.sh/?id=513039673
https://crt.sh/?id=580426211
https://crt.sh/?id=361640157
https://crt.sh/?id=466316496
https://crt.sh/?id=350919676
https://crt.sh/?id=383231198
https://crt.sh/?id=647775681
https://crt.sh/?id=646672309
https://crt.sh/?id=628099759
https://crt.sh/?id=655729315
https://crt.sh/?id=313866232
https://crt.sh/?id=607359444
https://crt.sh/?id=648058958
https://crt.sh/?id=294693213
https://crt.sh/?id=415595321
https://crt.sh/?id=636118971
https://crt.sh/?id=311838049
https://crt.sh/?id=453946603
https://crt.sh/?id=646965742
https://crt.sh/?id=483230617
https://crt.sh/?id=353591184
https://crt.sh/?id=637861917
https://crt.sh/?id=628099767
https://crt.sh/?id=351184208
https://crt.sh/?id=476780543
https://crt.sh/?id=652938233
https://crt.sh/?id=399399422
https://crt.sh/?id=361202099
https://crt.sh/?id=370670521
https://crt.sh/?id=638266565
https://crt.sh/?id=329243240
https://crt.sh/?id=645412148
https://crt.sh/?id=400250896
https://crt.sh/?id=332569541
https://crt.sh/?id=649319063
https://crt.sh/?id=575894818
https://crt.sh/?id=493200335
https://crt.sh/?id=367644363
https://crt.sh/?id=333174828
https://crt.sh/?id=313293734
https://crt.sh/?id=472607241
https://crt.sh/?id=317697755
https://crt.sh/?id=393100018
https://crt.sh/?id=487390910
https://crt.sh/?id=649151307
https://crt.sh/?id=414233231
https://crt.sh/?id=638167234
https://crt.sh/?id=647142081
https://crt.sh/?id=485308085
https://crt.sh/?id=477906365
https://crt.sh/?id=637024240
https://crt.sh/?id=351036926
https://crt.sh/?id=629080450
https://crt.sh/?id=593296009
https://crt.sh/?id=299223049
https://crt.sh/?id=400250962
https://crt.sh/?id=646734937
https://crt.sh/?id=334515010
https://crt.sh/?id=364182257
https://crt.sh/?id=628065345
https://crt.sh/?id=575859168
https://crt.sh/?id=608783998
https://crt.sh/?id=518756564
https://crt.sh/?id=638246372
https://crt.sh/?id=630656051
https://crt.sh/?id=638276124
https://crt.sh/?id=345269079
https://crt.sh/?id=390695558
https://crt.sh/?id=391181980
https://crt.sh/?id=541881539
https://crt.sh/?id=527912576
https://crt.sh/?id=376786860
https://crt.sh/?id=277668906
https://crt.sh/?id=341959410
https://crt.sh/?id=533096032
https://crt.sh/?id=630391164
https://crt.sh/?id=464619063
https://crt.sh/?id=533096028
https://crt.sh/?id=495988339
https://crt.sh/?id=406072859
https://crt.sh/?id=655635610
https://crt.sh/?id=341825000
https://crt.sh/?id=637847968
https://crt.sh/?id=463229890
https://crt.sh/?id=404381057
https://crt.sh/?id=404417702
https://crt.sh/?id=662339663
https://crt.sh/?id=294771669
https://crt.sh/?id=393048707
https://crt.sh/?id=551280884
https://crt.sh/?id=636296730
https://crt.sh/?id=463207033
https://crt.sh/?id=483230616
https://crt.sh/?id=393048838
https://crt.sh/?id=507631980
https://crt.sh/?id=614497709
https://crt.sh/?id=318508629
https://crt.sh/?id=429936322
https://crt.sh/?id=475070902
https://crt.sh/?id=470338365
https://crt.sh/?id=621053808
https://crt.sh/?id=470113366
https://crt.sh/?id=542438806
https://crt.sh/?id=311837682
(In reply to Jeremy Rowley from comment #0)

> On 2018/08/07 at 17:00 UTC, a customer submitted a request for information
> about our validation process for the verification of four of their domains.
> Upon investigation, 

What triggered the investigation? When did the investigation begin? Helping understand this can help make the difference between "We got lucky" and "Systemic controls such as X, Y, Z noticed this, and all CAs can benefit from implementing such controls"

> 2.	A timeline of the actions your CA took in response. A timeline is a
> date-and-time-stamped sequence of all relevant events. This may include
> events before the incident was reported, such as when a particular
> requirement became applicable, or a document changed, or a bug was
> introduced, or an audit was done.
> From approximately February through April 2018, DigiCert permitted some
> legacy Symantec customers to use Method 1 to validate their domains. Use of
> the method was subject to manager approval and reserved only for those
> companies that had urgent replacement deadlines that could not be met with
> an alternative validation method. Under this process, prior to approval, the
> validation staff was required to match the WHOIS company information and
> obtain approval using the WHOIS email address. 

Can you expand a bit more to discuss what these controls look like. 

"Prior to April 2018, domain validation was performed using the following steps:
1) A CSR is submitted containing domains
2) For each domain, our systems perform a lookup for the WHOIS information, displaying it to the validation staff.
3) The validation staff copies or transcribes this text into creating an email order using the WHOIS email to obtain authorization to issue on the basis of the Company Match
4) A random value is ..."

Or something to that effect? Help us understand how the existing system of controls were designed.

> Around April, 

"Around April" is very precisely date-and-time-stamped :)

> this process was modified to include a BR-compliant Random
> Value that the validation staff sent using the WHOIS contact information.
> Use of the random value indicated acceptance.  Adding the random value
> effectively transformed the validation from Method 1 to Method 2. The email
> could include multiple domains with the understanding that the WHOIS contact
> information had to match each domain listed. 

Here again, with the above details about how the system 'used' to work, it can be clearer how the system 'does' work.

> We believe that in some cases either the validation staff failed to match
> the WHOIS contact information for each domain listed, approving the
> certificate solely based on the existing verified registrant info, or the
> system did not check whether the WHOIS contact information matched the email
> address used in the original confirmation. 

Here, understanding the exact processes above help understand why it failed. "We believe" doesn't inspire strong confidence, because it suggests that processes or controls are missing from the workflow, or that the binding between requests and domains is a manual process that permits human error, etc.

This is about describing the system architecture to help identify where things went right - and wrong.

> On, August 7, 2018, a customer requested the audit trail of a certificate
> issued using our new process. 

Is this typical? Was this customer reporting unauthorized issuance?

> Upon review, validation management discovered
> the validation was improper because the previously verified email contact
> information did not match the WHOIS contact information.  This discovery
> created an escalation up to management.

When was it escalated? When was it acknowledged? -and-timestamp helps understand what the processes and response times are.

Postmortems help identify where things go right as much as they identify where things went wrong - it's often the things going right that prevent them from being far worse than they could have been, and serve as valuable tools to understand how to build and improve safe systems, which is part of the goal of these processes :)

> We’re currently investigating and will post an update when we know the
> number of certificates and more about what went wrong. For now, we know the
> number of impacted certificates is just under 2,500. We should have a
> clearer picture shortly, after we have conducted a manual review of all
> 2,500 certificates.

What does your manual review entail? By understanding all the above facts, and how the manual review is being done, we can better have confidence that it's "just" those 2,500 certificates.

While it's very good to be responsive and identify the issue, it's also important to be thorough, and by sharing information, we build confidence in the thoroughness.
> What triggered the investigation? When did the investigation begin? Helping understand this can help make the difference between > "We got lucky" and "Systemic controls such as X, Y, Z noticed this, and all CAs can benefit from implementing such controls"

Investigation was triggered by a customer requesting information about the validation process with one of their certs. We started investigating on 08/08/2018 (the day after the customer submitted the request). The one day delay was because we were getting more info on what the customer was actually asking. The controls didn't pick up the issue because everything looked right from the document-side.


> Can you expand a bit more to discuss what these controls look like. 
> 
> "Prior to April 2018, domain validation was performed using the following steps:
> 1) A CSR is submitted containing domains
> 2) For each domain, our systems perform a lookup for the WHOIS information, displaying it to the validation staff.
> 3) The validation staff copies or transcribes this text into creating an email order using the WHOIS email to obtain authorization to issue on the basis of the Company Match
> 4) A random value is ..."
> 
> Or something to that effect? Help us understand how the existing system of controls were designed.

Sure. Note this is the system for legacy Symantec orders which is slightly different than DigiCert certs. That's not intended to be an excuse, but it means this explanation doesn't apply to certs issued through legacy DigiCert systems. We're working on merging the two, but the progress is slow.

The system in question used the following process:
1) Customer submits a domain verification request through their Symantec portal
2) System kicks off email to BR-approved email address
3) If WHOIS is not parse-able (some ccTLDs) or requires captcha, validation staff is alerted to pending (assuming customer does not use a DNS or file based method)
4) Validation staff accesses WHOIS and screenshots information
5) Validation staff uploads WHOIS screenshot and records WHOIS email addresses
6) Validation staff can consolidate orders with the same WHOIS email into a single approval email
7) Validation staff emails WHOIS contact to verify the approval. Under method 1, the validation staff confirmed the org information in WHOIS. A random value was not included (since validation was method 1) 

Added/replaced in April:
7) System generates a random value (per BRs)
8) System kicks off email to the specified WHOIS address(with link to use random value)
9) Customer clicks link, verifies approval, and gets cert

What happened is two fold:
a) Some validation staff did not use the WHOIS contact to obtain approval. Instead they did a "real" method 1 and confirmed the org information in WHOIS matched what was verified during the OV/EV process. However, because the system required a WHOIS email address, an email address for the customer was specified that did not necessarily conform to what WHOIS showed.
b) When we converted the handful of customers verified using method 1 to method 2, we resent the confirmation email with a random value to the specified WHOIS address. Because the WHOIS information was not accurately recorded, the validation email went to an inappropriate email address.

> "Around April" is very precisely date-and-time-stamped :)

Yeah. I couldn't find the exact date we rolled the system random value when I wrote the original disclosure. Turns out the actual dates as March 1, 2018. On that same date, we restricted second approval to management. Previously second approval permitted any two staff to confirm the email address recorded as accurate. This lead to some interesting email addresses approved by validation staff. We restricted this to approved roles and expanded it as people received training.

> Is this typical? Was this customer reporting unauthorized issuance?

Customer requests about validation are not typical. The customer wasn't reporting unauthorized issuance. The customer approved the order through their account but never approved the domain verification and was wondering what process was in place. The request was more curiosity on the process than a complaint about mis-issuance.

> Here again, with the above details about how the system 'used' to work, it can be clearer how the system 'does' work.

I added details above. Let me know if you want more info about any part of that.

> Here, understanding the exact processes above help understand why it failed. "We believe" doesn't inspire strong confidence, > because it suggests that processes or controls are missing from the workflow, or that the binding between requests and domains > is a manual process that permits human error, etc.
> 
> This is about describing the system architecture to help identify where things went right - and wrong.

Right. I said "we believe" because we were still investigating the processes. I have a lot more info now that I can share. The architecture prior to March 1 was poor. We rolled out a significant update on March 1 that locked down permissions and automated the process more.

> When was it escalated? When was it acknowledged? -and-timestamp helps understand what the processes and response times are.

I escalated it to Dev on Aug 13. Prior to that, we were looking through the audit logs of the system to see what we could find out. I can look at my email to find the exact time, but it was in the afternoon. Monday morning was when I was convinced there was a problem.

> Postmortems help identify where things go right as much as they identify where things went wrong - it's often the things going > right that prevent them from being far worse than they could have been, and serve as valuable tools to understand how to build > and improve safe systems, which is part of the goal of these processes :)

Agreed. Happy to answer any questions you have. 

> What does your manual review entail? By understanding all the above facts, and how the manual review is being done, we can better have confidence that it's "just" those 2,500 certificates.

> While it's very good to be responsive and identify the issue, it's also important to be thorough, and by sharing information, > we build confidence in the thoroughness.

Yeah - we were still in investigation phase when I posted. Sorry about the lack of details there. 

The manual review process is going through every validation completed using this system and looking at the documentation to see what was uploaded by the validation staff. This is an exception system so it only applies where the normal, automated WHOIS failed to send an email with a random value (which is why there aren't a billion certs) and where no other method was usable by the customer.

The manual review involves someone trusted (generally an internal auditor) opening up each record, looking at the email address used and ensuring it matched what was in WHOIS. If the email address didn't match, the cert was considered invalid. The certs listed above are the ones where the WHOIS email didn't match the email where the approval was sent.
Hi Jeremy,

> On August 13, 2018, we stopped all issuance based on the process that
> converted Method 1 validations to Method 2 validations. 

in your certificate list you seem to have included certificates issued on or after 13/8/2018. Is this list correct or did some certificates slipped by the cutoff date?

for example
https://crt.sh/?id=649151307
https://crt.sh/?id=646734937
https://crt.sh/?id=647142081
Flags: needinfo?(jeremy.rowley)
Sorry, we stopped permitting that method for additional validations. Existing validations were not invalidated until we went through each one. These were all certs where the validation completed prior to the cutoff date but didn't issue. We've since invalidated all the validations that lacked random values.
Flags: needinfo?(jeremy.rowley)
Jeremy: please provide periodic updates on remediation status and a target completion date if possible. Also, I'm interested in learning what additional actions DigiCert has decided to take, especially in respect to better detection and prevention of this type of issue in the future.
Sure thing. We're currently working on the revocation/recheck. I'll have an update on that next week. I'll post what updates we're working on for better detection and prevention at the same time
Sorry for the delay on this. Everything was revalidated and approved using a BR-compliant method. As suspected, exactly zero certs were revoked. 

For better prevention, we're implementing tools that will eliminate some of the manual process that went into the system where this was used. It's mostly already there because Method 1 and Method 5 are no longer allowed and because GDPR eliminated a lot of WHOIS records. However, we want to improve that system to where the process is more automated. Still designing some of this, but generally the validation staff will be more restricted on what WHOIS information can be input.

For detection, we're leaning towards using machine learning to better identify when a document is a WHOIS-like document. This way we can detect when a document is not an actual screen shot of WHOIS but something else.
Status: UNCONFIRMED → RESOLVED
Closed: 10 months ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.