1654544 - GlobalSign: Use of Domain Validation Random Value for more than 30 days

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Arvid Vermote]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Firefox/68.0

Comment 1

[Arvid Vermote]

How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

The Baseline Requirements specify that CAs must not use a random value for more than 30 days for domain validation starting June 1, 2020. From July 1, some actively used random values were not updated within the 30 day period which permitted 78 domains to be validated using an expired random value over a period of approximately one day. These improperly validated domains were used for approximately the next 2 weeks (until 2020/07/15 08:00 GMT) to issue 101 certificates. All certificates were revoked prior to 2020/07/17 13:30 GMT

A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

Time (GMT)	Activity
2020/05/31 07:04	We reset all random values that might be re-used within our legacy CloudSSL 1.0 product. All other products already complied with the 30 day random value limit
2020/06/01	The requirement that random values must be limited to 30 days use came into effect.
2020/06/28 07:00	All random values were planned to be reset via the same script as was run on 2020/05/31, however, the script execution failed.
2020/06/29 19:46	It was identified the CloudSSL 1.0 random values were not reset as expected due the failure of the aforementioned script.
2020/06/30 22:14	The issue is further troubleshooted and the errors at the basis of the failure are being worked upon.
2020/07/01 02:08	The development team runs the script to update the CloudSSL 1.0 random values.
2020/07/01 11:00	Compliance team starts investigation to identify certificates that have been issued including domains that had a random value > 30 days.
2020/07/02 10:00	Compliance team pauses investigation to focus on remediation plans, mass-replacement activities and revocations in the context of https://bugzilla.mozilla.org/show_bug.cgi?id=1649937
2020/07/14 13:00	Investigation completed, domains & affected certificates identified.
2020/07/15 08:00	All domains that were validated with expired RVs were reset to expired.
2020/07/17 13:30	All certificates that were issued relying on a random value > 30 days have been revoked.

Whether your CA has stopped, or has not yet stopped, certificate issuance or the process giving rise to the problem or incident. A statement that you have stopped will be considered a pledge to the community; a statement that you have not stopped requires an explanation.

We have stopped issuance of certificates with improperly validated domains as of 2020/07/15 08:00.

In a case involving certificates, a summary of the problematic certificates. For each problem: the number of certificates, and the date the first and last certificates with that problem were issued. In other incidents that do not involve enumerating the affected certificates (e.g. OCSP failures, audit findings, delayed responses, etc.), please provide other similar statistics, aggregates, and a summary for each type of problem identified. This will help us measure the severity of each problem.

The problematic certificates were issued between 2020/06/30 07:04:56 GMT and 2020/07/15 08:00 that contain any of the following SAN:

*.migrated1593555188066.sharedtodedicated.incaptest.de
*.d1593555188066.sontoson.incaptest.de
*.d1593467429680.sontoson.incaptest.de
for.anna.incaptest.co
ssladdafterremoved1589830186470.1.incaptest.co
*.presents.migration.incaptest.co
*.aca.as.prod.asc.sabre.com
*.atlantic-hotels.de
*.parmalat.net.au
inspiredhealing.org
www.covonelaw.com
*.exness.partners
exness.trade
exness.partners
americanlawncarevail.com
*.joshchang.com
cardleorewards.com
*.gkgapps.com
doncrosby.net
*.justusfiles.com
*.moderna.design
*.cypmh-model.nhs.uk
karleyhall.com
riverr.co.uk
*.riverr.co.uk
*.datopay.com
*.exness.trade
exness.broker
*.csinyc.com
joshchang.com
peekaboonwa.com
*.partnersinwealth.org
*.themoja.com
*.loanmart.com
*.wf77.net
covonelaw.com
killtheflashover.com
*.bizpocket.ca
*.berlitz.de
*.berlitz.de
*.peekaboonwa.com
*.doncrosby.net
*.mirainbarrel.com
themoja.com
*.karleyhall.com
*.hallen.com
*.americanlawncarevail.com
*.go.idf.il
go.idf.il
*.ipeman.com
ipeman.com
*.cromwelloffer.com.au
cromwelloffer.com.au
*.inspiredhealing.org
*.killtheflashover.com
bizpocket.ca
*.exness.broker
*.vietnamairlines.com
*.cardleorewards.com
*.evvolabs.com
mirainbarrel.com
partnersinwealth.org
justusfiles.com
moderna.design
*.esmuleapi.zurich.com
*.verj.co.uk
verj.co.uk
enablemart.com
*.lindt-nordics.com
lindt-nordics.com
*.enablemart.com
malzonitecnomedit.morescreens.eu

In a case involving certificates, the complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem. In other cases not involving a review of affected certificates, please provide other similar, relevant specifics, if any.

This is the list of certificates:

https://crt.sh/?q=21f1d99ee77b68b1e116a43f506db875c125355ae55c995ffc3f4f73f2d8ed8f
https://crt.sh/?q=a73c1387e0c5714a096d0856a4952fa03013adb36edb6aa19a56e50dcd1d1d5b
https://crt.sh/?q=dce7a4b8b3ec9180900600ce7acc19e60e3ef1c6da1d8edd7b4fac0ab1c6813b
https://crt.sh/?q=789fd64244bfbdf5a494dd1801f8b5cfd9fc09519c2a45d75bd936b9ba4d2e6f
https://crt.sh/?q=e5575cbcf1930edc6e90549cdee36c614c64350be38efcd7000702d1ca1d6a12
https://crt.sh/?q=2f2a9ece02d5e3d1d9fed4c9edfd1913facb25af912e355e25da8dcf90365d10
https://crt.sh/?q=29b1300c69102a161b6d89c606176dae93e3c0462e042961c9ef3f0945ada51f
https://crt.sh/?q=5df8eac0432fc8a1e65b8d357440f677e6fa07ed323183ea631a87cfdd20405d
https://crt.sh/?q=d40fbd3e110d8b3e3bc997c33116f7e1b9e37d98ec2a20adc4fd2b47d3167667
https://crt.sh/?q=df21894a203f9faa1c4c40f890abee63540c58e60864bce52404f940d3c755d5
https://crt.sh/?q=78928acad447b8c5df932e1503df817c6e51985d8a8ea58fef1ed6379c5a5f43
https://crt.sh/?q=c2bc22d4542831ec6fd8ed4a2b17aade3b1ebdf6776ef01ea99566aad4ac1763
https://crt.sh/?q=93d090289ba703db47d36eb273d1837a2d558b186baa33c27731aff0818b72e0
https://crt.sh/?q=de22238c84e9b3c15e5ff2c5a7fc241836e81f06ee64826b1011d37b04d5cf4b
https://crt.sh/?q=fa3f5f716c21f68b23c834607ab756c5e8aac8ff0a666627e6d5af5d49bc8ba6
https://crt.sh/?q=6256b72563c6340a6d7b52a4623d9edac6ff0755d66ae88395c6bba07e26603b
https://crt.sh/?q=a20bd6b940177e9507d1ae8eb2c3dca5ca2783eca5c6ff8cf4c73425c3ec21bc
https://crt.sh/?q=fb1feb1664ba5add200b9649a1b6438097b9e1a6e0a21e893ce16447e8526b3c
https://crt.sh/?q=89e32007d973fe6d4810af20a7fc38d78a2693185d2d519b07aefe8109bacd71
https://crt.sh/?q=e0bbab0c5da6e5030529f62c271fb9361d9b4baea7d14c8d55915330f8f78d47
https://crt.sh/?q=3e54b53227319cd9a9cb13bc2b3fb31bd5a74327514c446a3f1d193bbb9254c5
https://crt.sh/?q=b1377e5e7b3fcbc8e197e95d71295e19fcd3b136f30ffe4e04b9e66c9a3501ea
https://crt.sh/?q=3bf3be69fa6e0c61c3fea9e3ea38b9754591251eba6626bbced7b408e887176f
https://crt.sh/?q=26d2d0793940d2e7fc8d009510765f11c2c6de9b86033a4b23166fbad9d6386e
https://crt.sh/?q=b1bf3dc95eed462d155d7a3b9b7ae05f977e5e2cf6d9279883c0aac403b21ed9
https://crt.sh/?q=bacc6e8d2e006b0e106ab1eb8788a62554ca56d915324d41c4ab3912e2fe2cfd
https://crt.sh/?q=0e0c157752ab6012681bdf00998dce43adfeaf5c681d098925dcab1e8d65013b
https://crt.sh/?q=7c194338191329fdaa2b934e3fcbb1724184f84f9668327ab9a9024e381527d0
https://crt.sh/?q=5fbfce13c72c00c07b90fc54131525913462bb00dceb9999401734ab87239ea2
https://crt.sh/?q=1e0ffbaacb165285af36303fe12744b9d626e004b837bc0997ee863af2f1e70f
https://crt.sh/?q=95df4f9f5a8bccf698dd004530995712c5b61e4018c47350dcae3ebd0ceb2d2a
https://crt.sh/?q=e7a95dc8f78b1cfea73bdaed147e1c39568b3fc58b1942a313b4f27758028adc
https://crt.sh/?q=fe729a0975644092453ebf5d8f735385d708a37ea022b3f252fc012e83c5560a
https://crt.sh/?q=8c80ed86206eb1631e0e0e2243f16ad0b23c75ba475ec2f788572c342d77f16c
https://crt.sh/?q=a6c807df165746d79fa2c16144160c151594b53b745e2805c1fc0b98bf8d73ef
https://crt.sh/?q=40bb0fcf8c0691ea6b61b52b90e63ee69cca097d7927a7918e8abb2e1e2e1cbd
https://crt.sh/?q=7e931490bac71042be435c5d6d904081bcc175ce29cc021b2251b477229124db
https://crt.sh/?q=5f63c559b7a5ca0490243ae8d2b80e27d6f1ea6732d92094032b6d5cfe0c3d24
https://crt.sh/?q=7d73b7cd777475b949a991e544870c49a70cad17f720a15d5058ef8807a4c84e
https://crt.sh/?q=b598257f0b0ba08157ad03a23b4f7503529fa16c831a960405b8cabc227c3bdb
https://crt.sh/?q=bcb089a7e689e8cf31a0084d1a9fa8fe1d1356b3681b053c086e2e2b76051dd7
https://crt.sh/?q=a90579fa87c74b3d51e2f5c377e891f94e0fdfd218132f10d65ce59260961c5f
https://crt.sh/?q=bc5d8a103d562ef27d160f2b8b15fc957de3dd99712ece9a3925bcf7fca8784e
https://crt.sh/?q=a17e4b100ba093ae40789942045c9e18c36886db2b467d58fa683d3490259fdc
https://crt.sh/?q=71b7abfe46c4a6008edb3c0e3a5bb704bcae00da772b01a54e3e3f7bde1eaf9d
https://crt.sh/?q=157ff8fd66258c1fc22f2b65a3a856b88e76e4245a0f820044cc67bb3ba85d3a
https://crt.sh/?q=22e74256c94e05ab8d15739cb041e6de0e940f60e4839035594758711bd8b1f0
https://crt.sh/?q=9ef00853304b69a3bc6341c507b75e4c01d3bcd9f3f33d9f87ff5d43f0c9d7a8
https://crt.sh/?q=d118b966a6aa9b8789813cc6099a5ac3bf7829e0cbd16eb3e2fb9bf54c229309
https://crt.sh/?q=fb5d6fd8a6940a3c9b109d98c8580284913020f2c671d82efd2d0b560272d76f
https://crt.sh/?q=7ce9bddbfe3e5288b11b05491a4123e2de605937e2d4c3f7b3a508a141562d30
https://crt.sh/?q=4806829d1adc34de43c1861abcdbd8d4718b577cb633a1ce89bdf05d39a98d6c
https://crt.sh/?q=a7a12e0e4a407d10918d733f45d394bd21c8f29b1690a8138325f7081dbaca4e
https://crt.sh/?q=127862f3bf608f9fc29239394535ae508125d05a6ad2d0a428acee08a2aa3524
https://crt.sh/?q=670830fa2dbf24f1cd138568cd4af037278aa3a39f9f2c711a3f1b0a01a4bfb7
https://crt.sh/?q=da97d24fd6babd41a7203e543a500de9e95b31b15e945f1345c54edafceead54
https://crt.sh/?q=6336f7ab9da9b1ff48c05d0761c40ae1dc2971c967d8b9234d49b6e1a9da5cef
https://crt.sh/?q=8632cab96c15b4b6f902ab4904ab35ce9b21808b193e5ca018a8f6e1f0fb4d03
https://crt.sh/?q=f85ea4752fe8af1be5d1478e35205583fa1a5fcfb5800916dcfc379e72416a93
https://crt.sh/?q=0f1c65c756c0ed3027ba3cd317cf87882da9a1023c2def860e0e94643553158e
https://crt.sh/?q=864242a6d43598baf4dadefb4cfc4ccd970566245b0bd925e891a9a7adf6a0fe
https://crt.sh/?q=b626120ffd3eda9c5f60a07514e3c9c78af9a5216fd2e186b63ab6885d6af533
https://crt.sh/?q=846e3933afe13dff7ed242b5dfa1ed1bbeb135193135eaf4198732bbce97669b
https://crt.sh/?q=f04fef738bd1b51f2bbc34eadb32cebc1a25566f31865b1e26513130c71f93cb
https://crt.sh/?q=202885df3854fa895f4ecf552ab3c055f66632af10d41ad1435fdeeb8375ba6b
https://crt.sh/?q=618b2910557546a97975436b223f5aacf8789b676db165137b26c233943dccd6
https://crt.sh/?q=72d8d3b5708da3b2a86ec7e3a5c1d62d949fa3745d895e867a66c0654961af39
https://crt.sh/?q=33bf8f8af7091c96897f211b0def5c131ff227b26f6e98d318d47041eb4b29ca
https://crt.sh/?q=3264afb0c612369a6f81e2bdb4683b988dca0c1c354cdf9c9a82244251ec7bc1
https://crt.sh/?q=fcf94837cef87aa4cdae0bf632a7464388d74d8c44e4737b81862e9a761f7b6f
https://crt.sh/?q=cbdd282e775e81a1581362f2c00ce68d3de94615596ae6323e93f51bd0d87ba2
https://crt.sh/?q=4ea1bd4056827e7d4d2fa12786744772b5f948ba87d0473169f873eca2f6ac53
https://crt.sh/?q=1c27e9ca87da77b43c2baeebc721ae703fa1c9e648a2e2cef1f1148db6ca9cf8
https://crt.sh/?q=fb148c0ed27fb7e7968db6bf98eba0e187f79542479409855fa5c3580b0dadd0
https://crt.sh/?q=cac83c796d57202d4f466974c07800e5cb5cfef5d116c254a9dd7a09ac183210
https://crt.sh/?q=fbfe89857777df3fda3d0bff3b8930e24d004d03fdb2ea889491ff34898397fc
https://crt.sh/?q=fb2db0fa8021754ad6f7b8b1c1b4428d55f228726966cb1289f763ced74385e9
https://crt.sh/?q=8ba6ea77fc65328a299bdf97931368d1a488ee8677214d29481f2f3f6647ba5f
https://crt.sh/?q=5bc52c7ae937ef1e2f310dff35a04c04b4df8189552069743cd87009b2bc91b4
https://crt.sh/?q=3622f24a7abe40aa1ac4b34c23ff52ef219ab5016fcd97421569370510dae1a3
https://crt.sh/?q=ec94579399782784295b52b568fb43b3b2ab2e8d4e074f12100ce33ca66416d9
https://crt.sh/?q=3396acc8e0589673ba19f8943cee94219d921550e5457ed0ac752b5a9afa1a5b
https://crt.sh/?q=27db86be8256011137914b5cd525d815befccdeadf55d11ee1d6e18aa79152b6
https://crt.sh/?q=e2073b761fa9eeb042748a7fc2ee20abcbab71b9f726899b19bfe98fd7d55809
https://crt.sh/?q=4f69879d66116c7ffd3775c5a20be4ada299a87c6bd1f4fe99caf9bceba04657
https://crt.sh/?q=e23817a46a60ea32b579a965f3e61e329382e8f08b29ce5fdc3396d35e2a6bfc
https://crt.sh/?q=ed4e5ec51a7af385f3c0876b58654b78c70597104f04f2fe876ac2655c0e3f0d
https://crt.sh/?q=613f6b0bb5dc467b89efaf4153d2f1fe03706011cfa777f8980f79db94712a7d
https://crt.sh/?q=fe5332e92f7bfd67596a3936973e8eefd72cb0cf20dfaa1cdb5ba39a816ae917
https://crt.sh/?q=bd39b3e8caf292fa6f4c97b11c386dbd2350a8288cd5d32d65d23c3a1083bca2
https://crt.sh/?q=a41586c1a8162bbae90ecf80a504f6d253051acb8740a0f45dfd9fab05164d5d
https://crt.sh/?q=13a78f8e4f404c4654522bba2665e7260da1b5155f0aad6460d516ef3c5e19f9
https://crt.sh/?q=2888fab345fb6cca8e626d2b012be4e487f4ac72b1e4d2f12e3953766fb18adb
https://crt.sh/?q=fb170b01ac81fe77d6c566da696e3e250a79a54aa8f3e62b8c6f6be3018d4272
https://crt.sh/?q=1cc64fc9ee67e356109be81abd37539a5b7829af9c73e54c0bf3b3bddbe90fd4
https://crt.sh/?q=e8cd385b5db9167717615ea66d2cb3e66a40a7f3357d39752b7b1b112423016e
https://crt.sh/?q=7ff464297ebff4469917ff1bf3cab654bc824755570548da9fe2094e3a6b15e9
https://crt.sh/?q=eef82945d0aebb5c6f69864d3bd04ff0928434771c5150e26323191a0a43c68a
https://crt.sh/?q=13f82337ccdb04a30da6e1f5a20249f6a34807e10ee043f8b9448a6d568f2a4e
https://crt.sh/?q=fb520d5828463c655cd16b290aff5ac152e3961c81931c714822dfb6a3647872
https://crt.sh/?q=9705efeb178cd55fece3226d5b134f4ad79d24b932e3d8daef4b781e01e1fce3

Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

The scheduled job to update the random values did not run and there were no secondary controls in place to immediately identify the failure and properly escalate it to the development and compliance teams for timely resolution.

List of steps your CA is taking to resolve the situation and ensure that such situation or incident will not be repeated in the future, accompanied with a binding timeline of when your CA expects to accomplish each of these remediation steps.

Since the failure, we have put redundant manual controls in place to make sure that the job, which is scheduled to run every 28 days, is run before the 30 day maximum period. Additionally, we've established monitoring & escalation procedures should the reset job fail again.

We are underway to move all the remaining legacy CloudSSL 1.0 customers to different offerings. By the end of August there will be just one larger customer and one small customer using this and they are both actively integrating with our new platform. The last large customer is expected to move away from CloudSSL 1.0 by Q2 2021.

We'll create a seperate ticket for not timely revoking the certificates per the Baseline Requirements #4.9.1.1.

Comment 2

[Ryan Sleevi]

Why was this not reported as soon as the incident had been identified? Prompt reporting is an essential component of ensuring trustworthy behaviour, by making sure that potential incidents are flagged for all CAs to examine, and to help ensure the CA is prioritizing correctly.

I don’t see questions 6 and 7 looking at the systemic failures here, and I think a more thoughtful examination is needed. Remember, the goal is not to feel good that “we fixed the bug”, but to understand the system that made the bug possible. I see multiple systemic failures here, including the failure to timely reporting and the failure to adequately staff for incident response, and I want to make sure there is a more comprehensive plan going forward.

I am deeply concerned that a known incident was ignored and scope left undetermined because y’all got busy. I can understand the complexities that multiple incidents present, but the default expectation for CAs is “fail closed”, not “fail open”. At this time, I would say this incident report is incomplete, as it only touches lightly on the surface issue without examining systemic issues, including that of the design and development of the CloudSSL product.

Comment 3

[Arvid Vermote]

We have detailed the details on our untimely action in https://bugzilla.mozilla.org/show_bug.cgi?id=1654545

Regarding 6 and 7: We've permitted the same random number to be used for multiple domains as part of the certificate request, and until June 1 per the clause in 3.2.2.4.6 and 3.2.2.4.7: (ii) if the Applicant submitted the Certificate request, the timeframe permitted for reuse of validated information relevant to the Certificate (such as in Section 4.2.1 of these Guidelines or Section 11.14.3 of the EV Guidelines).

Now that the limit is set at 30 days, we need to change the value at least every 30 days for those certificate requests. We permit certificate requests to be used to issue multiple certificates per 4.1.2, but now that the limit on random value re-use is 30 days, we need to regularly update the value used for domains for that certificate request. As we wind down this product, we've scripted the random value update for these orders to happen every 28 days and to verify the change was successful prior to the 30 day requirement. There are multiple individuals tracking the status of the script and are confident that all values are properly updated.

Comment 4

[Ryan Sleevi]

Thanks Arvid.

I'm hoping you could detail more of your system, although I realize it's a legacy system being phased out. This can help benefit other CAs in understanding designs and implementations that can lead to compliance issues.

For example, inferring from what you've shared, it sounds like at the time a request is generated, you generate a random value and a date. Rather than having the validation software check that the value is used within the range specified by the date, it sounds like you just check that the value is, well, the value. Separately, on your backend, you have a system that periodically examines the dates, and refreshes the value as appropriate. Is that... roughly correct?

The cause of the failure was an issue with the script running, which did not cause the values to be updated, and thus your system accepted the requests because the random values matched, even though they were outside of the date. Your fix is to run the update script at a cadence more frequent than the reuse period (i.e. every 28 days) to mitigate.

Does that sound roughly correct? Could you share any more relevant details about this system? And do you have anything you can share about how the 'new' system (that replaced CloudSSL 1.0) works regarding validation?

Comment 5

[Arvid Vermote]

We have the concept of Certificate Orders. When a new order is created based on a certificate request we generate a new Random Value (RV). Customers can update that Order over time by adding and removing SANs and they use the RV for that purpose. This creates a new certificate under that Certificate Order. Each time the user requests a new certificate in this way, all certificate data is verified to be accurate within the prescribed time limits, but since the data was collected for the initial order, it's typically re-usable for the entire duration of the "order", which is the same as the validity period of the certificate (up to 825 days currently).

Until the June date, there was no reason to set a date when the RV was created or when it expired within orders. Since this product is being EOLed, we didn't want to expend too much effort in tracking the usage periods of the RV at the point of use (validation) since there are many different validation methods that would need to be updated and tested. This lead us to the solution of globally updating all RVs for all customers and all orders every 28 days. We split it into a couple of different batches, but we update them all every 4 weekends.

In our newer products we more cleanly separate domain validation from issuance. For each domain the customer wants to have pre-validated against their identity, they request the domain (one at a time) and receive a RV unique to that domain. They validate that domain within 30 days, or else they need to "start over" and request a new RV. The validation log / record of the domain includes among others both the location & value of the RV as well as a specific check that it was created within the past 30 days.

Comment 6

[Arvid Vermote]

Can it be confirmed whether more information is required or this bug can be closed?

Comment 7

[Ben Wilson]

I intend to close this bug on Monday, 7-Sept-2020, unless there are objections.