1654545 - GlobalSign: Failure to revoke noncompliant certificates within 5 days

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Arvid Vermote]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Firefox/68.0

Comment 1

[Arvid Vermote]

Creating this bug to capture GlobalSign not revoking the certificates affected by https://bugzilla.mozilla.org/show_bug.cgi?id=1654544 within the 5 days time frame as set forth by section #4.9.1.1 of the SSL Baseline Requirements. We will post the incident report later.

Comment 2

[Ryan Sleevi]

Arvid: Please provide something more concrete than “later”

Comment 3

[Arvid Vermote]

Should be by July 27 2020 the latest.

Comment 4

[Arvid Vermote]

How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

We became aware of this problem when we were handling compliance issue https://bugzilla.mozilla.org/show_bug.cgi?id=1654544 and we established we didn't act and handle within the timeline as set forth by the Baseline Requirements 4.9.1.1. which stipulates the certificates must have been revoked within 5 days after we became aware the certificates were not issued in accordance with the Baseline Requirements, whereas the last affected certificates were revoked 19 days after becoming aware.

A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

Refer to https://bugzilla.mozilla.org/show_bug.cgi?id=1654544.

Time (UTC)	Activity
2020/06/29 19:46	It was identified the CloudSSL 1.0 random values were not reset as expected due the failure of the aforementioned script.
2020/06/30 22:14	The issue is further troubleshooted and the errors at the basis of the failure are being worked upon.
2020/07/01 02:08	The development team runs the script to update the CloudSSL 1.0 random values.
2020/07/01 11:00	Compliance team starts investigation to identify certificates that have been issued including domains that had a random value > 30 days.
2020/07/02 10:00	Compliance team pauses investigation to focus on remediation plans, mass-replacement activities and revocations in the context of https://bugzilla.mozilla.org/show_bug.cgi?id=1649937
2020/07/14 13:00	Investigation completed, domains & affected certificates identified.
2020/07/15 08:00	All domains that were validated with expired RVs were reset to expired.
2020/07/17 13:30	All certificates that were issued relying on a random value > 30 days have been revoked.

Whether your CA has stopped, or has not yet stopped, certificate issuance or the process giving rise to the problem or incident. A statement that you have stopped will be considered a pledge to the community; a statement that you have not stopped requires an explanation.

As detailed in https://bugzilla.mozilla.org/show_bug.cgi?id=1654544 GlobalSign has resolved the bug that led to the compliance failure.

In a case involving certificates, a summary of the problematic certificates. For each problem: the number of certificates, and the date the first and last certificates with that problem were issued. In other incidents that do not involve enumerating the affected certificates (e.g. OCSP failures, audit findings, delayed responses, etc.), please provide other similar statistics, aggregates, and a summary for each type of problem identified. This will help us measure the severity of each problem.

This is the list of certificates:

https://crt.sh/?q=21f1d99ee77b68b1e116a43f506db875c125355ae55c995ffc3f4f73f2d8ed8f
https://crt.sh/?q=a73c1387e0c5714a096d0856a4952fa03013adb36edb6aa19a56e50dcd1d1d5b
https://crt.sh/?q=dce7a4b8b3ec9180900600ce7acc19e60e3ef1c6da1d8edd7b4fac0ab1c6813b
https://crt.sh/?q=789fd64244bfbdf5a494dd1801f8b5cfd9fc09519c2a45d75bd936b9ba4d2e6f
https://crt.sh/?q=e5575cbcf1930edc6e90549cdee36c614c64350be38efcd7000702d1ca1d6a12
https://crt.sh/?q=2f2a9ece02d5e3d1d9fed4c9edfd1913facb25af912e355e25da8dcf90365d10
https://crt.sh/?q=29b1300c69102a161b6d89c606176dae93e3c0462e042961c9ef3f0945ada51f
https://crt.sh/?q=5df8eac0432fc8a1e65b8d357440f677e6fa07ed323183ea631a87cfdd20405d
https://crt.sh/?q=d40fbd3e110d8b3e3bc997c33116f7e1b9e37d98ec2a20adc4fd2b47d3167667
https://crt.sh/?q=df21894a203f9faa1c4c40f890abee63540c58e60864bce52404f940d3c755d5
https://crt.sh/?q=78928acad447b8c5df932e1503df817c6e51985d8a8ea58fef1ed6379c5a5f43
https://crt.sh/?q=c2bc22d4542831ec6fd8ed4a2b17aade3b1ebdf6776ef01ea99566aad4ac1763
https://crt.sh/?q=93d090289ba703db47d36eb273d1837a2d558b186baa33c27731aff0818b72e0
https://crt.sh/?q=de22238c84e9b3c15e5ff2c5a7fc241836e81f06ee64826b1011d37b04d5cf4b
https://crt.sh/?q=fa3f5f716c21f68b23c834607ab756c5e8aac8ff0a666627e6d5af5d49bc8ba6
https://crt.sh/?q=6256b72563c6340a6d7b52a4623d9edac6ff0755d66ae88395c6bba07e26603b
https://crt.sh/?q=a20bd6b940177e9507d1ae8eb2c3dca5ca2783eca5c6ff8cf4c73425c3ec21bc
https://crt.sh/?q=fb1feb1664ba5add200b9649a1b6438097b9e1a6e0a21e893ce16447e8526b3c
https://crt.sh/?q=89e32007d973fe6d4810af20a7fc38d78a2693185d2d519b07aefe8109bacd71
https://crt.sh/?q=e0bbab0c5da6e5030529f62c271fb9361d9b4baea7d14c8d55915330f8f78d47
https://crt.sh/?q=3e54b53227319cd9a9cb13bc2b3fb31bd5a74327514c446a3f1d193bbb9254c5
https://crt.sh/?q=b1377e5e7b3fcbc8e197e95d71295e19fcd3b136f30ffe4e04b9e66c9a3501ea
https://crt.sh/?q=3bf3be69fa6e0c61c3fea9e3ea38b9754591251eba6626bbced7b408e887176f
https://crt.sh/?q=26d2d0793940d2e7fc8d009510765f11c2c6de9b86033a4b23166fbad9d6386e
https://crt.sh/?q=b1bf3dc95eed462d155d7a3b9b7ae05f977e5e2cf6d9279883c0aac403b21ed9
https://crt.sh/?q=bacc6e8d2e006b0e106ab1eb8788a62554ca56d915324d41c4ab3912e2fe2cfd
https://crt.sh/?q=0e0c157752ab6012681bdf00998dce43adfeaf5c681d098925dcab1e8d65013b
https://crt.sh/?q=7c194338191329fdaa2b934e3fcbb1724184f84f9668327ab9a9024e381527d0
https://crt.sh/?q=5fbfce13c72c00c07b90fc54131525913462bb00dceb9999401734ab87239ea2
https://crt.sh/?q=1e0ffbaacb165285af36303fe12744b9d626e004b837bc0997ee863af2f1e70f
https://crt.sh/?q=95df4f9f5a8bccf698dd004530995712c5b61e4018c47350dcae3ebd0ceb2d2a
https://crt.sh/?q=e7a95dc8f78b1cfea73bdaed147e1c39568b3fc58b1942a313b4f27758028adc
https://crt.sh/?q=fe729a0975644092453ebf5d8f735385d708a37ea022b3f252fc012e83c5560a
https://crt.sh/?q=8c80ed86206eb1631e0e0e2243f16ad0b23c75ba475ec2f788572c342d77f16c
https://crt.sh/?q=a6c807df165746d79fa2c16144160c151594b53b745e2805c1fc0b98bf8d73ef
https://crt.sh/?q=40bb0fcf8c0691ea6b61b52b90e63ee69cca097d7927a7918e8abb2e1e2e1cbd
https://crt.sh/?q=7e931490bac71042be435c5d6d904081bcc175ce29cc021b2251b477229124db
https://crt.sh/?q=5f63c559b7a5ca0490243ae8d2b80e27d6f1ea6732d92094032b6d5cfe0c3d24
https://crt.sh/?q=7d73b7cd777475b949a991e544870c49a70cad17f720a15d5058ef8807a4c84e
https://crt.sh/?q=b598257f0b0ba08157ad03a23b4f7503529fa16c831a960405b8cabc227c3bdb
https://crt.sh/?q=bcb089a7e689e8cf31a0084d1a9fa8fe1d1356b3681b053c086e2e2b76051dd7
https://crt.sh/?q=a90579fa87c74b3d51e2f5c377e891f94e0fdfd218132f10d65ce59260961c5f
https://crt.sh/?q=bc5d8a103d562ef27d160f2b8b15fc957de3dd99712ece9a3925bcf7fca8784e
https://crt.sh/?q=a17e4b100ba093ae40789942045c9e18c36886db2b467d58fa683d3490259fdc
https://crt.sh/?q=71b7abfe46c4a6008edb3c0e3a5bb704bcae00da772b01a54e3e3f7bde1eaf9d
https://crt.sh/?q=157ff8fd66258c1fc22f2b65a3a856b88e76e4245a0f820044cc67bb3ba85d3a
https://crt.sh/?q=22e74256c94e05ab8d15739cb041e6de0e940f60e4839035594758711bd8b1f0
https://crt.sh/?q=9ef00853304b69a3bc6341c507b75e4c01d3bcd9f3f33d9f87ff5d43f0c9d7a8
https://crt.sh/?q=d118b966a6aa9b8789813cc6099a5ac3bf7829e0cbd16eb3e2fb9bf54c229309
https://crt.sh/?q=fb5d6fd8a6940a3c9b109d98c8580284913020f2c671d82efd2d0b560272d76f
https://crt.sh/?q=7ce9bddbfe3e5288b11b05491a4123e2de605937e2d4c3f7b3a508a141562d30
https://crt.sh/?q=4806829d1adc34de43c1861abcdbd8d4718b577cb633a1ce89bdf05d39a98d6c
https://crt.sh/?q=a7a12e0e4a407d10918d733f45d394bd21c8f29b1690a8138325f7081dbaca4e
https://crt.sh/?q=127862f3bf608f9fc29239394535ae508125d05a6ad2d0a428acee08a2aa3524
https://crt.sh/?q=670830fa2dbf24f1cd138568cd4af037278aa3a39f9f2c711a3f1b0a01a4bfb7
https://crt.sh/?q=da97d24fd6babd41a7203e543a500de9e95b31b15e945f1345c54edafceead54
https://crt.sh/?q=6336f7ab9da9b1ff48c05d0761c40ae1dc2971c967d8b9234d49b6e1a9da5cef
https://crt.sh/?q=8632cab96c15b4b6f902ab4904ab35ce9b21808b193e5ca018a8f6e1f0fb4d03
https://crt.sh/?q=f85ea4752fe8af1be5d1478e35205583fa1a5fcfb5800916dcfc379e72416a93
https://crt.sh/?q=0f1c65c756c0ed3027ba3cd317cf87882da9a1023c2def860e0e94643553158e
https://crt.sh/?q=864242a6d43598baf4dadefb4cfc4ccd970566245b0bd925e891a9a7adf6a0fe
https://crt.sh/?q=b626120ffd3eda9c5f60a07514e3c9c78af9a5216fd2e186b63ab6885d6af533
https://crt.sh/?q=846e3933afe13dff7ed242b5dfa1ed1bbeb135193135eaf4198732bbce97669b
https://crt.sh/?q=f04fef738bd1b51f2bbc34eadb32cebc1a25566f31865b1e26513130c71f93cb
https://crt.sh/?q=202885df3854fa895f4ecf552ab3c055f66632af10d41ad1435fdeeb8375ba6b
https://crt.sh/?q=618b2910557546a97975436b223f5aacf8789b676db165137b26c233943dccd6
https://crt.sh/?q=72d8d3b5708da3b2a86ec7e3a5c1d62d949fa3745d895e867a66c0654961af39
https://crt.sh/?q=33bf8f8af7091c96897f211b0def5c131ff227b26f6e98d318d47041eb4b29ca
https://crt.sh/?q=3264afb0c612369a6f81e2bdb4683b988dca0c1c354cdf9c9a82244251ec7bc1
https://crt.sh/?q=fcf94837cef87aa4cdae0bf632a7464388d74d8c44e4737b81862e9a761f7b6f
https://crt.sh/?q=cbdd282e775e81a1581362f2c00ce68d3de94615596ae6323e93f51bd0d87ba2
https://crt.sh/?q=4ea1bd4056827e7d4d2fa12786744772b5f948ba87d0473169f873eca2f6ac53
https://crt.sh/?q=1c27e9ca87da77b43c2baeebc721ae703fa1c9e648a2e2cef1f1148db6ca9cf8
https://crt.sh/?q=fb148c0ed27fb7e7968db6bf98eba0e187f79542479409855fa5c3580b0dadd0
https://crt.sh/?q=cac83c796d57202d4f466974c07800e5cb5cfef5d116c254a9dd7a09ac183210
https://crt.sh/?q=fbfe89857777df3fda3d0bff3b8930e24d004d03fdb2ea889491ff34898397fc
https://crt.sh/?q=fb2db0fa8021754ad6f7b8b1c1b4428d55f228726966cb1289f763ced74385e9
https://crt.sh/?q=8ba6ea77fc65328a299bdf97931368d1a488ee8677214d29481f2f3f6647ba5f
https://crt.sh/?q=5bc52c7ae937ef1e2f310dff35a04c04b4df8189552069743cd87009b2bc91b4
https://crt.sh/?q=3622f24a7abe40aa1ac4b34c23ff52ef219ab5016fcd97421569370510dae1a3
https://crt.sh/?q=ec94579399782784295b52b568fb43b3b2ab2e8d4e074f12100ce33ca66416d9
https://crt.sh/?q=3396acc8e0589673ba19f8943cee94219d921550e5457ed0ac752b5a9afa1a5b
https://crt.sh/?q=27db86be8256011137914b5cd525d815befccdeadf55d11ee1d6e18aa79152b6
https://crt.sh/?q=e2073b761fa9eeb042748a7fc2ee20abcbab71b9f726899b19bfe98fd7d55809
https://crt.sh/?q=4f69879d66116c7ffd3775c5a20be4ada299a87c6bd1f4fe99caf9bceba04657
https://crt.sh/?q=e23817a46a60ea32b579a965f3e61e329382e8f08b29ce5fdc3396d35e2a6bfc
https://crt.sh/?q=ed4e5ec51a7af385f3c0876b58654b78c70597104f04f2fe876ac2655c0e3f0d
https://crt.sh/?q=613f6b0bb5dc467b89efaf4153d2f1fe03706011cfa777f8980f79db94712a7d
https://crt.sh/?q=fe5332e92f7bfd67596a3936973e8eefd72cb0cf20dfaa1cdb5ba39a816ae917
https://crt.sh/?q=bd39b3e8caf292fa6f4c97b11c386dbd2350a8288cd5d32d65d23c3a1083bca2
https://crt.sh/?q=a41586c1a8162bbae90ecf80a504f6d253051acb8740a0f45dfd9fab05164d5d
https://crt.sh/?q=13a78f8e4f404c4654522bba2665e7260da1b5155f0aad6460d516ef3c5e19f9
https://crt.sh/?q=2888fab345fb6cca8e626d2b012be4e487f4ac72b1e4d2f12e3953766fb18adb
https://crt.sh/?q=fb170b01ac81fe77d6c566da696e3e250a79a54aa8f3e62b8c6f6be3018d4272
https://crt.sh/?q=1cc64fc9ee67e356109be81abd37539a5b7829af9c73e54c0bf3b3bddbe90fd4
https://crt.sh/?q=e8cd385b5db9167717615ea66d2cb3e66a40a7f3357d39752b7b1b112423016e
https://crt.sh/?q=7ff464297ebff4469917ff1bf3cab654bc824755570548da9fe2094e3a6b15e9
https://crt.sh/?q=eef82945d0aebb5c6f69864d3bd04ff0928434771c5150e26323191a0a43c68a
https://crt.sh/?q=13f82337ccdb04a30da6e1f5a20249f6a34807e10ee043f8b9448a6d568f2a4e
https://crt.sh/?q=fb520d5828463c655cd16b290aff5ac152e3961c81931c714822dfb6a3647872
https://crt.sh/?q=9705efeb178cd55fece3226d5b134f4ad79d24b932e3d8daef4b781e01e1fce3

In a case involving certificates, the complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem. In other cases not involving a review of affected certificates, please provide other similar, relevant specifics, if any.Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

Refer to the list above.

List of steps your CA is taking to resolve the situation and ensure that such situation or incident will not be repeated in the future, accompanied with a binding timeline of when your CA expects to accomplish each of these remediation steps.

Historically, WebPKI incident management was handled by multiple subject matter experts within GlobalSign (product management, incident management, compliance, key management and security). Since April 02 2020 the handling of WebPKI incidents was further formalized and ownership assigned to the PKI Governance Policy Authority "PACOM1 - PKI Governance", consisting among others of four senior PKI compliance operators in charge of overseeing and handling WebPKI incidents (further "compliance incident team").

By applying a formal "Root program incident management" procedure, the compliance incident team ensures compliance incidents are handled, revocation activities are timely, and incidents reports are provided to the relevant ecosystem stakeholders. Apart from being subject to the "Root program incident management" process, compliance incidents are also subject to the general GlobalSign incident management process, where a separate incident management team also monitors each compliance incident and the progress and actions taken by the compliance incident team.

At 2020/07/02 10:00 UTC, the investigation of compliance issue https://bugzilla.mozilla.org/show_bug.cgi?id=1654544 was paused to focus on remediation plans, mass-replacement activities and revocations in the context of OCSP EKU incident https://bugzilla.mozilla.org/show_bug.cgi?id=1649937. The magnitude of handling with the replacement of some and remediation planning of the 99 issuing CA affected by the OCSP EKU incident was unprecedented and consumed the compliance incident team normally handling incidents as the one that resulted in this revocation delay.

Once the OCSP EKU situation stabilized the senior PKI compliance operators resumed activities for https://bugzilla.mozilla.org/show_bug.cgi?id=1654544 and proceeded with revocation activities. Realizing this is not in line with internal and external expectations and not only the Baseline Requirements #4.9.1.1 but also Root Program requirements and general community expectations we are now taking following actions to ensure proper and timely incident response in the future, even if other (major) incidents are ongoing:

• The compliance incident team will be expanded to include additional PKI compliance and security operators to ensure more bandwidth to increase our capability for timely handling compliance incidents
• The general incident management process (owned and coordinated by a separate team) will be further tailored so that for compliance incidents they perform additional oversight on the compliance incident team in order to ensure they do not only execute in line with the general incident management procedure but also the "Root program incident management" process which incorporates specific timelines and requirements such as https://wiki.mozilla.org/CA/Responding_To_An_Incident#Immediate_Actions
• Inclusion of trigger for fail-safe mechanism in the general incident management process including immediate escalation of compliance incidents and authorization to stop issuance to the CISO (and backup PKI Governance manager) and CTO and COO should the CISO be unavailable.

The necessary changes to facilitate the above will be completed by August 14 2020.

Comment 5

[Ryan Sleevi]

Thanks for the update and the detailed description. GlobalSign is definitely providing a model for providing a clear understanding of operations and the steps being taken.

I've left the N-I on Bug 1654544, which I think is where we can continue the "late reporting" aspect. I think that, had that objective been met through timely reporting, it would have been easier to reconcile this issue.

While I can understand the "emergency stop" for production, it's not clear to me how that could have facilitated further investigation, either in parallel (which I understand your hiring to be about facilitating), or more expediently (e.g. better tooling to facilitate quicker investigations to allow serial execution). Similarly, it's not clear what the oversight activity would produce, if it determined a timeline was at risk of being violated? Would it trigger the revocation of any potentially suspect certs? A refocus on the original issue? Something else?

Put differently: I want to be understanding towards situations in which multiple incidents arise, and they have to be triaged and focused on appropriately. Provided there are prompt incident reports, this helps share the triage priorities with the community and add transparency (and that's on Bug 1654544). When you're at risk of exceeding a deadline, though, it's not clear to me what the expectation will be, or whether there's any opportunity to reduce the risk of exceeding deadlines due to investigative load.

Comment 6

[Arvid Vermote]

Expanding the compliance incident team will also ensure parallel and timely investigation, revocation, resolution and reporting of compliance incidents. To be clear, we are not hiring additional resources for now but training existing non-senior PKI compliance and security resources to be part of the compliance incident team and being capable of analyzing certificate problem reports, populations and extrapolating certificates to be revoked.

The additional oversight will ensure senior management being informed on compliance incidents close to exceeding acceptable timings / deadlines. Senior management has the authority to call for immediate revocations, refocus of resources in any department, shutting down systems / APIs / customer accounts to ensure compliance deadlines are met.

Comment 7

[Arvid Vermote]

All the actions described in the report have been completed mid-August. Is any more information required or can this bug be closed? Thank you.

Comment 8

[Ben Wilson]

I think this bug can be closed. I'll schedule this to be closed on or about 18-Sept-2020 unless there are additional issues to discuss.