946351 - Misissued Google certificates from DCSSI

Status: RESOLVED FIXED

(NSS :: CA Certificates Code, task, P1)

Comment 1

[Curtis Koenig [:curtisk-use curtis.koenig+bzATgmail.com]]]

Attachment: dcssi-gstatic-chain

Comment 2

[Curtis Koenig [:curtisk-use curtis.koenig+bzATgmail.com]]]

Attachment: dcssi-gstatic-chain-2

Comment 4

[Daniel Veditz [:dveditz]]

Moving to NSS component so we can add release-blocking flags

Comment 5

[Curtis Koenig [:curtisk-use curtis.koenig+bzATgmail.com]]]

There are 2 questions that come to mind for me
1) Do we chemspill a 25.0.2 given the site and timeframe for public disclosure? Also Chrome has already patched
2) What is the window for respin on 26.0 final?

Comment 6

[Lukas Blakk [:lsblakk] use ?needinfo]

(In reply to Curtis Koenig [:curtisk] from comment #5)
> There are 2 questions that come to mind for me
> 1) Do we chemspill a 25.0.2 given the site and timeframe for public
> disclosure? Also Chrome has already patched
> 2) What is the window for respin on 26.0 final?

Chempsilling a 25.0.2 will take a significant amount of resources and might put the 26.0 release at risk, if we can do anything to hold this until Tuesday Dec 10th for the 26.0 release we might be able to avoid the need for a 25.0.2

A respin of 26.0 could definitely start today, be with QA Thurs/Fri and ready for shipping on Tuesday.

Comment 7

[Daniel Veditz [:dveditz]]

We need to get this blocked in next week's releases (all of them). Given the proximity of the release a chemspill is probably overkill and too disruptive. AIUI this does mean we have to respin what we thought were our final Fx26 and ESR24.2 builds

Comment 8

[Kathleen Wilson]

More info from Google:

At the moment, my best guess is that this is another TURKTRUST like
event: where a MITM proxy has been installed with full, public
authority. That's a major breach of the CA's process, to be sure, but
it's not a major attack against users. We don't have evidence of a
widespread attack, although it is odd that the certificate was
observed outside of France.

Comment 9

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

Aloha. Let me look at the certificates that AGL has given us so I can figure out what should be blocked.

Comment 10

[Kathleen Wilson]

All, Would it be OK for me to create a separate bug for the actual code changes?

I'm concerned that the content so far in this bug may prevent us from opening it up to the public.

Comment 11

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

(In reply to Kathleen Wilson from comment #10)
> All, Would it be OK for me to create a separate bug for the actual code
> changes?
> 
> I'm concerned that the content so far in this bug may prevent us from
> opening it up to the public.

When we created the hidden security bug system, we did so with the promise that security bugs would get opened up. I don't see anything in this bug that would warrant keeping it hidden after we've shipped releases.

Comment 13

[Curtis Koenig [:curtisk-use curtis.koenig+bzATgmail.com]]]

If necessary we could make comment 1 private when we unhide this bug, we should be fine doing all we need to do here.

Comment 15

[Kai Engert (:KaiE:)]

Attachment: intermediate cert 1

According to comment 3, google has blocked "intermediate 1" from the list in the previous comment. I'm attaching a separate copy of that cert.

Comment 16

[Kai Engert (:KaiE:)]

Attachment: patch v1

This patch adds a distrust record for "intermediate 1" and inserts it below the root that it chains to.

Comment 17

[Camilo Viecco (:cviecco)]

Kai, would it better to block intermediate2 instead? From the looks of the certs intermediate 1 is an MITM device. While intermediate 2 (O=DG Tr\xC3\xA9sor, CN=AC DG Tr\xC3\xA9sor SSL) is the one that issued the cert for the MITM and they should be the ones feeling the pain for missisuance. Furthermore we dont know how many other CAs have been signed that have those same properties.

Comment 21

[Al Billings [:abillings - ex-MoCo]]

Do we need to assign a CVE here?

Comment 22

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

(In reply to Camilo Viecco (:cviecco) from comment #17)
> Kai, would it better to block intermediate2 instead? From the looks of the
> certs intermediate 1 is an MITM device. While intermediate 2 (O=DG
> Tr\xC3\xA9sor, CN=AC DG Tr\xC3\xA9sor SSL) is the one that issued the cert
> for the MITM and they should be the ones feeling the pain for missisuance.
> Furthermore we dont know how many other CAs have been signed that have those
> same properties.

This bug isn't about punishing anybody; there will be plenty of discussion regarding that kind of thing on dev.security.policy.

Kai's patch is definitely the least disruptive to end-users, while resolving the issue for the specific intermediate we know about. However, I do think we should acknowledge that there are limitations to the approach here. Obviously, the same entity that mis-issued the Google end-entity certificates could have more certificates, either with different keys, different issuers, different serial numbers, and/or different subject names. Also, the fact that this happened once is good evidence that it has happened or is likely to happen again within the DCSSI hierarchy. If we could find out more information about what the scope of impact would be on blocking one of the higher-level intermediates, we could better consider doing that. However, I suggest we check in Kai's patch now and do the NSS 3.15.4 release, while we try to assemble that information. We can broaden the scope of the blocking as necessary.

Comment 24

[Daniel Veditz [:dveditz]]

I agree w/Camilo. Why block the MITM device and not the 2nd level intermediate that issued the mis-used intermedate? Going by the subject names they are both the same DG Trésor organization (which appears to be part of the French gov't, http://www.tresor.economie.gouv.fr/, although the cert on that site chains to Certplus, not DCSSI)

Comment 25

[Kai Engert (:KaiE:)]

(In reply to Brian Smith from comment #22)
> ... However, I suggest we check in Kai's patch now and do the NSS 3.15.4
> release...

We might actually call it 3.15.3.1
We'll mention it in this bug once it's done.

Comment 26

[Kai Engert (:KaiE:)]

I prefer not to work tonight (11 pm at my place right now).
I'm happy to produce the release early in the morning (let me know your deadline by the hour).
I'm happy to land into branches, if you tell me, where you want it.

Since there is ongoing discussion, please get to final agreements what you'd like, and leave clear instructions in this bug which one you want blocked (using the descriptions from comment 14).

If you prefer to distrust an intermediate higher in the chain, no problem, I don't think it needs another review. I'd repeat the tests that Brian demonstrated in comment 19 + 20.

Comment 28

[Gervase Markham [:gerv]]

BTW, my recommendation is that we distrust the intermediate labelled 2, not 1. Intermediate 1 was issued to be placed in a specific MITM device. We don't know how many more devices there are out there. And if the controllers of the device have control of intermediate 2, which they used to create 1, then they can just create a replacement for it as soon as we block, if they choose. 

Intermediate 2 is also the SSL intermediate issued from a signature authentication intermediate (3), which seems suspicious and wrong. So I think blocking intermediate 2 is the right choice.

Given the scoping of this cert chain and the ownership of the parts, I think it is highly unlikely that blocking 2 will have significant collateral damage outside the French government, and maybe not even inside.

Gerv

Comment 29

[Gervase Markham [:gerv]]

The DGTPE distributes some certs from here: http://crl2.dgtpe.fr/ .

* "AC de signature et authentification de la DGTPE" is sub3.

* "AC racine de la DGTPE" is sub4.

* "AC racine du MINEFI" has the same public key modulus as sub5, but is self-signed, whereas sub5 is signed by the DCSSI. I presume therefore that sub5 is a cross-cert used to provide public trust to the otherwise-untrusted MINEFI root.

sub3, labelled as for signature authentication, has the following EKU:
            X509v3 Key Usage: critical
                Certificate Sign, CRL Sign

:-|

Gerv

Comment 30

[Gervase Markham [:gerv]]

Revocation:

End-Entity: no CRL or OCSP
Sub1: no CRL or OCSP
Sub2: no CRL or OCSP
Sub3: CRL: no revoked certificates
Sub4: CRL: no revoked certificates
Sub5: CRL: 1 revoked certificate; serial number (37D7BC5B) does not match Sub4.
Root CA: no CRL or OCSP

Gerv

Comment 31

[Daniel Veditz [:dveditz]]

Gerv: lack of OSCP and CRL was an issue when DCSSI was first going through the inclusion process according to the old bug. This morning Camilo checked one of their CRLs (not sure which one) and it had a 2011 date and a next update in 2014. Doesn't really sound like it was taken seriously.

Comment 33

[Gervase Markham [:gerv]]

Dan is referring to bug 368970.

Gerv

Comment 34

[Camilo Viecco (:cviecco)]

expanding comment 30:
intermediate 1: No CRL URL
intermediate 2: No CRL URL
intermediate 3:  http://crl1.dgtpe.fr/AC_Racine_DGTPE.crl -> Last Update: Sep  9 09:28:02 2011 GMT;  Next Update: Sep 13 09:28:02 2014 GMT (no revoked certs) (3 years between updates)
intermeduate 4:  http://www.icp.minefi.gouv.fr/ac-racine.crl -> Last Update: Jun  4 00:00:00 2013 GMT; Next Update: Jun  4 00:00:00 2015 GMT. (2 years between updates).
intermeiate 5: URI:http://www.icp.minefi.gouv.fr/igca.crl ->  Last Update: Dec  1 09:00:00 2013 GMT; Next Update: Jan  8 09:00:00 2014 GMT (finaly something that seems to be at least working).

Camilo

Comment 35

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

Lack of OCSP and CRL in anything except the end-entity certificate doesn't mean much for Firefox because we basically never check those for revocation anyway.

However, Gerv has made a very good point that the most immediate intermediate, labeled "intermediate 1" in comment 14, was probably self-issued within the organization that controls the intermediate labeled "2" in comment 14: The French Treasury office, since both of them refer to "Trésor" in their subject DNs. Also, I found https://www.tresor.economie.gouv.fr/, a public website, which doesn't even chain to DCSSI. So, I think that it makes sense to block the "2" intermediate instead of the first intermediate, according to the logic in comment 28.

So, I will prepare a new patch.

Comment 36

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

Attachment: distrust-ac-dg-tresor-ssl.patch

Distrust "intermediate 2", CN="AC DG Tresor SSL"

I ran the vfychain commands above and they gave the same results. I also ran:

  vfychain     -u 3 -a 2.crt -a 3.crt -a 4.crt -a 5.crt -a 6.crt
  vfychain -p  -u 3 -a 2.crt -a 3.crt -a 4.crt -a 5.crt -a 6.crt
  vfychain -pp -u 3 -a 2.crt -a 3.crt -a 4.crt -a 5.crt -a 6.crt
  vfychain     -u 3          -a 3.crt -a 4.crt -a 5.crt -a 6.crt
  vfychain -p  -u 3          -a 3.crt -a 4.crt -a 5.crt -a 6.crt

These resulted in the same results as above:

  Chain is bad!
  PROBLEM WITH THE CERT CHAIN:
  CERT 0. CN=AC DG Trésor SSL,O=DG Trésor,C=FR :
    ERROR -8171: Peer's certificate has been marked as not trusted by the user.

However, this command (using libpkix to verify "intermedate 2" as an SSL CA):

  vfychain -pp -u 3 -a 3.crt -a 4.crt -a 5.crt -a 6.crt

gives:

  Chain is bad!

So, libpkix doesn't provide an error in the verify log for that case only. Not sure why, but I don't think this should block landing this patch.

Comment 37

[Wan-Teh Chang]

Comment on attachment 8342768 [details] [diff] [review]
distrust-ac-dg-tresor-ssl.patch

r=wtc.

Comment 38

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

Attachment: release-numbering.patch

I just saw wtc's most recent email on the nss-dev thread. I would have preferred to release this as NSS 3.15.4 instead of NSS 3.15.3.1 but we already got agreement to do NSS 3.15.3.1 and I'd already pushed the patch to the NSS 3.15.3 release branch, so let's stick to the plan we agreed to previously. From now on, we can do things the new way.

Comment 39

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

Attachment: mozilla-beta: Update to NSS 3.15.3.1

This is:
$ python client.py --repo=<path-to-nss> update_nss NSS_3_15_3_1_RTM
$ hg commit -m "Update to NSS 3.15.3.1 (NSS_3_15_3_1_RTM), r=me, a=lsblakk"

Comment 40

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

Comment on attachment 8342768 [details] [diff] [review]
distrust-ac-dg-tresor-ssl.patch

default: http://hg.mozilla.org/projects/nss/rev/5a7944776645
NSS_3_15_3_RELEASE_BRANCH: http://hg.mozilla.org/projects/nss/rev/0185cbadb166

Comment 41

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

Comment on attachment 8342847 [details] [diff] [review]
release-numbering.patch

update NSS versions: http://hg.mozilla.org/projects/nss/rev/fb769b6652a2
tag NSS_3_15_3_1_RTM: http://hg.mozilla.org/projects/nss/rev/5dcf1ad7a98f

Comment 42

[Lukas Blakk [:lsblakk] use ?needinfo]

We'll need to land this to mozilla-release branch so we can do a build #2 of FF26.0 RC first thing in the morning.  a=me if someone comes around who can do this in a different timezone, otherwise will check in early tomorrow.

Comment 43

[Carsten Book [:Tomcat]]

(In reply to Lukas Blakk [:lsblakk] from comment #42)
> We'll need to land this to mozilla-release branch so we can do a build #2 of
> FF26.0 RC first thing in the morning.  a=me if someone comes around who can
> do this in a different timezone, otherwise will check in early tomorrow.

cc'ing ryan, not sure if also this stuff

Comment 44

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

https://hg.mozilla.org/releases/mozilla-release/rev/b760f9a3469f

Comment 45

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

Landed on mozilla-central and mozilla-aurora with the update to NSS 3.15.4 beta 7:
https://bugzilla.mozilla.org/show_bug.cgi?id=898431#c45

Comment 46

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

https://hg.mozilla.org/releases/mozilla-esr24/rev/ded0d9aef08b

Comment 47

[Kai Engert (:KaiE:)]

Comment on attachment 8342847 [details] [diff] [review]
release-numbering.patch

Unfortunately this didn't increment the version number of the internal root module.

Some software vendors package the contents of the root module (contents of file certdata.txt) independently from the rest of NSS (e.g. Fedora ships a package ca-certificates), and therefore a unique identifier is required.

Comment 48

[Kai Engert (:KaiE:)]

Attachment: supplemental release numbering patch (root module version) for release branches

Unfortunately the NSS_3_15_3_1_RTM tag has already been created :(

This patch isn't a functional change, but I'd like to include it in the NSS release to avoid ambiguities.

Can I delete and recreate the NSS release tag?

Comment 49

[Kai Engert (:KaiE:)]

Comment on attachment 8342958 [details] [diff] [review]
supplemental release numbering patch (root module version) for release branches

Checked in to NSS 3.15.3 release branch:
https://hg.mozilla.org/projects/nss/rev/e826416013cc

Comment 50

[Kai Engert (:KaiE:)]

Comment on attachment 8342847 [details] [diff] [review]
release-numbering.patch

This is fine, but insufficient (see my comments about the root module version).

r=kaie

Comment 51

[Ryan VanderMeulen [:RyanVM]]

(In reply to Brian Smith (:briansmith, was :bsmith; Please NEEDINFO? me if you want a response) from comment #44)
> https://hg.mozilla.org/releases/mozilla-release/rev/b760f9a3469f

a+ to uplift to b2g26? We're not auto-merging now that Fx26 is off m-b.

Comment 52

[Kai Engert (:KaiE:)]

Comment on attachment 8342958 [details] [diff] [review]
supplemental release numbering patch (root module version) for release branches

Pushed to esr24 and release branches:
https://hg.mozilla.org/releases/mozilla-esr24/rev/f00201c126c9
https://hg.mozilla.org/releases/mozilla-release/rev/a79174df6ea3

Also pushed to NSS 3.15.3 release branch:
https://hg.mozilla.org/projects/nss/rev/e826416013cc

Comment 53

[Kai Engert (:KaiE:)]

Attachment: mozilla-beta: Update to NSS 3.15.3.1 and fix the root module version number [beta26 already covered]

I'm attaching an updated version of Brian's patch for mozilla-beta, which has one added change, the root module version number update.

Comment 54

[Lukas Blakk [:lsblakk] use ?needinfo]

Ryan - yes to b2g26

Comment 55

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-b2g26_v1_2/rev/7a0b5d1f3e6e

This has Kai's follow-up folded in.

Comment 56

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

Update configure.in to require NSS 3.15.3.1:
https://hg.mozilla.org/releases/mozilla-esr24/rev/366323a872a7

Comment 57

[Daniel Veditz [:dveditz]]

Probably should have updated release to require 3.15.3.1, too. Too late for that I guess (builds have started). the Beta and Release repos still use 3.15 as the minimum. Aurora uses 3.15.4

Comment 58

[Kathleen Wilson]

Snippets of emails from DCSSI...

First response:
Thank you for your email. We take very seriously the potential risks of misuse of certificate in the IGC/A CA hierarchy.
We have been looking into the issue since we received the first notification from Google.
We will let you know as soon as we have something to report.

Later response:
This anomaly has not for origin an attack but an infortunate and unacceptable configuration - as it is prohibited by our certification policy - that our teams and the concerned actors are at the moment handling : revocation and removal of the concerned CA.

Gerv has also had a couple of phone calls with them, regarding their findings and progress.

Comment 59

[Kathleen Wilson]

From DCSSI:
following our previous message, please find below the actions we have engaged in order to fullfill back the security policy of the IGC/A :

At 10:30 am (UTC+1), a revocation has been requested...

An updated CRL is available at the following url : http://crl1.dgtpe.fr/

In the afternoon, technical teams deleted inconsistent certificates and their associated keys on the equipments they were installed in. The whole key pairs (public and private keys) associated to the certificates were thus deleted.

As a precautionary measure and to ensure the security if the generated certificates are still present in the navigators, it was asked that the control capability of the LCR be activated on the clients - all clients behind the targeted proxy -.

Comment 60

[Adam Langley]

Note: DCSSI have indicated that they have revoked at the second intermediate. I'll be updating Chrome's block to reflect that.

Comment 61

[Gervase Markham [:gerv]]

I spoke to a representative of DCSSI twice this morning French time, which is what prompted the emails from which Kathleen has quoted. I now also have contact info for Stephane Dubreuil, the head of their response team, and their team email alias has been CCed on this bug.

I think we are taking the correct immediate steps - there seems to be consensus between us, DCSSI and Chrome on which cert to block right now. At some point, we will need to release an appropriate statement, and after that, consider what additional action, if any, would be appropriate.

Regarding revocation, I would expect to see sub2 appear on the CRL for sub3, which is http://crl1.dgtpe.fr/AC_Racine_DGTPE.crl . However, at time of writing, this CRL still says "no revoked certificates". And it still has Last Update and Next Update times which are 3 years apart (Sept 2011 and Sept 2014).

Gerv

Comment 62

[Camilo Viecco (:cviecco)]

Gerv: after a:
wget http://crl1.dgtpe.fr/ac-racine.crl && wget http://crl1.dgtpe.fr/AC_Racine_DGTPE.crl && wget http://crl1.dgtpe.fr/AC_DGTPE_Signature_Authentification.crl && wget http://crl1.dgtpe.fr/AC_DGTPE_Chiffrement.crl
(get all their certs) and a

ls *.crl|sort|xargs -I % openssl crl -text -inform der -in % |grep -A 4 -iP 'update|31da7'
>
        Last Update: Dec  5 14:55:02 2013 GMT
        Next Update: Dec 12 14:55:02 2013 GMT
        CRL extensions:
            X509v3 Authority Key Identifier: 
                keyid:E9:DB:90:8F:FD:5B:99:E4:15:3B:F0:62:5C:A7:E4:0D:58:E8:87:99

--
    Serial Number: 031DA7
        Revocation Date: Dec  5 09:26:54 2013 GMT
        CRL entry extensions:
            X509v3 CRL Reason Code: 
                Cessation Of Operation


you can see that the intermediate 2 is correctly revoked by int3 however this seems of little use as intermediate2 (the one we blocked) had no CRLDP. And thus clients have no mechanism to check for this revocation. Only the crls:
AC_DGTPE_Chiffrement.crl and AC_DGTPE_Signature_Authentification.crl have changed as of a few minutes ago.

Comment 63

[juan becerra [:juanb]]

Camilo is helping me verify this on Fx26, and he noticed something unexpected that he is looking into.

Comment 64

[Daniel Veditz [:dveditz]]

Given the contained nature of this incident the initial sec-critical guess is a little overstated. Appropriate when we thought we might have a hacked CA or bogus Google certs on the open internet. I'm lowering this to sec-high which is our more normal rating for compromise of user data.

Comment 65

[Robert Relyea]

Comment on attachment 8342847 [details] [diff] [review]
release-numbering.patch

belated r+.

Comment 66

[Gervase Markham [:gerv]]

:juanb, :camilo: what did you find out?

Gerv

Comment 67

[Kai Engert (:KaiE:)]

Comment on attachment 8343045 [details] [diff] [review]
mozilla-beta: Update to NSS 3.15.3.1 and fix the root module version number [beta26 already covered]

I realize this patch is unnecessary, because version 26 had already been uplifted to the mozilla-release branch, meaning the current contents of beta 26 are obsolete... And beta will soon get the uplift to 27.

Comment 68

[Kathleen Wilson]

Can we make this bug public now?

Comment 69

[Adam Langley]

Kathleen: I think the leaf certificate and the first intermediate contain strings that give details that should be kept confidential, if possible.

Comment 70

[juan becerra [:juanb]]

(In reply to Gervase Markham [:gerv] from comment #66)
> :juanb, :camilo: what did you find out?
> 
> Gerv

To verify it manually Camilo had suggested something like:
- Import the certificates, one by one, top to bottom, except the last one (the root)
- Go to Options/Advanced/Certificates/View Certificates and select the Authorities tab
- Scroll down and select the certificate under "DG Trésor" and click on View

We had not expected to see an entry in the "This certificate has been verified for the following uses:" field set to SSL Certificate Authority, but that's what we saw in both Fx26b10 and Fx26RCb2 (the one with the fix).

We consulted with briansmith who said this was due to bug 829677, but that it was only a problem of showing this information about the certificate vs actually it being valid, and that users were not at risk.

Comment 71

[Al Billings [:abillings - ex-MoCo]]

(In reply to Kathleen Wilson from comment #68)
> Can we make this bug public now?

Shouldn't we release first?

Comment 72

[Matt Wobensmith [:mwobensmith][:matt:]]

I ran the same tests in vfychain that Brian did in comment 19 and confirmed that the cert has been explicitly untrusted, in FF 24.2.0esr, 26, 27 and 28.

Comment 73

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

Attachment: anssi-chain-from-agl.txt

This is the certificate chain information that Adam Langley publicly shared here:
https://www.imperialviolet.org/binary/anssi-chain.txt

Adam: Thank you very much for sharing the complete certificate chains, and thanks for telling us which certificates you feel comfortable with sharing publicly. I believe I've hidden all the attachments that need to be hidden before the bug is opened up.

Comment 74

[Adam Langley]

Brian: sadly the vendor that the French government are using for filtering and the location of one of their offices are scattered in the comments on this bug. See #3, #14, #18-20, 23, 27, 32.

Comment 75

[Kathleen Wilson]

Doesn't Microsoft's security advisory already publish this info?
http://technet.microsoft.com/en-us/security/advisory/2916652

Comment 76

[Adam Langley]

Kathleen: I don't believe so, unless there's something inside the update itself.

Comment 77

[Daniel Veditz [:dveditz]]

(In reply to Kathleen Wilson from comment #75)
> Doesn't Microsoft's security advisory already publish this info?
> http://technet.microsoft.com/en-us/security/advisory/2916652

They published the government department involved, but not the device vendor. I don't know why the vendor needs to be private, but then again knowing which of several possible vendors it is doesn't affect our understanding of what we think happened. I've hidden the comments Adam noted in comment 74. The comments were mostly procedural or extracts from the various keys and don't alter an understanding of the timeline, but losing comment 27 hurts a bit. I'll reproduce a sanitized version below:

(Gervase Markham [:gerv] from comment #27)
> I am currently, conveniently, in France, where it is 11.20pm, and will be
> making urgent enquiries with DCSSI in the morning.
> 
> In the mean time, here are some immediate things I have discovered.
> 
> end entity cert (b):
> [...]
> 
> This certificate was probably issued by an MITM device in the normal course
> of its workings...
> 
> intermediate 1:
> [...]
> 
> This intermediate probably resides in an MITM device created by [...]
> 
> 2:
> Serial Number: 204199 (0x31da7)
> Issuer: C=FR, O=DGTPE, CN=AC DGTPE Signature Authentification
> Subject: C=FR, O=DG Tr\xC3\xA9sor, CN=AC DG Tr\xC3\xA9sor SSL
> 
> This intermediate belongs to the Direction Generale (DG) du Tresor - the
> Treasury Department of the French Government. It appears to have been issued
> for SSL use, but from a higher intermediate issued for signature
> authentication.
> 
> 3:
> Serial Number: 6 (0x6)
> Issuer: C=FR, O=DGTPE, CN=AC Racine DGTPE
> Subject: C=FR, O=DGTPE, CN=AC DGTPE Signature Authentification
> 
> This intermediate belongs to the DGTPE - Direction Générale du Trésor et de
> la Politique Economique (General Directorate of Treasury and Economic
> Policy). Presumably this contains the Treasury Department. According to the
> CN, it is designed to be used for signature authentication.
> 
> 4:
> Serial Number: 30:30:30:31:31:31:35:39:38:32:31:36:30:37:35:32
> Issuer: C=FR, O=MINEFI, OU=AGENCE AUTORITE, CN=MINEFI-AUTORITE DE
> CERTIFICATION RACINE
> Subject: C=FR, O=DGTPE, CN=AC Racine DGTPE
> 
> This certificate is labelled as the DGTPE Root CA (AC Racine), even though
> it's an intermediate in this chain. 
> 
> 5:
> Serial Number: 11:21:d1:f6:1f:30:2d:07:1d:f1:03:80:aa:7b:2b:cf:b2:b4
> Issuer: C=FR, ST=France, L=Paris, O=PM/SGDN, OU=DCSSI,
> CN=IGC/A/emailAddress=[e-mail]
> Subject: C=FR, O=MINEFI, OU=AGENCE AUTORITE, CN=MINEFI-AUTORITE DE
> CERTIFICATION RACINE
> 
> This certificate belongs to MINEFI, the Ministry of Economics, Finance and
> Industry.
> 
> root:
> Serial Number: 245102874772 (0x3911451094)
> Issuer: C=FR, ST=France, L=Paris, O=PM/SGDN, OU=DCSSI,
> CN=IGC/A/emailAddress=[e-mail]
> Subject: C=FR, ST=France, L=Paris, O=PM/SGDN, OU=DCSSI,
> CN=IGC/A/emailAddress=[e-mail]
> 
> This root belongs to the DCSSI, Direction Centrale de la Sécurité des
> Systèmes d'Information - the Central Information Systems Security Division
> of the French Government.
> 
> 
> So here's my current understanding: the DCSSI (root) issued a sub (5) to
> MINEFI, who issued one (4) to the DGTPE, who issued an intermediate (3) for
> signature authentication. Someone decided to use that to issue a sub (2) to
> the Treasury Department for SSL use. They used that to issue another sub (1)
> to put in an MITM device, location unknown. That sub issued the end entity
> certificates as part of its normal MITM operation.
> 
> So the next point of investigation is to have a more detailed look at the
> certs and see what the key usage is on that sub ostensibly intended for
> signature authentication.
> 
> Gerv