1397957 - DigiCert / CTJ: Metadata in OU fields, Reserved IP Address

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Kathleen Wilson]

This bug has been separated out from Bug #1389172 to request an incident report specific to this subCA.

CTJ
  a) Metadata in OU fields
    - See Bug #1389172: Comments 0, 1, 4, 9
    - Remediation
      - 2017-08-16 - CTJ systems patched
      - 2018-08-XX - CTJ migrates to a new system (See Comment #1)
  b) Reserved IP Address
    - See Bug #1389172: Comments 3, 6, 9
    - Remediation
      - None performed; no root cause analysis performed


For the problems listed above, please provide an incident report as described here:

https://wiki.mozilla.org/CA/Responding_To_A_Misissuance#Incident_Report

Comment 1

[Jeremy Rowley]

Hey Kathleen - I thought I posted about (b)  to the Mozilla mailing. If not, I apologize for the oversight. The info is below.

(a) As mentioned, this was previously discussed on the Mozilla mailing list where the decision was made to patch and not revoke the certs https://groups.google.com/forum/#!msg/mozilla.dev.security.policy/DgeLqKMzIds/E9--gj5WEAAJ).  I thought this issue was closed. Just in case it's not:

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, via a discussion in mozilla.dev.security.policy, or via a Bugzilla bug), and the date.
- The issue was posted to the Mozilla dev list a while ago.

2. A timeline of the actions your CA took in response.
- We  immediately responded and contacted CTJ in the other thread.  Engaged with the community, discussing the lack of a security risk. The ultimate conclusion of the discussion was that CTJ needed to patch their systems but not revoke the certificates. The fix was to turn OU into a validated field that is reviewed by validation staff prior to issuance.

3. Confirmation that your CA has stopped issuing TLS/SSL certificates with the problem.
- Confirmed. CJT has patched their system to detect whether the OU field has metadata and prevents issuance if non-validated information is present.

4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.
- Lots of certs with the one problem. The corpus of certificates was identified in the older discussion and bug. The last one I could find was Jan 2017 (75591249). The first one was probably as early as when the BRs were adopted.

5. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.
- This is in the other bug. Do you want me replicate it here?

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
- OU is traditionally just a dumping ground for non-validated information. Although the BRs adopted the requirement that no other field should contain meta-data, the requirement does not squarely apply to the OU field under the language (as it's not an "Other Subject" field.  However, rather than fight over nuances with respect to the OU, CTJ patched the issue and put safeguards in place to prevent re-occurrence.

7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.
- CTJ is migrating to a DigiCert hosted solution.  This is expected to take approximately a year (Mid-2018?) CTJ is a great CA and responds promptly to all communication. The patch plus the move to validated information resolved the issue.

B) This cert was not actually mis-issued.  Instead, the cert was not properly revoked in accordance with the BR requirements for reserved IP addresses. Should it be a different problem report as "mis-issuance" is misleading? From CTJ:

Number of affected certificates:
One.  After we received the revocation request from DigiCert, we conducted an investigation to see how many certificate have the same issue.  We confirmed that no other active certificates except the one that was pointed out by Jonathan contain reserved IP address.  The CN of this certificate is g2-sanfull01.ctjssl.info, and this is for our own use.

Cause of missing the revocation:
The last scan we conducted to find out certificates with reserved IP address was February 2016.  At that time, we found out one certificate that were active and contained reserved IP.  This certificate is g2-sanfull01.ctjssl.info, the same certificate that is pointed out in the forum.  Also, we had already stopped issuing new or renewal certificates containing reserved IP address at that time.  So this would mean we could pay attention to only one certificate of our own to be revoked.  This is why we did not implement additional scanning after February, 2016.
 
We planned to revoke later; however, we forgot to do the revocation.  Therefore, the cause of this issue is human error and lack of supervising.  Our administrators forgot to register the revocation task on their task schedule and the manager did not follow.
 
Remediation actions:
When any provisions of the industry standards such as BR become effective, we will re-check our compliance by scanning our certificates.  Although scanning has been completed in the past, we will check again one month before the effective date.  Also, we already are checking crt.sh daily to see compliance of our newly issued or renewed certificates.

- Again, CTJ is migrating to a hosted CA, which will eventually resolve all of these issues. CTJ hasn't repeated any of these mis-issuance or had problems since we closed (a) previously.

Comment 2

[Ryan Sleevi]

(In reply to Jeremy from comment #1)
> 2. A timeline of the actions your CA took in response.
> - We  immediately responded and contacted CTJ in the other thread. 

This is not a timeline.

> 3. Confirmation that your CA has stopped issuing TLS/SSL certificates with
> the problem.
> - Confirmed. CJT has patched their system to detect whether the OU field has
> metadata and prevents issuance if non-validated information is present.

When? A complete answer in #2 would have avoided this question :)

> 4. A summary of the problematic certificates. For each problem: number of
> certs, and the date the first and last certs with that problem were issued.
> - Lots of certs with the one problem. The corpus of certificates was
> identified in the older discussion and bug. The last one I could find was
> Jan 2017 (75591249). The first one was probably as early as when the BRs
> were adopted.

This is not a complete answer.

> 5. The complete certificate data for the problematic certificates. The
> recommended way to provide this is to ensure each certificate is logged to
> CT and then list the fingerprints or crt.sh IDs, either in the report or as
> an attached spreadsheet, with one list per distinct problem.
> - This is in the other bug. Do you want me replicate it here?

Yes please.


> 7. List of steps your CA is taking to resolve the situation and ensure such
> issuance will not be repeated in the future, accompanied with a timeline of
> when your CA expects to accomplish these things.
> - CTJ is migrating to a DigiCert hosted solution.  This is expected to take
> approximately a year (Mid-2018?) CTJ is a great CA and responds promptly to
> all communication. The patch plus the move to validated information resolved
> the issue.

Given that an incomplete answer was provided in #2, this statement cannot be independently verified as to how effectively the CA responds.

> B) This cert was not actually mis-issued.  Instead, the cert was not
> properly revoked in accordance with the BR requirements for reserved IP
> addresses. Should it be a different problem report as "mis-issuance" is
> misleading? From CTJ:

> We planned to revoke later; however, we forgot to do the revocation. 
> Therefore, the cause of this issue is human error and lack of supervising. 
> Our administrators forgot to register the revocation task on their task
> schedule and the manager did not follow.
>  
> Remediation actions:
> When any provisions of the industry standards such as BR become effective,
> we will re-check our compliance by scanning our certificates.  Although
> scanning has been completed in the past, we will check again one month
> before the effective date.  Also, we already are checking crt.sh daily to
> see compliance of our newly issued or renewed certificates.

Note that these proposed mitigations do not address the cause of this failure - which was human error and a lack of supervising. What changes have been done to ensure prompt adherence?

That is, the substance of the reply can be read as "We'll do it right next time", but provides no assurances as to how that can be considered, in light of the failure to do it right this time.

Comment 3

[Jeremy Rowley]

With more detail this time:
(a) META DATA IN THE OU 

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, via a discussion in mozilla.dev.security.policy, or via a Bugzilla bug), and the date.
- The issue was posted to the Mozilla dev list on Aug 9, 2017. 

2. A timeline of the actions your CA took in response.
- We  investigated to see what was going on that same day, responding to Jonathan. We also contacted CTJ. We posted some initial findings to that mailing list. We determined this was not a direct violation of the BRs as it states "All other subject attributes MUST contain information that has been verified by the CA". OU is not an "other subject attribute" and is defined in the section immediately prior to this. As such, we have not revoked these certificates.

3. Confirmation that your CA has stopped issuing TLS/SSL certificates with the problem.
- Confirmed. CTJ has patched their system to detect whether the OU field has metadata and prevents issuance if non-validated information is present. Although I don't think this is actually a violation per the above explanation, we do see value in preventing meta-data from all subject fields, not just other subject fields. 

4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.
- Unknown. Customers can enter data they want in the OU Field. These are screened for addresses and names. However, CTJ did patch their systems to prevent this early in 2017. The last cert I could find with metadata in the OU was Jan 2017 (75591249).

5. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.
This is the list of all certs with metadata in the OU:
53176869
57830766
65616693
117197299
93295619
104632894
21853640
22198796
30236343
30547236
32078607
34809638
39655364
40493955
43400295
45247984
47645844
45815760
40840774
48674276
40627018
48832585
57830776
57830777
57830779
57830781
61135216
57785675
57785694
53176900
53176901
53176903
64152048
78648920
53176895
53176898
57830773
102832536
51567050
132148480
110336219
111461024
80230338
81246678
91907174
91907175
110354478
110354479
78766827
79544403
78494050
80022464
15522999
16058171
20501110
20501178
21202825
21202827
21202829
20501158
20501137
20686696
20501187
20501201
20501217
20686693
20835061
21202831
21202833
21202835
21202836
26829282
11658435
11723002
11020877
11758671
12502488
12502489
13247204
13247205
13388567
42698962
16519562
19240755
20479086
20103452
21192792
54658744
21477087
42505252
29807378
57881547
30238735
30238738
30238741
30351648
47300904
54163587
54163620
54163639
51576885
56762416
70713886
73109181
78277867
73090743
77286114
75591249

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
- OU is traditionally just a dumping ground for non-validated information. Although the BRs adopted the requirement that no other field should contain meta-data, the requirement does not squarely apply to the OU field under the language (as it's not an "Other Subject" field.  However, rather than fight over nuances with respect to the OU, CTJ patched the issue and put safeguards in place to prevent re-occurrence.

7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.
- CTJ is migrating to a DigiCert hosted solution.  This is expected to take approximately a year (Mid-2018?) CTJ is a great CA and responds promptly to all communication. The patch plus the move to validated information resolved the issue.

Comment 4

[Jeremy Rowley]

On 6 - this was patched early 2017. I think it was in April when the previous discussion happened. I've pinged Masaru-san so he can chime in on the exact data of the patch. 

For B) I'll let Masura-san provide a comment on what they did to prevent missing certs in the future. I'm not sure how they changed their policy to prevent forgetting to revoke a cert.

Comment 5

[Masaru Sakamoto]

Hi Ryan, Jeremy

Thanks for giving us an opportunity to answer your questions.


(a) META DATA IN THE OU 

3. Confirmation that your CA has stopped issuing TLS/SSL certificates with the problem.
- Confirmed. CTJ has patched their system to detect whether the OU field has metadata and prevents issuance if non-validated information is present. Although I don't think this is actually a violation per the above explanation, we do see value in preventing meta-data from all subject fields, not just other subject fields. 

CTJ: Apr.9, 2017. 


4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.
- Unknown. Customers can enter data they want in the OU Field. These are screened for addresses and names. However, CTJ did patch their systems to prevent this early in 2017. The last cert I could find with metadata in the OU was Jan 2017 (75591249).

CTJ: The last certificate with metadata in OU field was issued on March. 30, 2017.  The first one since the BRs were adopted was issued on July 2, 2012.


(b) Reserved IP Address

Note that these proposed mitigations do not address the cause of this failure - which was human error and a lack of supervising. What changes have been done to ensure prompt adherence?

CTJ: In addition to the remediation actions mentioned already, we changed work flow as following.

Before remediation:
Same person conducted his/her checking activities and registered issue(s) if found on an issue-tracking sheet. In this case, if this person missed registering, the manager could not realize the founded issue because the manager only checked the sheet. 

After remediation:
We assign different persons for checking and registering.  And, the outcomes of their works are reviewed by each other. Then, their manager finally checks both outcomes.


Please let me know if you have any questions.
 
Kind regards,
Mo (Masaru)

Comment 6

[Ryan Sleevi]

Below is my summary of the issues and remediation plan:

1) Metadata in OU fields
  - See Bug #1389172: Comments 0, 1, 4, 9
  - See Comment #5
  - Remediation
    - 2017-04-09 - CTJ systems patched
    - 2018-08-XX - CTJ migrates to a new system (See Comment #1)
2) Reserved IP Address
  - See Bug #1389172: Comments 3, 6, 9
  - See Comment #1, #5
  - Remediation
    - 2017-09-07 - CTJ has updated procedures to include scanning one month before the effective date of BR provisions to ensure compliance
    - 2017-09-11 - CTJ performs two-party validation of IP addresses to ensure they're not reserved

Is this a correct summary?

Comment 7

[Jeremy Rowley]

A couple of modifications:

1) Metadata in OU fields
- Remediation
    - 2017-04-09 - CTJ systems patched
    - 2018-12-31 - CTJ will migrate to the new system.  The primary hold up is DigiCert's lack of support for the Japanese language and expected Japanese process flow. We are currently working on implementing both. Once complete, the transition will begin.

2) Reserved IP Address
- Remediation
    - 2017-09-11 - Implementation of a stricter separation of duties and group responsibility for cross-checking work completion.

CTJ implemented a fix in its systems to prevent reserved IP addresses in accordance with the appropriate timeline. However, a single validation staff member was assigned the task of revoking the certificate. Now, multiple individuals receive the assignment for revoking certificates. Validation staff is still responsible for revoking certs when required by the BRs, but different persons are assigned for the revocation and checking for completing the revocation. The validation staff reviews each other regularly to ensure all steps are carried out. Both staff members submit reports to their managers who double checks the outcome.

Comment 8

[Firefox Bug Husbandry Bot]

Changing QA contact per https://bugzilla.mozilla.org/show_bug.cgi?id=1438254

Comment 9

[Ryan Sleevi]

Jeremy: Can you confirm the steps in Comment #7 were completed?

Comment 10

[Jeremy Rowley]

Unfortunately not fully migrated. They are using our systems for some certs, but not all. I'm not sure how many are issued by CTJ as a separate sub CA compared to issuance through our new system (trying to find this out which is the delayed response). The blocker has been Japanese support in our system. CTJ has improved their systems to the point they no longer are having issues.

Comment 11

[Ryan Sleevi]

I'm slightly concerned, given that the last substantial update was 2017-10-17 in Comment #7, which itself was a push back from the dates in Comment #1.

Do you have concrete dates for migration? It seems like a weekly update, per https://wiki.mozilla.org/CA/Responding_To_An_Incident#Keeping_Us_Informed , may be appropriate.

Comment 12

[Jeremy Rowley]

The blocker is currency tbh. Considering CTJ's clean record since this incident, do we need to migrate them? That was originally part of the plan, but I'm wondering if that's a Mozilla requirement or something we said that shot us in the foot. If it's a Mozilla expectation, we can force it faster. I'll have Mo chime in as well about the migration effort.

Comment 13

[Ryan Sleevi]

Thanks Jeremy. I defer to Wayne. Based on this and Comment #7, I think where we stand for this issue are:

1. Metadata in OU fields

• See Bug #1389172: Comments 0, 1, 4, 9
• See Comment #5, Comment #10
• Remediation
  • 2017-04-09 - CTJ systems patched

2. Reserved IP Address

• See Bug #1389172: Comments 3, 6, 9
• See Comment #1, Comment #5
• Remediation
  • 2017-09-11 - Implementation of a stricter separation of duties and group responsibility for cross-checking work completion.

Comment 14

[Wayne Thayer]

Migration away from the CTJ system is not a Mozilla requirement, but it is a sensible remediation action. Is this still really your plan to ultimately address CTJ compliance? If so, then that is an acknowledgement that some risk still exists in the legacy CTJ system, even though there have been no new incidents, and I'd prefer you to just provide a new date for completing that change. If migration is no longer the plan and the CTJ system is not slated for retirement in the near future, then I would like some more information before we consider remediation to be completed. In the context of the CTJ system, what steps have been taken to ensure that this type of problem will not be repeated? For example, has pre-issuance linting been implemented?

Comment 15

[Jeremy Rowley]

Still planned, but let me talk to CTJ on when they can complete the migration effort. We've finished the language conversion to Japanese recently so most blockers should be complete. I pinged Magura-san here and in email (yesterday). Should have an answer from him tomorrow.

Comment 16

[Masaru Sakamoto]

We have integrated pre-issuance checking via the established certificate linting program into our issuance pipeline. It has been working effectively since March 30th, 2018.

Now that DigiCert has completed Japanese support, we would like to accelerate the migration. But I would like to emphasize our pre-issuance linting.

Mo (Masaru)
Cybertrust Japan

Comment 17

[Ryan Sleevi]

Thanks. Then the summary is:

1. Metadata in OU fields

• See Bug #1389172: Comments 0, 1, 4, 9
• See Comment #5, Comment #10
• Remediation
  • 2017-04-09 - CTJ systems patched
  • 2018-03-30 - CTJ systems integrating pre-issuance linting

2. Reserved IP Address

• See Bug #1389172: Comments 3, 6, 9
• See Comment #1, Comment #5
• Remediation
  • 2017-09-11 - Implementation of a stricter separation of duties and group responsibility for cross-checking work completion.