Closed Bug 1397957 Opened 8 years ago Closed 6 years ago

DigiCert / CTJ: Metadata in OU fields, Reserved IP Address

Categories

(CA Program :: CA Certificate Compliance, task)

Product:
CA Program
Component:
CA Certificate Compliance
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: kathleen.a.wilson, Assigned: jeremy.rowley)

References

Details

(Whiteboard: [ca-compliance] [ov-misissuance])

This bug has been separated out from Bug #1389172 to request an incident report specific to this subCA. CTJ a) Metadata in OU fields - See Bug #1389172: Comments 0, 1, 4, 9 - Remediation - 2017-08-16 - CTJ systems patched - 2018-08-XX - CTJ migrates to a new system (See Comment #1) b) Reserved IP Address - See Bug #1389172: Comments 3, 6, 9 - Remediation - None performed; no root cause analysis performed For the problems listed above, please provide an incident report as described here: https://wiki.mozilla.org/CA/Responding_To_A_Misissuance#Incident_Report
Hey Kathleen - I thought I posted about (b) to the Mozilla mailing. If not, I apologize for the oversight. The info is below. (a) As mentioned, this was previously discussed on the Mozilla mailing list where the decision was made to patch and not revoke the certs https://groups.google.com/forum/#!msg/mozilla.dev.security.policy/DgeLqKMzIds/E9--gj5WEAAJ). I thought this issue was closed. Just in case it's not: 1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, via a discussion in mozilla.dev.security.policy, or via a Bugzilla bug), and the date. - The issue was posted to the Mozilla dev list a while ago. 2. A timeline of the actions your CA took in response. - We immediately responded and contacted CTJ in the other thread. Engaged with the community, discussing the lack of a security risk. The ultimate conclusion of the discussion was that CTJ needed to patch their systems but not revoke the certificates. The fix was to turn OU into a validated field that is reviewed by validation staff prior to issuance. 3. Confirmation that your CA has stopped issuing TLS/SSL certificates with the problem. - Confirmed. CJT has patched their system to detect whether the OU field has metadata and prevents issuance if non-validated information is present. 4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued. - Lots of certs with the one problem. The corpus of certificates was identified in the older discussion and bug. The last one I could find was Jan 2017 (75591249). The first one was probably as early as when the BRs were adopted. 5. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem. - This is in the other bug. Do you want me replicate it here? 6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now. - OU is traditionally just a dumping ground for non-validated information. Although the BRs adopted the requirement that no other field should contain meta-data, the requirement does not squarely apply to the OU field under the language (as it's not an "Other Subject" field. However, rather than fight over nuances with respect to the OU, CTJ patched the issue and put safeguards in place to prevent re-occurrence. 7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things. - CTJ is migrating to a DigiCert hosted solution. This is expected to take approximately a year (Mid-2018?) CTJ is a great CA and responds promptly to all communication. The patch plus the move to validated information resolved the issue. B) This cert was not actually mis-issued. Instead, the cert was not properly revoked in accordance with the BR requirements for reserved IP addresses. Should it be a different problem report as "mis-issuance" is misleading? From CTJ: Number of affected certificates: One. After we received the revocation request from DigiCert, we conducted an investigation to see how many certificate have the same issue. We confirmed that no other active certificates except the one that was pointed out by Jonathan contain reserved IP address. The CN of this certificate is g2-sanfull01.ctjssl.info, and this is for our own use. Cause of missing the revocation: The last scan we conducted to find out certificates with reserved IP address was February 2016. At that time, we found out one certificate that were active and contained reserved IP. This certificate is g2-sanfull01.ctjssl.info, the same certificate that is pointed out in the forum. Also, we had already stopped issuing new or renewal certificates containing reserved IP address at that time. So this would mean we could pay attention to only one certificate of our own to be revoked. This is why we did not implement additional scanning after February, 2016. We planned to revoke later; however, we forgot to do the revocation. Therefore, the cause of this issue is human error and lack of supervising. Our administrators forgot to register the revocation task on their task schedule and the manager did not follow. Remediation actions: When any provisions of the industry standards such as BR become effective, we will re-check our compliance by scanning our certificates. Although scanning has been completed in the past, we will check again one month before the effective date. Also, we already are checking crt.sh daily to see compliance of our newly issued or renewed certificates. - Again, CTJ is migrating to a hosted CA, which will eventually resolve all of these issues. CTJ hasn't repeated any of these mis-issuance or had problems since we closed (a) previously.
Flags: needinfo?(masaru.sakamoto)
Flags: needinfo?(ben.wilson)
(In reply to Jeremy from comment #1) > 2. A timeline of the actions your CA took in response. > - We immediately responded and contacted CTJ in the other thread. This is not a timeline. > 3. Confirmation that your CA has stopped issuing TLS/SSL certificates with > the problem. > - Confirmed. CJT has patched their system to detect whether the OU field has > metadata and prevents issuance if non-validated information is present. When? A complete answer in #2 would have avoided this question :) > 4. A summary of the problematic certificates. For each problem: number of > certs, and the date the first and last certs with that problem were issued. > - Lots of certs with the one problem. The corpus of certificates was > identified in the older discussion and bug. The last one I could find was > Jan 2017 (75591249). The first one was probably as early as when the BRs > were adopted. This is not a complete answer. > 5. The complete certificate data for the problematic certificates. The > recommended way to provide this is to ensure each certificate is logged to > CT and then list the fingerprints or crt.sh IDs, either in the report or as > an attached spreadsheet, with one list per distinct problem. > - This is in the other bug. Do you want me replicate it here? Yes please. > 7. List of steps your CA is taking to resolve the situation and ensure such > issuance will not be repeated in the future, accompanied with a timeline of > when your CA expects to accomplish these things. > - CTJ is migrating to a DigiCert hosted solution. This is expected to take > approximately a year (Mid-2018?) CTJ is a great CA and responds promptly to > all communication. The patch plus the move to validated information resolved > the issue. Given that an incomplete answer was provided in #2, this statement cannot be independently verified as to how effectively the CA responds. > B) This cert was not actually mis-issued. Instead, the cert was not > properly revoked in accordance with the BR requirements for reserved IP > addresses. Should it be a different problem report as "mis-issuance" is > misleading? From CTJ: > We planned to revoke later; however, we forgot to do the revocation. > Therefore, the cause of this issue is human error and lack of supervising. > Our administrators forgot to register the revocation task on their task > schedule and the manager did not follow. > > Remediation actions: > When any provisions of the industry standards such as BR become effective, > we will re-check our compliance by scanning our certificates. Although > scanning has been completed in the past, we will check again one month > before the effective date. Also, we already are checking crt.sh daily to > see compliance of our newly issued or renewed certificates. Note that these proposed mitigations do not address the cause of this failure - which was human error and a lack of supervising. What changes have been done to ensure prompt adherence? That is, the substance of the reply can be read as "We'll do it right next time", but provides no assurances as to how that can be considered, in light of the failure to do it right this time.
With more detail this time: (a) META DATA IN THE OU 1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, via a discussion in mozilla.dev.security.policy, or via a Bugzilla bug), and the date. - The issue was posted to the Mozilla dev list on Aug 9, 2017. 2. A timeline of the actions your CA took in response. - We investigated to see what was going on that same day, responding to Jonathan. We also contacted CTJ. We posted some initial findings to that mailing list. We determined this was not a direct violation of the BRs as it states "All other subject attributes MUST contain information that has been verified by the CA". OU is not an "other subject attribute" and is defined in the section immediately prior to this. As such, we have not revoked these certificates. 3. Confirmation that your CA has stopped issuing TLS/SSL certificates with the problem. - Confirmed. CTJ has patched their system to detect whether the OU field has metadata and prevents issuance if non-validated information is present. Although I don't think this is actually a violation per the above explanation, we do see value in preventing meta-data from all subject fields, not just other subject fields. 4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued. - Unknown. Customers can enter data they want in the OU Field. These are screened for addresses and names. However, CTJ did patch their systems to prevent this early in 2017. The last cert I could find with metadata in the OU was Jan 2017 (75591249). 5. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem. This is the list of all certs with metadata in the OU: 53176869 57830766 65616693 117197299 93295619 104632894 21853640 22198796 30236343 30547236 32078607 34809638 39655364 40493955 43400295 45247984 47645844 45815760 40840774 48674276 40627018 48832585 57830776 57830777 57830779 57830781 61135216 57785675 57785694 53176900 53176901 53176903 64152048 78648920 53176895 53176898 57830773 102832536 51567050 132148480 110336219 111461024 80230338 81246678 91907174 91907175 110354478 110354479 78766827 79544403 78494050 80022464 15522999 16058171 20501110 20501178 21202825 21202827 21202829 20501158 20501137 20686696 20501187 20501201 20501217 20686693 20835061 21202831 21202833 21202835 21202836 26829282 11658435 11723002 11020877 11758671 12502488 12502489 13247204 13247205 13388567 42698962 16519562 19240755 20479086 20103452 21192792 54658744 21477087 42505252 29807378 57881547 30238735 30238738 30238741 30351648 47300904 54163587 54163620 54163639 51576885 56762416 70713886 73109181 78277867 73090743 77286114 75591249 6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now. - OU is traditionally just a dumping ground for non-validated information. Although the BRs adopted the requirement that no other field should contain meta-data, the requirement does not squarely apply to the OU field under the language (as it's not an "Other Subject" field. However, rather than fight over nuances with respect to the OU, CTJ patched the issue and put safeguards in place to prevent re-occurrence. 7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things. - CTJ is migrating to a DigiCert hosted solution. This is expected to take approximately a year (Mid-2018?) CTJ is a great CA and responds promptly to all communication. The patch plus the move to validated information resolved the issue.
On 6 - this was patched early 2017. I think it was in April when the previous discussion happened. I've pinged Masaru-san so he can chime in on the exact data of the patch. For B) I'll let Masura-san provide a comment on what they did to prevent missing certs in the future. I'm not sure how they changed their policy to prevent forgetting to revoke a cert.
Hi Ryan, Jeremy Thanks for giving us an opportunity to answer your questions. (a) META DATA IN THE OU 3. Confirmation that your CA has stopped issuing TLS/SSL certificates with the problem. - Confirmed. CTJ has patched their system to detect whether the OU field has metadata and prevents issuance if non-validated information is present. Although I don't think this is actually a violation per the above explanation, we do see value in preventing meta-data from all subject fields, not just other subject fields. CTJ: Apr.9, 2017. 4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued. - Unknown. Customers can enter data they want in the OU Field. These are screened for addresses and names. However, CTJ did patch their systems to prevent this early in 2017. The last cert I could find with metadata in the OU was Jan 2017 (75591249). CTJ: The last certificate with metadata in OU field was issued on March. 30, 2017. The first one since the BRs were adopted was issued on July 2, 2012. (b) Reserved IP Address Note that these proposed mitigations do not address the cause of this failure - which was human error and a lack of supervising. What changes have been done to ensure prompt adherence? CTJ: In addition to the remediation actions mentioned already, we changed work flow as following. Before remediation: Same person conducted his/her checking activities and registered issue(s) if found on an issue-tracking sheet. In this case, if this person missed registering, the manager could not realize the founded issue because the manager only checked the sheet. After remediation: We assign different persons for checking and registering. And, the outcomes of their works are reviewed by each other. Then, their manager finally checks both outcomes. Please let me know if you have any questions. Kind regards, Mo (Masaru)
Flags: needinfo?(masaru.sakamoto)
Below is my summary of the issues and remediation plan: 1) Metadata in OU fields - See Bug #1389172: Comments 0, 1, 4, 9 - See Comment #5 - Remediation - 2017-04-09 - CTJ systems patched - 2018-08-XX - CTJ migrates to a new system (See Comment #1) 2) Reserved IP Address - See Bug #1389172: Comments 3, 6, 9 - See Comment #1, #5 - Remediation - 2017-09-07 - CTJ has updated procedures to include scanning one month before the effective date of BR provisions to ensure compliance - 2017-09-11 - CTJ performs two-party validation of IP addresses to ensure they're not reserved Is this a correct summary?
A couple of modifications: 1) Metadata in OU fields - Remediation - 2017-04-09 - CTJ systems patched - 2018-12-31 - CTJ will migrate to the new system. The primary hold up is DigiCert's lack of support for the Japanese language and expected Japanese process flow. We are currently working on implementing both. Once complete, the transition will begin. 2) Reserved IP Address - Remediation - 2017-09-11 - Implementation of a stricter separation of duties and group responsibility for cross-checking work completion. CTJ implemented a fix in its systems to prevent reserved IP addresses in accordance with the appropriate timeline. However, a single validation staff member was assigned the task of revoking the certificate. Now, multiple individuals receive the assignment for revoking certificates. Validation staff is still responsible for revoking certs when required by the BRs, but different persons are assigned for the revocation and checking for completing the revocation. The validation staff reviews each other regularly to ensure all steps are carried out. Both staff members submit reports to their managers who double checks the outcome.
Flags: needinfo?(ben.wilson)
Changing QA contact per https://bugzilla.mozilla.org/show_bug.cgi?id=1438254
QA Contact: gerv → wthayer
Whiteboard: [ca-compliance] → [ca-compliance] - Next Update - 01-January 2019
Jeremy: Can you confirm the steps in Comment #7 were completed?
Flags: needinfo?(jeremy.rowley)

Unfortunately not fully migrated. They are using our systems for some certs, but not all. I'm not sure how many are issued by CTJ as a separate sub CA compared to issuance through our new system (trying to find this out which is the delayed response). The blocker has been Japanese support in our system. CTJ has improved their systems to the point they no longer are having issues.

Flags: needinfo?(jeremy.rowley)

I'm slightly concerned, given that the last substantial update was 2017-10-17 in Comment #7, which itself was a push back from the dates in Comment #1.

Do you have concrete dates for migration? It seems like a weekly update, per https://wiki.mozilla.org/CA/Responding_To_An_Incident#Keeping_Us_Informed , may be appropriate.

Flags: needinfo?(jeremy.rowley)

The blocker is currency tbh. Considering CTJ's clean record since this incident, do we need to migrate them? That was originally part of the plan, but I'm wondering if that's a Mozilla requirement or something we said that shot us in the foot. If it's a Mozilla expectation, we can force it faster. I'll have Mo chime in as well about the migration effort.

Flags: needinfo?(jeremy.rowley) → needinfo?(masaru.sakamoto)

Thanks Jeremy. I defer to Wayne. Based on this and Comment #7, I think where we stand for this issue are:

  1. Metadata in OU fields
  1. Reserved IP Address
  • See Bug #1389172: Comments 3, 6, 9
  • See Comment #1, Comment #5
  • Remediation
    • 2017-09-11 - Implementation of a stricter separation of duties and group responsibility for cross-checking work completion.
Flags: needinfo?(wthayer)

Migration away from the CTJ system is not a Mozilla requirement, but it is a sensible remediation action. Is this still really your plan to ultimately address CTJ compliance? If so, then that is an acknowledgement that some risk still exists in the legacy CTJ system, even though there have been no new incidents, and I'd prefer you to just provide a new date for completing that change. If migration is no longer the plan and the CTJ system is not slated for retirement in the near future, then I would like some more information before we consider remediation to be completed. In the context of the CTJ system, what steps have been taken to ensure that this type of problem will not be repeated? For example, has pre-issuance linting been implemented?

Flags: needinfo?(wthayer) → needinfo?(jeremy.rowley)

Still planned, but let me talk to CTJ on when they can complete the migration effort. We've finished the language conversion to Japanese recently so most blockers should be complete. I pinged Magura-san here and in email (yesterday). Should have an answer from him tomorrow.

Flags: needinfo?(jeremy.rowley)

We have integrated pre-issuance checking via the established certificate linting program into our issuance pipeline. It has been working effectively since March 30th, 2018.

Now that DigiCert has completed Japanese support, we would like to accelerate the migration. But I would like to emphasize our pre-issuance linting.

Mo (Masaru)
Cybertrust Japan

Flags: needinfo?(masaru.sakamoto)

Thanks. Then the summary is:

  1. Metadata in OU fields
  • See Bug #1389172: Comments 0, 1, 4, 9
  • See Comment #5, Comment #10
  • Remediation
    • 2017-04-09 - CTJ systems patched
    • 2018-03-30 - CTJ systems integrating pre-issuance linting
  1. Reserved IP Address
  • See Bug #1389172: Comments 3, 6, 9
  • See Comment #1, Comment #5
  • Remediation
    • 2017-09-11 - Implementation of a stricter separation of duties and group responsibility for cross-checking work completion.
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
Whiteboard: [ca-compliance] - Next Update - 01-January 2019 → [ca-compliance]
Product: NSS → CA Program
Whiteboard: [ca-compliance] → [ca-compliance] [ov-misissuance]
You need to log in before you can comment on or make changes to this bug.