1899466 - Chunghwa Telecom: Controversial Values within Extension (2.5.29.9, subjectDirectoryAttributes)

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Leo Fang]

Attachment: affectedCertList-12911-records-20240529.csv

Incident Report

How became aware of the problem.

GTLSCA received an inquiry from Bugzilla 1887096 regarding the 2.5.29.9 (subjectDirectoryAttributes) extension. We had included a potentially "controversial" value in this extension. To comply with BR, we decided to remove this extension to eliminate controversy.

Impact

A total of 12,911 certificates are affected.

Root Cause

We included a potentially "controversial" value in the 2.5.29.9 (subjectDirectoryAttributes) extension of our certificates. To comply with BR, we decided to remove this extension to eliminate controversy.

TimeLine

All times are UTC+8.

2024-05-23

• 12:00 Investigated, and reviewed the cause.
• 14:46 Stopped issuing certificates.
• 16:00 Engineers provided adjustment solutions, updated CPS in the new version
• 18:00 Adjusted the program, tested issuance
• 18:30 Resumed normal certificate issuance

As of 18:30 on 5/23, normal certificate issuance has resumed, and subsequent certificates will no longer contain the extension (2.5.29.9 subjectDirectoryAttributes).

Certificates issued after the removal of the extension are as follows:
https://crt.sh/?id=13148763476
https://crt.sh/?id=13155909251

2024-05-24

• Response to first actions taken.

2024-05-27

• Report to the supervisor for a decision.
• Decide to revoke certificates containing the controversial extension.

2024-05-28

• Assess the scope of impact and develop a timeline for certificate revocation.
• Plan the reissuance process and notify the responsible units.

2024-05-29

• Begin reissuing certificates.

Action Items

Action Item	Status	Due Date
Stop issuing and remove the controversial extension	Finished	2024-05-23
Response to first actions taken	Finished	2024-05-24
Report to the supervisor for a decision	Finished	2024-05-27
Schedule and discuss a meeting	Finished	2024-05-28
Reissue certificates	Started	2024-05-31
Contact subscribers to replace certificates	Not yet started	2024-07-14
Revoke all affected certificates	Not yet started	2024-07-14

Appendix

List of 12,911 affected certificates to be revoked.

Comment 1

[Wayne]

(In reply to Leo Fang from comment #0)

2024-05-27

• Report to the supervisor for a decision.
• Decide to revoke certificates containing the controversial extension.
  ...

Action Items

Action Item	Status	Due Date
Revoke all affected certificates	Not yet started	2024-07-14

Appendix

List of 12,911 affected certificates to be revoked.

So from the outset a timeline has been generated to handle revocation in 49 days. Chunghwa Telecom are aware that by their own Certificate Policy this must be done within 5 days? Given a delayed revocation is inevitable can you please prepare a per-subscriber breakdown of what is stopping revocation within this timeframe?

I presume this is a preliminary report as it does not conform with the CCADB incident report template and is lacking information.

Comment 2

[Leo Fang]

(In reply to Wayne from comment #1)

So from the outset a timeline has been generated to handle revocation in 49 days. Chunghwa Telecom are aware that by their own Certificate Policy this must be done within 5 days? Given a delayed revocation is inevitable can you please prepare a per-subscriber breakdown of what is stopping revocation within this timeframe?

I presume this is a preliminary report as it does not conform with the CCADB incident report template and is lacking information.

Thank you for reminding. Because some business information systems of government agencies have SLAs for continuous operation, which may not be completed within 5 days, the certificate replacement time needs to be determined based on the nature of each system. We will contact each user to complete the certificate revocation as soon as possible.

Comment 3

[Tim Callan]

It is worth noting that in 2014 it came to light that Chunghwa Telecom was misusing the 2.16.886 OID arc in Certificate Policy OIDs: (1 2 3 4 5 6 7). Taiwan CA, which appears to have had a similar problem, solved it by requesting an OID arc under IANA PEN (Private Enterprise Numbers) to use instead of 2.16.886 and advised Chunghwa Telecom to do the same. A PEN was registered, but it is disappointing to see that Chunghwa Telecom only applied the “the 2.16.886 arc has never been assigned to Taiwan, so you cannot use it” advice to Certificate Policy OIDs in particular.

Comment 4

[amir]

The root cause of this issue is extremely lacking.

Why did you include this in the first place? What problem was this extension solving that the certificate itself couldn’t solve?

Comment 5

[Tim Callan]

(In reply to Leo Fang from comment #2)

Thank you for reminding. Because some business information systems of government agencies have SLAs for continuous operation, which may not be completed within 5 days, the certificate replacement time needs to be determined based on the nature of each system. We will contact each user to complete the certificate revocation as soon as possible.

I will point out than in bug 1892419 comment 8 Amir stated,

I think this is a deeply concerning statement. There are a few consequences to this statement:

1. If mass revocation is required in the future, you're going to fail the mass revocation again.

Now here we are, less than a month later, and Chunghwa Telecom is informing us that another failed mass revocation is coming. This isn’t surprising considering that none of the action items in bug 1892419 comment 0 actually addresses the problem of delayed revocation and that in its handling of bug 1892419 and bug 1887096 Chunghwa Telecom fails to take responsibility for its deliberate choice not to revoke misissued certificates on time.

Your Subscribers went into contract with you, presuming they signed the Subscriber Agreement, knowing that a 24-hour or 5-day revocation event is possible at any time. If they were aware of government regulations preventing that, they should have highlighted this to the CA, who in turn should have informed these Subscribers that public certificates did not meet their business requirements. Signing a contract to which Subscribers knew they could not adhere is a near-ubiquitous theme within delayed revocation bugs. We cannot allow that to prevent CAs from meeting their responsibilities.

Comment 6

[Wayne]

It has been over a week and multiple questions are still outstanding in this incident.

Comment 7

[amir]

Chunghwa:

Your lack of response here has now created another incident that I'd expect to be filed.

1. Your existing failure to revoke on time, since apparently you're not planning to finish revoking until July?
2. Another incident for why you're unable to respond to the requests of the community in a timely manner.

Beyond that, I want to reiterate my question:

In 1887096 I found that Chunghwa has been misusing this OID value since at least, July 30, 1998

Can you please provide me with this information:

1. Please do a look into your historical documents, and see if Chunghwa was aware they were misusing this OID space?
2. I assume that you know that CAs are required to self-audit themselves yearly. Have you consistently done that? Was this ever brought up by any persons at currently, or previously at Chunghwa?
3. Have any of your auditors every questioned you about the use of this OID or extension? Did they raise any concerns about it privately with you?

Comment 8

[Leo Fang]

Attachment: revoke0613-crtsh-4321.csv

Update:

• 6/13 revoked 4,306 certificates
• 6/13 expired 15 certificates

Total affected certificates: 12,911
Total certificates revoked: 4,306 (33.35%)
Total certificates expired: 15 (0.12%)
Remaining: 8,590 (66.53%)

We have received a lot of valuable feedback recently and are currently in discussions with government agencies. We expect to response next week. Thank you.

Comment 9

[amir]

I do not understand what relevance you chatting with government agencies has here? Your compliance requirements to public trust take precedence over your discussions with government agencies. Do you not agree?

You’re required to open the incident within 72 hours and have the full report ready within two weeks.

Are you saying you’re not going to do that? I hope we get a response as of why you’re not following the incident response procedure here - the procedure is not up for debate.

Comment 10

[Wayne]

This is not the place for a delayed revocation incident. You need to create a new report for that and get the necessary information which is not in your current comment. There have been more than enough delayed revocation incidents lately that should have already been read that this should not be a surprise.

It is extremely disheartening as there are multiple outstanding incidents that I can see:

• Delayed revocation for this issue alone, I explicitly gave advice on this 16 days ago
• Report not raised within 72h for the Delayed Revocation issue
• Incident for the OID misuse and further revocations
• Report not raised within 72h for the OID incident mentioned

And presumably another delayed revocation for the OID incident at the rate we are going. I'm sure there are more but Chunghwa Telecom need to be moving with haste to resolve this, not conferring with a government agency that has no relevancy to this issue at hand.

Comment 11

[Tsung-Min Kuo]

(In reply to Tim Callan from comment #5)

(In reply to Leo Fang from comment #2)

Thank you for reminding. Because some business information systems of government agencies have SLAs for continuous operation, which may not be completed within 5 days, the certificate replacement time needs to be determined based on the nature of each system. We will contact each user to complete the certificate revocation as soon as possible.

I will point out than in bug 1892419 comment 8 Amir stated,

I think this is a deeply concerning statement. There are a few consequences to this statement:

1. If mass revocation is required in the future, you're going to fail the mass revocation again.

Now here we are, less than a month later, and Chunghwa Telecom is informing us that another failed mass revocation is coming. This isn’t surprising considering that none of the action items in bug 1892419 comment 0 actually addresses the problem of delayed revocation and that in its handling of bug 1892419 and bug 1887096 Chunghwa Telecom fails to take responsibility for its deliberate choice not to revoke misissued certificates on time.

Your Subscribers went into contract with you, presuming they signed the Subscriber Agreement, knowing that a 24-hour or 5-day revocation event is possible at any time. If they were aware of government regulations preventing that, they should have highlighted this to the CA, who in turn should have informed these Subscribers that public certificates did not meet their business requirements. Signing a contract to which Subscribers knew they could not adhere is a near-ubiquitous theme within delayed revocation bugs. We cannot allow that to prevent CAs from meeting their responsibilities.

I agree with this reply. I think every CA should face up to this problem and take their responsibility, even they are just a subordinate CA, all these regulations and responsibility has been set forth in the BR and their CPS, they should know that.

Please give me couple days. I think it is necessary for the Root CA team to hold bilateral meetings with the GTLSCA CA team and carry out relevant reform plans.

Comment 12

[Tsung-Min Kuo]

(In reply to Wayne from comment #10)

This is not the place for a delayed revocation incident. You need to create a new report for that and get the necessary information which is not in your current comment. There have been more than enough delayed revocation incidents lately that should have already been read that this should not be a surprise.

And presumably another delayed revocation for the OID incident at the rate we are going. I'm sure there are more but Chunghwa Telecom need to be moving with haste to resolve this, not conferring with a government agency that has no relevancy to this issue at hand.

That's because the subscriber of GTLSCA are all government agencies in Taiwan, none of them are business or individual.

It is extremely disheartening as there are multiple outstanding incidents that I can see:

• Delayed revocation for this issue alone, I explicitly gave advice on this 16 days ago
• Report not raised within 72h for the Delayed Revocation issue
• Incident for the OID misuse and further revocations
• Report not raised within 72h for the OID incident mentioned

I am in represent of Root CA team to help GTLSCA CA team to response these questions and make sure they will take necessary actions.
In the viewpoint of Root CA team, I think the revocation should be executed as scheduled to comply with the BR, even if the subscriber window cannot be contacted. I will join the discussion and follow up this bug, and hold a bilateral meeting with the GTLSCA CA team to speed up the revocation.

Comment 13

[Leo Fang]

(In reply to amir from comment #9)
I do not understand what relevance you chatting with government agencies has here? Your compliance requirements to public trust take precedence over your discussions with government agencies. Do you not agree?

Yes, we do.

You’re required to open the incident within 72 hours and have the full report ready within two weeks.

Thank you for your reminder, amir. We opened a new incident report 1903066 describing delayed revocation .

Comment 14

[Leo Fang]

(In reply to Tim Callan from comment #3)

It is worth noting that in 2014 it came to light that Chunghwa Telecom was misusing the 2.16.886 OID arc in Certificate Policy OIDs: (1 2 3 4 5 6 7). Taiwan CA, which appears to have had a similar problem, solved it by requesting an OID arc under IANA PEN (Private Enterprise Numbers) to use instead of 2.16.886 and advised Chunghwa Telecom to do the same. A PEN was registered, but it is disappointing to see that Chunghwa Telecom only applied the “the 2.16.886 arc has never been assigned to Taiwan, so you cannot use it” advice to Certificate Policy OIDs in particular.

Thank you, this part has been removed immediately after confirming the issue.

Comment 15

[Leo Fang]

(In reply to amir from comment #4)

The root cause of this issue is extremely lacking.

Why did you include this in the first place? What problem was this extension solving that the certificate itself couldn’t solve?

The predecessor of GTLSCA is GCA. Our subscribers are only government agencies and do not include corporate or individual users. Later, in order to comply with BR regulations, GTLSCA was moved under CHT Root, but it is true that we did not check this wrong setting at the time.

Comment 16

[Leo Fang]

(In reply to amir from comment #7)

Chunghwa:

Beyond that, I want to reiterate my question:

In 1887096 I found that Chunghwa has been misusing this OID value since at least, July 30, 1998

Can you please provide me with this information:

1. Please do a look into your historical documents, and see if Chunghwa was aware they were misusing this OID space?

This issue can be traced back to the discussion in 2014. We already know that the value of BR cannot be controversial, so we removed it as soon as possible after the discussion, and we will not defend it.

2. I assume that you know that CAs are required to self-audit themselves yearly. Have you consistently done that? Was this ever brought up by any persons at currently, or previously at Chunghwa?

Since GTLSCA was established in 2019, it should have experienced the issues in 2014 and should not have this value. We asked RootCA and got the same answer (this value cannot be included). We will add this certificate profiling check during annual self-examination and align it with RootCA.

3. Have any of your auditors every questioned you about the use of this OID or extension? Did they raise any concerns about it privately with you?

Our external auditors have not asked this question. We will remind them to conduct key confirmation inspections of this project in the audit plan every subsequent year.

Comment 17

[Ben Wilson]

If this issue can be closed, then please indicate such in a comment here and add a "Need Info" for me ("Request information from triage owner").

Comment 18

[Leo Fang]

For this bug, all action items had been completed.
There are no further action items related to this bug.
We kindly request that it be closed.

Comment 19

[Ben Wilson]

I will close this on Wed. 11-Sept-2024.