Closed Bug 1815534 Opened 1 year ago Closed 2 days ago

e-commerce monitoring GmbH: SCT in precertificate

Categories

(CA Program :: CA Certificate Compliance, task)

Product:
CA Program
Component:
CA Certificate Compliance
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: agwa-bugs, Assigned: ca)

Details

(Whiteboard: [ca-compliance] [ov-misissuance] [external])

Attachments

(1 file)

e-commerce monitoring GmbH takes the allegation seriously and has analyzed the matter in detail. The appendix contains the exact timeline, the analysis and the findings. Regardless of the interpretation of RFC 6962 and the behavior of the ct-log server, e
4.17 KB, text/plain
Details

e-commerce monitoring GmbH has produced the following signed artifacts:

  1. Correctly-formed precertificate with serial number 10:45:67:49:88:c4:db:00:76:89:b9:
    https://crt.sh/?sha256=6f1fd13b3c72fd064e01b4fa810d44a06bde36f44876179a45bf752ae988d6cf

  2. Precertificate with serial number 10:45:67:49:88:c4:db:00:76:89:b9 containing an SCT extension,
    in violation of RFC6962 which only defines the SCT extension for final certificates, not precertificates:
    https://crt.sh/?sha256=c967a3109b953f8c14ffda5901db5f76d1bc0b6f84a3b6163ce14cd16d12086a

  3. Certificate with serial number 10:45:67:49:88:c4:db:00:76:89:b9 containing SCTs with invalid signatures:
    https://crt.sh/?sha256=5ee21e3bb051f280dec1339e83f60969919837da84a554007553edfb77c74a5f

Certificate 3 matches precertificate 1 according to the algorithm in RFC 6962, but
its embedded SCTs are for precertificate 2.

Assignee: nobody → ca
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true
Whiteboard: [ca-compliance] [ov-misissuance]

Thank you Andrew, we will have a look at this and get back soon.

Hi Andrew,

Thank you for your message.

e-commerce monitoring GmbH takes the allegation seriously and has analyzed the matter in detail. The appendix contains the exact timeline, the analysis and the findings. Regardless of the interpretation of RFC 6962 and the behavior of the ct-log server, e-commerce monitoring GmbH is also working to improve the issuing of certificates.

https://bugzilla.mozilla.org/attachment.cgi?id=9316626

Best regards
Hans

From attachment 9316626 [details]:

RFC 6962 don't say anything about using SCTs in pre-certificates.
Regardless, I agree that SCT entries in pre-certificates don't make sense.

Hi Hans.

Are you claiming that this bug describes behaviour that, although unwanted, is fully compliant with the Mozilla Root Store Policy and is therefore not an incident?

RFC 6962 says nothing about the content of further extensions of a pre-certificate.
...
In future, we will therefore no longer only trust the operators of the ct-log servers, but have taken our own measures to avoid unwanted behavior.

In what way(s) did you "trust the operators of the ct-log servers"?

The Abstract of RFC6962 explains that CT logs are designed to enable "anyone to...notice the issuance of suspect certificates". This requires submission of a suspect certificate to be accepted by the log, not rejected.

RFC6962 section 3 requires that "When a valid certificate is submitted to a log, the log MUST immediately return a Signed Certificate Timestamp (SCT)", so ISTM that the logs behaved correctly by returning SCTs when you submitted https://crt.sh/?sha256=c967a3109b953f8c14ffda5901db5f76d1bc0b6f84a3b6163ce14cd16d12086a. (In RFC6962 section 3.1 it becomes clear that "valid certificate" is essentially shorthand for "valid signature chain leading back to a trusted root CA certificate").

Hi all,

Thanks for the reply. In my first statement, I only stated the facts. As already explained in detail, we also take the matter seriously and have therefore taken measures to prevent such undesirable situations in the future. In particular, there will be extended checks on the response of the ct-log server (this is what "trust ct-log server" means). There are certainly different interpretations of the interpretation of RFC6962, which I respect and which we will in any case take into account.

Now it is time to look to the future and approach the matter soberly. Incidents should serve to ensure that all sides learn from them and improve their services. In this sense, I am open to any constructive suggestion.

Best Regards
Hans

Hi Hans. Let me explain what I believe is the non-compliant behaviour that makes this bug an Incident.

MRSP 5.4 says:
"if a precertificate implies the existence of a final certificate that does not comply with this policy, it is considered misissuance of the final certificate, even if the certificate does not actually exist."

The correctly formed precertificate identified in comment 0 implies the existence of a corresponding "final" certificate. The SCTs embedded in this "final" certificate don't match the corresponding precertificate, which is unhelpful, but I'm not sure that it's necessarily non-compliant. The only relevant requirement I've found is in BR 7.1.2.4, which says that "The CA SHALL NOT issue a Certificate that contains a...Certificate extension...not specified in Section 7.1.2.1, Section 7.1.2.2, or Section 7.1.2.3 unless the CA is aware of a reason for including the data in the Certificate." Are you aware of a "reason for including the" wrong SCT list "data in the Certificate"? It doesn't say that it has to be a good reason, so it's a pretty low bar. Maybe you could argue that the "reason" was that your CA system had a bug that you weren't previously aware of.

However, the other precertificate also implies the existence of a different, corresponding "final" certificate. Why does it have to be "different"? It's because the "final" certificate that we have observed doesn't match this precertificate according to the RFC6962 algorithm. Now, I'm guessing that you haven't actually issued this different "final" certificate, but nonetheless MRSP 5.4 operates on the basis that you have issued it.

The assumed-to-exist "final" certificate has the same issuer name and serial number as the "final" certificate that we have observed. These two "final" certificates are both in scope for the BRs/MRSP/RFC5280, which prohibit multiple "final" certificates from sharing the same issuer name and serial number.

Hi Rob,
I totally agree with your statements.

There is of course no intention of issuing another certificate with the same serial number. Therefore, there is no intention to use the now existing pre-certificate to issue a real certificate.

I would therefore like to come to a solution that meets the requirements and is correctly mapped in the framework of crt.sh. Do you have a suggestion on how we can solve this in short time? Would revoking the pre-certificate solve the problem?

I'm open to any suggestion.

Best Regards
Hans

Hi Hans. Sorry for the delayed reply.

I don't understand what "correctly mapped in the framework of crt.sh" is intended to mean, or why that would be relevant to this bug.

Would revoking the pre-certificate solve the problem?

Since all of the precertificates and certificates (both those observed and those presumed to exist) mentioned so far in this bug share the same issuer name and serial number, it's only possible to revoke all of them via CRL and/or OCSP. As far as the CRL and OCSP protocols are concerned, they are all one and same certificate.

The latest of the precertificates was definitely misissued (see my explanation in comment 6), and BR 4.9.1.1 says that the "CA SHOULD revoke a certificate within 24 hours and MUST revoke a Certificate within 5 days if ... 7. . The CA obtains evidence that the Certificate was misused; ... 12. The CA is made aware that the Certificate was not issued in accordance with these Requirements".

So ISTM that revocation is both necessary and should have already been done.

Dear Rob,

thank you for your comment. We get back with an answer during next week.

Best regards,
Daniel

Hi,
the certificate number 10:45:67:49:88:c4:db:00:76:89:b9 is revoked as commited.
Best
Hans

Does anyone have any additional questions, comments, or "lessons learned"?
Thanks,
Ben

Flags: needinfo?(bwilson)

(speaking solely as an individual, not as a representative of my employer)

Sorry, but it seems like we're missing a lot of analysis here. Although e-commerce monitoring GmbH provided a timeline and root cause analysis in the attached file, that write-up does not meet the expectations of an incident report (e.g. it does not provide a list of steps the CA is taking to ensure the incident will not reoccur in the future), and does not address the actual incident at hand.

It appears to me that two incident reports still need to be provided, addressing the following issues:

  1. Issuance of two different certificates with the same issuer and same serial number, a violation of RFC 5280 Section 4.1.2.2 and the BRs Section 7.1.2.4; and
  2. Failure to revoke a certificate within 5 days of being made aware that the certificate was issued in violation of the BRs, a violation of BRs Section 4.9.1.1 Paragraph 12.

Agree with Comment 12. This incident has really fallen short of expectations, both in the quality of the incident report and the timeliness of the response.

Flags: needinfo?(bwilson) → needinfo?(ca)

Thank you Rob, Aaron and Andrew for contributing to this Bug.
we are now finalzing incident reporting. I'll file this on the short term and post cross reference, where helpful.

Flags: needinfo?(ca)

just filed Bug 1830536 (e-commerce monitoring gmbh: certificate issued with two pre-certificates) https://bugzilla.mozilla.org/show_bug.cgi?id=1830536

Daniel,
Please provide a status on this bug.
Thanks, Ben

Flags: needinfo?(ca)

The three signed data structures deemed to be one and the same certificate have been revoked on the 30st of March 2023.

https://crt.sh/?sha256=6f1fd13b3c72fd064e01b4fa810d44a06bde36f44876179a45bf752ae988d6cf
https://crt.sh/?sha256=c967a3109b953f8c14ffda5901db5f76d1bc0b6f84a3b6163ce14cd16d12086a
https://crt.sh/?sha256=5ee21e3bb051f280dec1339e83f60969919837da84a554007553edfb77c74a5f

A number of improvements have since been made. They are all implemented at the program level so as part of our internal certificate linting process.

  • a certificate that contains a CT Precertificate Poison must not contain SCTs
  • a certificate that contains SCTs must not contain a Precertificate Poison extension
  • (if a certificate is not intended for TLS purposes it must not contain either)
  • the monitoring of the lists of usable ct servers has been improved
  • if for any reason no sufficient number of usable SCTs (currently 2x Google, 2x Non-Google) can be retrieved, the certificate is not issued
  • it is ensured that two different certificates can not share the same issuer and serial number, now also for the case that one of these certificates is only a presumed-to-exist one by virtue of a precertificate, as pointed out in Comment 6
  • for this last point - certificate issued with 2 precerts, so duplicate serial number - we have filed bug https://bugzilla.mozilla.org/show_bug.cgi?id=1830536 upon the community's wish to separate the disctinct issues.

We are convinced that with this improvements the underlying problem is remediated and it is ensured the incident will not occur again in the future.

Flags: needinfo?(ca)

Hi Daniel,

This update still does not meet the requirements of an incident report. Please see the instructions at https://www.ccadb.org/cas/incident-report#incident-reports on how to structure an incident report and what information needs to be included.

Flags: needinfo?(ca)

Hi Gable,
thank you for your interest in the matter. You have probably already read our report.
Can you please explain exactly what information you are missing or is it just a question of formatting.

Best regards

Bug 1830536 constitutes an incident report for the first issue identified in https://bugzilla.mozilla.org/show_bug.cgi?id=1815534#c12: two precertificates with the same serial number. That part is fine.

However, no incident report has been filed for the second issue identified in that comment: failure to revoke within 5 days of being made aware that the certificate was issued in violation of the BRs.

This bug was opened on Feb 7. You received confirmation that yes, this is definitely an incident, on Feb 16 (Comment 6), and then again on March 22 (Comment 8). You acknowledged that you were investigating again on March 23 (Comment 9). The certificates were finally revoked on March 30. Even in the loosest interpretation of this timeline, it's 7 days from notification (March 23) to revocation (March 30). In the strictest interpretation, it was 51 days.

The Baseline Requirements are clear:

Section 4.9.1.1: [T]he CA [...] MUST revoke a Certificate within 5 days [if...] The CA is made aware that the Certificate was not issued in accordance with these Requirements.

The CCADB Policy is also clear:

CA Owners should submit an additional, separate incident report when... Policy requires the revocation of one or more certificates by a certain deadline, such as those in BR section 4.9, but that deadline will not be or has not been met by the CA Owner.

It is disappointing to me that it took e-commerce monitoring GmbH 51 days to determine that the certificates were misissued and finally revoke them. It is also disappointing to me that e-commerce monitoring GmbH has still not filed an incident report for this delay, even after being explicitly told that doing so is necessary 6 months ago.

There is a new template for incident reporting that you should use - see https://www.ccadb.org/cas/incident-report#incident-report-template

Flags: needinfo?(hans.zeger)

(In reply to Aaron Gable from comment #20)

However, no incident report has been filed for the second issue identified in that comment: failure to revoke within 5 days of being made aware that the certificate was issued in violation of the BRs.

This bug was opened on Feb 7. You received confirmation that yes, this is definitely an incident, on Feb 16 (Comment 6), and then again on March 22 (Comment 8). You acknowledged that you were investigating again on March 23 (Comment 9). The certificates were finally revoked on March 30. Even in the loosest interpretation of this timeline, it's 7 days from notification (March 23) to revocation (March 30). In the strictest interpretation, it was 51 days.

The Baseline Requirements are clear:

Section 4.9.1.1: [T]he CA [...] MUST revoke a Certificate within 5 days [if...] The CA is made aware that the Certificate was not issued in accordance with these Requirements.

The CCADB Policy is also clear:

CA Owners should submit an additional, separate incident report when... Policy requires the revocation of one or more certificates by a certain deadline, such as those in BR section 4.9, but that deadline will not be or has not been met by the CA Owner.

Accordingly, I have opened another bug, and a separate incident report is required to be filed. See Bug #1862004.

Hi Aaron Gable,
hope You are well.

Can you please explain to me what "In the strictest interpretation, it was 51 days." is meant?

We submitted a report 4 days after Androw's report (see attached file) and then another addition a few days later (comment2). All points in the sense of https://www.ccadb.org/cas/incident-report#incident-report-template have been addressed.

Which point are you actually missing? Or is it just a matter of formatting?

Looking forward to hear from you.
Hans

(In reply to Aaron Gable from comment #20)

Bug 1830536 constitutes an incident report for the first issue identified in https://bugzilla.mozilla.org/show_bug.cgi?id=1815534#c12: two precertificates with the same serial number. That part is fine.

However, no incident report has been filed for the second issue identified in that comment: failure to revoke within 5 days of being made aware that the certificate was issued in violation of the BRs.

This bug was opened on Feb 7. You received confirmation that yes, this is definitely an incident, on Feb 16 (Comment 6), and then again on March 22 (Comment 8). You acknowledged that you were investigating again on March 23 (Comment 9). The certificates were finally revoked on March 30. Even in the loosest interpretation of this timeline, it's 7 days from notification (March 23) to revocation (March 30). In the strictest interpretation, it was 51 days.

The Baseline Requirements are clear:

Section 4.9.1.1: [T]he CA [...] MUST revoke a Certificate within 5 days [if...] The CA is made aware that the Certificate was not issued in accordance with these Requirements.

The CCADB Policy is also clear:

CA Owners should submit an additional, separate incident report when... Policy requires the revocation of one or more certificates by a certain deadline, such as those in BR section 4.9, but that deadline will not be or has not been met by the CA Owner.

It is disappointing to me that it took e-commerce monitoring GmbH 51 days to determine that the certificates were misissued and finally revoke them. It is also disappointing to me that e-commerce monitoring GmbH has still not filed an incident report for this delay, even after being explicitly told that doing so is necessary 6 months ago.

Hi Ben,

Please understand for me. The link points to a template that didn't exist in February. Should we enter the information from February here?

Best
Hans

(In reply to Ben Wilson from comment #21)

There is a new template for incident reporting that you should use - see https://www.ccadb.org/cas/incident-report#incident-report-template

Flags: needinfo?(hans.zeger)

Yes - please use the new form to report everything from February (including any relevant events from before and after February).
Thanks,
Ben

(In reply to Andrew Ayer from comment #13)

Agree with Comment 12. This incident has really fallen short of expectations, both in the quality of the incident report and the timeliness of the response.

Hi,
Thanks for response. Our report may not meet your formal expectations, but from our point of view it is a complete report that clearly presents our position.

Regrads
Hans

(In reply to Ben Wilson from comment #25)

Yes - please use the new form to report everything from February (including any relevant events from before and after February).
Thanks,
Ben

Hi,
to my understanding:
You mean not using the October 2023 template in February is an incident?

Or is it the lack of hashes in our statements? Maybe it's not about certificates that have been revoked for eight months, but about something else. Please to enlightenment.

Best regards
Hans

Incident Report

Summary

Issuance of one leaf certificate for which there are existing two pre-certificates

Impact

Reviewing our issuing Processes

Timeline

2023-02-02 14:04 UTC: After retrieving the SCTs using the precertificate, we found that the number and composition of the SCTs did not meet our specifications (2 instead of 3 SCTs and no SCTs from a Google ct-log-server). A detailed analysis revealed that some Google ct-log-servers changed the format of their urls at the turn of the year. This unused pre-certificate has also been revoked.

2023-02-07 Our software was converted to the new url format and the entire module was subjected to a comprehensive revision. No server end-entity certificates were issued during this conversion and revision process.

2023-02-07 10:23 UTC: An SCT query was made for "10:45:67:49:88:c4:db:00:76:89:b9" (in your report precertificate and response #1), which was insufficient according to our specifications (only one SCT instead of three). Since this SCT was suitable, it was stored internally.

2023-02-07 10:38 UTC: Shortly thereafter, another SCT query was made, the response of which met our specifications (in your report precertificate and response #2). These SCT values were stored internally.

2023-02-07 10:50 UTC: Based on the correct SCT entries from response #2, the end-entity certificate of authenticity was issued (in your report certificate and response #3). The end-entity certificate of authenticity contains only SCTs from response #2

2023-02-07 A subsequent review was carried out, primarily to avoid future disruptions due to unexpected changes in the url format and to speed up the issuing process of server end-entity certificates.

In the course of the review, we were also able to determine that the pre-certificate (for response #2) contained an SCT.

Root Cause Analysis

No Root affected

Lessons Learned

What went well

We had already implemented all the necessary checks (see below) to avoid the behavior described before Andrew Ayer (2023-02-07 19:33) reported this case to bugzilla.

We continuously analyze our operations with regard to potential deviations from legal, technical or other requirements and agreements, like CA/Browserforum-Requirements, RFC, ETSI or eIDAS-Regulation.

We improve compliance management on an ongoing basis, aiming to reduce the error rate to an absolute minimum of acceptable rest risk.

GLOBALTRUST also deem it as a higher goal to contribute with the „lessons learned“ to the benefit of the entire community. Therefore we are looking also beyond this singular case to apply a holistic approach and to describe the situation in the overall structure of a constantly changing system of processes, as transparently as possible.

We have now thoroughly examined the case and you will find below an incident report that has been prepared to the best of our knowledge and belief, and which we believe should meet the expectations of the community and the corresponding requirements. We look forward to your constructive feedback.

What didn't go well

The undesired behavior resulted from the need to run multiple ct-log queries. However, the same certificate content including the same modulus, the same algorithm and the same serial number was used in all queries. The SCTs are no part of the certificate.

RFC 6962 requires the extension 1.3.6.1.4.1.11129.2.4.3 with value NULL. All pre-certificates mentioned in the bug contain this value.

RFC 6962: "The Precertificate is constructed from the certificate to be issued by adding a special critical poison extension (OID 1.3.6.1.4.1.11129.2.4.3, whose extnValue OCTET STRING contains ASN.1 NULL data (0x05 0x00)) to the end-entity TBSCertificate" (3.1.)

Another requirement for the pre-certificate is that it comes from the same certificate chain as the end-entity certificate (3.1.). This requirement was also met.

RFC 6962 says nothing about the content of further extensions of a pre-certificate. OID 1.3.6.1.4.1.11129.2.4.3 ensures that this pre-certificate - regardless of its other attributes and extensions - cannot be validated by an X.509v3 client: "cannot be validated by a standard X.509v3 client" (3.1.).

Apparently the ct-log operators take this position. Otherwise it would not be comprehensible that a precertificate request that does not conform to RFC6962 is answered with correct SCTs. Such behavior would be somewhat surprising and unusual.

RFC 6962: "When a valid certificate is submitted to a log, the log MUST immediately return a Signed Certificate Timestamp (SCT)." (3.)

Where we got lucky

All customer data and cryptographic procedures/methods of all certificates were correct at all times.

Action Items

Action Item Kind Due Date
Activation of additional checks when creating pre-certificates on the topics of duplicate/identical information prevent 2023-02-07 15:00 UTC
Additional checks for the availability of ct servers prevent 2023-02-07 15:00 UTC
Additional user interface for employees to identify faulty ct servers prevent 2023-02-07 15:00 UTC
Additional checking of the certificates published under crt.sh prevent 2023-02-07 15:00 UTC
Additional automated reporting procedure for missing ct entries prevent 2023-02-07 15:00 UTC
Extraordinary verification of other issued certificates prevent 2023-02-08 15:00 UTC
Since no further abnormalities were found, the case was closed prevent 2023-02-09 15:35 UTC
support ongoing discussion at Bugzilla discussion 2023-02-13 8:00 UTC
additional informations issued at https://bugzilla.mozilla.org/show_bug.cgi?id=1830536 comment 1 discussion 2023-04-28 15:18 UTC
support extra information at Bugzilla discussion 2023-10-03 11:07 UTC
answer questions at Bugzilla discussion 2023-10-31 16:11 UTC
send this report at Bugzilla discussion 2023-11-05 15:30 UTC

Appendix

Details of affected certificates

Correctly-formed precertificate with serial number 10:45:67:49:88:c4:db:00:76:89:b9:
https://crt.sh/?sha256=6f1fd13b3c72fd064e01b4fa810d44a06bde36f44876179a45bf752ae988d6cf

Precertificate with serial number 10:45:67:49:88:c4:db:00:76:89:b9 containing an SCT extension,
in violation of RFC6962 which only defines the SCT extension for final certificates, not precertificates:
https://crt.sh/?sha256=c967a3109b953f8c14ffda5901db5f76d1bc0b6f84a3b6163ce14cd16d12086a

Certificate with serial number 10:45:67:49:88:c4:db:00:76:89:b9 containing SCTs with invalid signatures:
https://crt.sh/?sha256=5ee21e3bb051f280dec1339e83f60969919837da84a554007553edfb77c74a5f

Certificate 3 matches precertificate 1 according to the algorithm in RFC 6962, but
its embedded SCTs are for precertificate 2.

With this updated incident report, I am thinking that this bug can be closed now.

Flags: needinfo?(ca) → needinfo?(bwilson)

With this updated incident report, I am thinking that this bug can be closed now.

I don't think this incident report goes far enough, and there's misunderstandings that need to be resolved still.

Impact
Reviewing our issuing Processes

What does this mean?

Root cause Analysis
No root affected.

That's not what "Root cause Analysis" means. We're not asking if your root is safe or not, it's "what is the root cause of this incident?"

All customer data and cryptographic procedures/methods of all certificates were correct at all times.

No, they weren't. That's why you had this incident.

e-commerce monitoring GmbH, based on this incident report, a few things are clear to me:

  1. You have completely ignored Bugzilla since you've become a CA. It is expected of CAs to monitor this component for other incidents, and apply learnings from them. It was, and is, unacceptable to not know what is expected of incident reports.

  2. Instead of attempting to solve this problem, and provide a real incident report, you've instead sought to defend your short comings and put blame on other systems:

  1. The action items are severely lacking. How are you going to prevent this issue, and issues of similar caliber going into the future?

  2. https://bugzilla.mozilla.org/show_bug.cgi?id=1862004 -- You then filed this:

Apparently some participants in the discussion forum did not understand the European approach to the matter. It is a matter of course for us to clearly state different positions and only then take the necessary actions. It is possible that some people do not see differences of opinion this way.

As a global & public CA, you are required to follow the rules and procedures of the CA&B and root programs no matter your jurisdiction. There are no special "European approaches" that are permitted here.

(In reply to Ben Wilson from comment #29)

With this updated incident report, I am thinking that this bug can be closed now.

Ben, I worry that closing this with such a poor-quality incident report sets a bad precedent for future reporters. Is Mozilla sure they want to signal this is the appropriate level of response for future incidents?

Daniel,
Please provide a response to Comment #30 and a more thorough Incident Report.
Thanks,
Ben

Flags: needinfo?(bwilson) → needinfo?(ca)

(In reply to ryan_hurst from comment #31)

(In reply to Ben Wilson from comment #29)

With this updated incident report, I am thinking that this bug can be closed now.

Ben, I worry that closing this with such a poor-quality incident report sets a bad precedent for future reporters. Is Mozilla sure they want to signal this is the appropriate level of response for future incidents?

In the context of incident precedents, it is probably also useful to look at patterns and how they may relate to current and future issues. To that end here is the e-commerce monitoring application bug - https://bugzilla.mozilla.org/show_bug.cgi?id=1440271#c45

I wanted to add that I agree with Ryan's and Amir's interpretation of the 120 hours for 5 days.

Perhaps a CA or Browser representative can create a CA/B Forum Ballot to clarify this further in the Baseline Requirements, either by defining "day" globally to mean "24 hours", or by replacing all instances of days to their hour-form-equivalent, where it's not already specified.

I think Antonis' Comment #34 was meant to be filed in Bug #1865080.

Ben, you're right. So many tabs open lately :)

A significant amount of time has passed with questions left unanswered. As stated in the incident report guidance on ccadb.org: “Once the report is posted, CA Owners should respond promptly to questions that are asked, and in no circumstances should a question linger without a response for more than one week, even if the response is only to acknowledge the question and provide a later date when an answer will be delivered.”

Dear All,

Thank you for your feedback and your clear update requests. It helps e-commerce monitoring GmbH to improve our understanding of the required level of incident documentation. e-commerce monitoring GmbH also welcomes this opportunity to enhance our incident handling in the future in order to better meet the community’s expectations.
We will provide updates to this bug no later than Friday 16th.
Thank you for your patience.

Best regards,
Daniel Zens

Flags: needinfo?(ca)
Whiteboard: [ca-compliance] [ov-misissuance] → [ca-compliance] [ov-misissuance] [external]

Incident Report

Summary

This is an updated report.

On 2023-02-07 e-commerce monitoring GmbH has issued multiple signed data structures with the serial number 10:45:67:49:88:c4:db:00:76:89:b9: One containing a pre-certificate Poison Extension and SCTs at the same time.

Impact

This combination is violating RFC6962. Affected certificates: Depending on the counting method, between one and four data structures are affected (one of which is only intended/presumed-to-exist under MRSP 5.4). As they all share the same serial number, you can argue they are the same certificate. Applicable Policy: OV SSL. The certificate was already in use and had to be revoked (delayed, filed separately from this bug) and had to be replaced.

Timeline

2023-01-03

  • 10:23 UTC: An update was made to our issuance tool in order to use the then-current CT log servers

2023-02-07

  • 10:23 UTC: An SCT query was made for "10:45:67:49:88:c4:db:00:76:89:b9" which was insufficient according to our specifications (only one SCT instead of three). Since this SCT was suitable, it was stored internally. (SCT response 1)
  • 10:38 UTC: Another SCT query was made, the response of which met our specifications These SCT values were stored internally. (SCT response 2)
  • 10:50 UTC: Based on the correct SCT entries from SCT response 2, the leaf certificate was issued The SCTs were used as embedded SCTs.
  • 14:04 UTC: Internal controls showed that two pre-certificates were issued for one leaf certificate. (10:45:67:49:88:c4:db:00:76:89:b9)
  • 14:30 UTC (rounded) A detailed analysis revealed that some Google ct-log-servers changed the format of their URLs at the turn of the year 2022-2023. Our software was converted to the new url format and the entire module was subjected to a comprehensive revision. No server end-entity certificates were issued during this conversion and revision process.
  • 15:00-16:00 UTC (rounded) A subsequent review was carried out, primarily to avoid future disruptions due to unexpected changes in the URL format and to speed up the issuing process of server end-entity certificates.
  • 19:33 UTC This Bug was filed

2023-02-08

  • 9:30 UTC Internal acknowledgement of this bug. Forward to the responsible staff.
  • 10:00-11:30 UTC (rounded) internal discussion, coordination of the next steps. A deeper post issuance lint tests was carried out, which did not show any warning nor error. However it can be confirmed the leaf certificate includes the inappropriate SCTs (as in https://bugzilla.mozilla.org/show_bug.cgi?id=1815534#c0). It has been discussed the certificate is likely problematic, however it also could be confirmed it is successfully used by the subscriber.
  • 14:25 UTC first response to this bug, more detailed response later the same day. Ongoing discussions for the next days in in this bug and internally with our staff.

2023-03-22

  • 20:30 UTC confirmation by community members that revocation is necessary and should already have been done.

2023-03-28

  • 10:28 UTC A correct certificate was issued in order to replace the revoked.

2023-04-28

  • Communication of our improvements. Here is an update to provide more detail to the applicable improvements and leave out points that do not apply to this bug:
  • A certificate that contains a CT Precertificate Poison must not contain SCTs, and vice versa, a certificate that contains SCTs must not contain a Precertificate Poison extension
  • The monitoring of the lists of usable ct servers has been improved. Preparation of a script that checks our list of used ct servers against the authentic JSON list of ct servers via our monitoring software, rather than manually reviewing every quarter.
  • If for any reason no sufficient number of usable SCTs (then current: 2x Google, 2x Non-Google) can be retrieved, the certificate is not issued

Root Cause Analysis

  • At that time, the question of whether CT servers were still usable or qualified had to be checked completely manually and was scheduled on a quarterly basis. Although manually, this had worked well until then.
  • At that time, our issuance system was configured in a way to store SCT results and even if no corresponding leaf is issued, the SCTs were used for the next. This is obviously the reason why it was possible to issue a leaf certificate containing embedded SCTs retrieved for certificate A, though the ones for certificate B should have been used.
  • No controls were implemented to detect presumed-to-exist certificates i.e. if a precertificate is issued but no corresponding leaf follows

Lessons Learned

What went well

  • e-commerce monitoring GmbH has quickly fixed the bug and analyzed the underlying causes and has taken measures to prevent the incident from occurring again.

What didn't go well

  • It would have been better to assign a higher priority to incident documentation and its communication.

Where we got lucky

  • Only one case was affected.

Action Items

No pending action items

Appendix

Details of affected certificates

Correctly-formed pre-certificate with serial number 10:45:67:49:88:c4:db:00:76:89:b9:
https://crt.sh/?sha256=6f1fd13b3c72fd064e01b4fa810d44a06bde36f44876179a45bf752ae988d6cf

Precertificate with serial number 10:45:67:49:88:c4:db:00:76:89:b9 containing an SCT extension,
in violation of RFC6962 which only defines the SCT extension for final certificates, not precertificates:
https://crt.sh/?sha256=c967a3109b953f8c14ffda5901db5f76d1bc0b6f84a3b6163ce14cd16d12086a

Certificate with serial number 10:45:67:49:88:c4:db:00:76:89:b9 containing SCTs with invalid signatures:
https://crt.sh/?sha256=5ee21e3bb051f280dec1339e83f60969919837da84a554007553edfb77c74a5f

As no further questions have been raised on the specific issue, we ask that the ticket be closed.

Flags: needinfo?(bwilson)

I'll close this incident on or about Wed. 17-Apr-2024, unless there are additional comments or questions.

Status: ASSIGNED → RESOLVED
Closed: 2 days ago
Flags: needinfo?(bwilson)
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: