1913387 - SSL_ERROR_BAD_CERT_DOMAIN with wildcard domain

Status: UNCONFIRMED

(Core :: Networking: DNS, defect, P2)

Description

[Max van Gent]

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:129.0) Gecko/20100101 Firefox/129.0

Steps to reproduce:

I tried to use https://users.api.nebula.app/

Actual results:

I got the following error

Websites prove their identity via certificates. Firefox does not trust this site because it uses a certificate that is not valid for users.api.nebula.app. The certificate is only valid for *.api.nebula.app.

Error code: SSL_ERROR_BAD_CERT_DOMAIN

Expected results:

The certificate should have been accepted because users.api.nebula.app is a subdomain of *.api.nebula.app.

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

The Bugbug bot thinks this bug should belong to the 'Core::Security: PSM' component, and is moving the bug to that component. Please correct in case you think the bot is wrong.

Comment 2

[Dana Keeler (she/her) [:keeler]]

I'm not seeing that error when I visit that site. Do you still see the error if you use a new profile?

Comment 3

[Max van Gent]

I do still see the error when using a different profile, but I don't see it when using a VPN ().

When it's working I get a cert chain of 3 certificates, namely

• *.api.nebula.app
• Amazon RSA 2048 M03
• Amazon Root CA 1

and when it's not working I get an extra cert added to the end, namely Starfield Services Root Certificate Authority - G2.
It also seems that then the Amazon Root CA 1 certs differ.

I also tested in chromium and there it works and there is no Starfield Services Root Certificate Authority - G2.

I exported both certificate chains, which I could send. I am not sure how to add those to my comment though.

I also tested it on my laptop on version 128 it works and then after I upgrade to 129 I get the error, so maybe the update is necessary to trigger it?

Comment 4

[Max van Gent]

Oh I just remembered that my VPN setup first passes it through a http proxy, so that means the ssl is probably not managed by firefox anymore.

Comment 5

[Dana Keeler (she/her) [:keeler]]

Can you attach the certificate you see when you see the error to this bug?

Comment 6

[Max van Gent]

Attachment: api-nebula-app-chain.pem

Comment 7

[Dana Keeler (she/her) [:keeler]]

That's the same certificate I'm seeing, and I'm not seeing the error. Since it worked in 128 but not 129, can you use https://mozilla.github.io/mozregression/ to narrow down when it started not working?

Comment 8

[Max van Gent]

Attachment: working certificate chain.pem

Okay I have tried it 3 times and I got 3 different results.

1. The commit closing https://phabricator.services.mozilla.com/D216001
2. The commit closing https://phabricator.services.mozilla.com/D213368
3. The commit closing https://phabricator.services.mozilla.com/D213179

I also thought maybe the "good" certificate chain might be helpful so I added it as attachment.

Note that the certificate chains start differing only at Amazon Root CA 1, I think this means that somehow the wrong Amazon Root CA 1 cert is used in the bad chain.

Comment 9

[Max van Gent]

Okay another small update, it suddenly started working when shift reloading, but after shift reloading a few more times it stopped again.

I'll see if I can set up a more reproducible environment for bisection over the weekend.

Comment 11

[David Mak]

Attachment: ZIP file containing the good (expected) and bad (error-inducing) PEM certs.

I would like to add my 2-cents to this issue.

I have my own LetsEncrypt domain, and I encountered the same issue as well. When accessing my own domain from the same local network, there are no issues; However, when accessing the same domain using mobile network via my phone, the SSL_ERROR_BAD_CERT_DOMAIN will occasionally occur. As with others, a couple of shift-reloads can temporarily fix the issue.

Attached are the good and bad certificates. It appears to me that the good cert has ISRG Root X1 as its root certificate, whereas the bad cert ends with LetsEncrypt's E6 cert; The E6 cert otherwise appears to be the same. This seems to be different from the certs attached for users.api.nebula.app, where the good cert appears to have a different valid duration and is self-verified compared to the bad cert.

Also, curl appears to work every single time. I believe the 0-2 levels of certificate corresponds to the expected chain of certs as appears in the good cert.

% curl -H 'Cache-Control: no-cache, no-store' -Lv https://media.derppening.com
* Host media.derppening.com:443 was resolved.
* IPv6: (none)
* IPv4: 219.77.232.229
*   Trying 219.77.232.229:443...
* Connected to media.derppening.com (219.77.232.229) port 443
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: none
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / x25519 / id-ecPublicKey
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=derppening.com
*  start date: Aug 16 15:59:02 2024 GMT
*  expire date: Nov 14 15:59:01 2024 GMT
*  subjectAltName: host "media.derppening.com" matched cert's "*.derppening.com"
*  issuer: C=US; O=Let's Encrypt; CN=E6
*  SSL certificate verify ok.
*   Certificate level 0: Public key type EC/prime256v1 (256/128 Bits/secBits), signed using ecdsa-with-SHA384
*   Certificate level 1: Public key type EC/secp384r1 (384/192 Bits/secBits), signed using sha256WithRSAEncryption
*   Certificate level 2: Public key type RSA (4096/152 Bits/secBits), signed using sha256WithRSAEncryption
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://media.derppening.com/
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: media.derppening.com]
* [HTTP/2] [1] [:path: /]
* [HTTP/2] [1] [user-agent: curl/8.9.1]
* [HTTP/2] [1] [accept: */*]
* [HTTP/2] [1] [cache-control: no-cache, no-store]
> GET / HTTP/2
> Host: media.derppening.com
> User-Agent: curl/8.9.1
> Accept: */*
> Cache-Control: no-cache, no-store
> 
* Request completely sent off
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
< HTTP/2 401 
< server: nginx
< date: Fri, 16 Aug 2024 17:38:31 GMT
< content-type: text/html
< content-length: 193
< x-plex-protocol: 1.0
< cache-control: no-cache
< 
* Connection #0 to host media.derppening.com left intact
<html><head><script>window.location = window.location.href.match(/(^.+\/)[^\/]*$/)[1] + 'web/index.html';</script><title>Unauthorized</title></head><body><h1>401 Unauthorized</h1></body></html>%

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:129.0) Gecko/20100101 Firefox/129.0

Please let me know if I can provide more system information.

Comment 12

[David Mak]

Upon running my domain using SSL Labs, I can see that there is an CRL ERROR: IOException occurred for the E6 cert. Could this be the cause of the issue?

Comment 13

[Dana Keeler (she/her) [:keeler]]

I'm pretty sure this is due to ECH. The HTTPS RRs for media.derppening.com and nebula.app indicate cloudflare-ech.com is the outer name that should be in the certificate presented by each server. However, the servers are presenting certificates that are valid for names that are presumably intended to be protected by ECH instead. Bug 1850248 is certainly at play here, because presenting the error this way is very confusing. That said, the ECH setup is wrong in each case - perhaps ECH was not actually intended to be used? Or if it is, then the server isn't set up to send the correct certificate.

Dennis, can you weigh in here?

Comment 16

[Max van Gent]

I did another more reproducible bisection and got consistent results. I got the following changeset.

This changeset enables network.dns.native_https_query and turning this off in about:config fixes the issue.

Another bisection with network.dns.native_https_query enabled gave the following pushlog.

Comment 17

[Andrew Moore]

(In reply to Max van Gent from comment #16)

I did another more reproducible bisection and got consistent results. I got the following changeset.

This changeset enables network.dns.native_https_query and turning this off in about:config fixes the issue.

Another bisection with network.dns.native_https_query enabled gave the following pushlog.

I can confirm that I'm seeing the exact same behaviour on openSUSE Tumbleweed 20240816 with Firefox 129.0.

Turning off network.dns.native_https_query solves the SSL_ERROR_BAD_CERT_DOMAIN error.

Comment 18

[Kershaw Chang [:kershaw]]

Hi Max,

Could you try to capture a http log that shows the SSL_ERROR_BAD_CERT_DOMAIN error?
Does the behavior change if DoH is enabled?

Thanks.

Comment 19

[Max van Gent]

Attachment: HTTP log

(In reply to Kershaw Chang [:kershaw] from comment #18)

Hi Max,

Could you try to capture a http log that shows the SSL_ERROR_BAD_CERT_DOMAIN error?

I uploaded the log with the default logging settings used.

Does the behavior change if DoH is enabled?

I assume you meant DNS over HTTPS. The error pops up again when setting it to Increased Protection or Max Protection and goes away when using Default Protection or turning it off. (This is with the network.dns.native_https_query flag disabled).

Comment 20

[Max van Gent]

I just got an email from nebula customer support that they put out a hotfix, so we may need to find a new testcase.

I could setup a test case if I knew what led to this error.

Comment 21

[Max van Gent]

The bug seems to work again.

Comment 22

[Valentin Gosu [:valentin] {{ PTO 3 July - 3 August }}]

I captured a profile when reproducing the bug https://share.firefox.dev/3YR8uOX
I also have a pcap file https://drive.google.com/drive/folders/1MyFyZmXnTEY-JxSj-TqxlGAqpjbjPA1Q?usp=sharing saved here.

I think Kershaw's suggestion might be accurate - that we connect to the IPs in the A record and attempt to use ECH, whereas the expectation might be to use the iphint in the HTTPS record.

Comment 23

[Dana Keeler (she/her) [:keeler]]

Seems like more of a networking issue based on comment 22.

Comment 24

[Sunil Mayya]

(In reply to Dana Keeler (she/her) [:keeler] from comment #23)

Seems like more of a networking issue based on comment 22.

Valentin, what would be the next steps for this bug?

Comment 25

[Valentin Gosu [:valentin] {{ PTO 3 July - 3 August }}]

I think Kershaw's suggestion might be accurate - that we connect to the IPs in the A record and attempt to use ECH, whereas the expectation might be to use the iphint in the HTTPS record.

Kershaw, would this be fixed by HE3?