Closed Bug 1885594 Opened 10 months ago Closed 7 months ago

WebRTC fails to connect via TURNS when a proxy is configured

Categories

(Core :: WebRTC: Networking, defect, P2)

Product:
Core
Component:
WebRTC: Networking
Version:
Firefox 123
Type:
defect

Tracking

()

Status:
RESOLVED FIXED
Milestone:
129 Branch
Tracking Flags:
Tracking Status
firefox129 --- fixed

People

(Reporter: bbaldino, Assigned: bwc, NeedInfo)

References

Details

Attachments

(5 files, 1 obsolete file)

FIREFOX_NOT_WORKING_does_not_establish_tls_first.pcapng
490.00 KB, application/octet-stream
Details
Firefox_Proxy_443_TLS_Webrtc_and_Proxy_debugs.zip
385.39 KB, application/zip
Details
turn_test.htm
276 bytes, text/html
Details
TURN_FAILS_PAC_FILE_HTTP_AND_HTTPS_PROXY_SQUID_LOGS_PCAP.zip
795.66 KB, application/zip
Details
Bug 1885594: Some fixes required when doing TLS through a web proxy. r?kershaw
48 bytes, text/x-phabricator-request
Details | Review

User Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:123.0) Gecko/20100101 Firefox/123.0

Steps to reproduce:

Firefox is configured to use a local squid proxy
Coturn server setup to listen on 443
Attempt to connect to the turn server via TLS

Actual results:

After establishing the http connection, instead of doing the TLS handshake, firefox jumps straight to the turn allocate (i.e. no TLS client hello is sent)

Expected results:

TLS handshake should be done before the turn allocate when connecting via turns

Attached file turn_test.htm —

Just to be clear, do we never try a TLS handshake at all?

Yeah when the proxy is enabled there's no Client Hello sent.

NI :bwc for following up.

Flags: needinfo?(docfaraday)

I am having no luck reproducing this with the off-the-shelf web-proxy offered by NordVPN. Does the local squid proxy have an FQDN?

Flags: needinfo?(docfaraday) → needinfo?(bbaldino)

I wonder if necko will not do a TLS handshake through a non-TLS web-proxy...

Trying a local squid proxy with whereby reveals that secure websockets are broken; the webrtc networking code handles proxies very similarly to websockets, so the root cause may be the same.

Yeah, we tested with a SQUID proxy with a FQDN. Has a valid certificate (Let Encrypt)

Tried HTTP_PROXY and HTTPS_PROXY.

SQUID only configured to allow TLS tunnel via the HTTPS_PROXY Port
SQUID configured to allow 443 / 80 outbound the HTTP_PROXY

THE HTTP Connect is successful and sets up the tunnel.
Firefox doesn't setup a TLS connection via that tunnel when the proxy is configured.

My Squid Conf:

http_port 0.0.0.0:3128 
https_port 0.0.0.0:3130 tls-cert=/etc/squid/ssl_cert/fullchain.pem tls-key=/etc/squid/ssl_cert/privkey.pem


acl localnet src 10.0.0.0/8    # RFC1918 possible internal network
acl localnet src 172.16.0.0/12 # RFC1918 possible internal network
acl localnet src 192.168.0.0/16 # RFC1918 possible internal network

acl public_ip src <public_ip/subnet_prefix> # My Public IP for testing

acl SSL_ports port 443
acl Safe_ports port 80          # http
acl Safe_ports port 21          # ftp
acl Safe_ports port 443         # https
acl Safe_ports port 70          # gopher
acl Safe_ports port 210         # wais
acl Safe_ports port 1025-65535  # unregistered ports
acl Safe_ports port 280         # http-mgmt
acl Safe_ports port 488         # gss-http
acl Safe_ports port 591         # filemaker
acl Safe_ports port 777         # multiling http
acl CONNECT method CONNECT


http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access allow localnet manager
http_access deny manager

# Only allow cachemgr access from localhost
http_access allow localnet

# Allow access from cisco public
http_access allow public_ip

# And finally deny all other access to this proxy
http_access deny all

coredump_dir /var/spool/squid

Is there some logging we ought to be enabling for the networking code to help debug this? Our logging definitely indicates that we're asking for TLS (we're supplying an https url here).

Flags: needinfo?(kershaw)

@bwc sorry, forgot to include the log settings that were used. Here they are:

export MOZ_LOG="timestamp,rotate:200,sync,jsep:5,sdp:5,signaling:5,mtransport:5,RTCRtpReceiver:5,RTCRtpSender:5,RTCDMTFSender:5,VideoFrameConverter:5,WebrtcTCPSocket:5,CamerasChild:5,CamerasParent:5,VideoEngine:5,ShmemPool:5,TabShare:5,MediaChild:5,MediaParent:5,MediaManager:5,MediaTrackGraph:5,cubeb:5,MediaStream:5,MediaStreamTrack:5,DriftCompensator:5,ForwardInputTrack:5,MediaRecorder:5,MediaEncoder:5,TrackEncoder:5,VP8TrackEncoder:5,Muxer:5,GetUserMedia:5,MediaPipeline:5,PeerConnectionImpl:5,WebAudioAPI:5,webrtc_trace:5,RTCRtpTransceiver:5,ForwardedInputTrack:5,HTMLMediaElement:5,HTMLMediaElementEvents:5,proxy:5"

# Set the file where the logs will be stored
export MOZ_LOG_FILE=~/Desktop/firefox_proxy_443_Webrtc_Debugs_3_14_proxy.log

/Applications/Firefox.app/Contents/MacOS/firefox -P "ProxyProfile" --private-window http://127.0.0.1:5500/turn_test.htm

I might be able to determine more with nsHttp:5 in the log modules. From what I can tell, for me the http code is doing TLS with TURN TLS servers, but maybe it is not on your end for some reason?

@bwc I don't think it's on our end since the pcap is not showing any attempt at a tls connection from firefox

Flags: needinfo?(bbaldino)

I cannot think of any good reason why we'd be doing TLS on my hardware (with nordvpn and whereby), but not in your scenario, but there really does seem to be some difference. It is puzzling. It might be that the logging is lying to me? Could you run with nsHttp:5 so I can compare?

(In reply to Byron Campen [:bwc] from comment #14)

I cannot think of any good reason why we'd be doing TLS on my hardware (with nordvpn and whereby), but not in your scenario, but there really does seem to be some difference. It is puzzling. It might be that the logging is lying to me? Could you run with nsHttp:5 so I can compare?

Yes, I think enabling http logging is the best way to debug this. Please also add proxy:5 into MOZ_LOG and select Logging to a file to create a log file.

Thanks.

Flags: needinfo?(kershaw) → needinfo?(bbaldino)
Attached file firefox_with_proxy_5_debug.zip (obsolete) —

Attached is 3 debug log files, including proxy:5, all networking debugs, and webrtc debugs enabled.

  1. Enabled logging, refreshed the page to test turn relay connectivity:
  2. Secure TURNS URL: turns:turn_server_fqdn:443?transport=tcp (username / password)
  3. Enable all logging
  4. Run connectivity test via WebRTC page
  5. Capture success or failure
  6. Stop Log

Attached Logs as well:

  1. Firefox, no proxy Connecting to coturn server over port 443 (Success, returned relay as candidate)
  2. Firefox, HTTP Proxy (3128) for HTTP/HTTPS, failed. Wireshark shows an SSL packet sent across the TUNNEL through the proxy after the HTTP CONNECT to the proxy (Never sends TLS Client Hello)
  3. Firefox HTTP(3128) & HTTPS(3130), failed. TURN Relay allocation fails.

My etc/turnserver.conf

listening-ip=<private_ip>
external-ip=<public_ip>/<private_ip>

realm=<MY REALM>

server-name=<FQDN>

cert=/etc/turnserver/cert.pem
pkey=/etc/turnserver/pkey.pem

# Default 3478
listening-port=''
# Default alt listening listening + 1
alt-listening-port=''

# Default 5349 tls
tls-listening-port=443
# tls port + 1
alt-tls-listening-port=''

secure-stun
no-tcp-relay

# Disables TCP relay allocations, only allowing TLS connections
no-tcp
no-udp

verbose

cipher-list="ECDHE+AESGCM:ECDHE+CHACHA20"
fingerprint
userdb=/var/lib/turn/turndb
lt-cred-mech

user=<test_username>:<test_password>

log-file=/var/log/turnserver/turn.log
dh-file=/etc/coturn/dhparam.pem
min-port=49152
max-port=65535

I've looked at the log (firefox_HTTP&HTTPS_Proxy_DEBUG_LOGS_FAILS_NO_TLS.txt-main.38594.moz_log.txt-main.38594.moz_log) and it shows that there is an error from NSS:

2024-04-10 00:27:45.476932 UTC - [Parent 38594: Socket Thread]: D/nsSocketTransport nsSocketInputStream::Read [this=1340b94c0 count=32768]
2024-04-10 00:27:45.477115 UTC - [Parent 38594: Socket Thread]: D/nsSocketTransport   calling PR_Read [count=32768]
2024-04-10 00:27:45.477292 UTC - [Parent 38594: Socket Thread]: D/nsSocketTransport   PR_Read returned [n=-1]
2024-04-10 00:27:45.477464 UTC - [Parent 38594: Socket Thread]: D/nsSocketTransport ErrorAccordingToNSPR [in=-5987 out=80004005]

-5987 is PR_INVALID_ARGUMENT_ERROR and it might be coming from here.
To help us debug this more, could you try to record another log with adding pipnss:5 into MOZ_LOG?

Moreover, the log also shows that the proxy's response to CONNECT request is broken. As the log below, the proxy response is only 7 bytes.

2024-04-10 00:27:45.470870 UTC - [Parent 38594: Socket Thread]: V/nsHttp nsHttpTransaction::ParseHead [count=7]
2024-04-10 00:27:45.471041 UTC - [Parent 38594: Socket Thread]: V/nsHttp nsHttpResponseHead::ParseVersion [version=]
2024-04-10 00:27:45.471209 UTC - [Parent 38594: Socket Thread]: V/nsHttp looks like a HTTP/0.9 response
2024-04-10 00:27:45.471376 UTC - [Parent 38594: Socket Thread]: V/nsHttp response status line needs default reason phrase
2024-04-10 00:27:45.471544 UTC - [Parent 38594: Socket Thread]: E/nsHttp Have status line [version=9 status=200 statusText=OK]
2024-04-10 00:27:45.471718 UTC - [Parent 38594: Socket Thread]: V/nsHttp nsHttpTransaction::HandleContent [this=142a98600 count=7]
2024-04-10 00:27:45.471890 UTC - [Parent 38594: Socket Thread]: V/nsHttp nsHttpTransaction::HandleContentStart [this=142a98600]
2024-04-10 00:27:45.472054 UTC - [Parent 38594: Socket Thread]: I/nsHttp http response [
2024-04-10 00:27:45.472228 UTC - [Parent 38594: Socket Thread]: E/nsHttp     OriginalHeaders
2024-04-10 00:27:45.472397 UTC - [Parent 38594: Socket Thread]: I/nsHttp ]

Normally, the proxy response should be like:

2024-04-10 00:25:56.111867 UTC - [Parent 38594: Socket Thread]: E/nsHttp nsHttpTransaction::ProcessData [this=1478d5100 count=39]
2024-04-10 00:25:56.112045 UTC - [Parent 38594: Socket Thread]: V/nsHttp nsHttpTransaction::ParseHead [count=39]
2024-04-10 00:25:56.112209 UTC - [Parent 38594: Socket Thread]: E/nsHttp nsHttpTransaction::ParseLine [HTTP/1.1 200 Connection established]
2024-04-10 00:25:56.112229 UTC - [Parent 38594: Main Thread]: D/nsHttp nsHttpHandler::NotifyObservers [this=131b21b00 chan=12a239440 event="http-on-examine-response"]
2024-04-10 00:25:56.112386 UTC - [Parent 38594: Socket Thread]: V/nsHttp nsHttpResponseHead::ParseVersion [version=HTTP/1.1 200 Connection established]
2024-04-10 00:25:56.112580 UTC - [Parent 38594: Socket Thread]: E/nsHttp Have status line [version=11 status=200 statusText=Connection established]
2024-04-10 00:25:56.112769 UTC - [Parent 38594: Socket Thread]: V/nsHttp nsHttpTransaction::HandleContent [this=1478d5100 count=0]
2024-04-10 00:25:56.112945 UTC - [Parent 38594: Socket Thread]: V/nsHttp nsHttpTransaction::HandleContentStart [this=1478d5100]
2024-04-10 00:25:56.113121 UTC - [Parent 38594: Socket Thread]: I/nsHttp http response [
2024-04-10 00:25:56.112954 UTC - [Parent 38594: Main Thread]: D/nsHttp nsHttpChannelAuthProvider::CheckForSuperfluousAuth? [this=147539dc0 channel=12a239af0]
2024-04-10 00:25:56.113297 UTC - [Parent 38594: Socket Thread]: E/nsHttp   HTTP/1.1 200 Connection established
2024-04-10 00:25:56.113653 UTC - [Parent 38594: Socket Thread]: E/nsHttp     OriginalHeaders
2024-04-10 00:25:56.113829 UTC - [Parent 38594: Socket Thread]: I/nsHttp ]

I am not sure if this is related, but could you also check if there is anything wrong at the proxy?

Apologies, I think that was one of the earlier tests I was doing before.
Here is the debug, including the added logging levels you asked for, as well as a PCAP taken at the same time.

You'll see firefox reaching out to other services via the PROXY via the HTTP Tunnel then setting up the TLS Connection through the tunnel, however when it comes to the coturn it does not.

If I modify my TURN Server to accept Incoming TCP relay requests, it "WORKS" via the PROXY.
However, when my TURN Server is configured to only accept TLS Connections to the RELAY it fails via the PROXY.

I've tried configuring a PAC file with the PROXY Server config.
System Settings detecting PROXY
Manual Proxy SETUP.

I take the same config, pac file, test and do it in a Chromium based browser (Chrome / Brave, what i've tested). I see the CONNECT, 200 Okay, TLS Handshake, TURN ALLOCATION with success.

My lab setup will be down for a few days April 17th and 18th. Let me know if you need anything else before then.

Attachment #9395916 - Attachment is obsolete: true

@kershaw did those log files get you the info you need?

Flags: needinfo?(bbaldino)

(In reply to bbaldino from comment #19)

@kershaw did those log files get you the info you need?

Sorry for the delay.

The log seems to suggest the problem is about WebrtcTCPSocket. This is what I found from the log.

  1. WebrtcTCPSocket was created and opened.
2024-04-12 02:49:14.682342 UTC - [Parent 6982: Main Thread]: D/WebrtcTCPSocket WebrtcTCPSocketParent::WebrtcTCPSocketParent 13a405e20
2024-04-12 02:49:14.682467 UTC - [Parent 6982: Main Thread]: D/WebrtcTCPSocket WebrtcTCPSocket::WebrtcTCPSocket 13ed39000
2024-04-12 02:49:14.682559 UTC - [Parent 6982: Main Thread]: D/WebrtcTCPSocket WebrtcTCPSocketParent::RecvAsyncOpen 13a405e20 to coturn.tacrepro.io:443
2024-04-12 02:49:14.682762 UTC - [Parent 6982: Main Thread]: D/WebrtcTCPSocket WebrtcTCPSocket::Open 13ed39000 remote-host=coturn.tacrepro.io local-addr=172.31.16.1 local-port=53201
  1. Then, it was closed right after.
2024-04-12 02:49:14.683672 UTC - [Parent 6982: Main Thread]: D/WebrtcTCPSocket WebrtcTCPSocket::Close 13ed39000
2024-04-12 02:49:14.683712 UTC - [Parent 6982: Main Thread]: D/WebrtcTCPSocket WebrtcTCPSocket::CloseWithReason 13ed39000 reason=0
  1. Necko successfully established the tunnel and transferred the connection to WebrtcTCPSocket, but it's already closed.
2024-04-12 02:49:15.172314 UTC - [Parent 6982: Socket Thread]: V/nsHttp nsHttpConnection::TakeTransport [14ab04300] calling StartLongLivedTCPKeepalives
2024-04-12 02:49:15.173128 UTC - [Parent 6982: Socket Thread]: D/nsSocketTransport Reset callbacks for tlsSocketInfo=14aa82a00 callbacks=13e862700
2024-04-12 02:49:15.173330 UTC - [Parent 6982: Socket Thread]: D/WebrtcTCPSocket WebrtcTCPSocket::OnTransportAvailable 13ed39000
2024-04-12 02:49:15.174023 UTC - [Parent 6982: Socket Thread]: D/WebrtcTCPSocket WebrtcTCPSocket::OnTransportAvailable 13ed39000 closed

@bwc, could you also take a look at the log? Could you comment if this behavior is expected?
Thanks.

Flags: needinfo?(docfaraday)
Priority: -- → P2
Severity: -- → S3

If the socket is being closed immediately, how are we sending anything at all on it? Are we sure this is one of the sockets that is sending unencrypted traffic to a TURNS server? I see other sockets making it much further (eg; WebrtcTCPSocketParent 13c7a3380).

Flags: needinfo?(docfaraday) → needinfo?(kershaw)

(In reply to Byron Campen [:bwc] from comment #21)

If the socket is being closed immediately, how are we sending anything at all on it? Are we sure this is one of the sockets that is sending unencrypted traffic to a TURNS server? I see other sockets making it much further (eg; WebrtcTCPSocketParent 13c7a3380).

Well, I can't tell which socket is the right one to look from the log.
I can only tell that necko didn't try to setup the secondary TLS because NS_HTTP_CONNECT_ONLY flag was set by WebrtcTCPSocket here. I think it means that necko only hands over the raw socket to WebrtcTCPSocket and the webrtc code needs to take care of TLS handshake by itself.

Flags: needinfo?(kershaw)

Ok, I'm trying to fix this, but it seems that we're never seeing the HTTP upgrade finish. This code (which we do reach) suggests that the only two valid upgrade scenarios are WebSocket, and NS_HTTP_CONNECT_ONLY:

https://searchfox.org/mozilla-central/source/netwerk/protocol/http/nsHttpChannel.cpp#8037-8046,8056-8057

Do we need to add another mode here?

Flags: needinfo?(kershaw)

(In reply to Byron Campen [:bwc] from comment #23)

Ok, I'm trying to fix this, but it seems that we're never seeing the HTTP upgrade finish. This code (which we do reach) suggests that the only two valid upgrade scenarios are WebSocket, and NS_HTTP_CONNECT_ONLY:

https://searchfox.org/mozilla-central/source/netwerk/protocol/http/nsHttpChannel.cpp#8037-8046,8056-8057

Do we need to add another mode here?

Yes, we probably need another flag for this.
Do you know how can I test this locally? I could try to write a patch at necko side.

Flags: needinfo?(kershaw) → needinfo?(docfaraday)

I've just been using an off-the-shelf "VPN" browser plugin (in my case, NordVPN), and testing on whereby. I've fixed a handful of other problems to reach the point where the next problem is the upgrade callback. I can push the work in progress to try, if you want to base your work on top of that. Or, if you'd prefer, I can test your work in progress.

Flags: needinfo?(docfaraday)

(In reply to Byron Campen [:bwc] from comment #25)

I've just been using an off-the-shelf "VPN" browser plugin (in my case, NordVPN), and testing on whereby. I've fixed a handful of other problems to reach the point where the next problem is the upgrade callback. I can push the work in progress to try, if you want to base your work on top of that. Or, if you'd prefer, I can test your work in progress.

Thanks. I haven't started writing my patch yet, so it would be good if I could base it on your work. By the way, is there a public TURN server available that I could use for testing? I would like to know how to test the WebrtcTCPSocket.

Flags: needinfo?(docfaraday)

This is a decent place to start when testing whether TURN TLS is working:

https://bugzilla.mozilla.org/show_bug.cgi?id=1891503#c0

Flags: needinfo?(docfaraday)

Ok, here's a WIP. It may be broken in ways I have not discovered yet.

hg.mozilla.org/try/rev/1c339908a12119043f81c37b070403be12e3c202

Sorry for the late response.
I've created bug 1896625 at necko side.

Depends on: 1896625

(In reply to Byron Campen [:bwc] from comment #28)

Ok, here's a WIP. It may be broken in ways I have not discovered yet.

hg.mozilla.org/try/rev/1c339908a12119043f81c37b070403be12e3c202

Looking at your WIP, I noticed that SetConnectOnly is removed.
I am not sure why, but I think we can keep it. This flag is still needed to instruct necko to upgrade the connection.
Please take a look at the patch in bug 1896625. I tried to make it possible to set up a TLS tunnel when an HTTPS proxy is in use. For HTTP proxies, the behavior remains unchanged.

Ok, I removed that because it was suppressing secondary TLS, but if we're not suppressing that anymore I'll try adding it back in.

Could you try a binary from this try push, and see if it works for you? It appears to be working for me using NordVPN/whereby.

https://treeherder.mozilla.org/jobs?repo=try&revision=0d592712b3df2e7a986409e612e4950501a17566

Flags: needinfo?(bbaldino)

(In reply to Byron Campen [:bwc] from comment #32)

Could you try a binary from this try push, and see if it works for you? It appears to be working for me using NordVPN/whereby.

https://treeherder.mozilla.org/jobs?repo=try&revision=0d592712b3df2e7a986409e612e4950501a17566

I pulled down and did a build based on the revision ID provided. If there was another way to test than that let me know. Long time fan first time compiling from sourcer.

I can get the turns:<turnaddress.test>:443?transport=tcp to work if the proxy is configured as HTTPS proxy.
Still doesn't work from my testing if its an HTTP Proxy, it won't build the TLS Tunnel through the HTTP Proxy.

DETAILS:
If I have the following Proxy configuration in my PAC file.

WORKS (SETS UP TLS with Proxy and TURN allocation succeeds):
function FindProxyForURL(url, host) {
if (url.substring(0, 5) == "https") {
return "HTTPS squid.tacrepro.io:3130";
}

However I can't get it to work if the configuration for a http PROXY only, it doesn't build the TLS tunnel through the HTTP Proxy.

DOES NOT WORK (IE: FF doesn't create the TLS Tunnel through the HTTP Proxy):
function FindProxyForURL(url, host) {
if (url.substring(0, 5) == "https") {
return "PROXY <http.proxy.com>:3128";
}

// If the URL is HTTP (not HTTPS), use port 3128
if (url.substring(0, 4) == "http") {
    return "PROXY <http.proxy.com>:3128";
}

(In reply to brian.m.pettis from comment #33)

(In reply to Byron Campen [:bwc] from comment #32)

Could you try a binary from this try push, and see if it works for you? It appears to be working for me using NordVPN/whereby.

https://treeherder.mozilla.org/jobs?repo=try&revision=0d592712b3df2e7a986409e612e4950501a17566

I pulled down and did a build based on the revision ID provided. If there was another way to test than that let me know. Long time fan first time compiling from sourcer.

To download a build you find the green 'B' that corresponds to the platform you want, and select it. Then, in the pane that pops up at the bottom of the window, there is a sub-pane on the right with some tabs (named "Artifacts and Debugging Tools", "Failure Summary", etc). Select the "Artifacts and Debugging Tools" tab. That'll display a list of download links. The download you want will vary by platform; MacOS will be "target.dmg", Windows will be "target.installer.exe", linux will be "target.tar.bz2".

I can get the turns:<turnaddress.test>:443?transport=tcp to work if the proxy is configured as HTTPS proxy.
Still doesn't work from my testing if its an HTTP Proxy, it won't build the TLS Tunnel through the HTTP Proxy.

DETAILS:
If I have the following Proxy configuration in my PAC file.

WORKS (SETS UP TLS with Proxy and TURN allocation succeeds):
function FindProxyForURL(url, host) {
if (url.substring(0, 5) == "https") {
return "HTTPS squid.tacrepro.io:3130";
}

However I can't get it to work if the configuration for a http PROXY only, it doesn't build the TLS tunnel through the HTTP Proxy.

DOES NOT WORK (IE: FF doesn't create the TLS Tunnel through the HTTP Proxy):
function FindProxyForURL(url, host) {
if (url.substring(0, 5) == "https") {
return "PROXY <http.proxy.com>:3128";
}

// If the URL is HTTP (not HTTPS), use port 3128
if (url.substring(0, 4) == "http") {
    return "PROXY <http.proxy.com>:3128";
}

Yeah, the patch over in bug 1896625 does not allow TLS tunneling through an HTTP proxy right now. I've asked whether there's a way we could allow that.

Assignee: nobody → docfaraday
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true
Pushed by kjang@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/e4b41a9b6988 Some fixes required when doing TLS through a web proxy. r=kershaw

https://hg.mozilla.org/mozilla-central/rev/e4b41a9b6988

Status: ASSIGNED → RESOLVED
Closed: 7 months ago
status-firefox129: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → 129 Branch
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: