1851570 - Proxy with basic authentication not working since Firefox 114+

Status: RESOLVED FIXED

(Core :: Networking, defect, P2)

Description

[ondrej.machulda]

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/117.0

Steps to reproduce:

Starting with Firefox 114, I'm unable to use proxy servers which requires basic authentication.

1. Setup proxy with some add-on (FoxyProxy, SwitchyOmega, SmartProxy - it doesn't work with any of them) (we use proxy https://our.proxy:port). This proxy requires authentication.

Actual results:

• Proxy which requires basic authentication does not work, the connection to any website times-out (NS_ERROR_NET_TIMEOUT).
• Using the same add-on in Chrome, basic auth dialog appears asking for username/password, and the proxy then works.
• Also using another proxy which does not need authentication works without an issue

Expected results:

Proxy requiring basic authentication should ask for password.

This works in Firefox 113 (respectively the addons have fileds in proxy settings for username/password, which now seems to have no effect). Its nor working in Firefox 114-116 .

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

The Bugbug bot thinks this bug should belong to the 'Core::Networking' component, and is moving the bug to that component. Please correct in case you think the bot is wrong.

Comment 2

[ondrej.machulda]

Also, from my other testing, this could be related to https protocol of the proxy. I was unable to set any proxy with https protocol (either using PAC file, or using addons). So maybe the issue not lies in the basic authentication, but in the https protocol.

Comment 3

[Kershaw Chang [:kershaw]]

Hi Reporter,

Could you try to use mozregression to help us find the regression range?

Thanks.

Comment 4

[ondrej.machulda]

Hi, I've just done that. Mozregression seems like a great tool, I never know it exists. I was quite impressed.

So the commit which breaks authenticated proxy is this:

2023-09-05T15:30:15.100000: DEBUG : Found commit message:
Bug 1289186 - wait for the server certificate to verify successfully before asking for a client auth certificate r=jschanck

If a TLS server asks for a client authentication certificate, no dialog asking
the user to select one should be shown until the server's certificate verifies
successfully.

Differential Revision: https://phabricator.services.mozilla.com/D175170

2023-09-05T15:30:15.100000: DEBUG : Did not find a branch, checking all integration branches
2023-09-05T15:30:15.102000: INFO : The bisection is done.
2023-09-05T15:30:15.103000: INFO : Stopped

As I read about certificates, maybe one more information - the proxy we use is our company proxy, with its https certificate signed by out CA (which is imported into system/browser trust).

Comment 5

[Kershaw Chang [:kershaw]]

Dana, could you take a look?
Thanks.

Comment 6

[Dana Keeler (she/her) [:keeler]]

Can you collect some logs using about:logging? timestamp,sync,nsHttp:5,nsSocketTransport:5,pipnss:4,certverifier:5 should be a good module string to use. Thanks!

Comment 7

[ondrej.machulda]

Hi, I e-mailed the log file to [e-mail] .

What I did:
0) Created clean Firefox profile and configured FoxyProxy extension with authenticated HTTPS proxy (the same settings works with the same extension in Chrome, and it did work before Firefox 114)

1. I turned on logging as requested
2. I tried opening a website which should be served through the proxy
3. I waited ~30 sec until the page timed out.

Let me know if there is anything more I can do to help.

Comment 8

[Valentin Gosu [:valentin] (he/him)]

I tried installing mitmproxy and pointing foxyproxy to it (without setting any authorization).
Pretty much everything stopped working. Even disabling the rule in foxyproxy prevented the user authentication dialog from popping up. I haven't dug any further but this happens even in Fx 110 - so not exactly a new issue.
I have shared the logs with Dana, but there doesn't seem to be any pipnss or certverifier message in the logs.
Ondrej, are you sure you set the logging modules as indicated in comment 6?

Comment 9

[ondrej.machulda]

(In reply to Valentin Gosu [:valentin] (he/him) from comment #8)

Ondrej, are you sure you set the logging modules as indicated in comment 6?

I may have use the default settings, sorry. I just e-mailed a new log, it now contains some pipnss and certverifier messages, so hopefully it will be useful now.

I tried installing mitmproxy and pointing foxyproxy to it (without setting any authorization).

The way it worked for me and colleagues in our company for years is using HTTPS proxy in FoxyProxy, and storing user credentials in addon settings. And if I remember correctly, then there was no pop-up when using the proxy, it just somehow internally provided the basic auth and worked. But I'd be happy with pop-up as well :).

Comment 10

[Valentin Gosu [:valentin] (he/him)]

https://drive.google.com/file/d/1_GoihuReO4YfIQ4CW0JDz76fAvgTW0Sg/view?usp=drive_link

Dana, could you take a look?

Comment 11

[Dana Keeler (she/her) [:keeler]]

That log seems to indicate that Firefox can't find any client authentication certificates to use, so it doesn't present one to the server and continues without one. I take it that's not the expected configuration?

Comment 12

[ondrej.machulda]

(In reply to Dana Keeler (she/her) (use needinfo) (:keeler for reviews) from comment #11)

That log seems to indicate that Firefox can't find any client authentication certificates to use, so it doesn't present one to the server and continues without one. I take it that's not the expected configuration?

No, the connection should not use client authentication certificate. There is just the basic authentication for the proxy.

Comment 13

[Kershaw Chang [:kershaw]]

(In reply to ondrej.machulda from comment #12)

(In reply to Dana Keeler (she/her) (use needinfo) (:keeler for reviews) from comment #11)

That log seems to indicate that Firefox can't find any client authentication certificates to use, so it doesn't present one to the server and continues without one. I take it that's not the expected configuration?

No, the connection should not use client authentication certificate. There is just the basic authentication for the proxy.

The log shows that the basic authentication for the proxy works.

2023-09-13 10:43:51.548685 UTC - [Parent 3247940: Socket Thread]: I/nsHttp http response [
2023-09-13 10:43:51.548692 UTC - [Parent 3247940: Socket Thread]: E/nsHttp   HTTP/1.0 200 Connection Established
2023-09-13 10:43:51.548695 UTC - [Parent 3247940: Socket Thread]: E/nsHttp   Proxy-agent: Apache
2023-09-13 10:43:51.548696 UTC - [Parent 3247940: Socket Thread]: E/nsHttp     OriginalHeaders
2023-09-13 10:43:51.548698 UTC - [Parent 3247940: Socket Thread]: E/nsHttp   Proxy-agent: Apache
2023-09-13 10:43:51.548700 UTC - [Parent 3247940: Socket Thread]: I/nsHttp ]

The problem happens during setting up the tunnel.

2023-09-13 10:44:22.436098 UTC - [Parent 3247940: Socket Thread]: V/nsHttp canceling transaction: tls handshake takes too long: tls handshake last 30946ms, timeout is 30000ms.
2023-09-13 10:44:22.436102 UTC - [Parent 3247940: Socket Thread]: V/nsHttp nsHttpConnection::CloseTransaction[this=7f2cde7c1d00 trans=7f2cdcbdf500 reason=804b000e]

For some reason, the connection was closed due to timeout.

Comment 14

[edgul]

It looks like a wireshark capture might be helpful with this, ondrej are you able to provide?

Comment 15

[ondrej.machulda]

Hi, I made two captures, one with Firefox (where the request via proxy times out), second one with Chrome (where the request via the same proxy works) and e-mailed them to [e-mail].

The behavior is different, though, as Chrome first (even before I started the capture, right after I enabled the proxy) pop-up basic auth window asking for proxy username and password, while Firefox won't (no matter if the password is set in FoxyProxy or is kept empty).

Just for the record - the previous behavior prior Firefox 114 was, that the password from FoxyProxy was being used (and there was no basic auth pop-up).

Comment 16

[Kershaw Chang [:kershaw]]

I think I've reproduced this locally with the following steps:

1. Setup a local proxy (I was using this one).
2. Configure Firefox to use the local proxy.
3. Install a certificate.
4. Visit https://client.badssl.com/
5. The connection was timeout (same as the log shown in comment #10).

Here is the log:

[Parent 65871: Socket Thread]: V/nsHttp proxy CONNECT succeeded! endtoendssl=1 onlyconnect=0
[Parent 65871: Socket Thread]: V/nsHttp 143484d00 new TLSFilterTransaction client.badssl.com 443
[Parent 65871: Socket Thread]: V/nsHttp nsHttpConnection 143484d00 SetupSecondaryTLS client.badssl.com 443
[Parent 65871: Socket Thread]: V/nsHttp TLSTransportLayer ctor this=[14ba31fc0]
[Parent 65871: Socket Thread]: V/nsHttp TLSTransportLayer::Init this=[14ba31fc0]
[Parent 65871: Socket Thread]: D/pipnss [14a3741c0] nsSSLIOLayerSetOptions: using TLS version range (0x0303,0x0304)
[Parent 65871: Socket Thread]: D/pipnss [14a3741c0] nsSSLIOLayerSetOptions: enabling TLS ECH Grease
[Parent 65871: Socket Thread]: D/pipnss [14a3741c0] Socket set up

[Parent 65871: Socket Thread]: D/pipnss [14a374a30] SSLGetClientAuthDataHook
[Parent 65871: Socket Thread]: D/pipnss FindClientCertificatesWithPrivateKeys
[Parent 65871: Socket Thread]: D/pipnss   module 'NSS Internal PKCS #11 Module'
[Parent 65871: Socket Thread]: D/pipnss     slot 'NSS Internal Cryptographic Services'
[Parent 65871: Socket Thread]: D/pipnss     (looking at non-internal slot)
[Parent 65871: Socket Thread]: D/pipnss     slot 'NSS User Private Key and Certificate Services'
[Parent 65871: Socket Thread]: D/pipnss     (looking at internal/builtin slot)
[Parent 65871: Socket Thread]: D/pipnss       provisionally adding 'CN=BadSSL Client Certificate,O=BadSSL,L=San Francisco,ST=California,C=US'
[Parent 65871: Socket Thread]: D/pipnss   module 'Builtin Roots Module'
[Parent 65871: Socket Thread]: D/pipnss     slot 'NSS Builtin Objects'
[Parent 65871: Socket Thread]: D/pipnss     (looking at internal/builtin slot)
[Parent 65871: Socket Thread]: D/pipnss       (no private keys)
[Parent 65871: Socket Thread]: D/pipnss   module 'OS Client Cert Module'
[Parent 65871: Socket Thread]: D/pipnss     slot 'OS Client Cert Slot'
[Parent 65871: Socket Thread]: D/pipnss     (looking at non-internal slot)
[Parent 65871: Socket Thread]: D/pipnss       provisionally adding 'C=DE,CN=Kershaw Chang'
[Parent 65871: Socket Thread]: D/pipnss       provisionally adding 'C=DE,CN=Kershaw Chang’s CA'
[Parent 65871: Socket Thread]: D/pipnss   returning:
[Parent 65871: Socket Thread]: D/pipnss     CN=BadSSL Client Certificate,O=BadSSL,L=San Francisco,ST=California,C=US
[Parent 65871: Socket Thread]: D/pipnss [14a3741c0] setting pending select client auth certificate

According to the log, it looks like mPendingSelectClientAuthCertificate was set, but never dispatched.

Dana, could you also take a look?
Thanks.

Comment 17

[Dana Keeler (she/her) [:keeler]]

What I'm seeing is it sets a pending event to select a client auth certificate, which doesn't run because certificate verification is pending. When certificate verification succeeds, necko is supposed to continue the connection (poll it again?), but it never does.

Comment 18

[edgul]

Seems like something we can act on, Kershaw can you verify that the P/S and priority-new is appropriate, thanks!

Comment 19

[Kershaw Chang [:kershaw]]

(In reply to Dana Keeler (she/her) (use needinfo) (:keeler for reviews) from comment #17)

What I'm seeing is it sets a pending event to select a client auth certificate, which doesn't run because certificate verification is pending. When certificate verification succeeds, necko is supposed to continue the connection (poll it again?), but it never does.

Actually, we did call Poll, but the poll flag was set to 0 here.
Then, the poll in lower level, which is TLSTransportLayer::Poll was not called, and the poll function of the real socket was never called. In the end, we timeout.

[Parent 4140: Socket Thread]: D/pipnss [10e6caa10] setting pending select client auth certificate
[Parent 4140: Socket Thread]: V/nsHttp nsHttpConnection::OnSocketReadable 14a5f9000 return due to inactive tunnel setup but incomplete NPN state
[Parent 4140: Socket Thread]: V/nsHttp nsHttpConnection::ResumeRecv [this=14a5f9000]
[Parent 4140: Socket Thread]: V/pipnss [10e6caa10] read -1 bytes
[Parent 4140: Socket Thread]: V/nsHttp TLSTransportLayer::InputStreamWrapper::AsyncWait [this=12661b3a8, callback=14a5f90f0]
[Parent 4140: Socket Thread]: V/pipnss [10e6caa10] polling SSL socket during certificate verification using lower 5
[Parent 4140: Socket Thread]: V/pipnss [10e6caa10] poll SSL socket returned 0

I also tried to let necko know when the server cert verification is done by calling a callback here. At this point, I tried to call poll again and I can see the client cert selection dialog shown. However, after I click the OK button, we are stuck again. I expect NSSSocketControl::SetHandshakeCompleted to be called, but it didn't.

Dana, I think I need some help. Do you maybe have an idea here?
Thanks.

Comment 20

[Dana Keeler (she/her) [:keeler]]

NSSSocketControl::SetHandshakeCompleted won't be called until the handshake has been completed, which at least requires both certificate validation and client auth certificate selection to be completed, and possibly also a poll operation, or maybe by driving the handshake? (NSSSocketControl::DriveHandshake)

Maybe we need to poll again when the client auth selection has been completed? (NSSSocketControl::ClientAuthCertificateSelected)

Comment 21

[Kershaw Chang [:kershaw]]

Attachment: Bug 1851570 - Allow necko to know when client auth is selected to drive TLS handshake, r=#necko

Comment 22

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/7e5a4e647b2c
Allow necko to know when client auth is selected to drive TLS handshake, r=necko-reviewers,keeler,valentin

Comment 23

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/7e5a4e647b2c

Comment 24

[BugBot [:suhaib / :marco/ :calixte]]

The patch landed in nightly and beta is affected.
:kershaw, is this bug important enough to require an uplift?

• If yes, please nominate the patch for beta approval.
• If no, please set status-firefox121 to wontfix.

For more information, please visit BugBot documentation.

Comment 25

[Ryan VanderMeulen [:RyanVM]]

Is this something we should consider backporting to ESR? It grafts mostly cleanly (just a minor test manifest conflict). Please nominate if yes.

Comment 26

[Kershaw Chang [:kershaw]]

(In reply to Ryan VanderMeulen [:RyanVM] from comment #25)

Is this something we should consider backporting to ESR? It grafts mostly cleanly (just a minor test manifest conflict). Please nominate if yes.

Yes, I think it's a good idea to uplift this. Thanks.

Comment 27

[Kershaw Chang [:kershaw]]

Attachment: Bug 1851570 - [for esr] Allow necko to know when client auth is selected to drive TLS handshake, r=#necko

Comment 28

[Kershaw Chang [:kershaw]]

Comment on attachment 9379961 [details]
Bug 1851570 - [for esr] Allow necko to know when client auth is selected to drive TLS handshake, r=#necko

ESR Uplift Approval Request

• If this is not a sec:{high,crit} bug, please state case for ESR consideration: Proxy which requires authentication won't work when the server asks for client certificate.
• User impact if declined: Same as above.
• Fix Landed on Version: 122
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): Should be low, since it's already landed a while ago and no regression.

Comment 29

[Dianna Smith [:diannaS]]

Comment on attachment 9379961 [details]
Bug 1851570 - [for esr] Allow necko to know when client auth is selected to drive TLS handshake, r=#necko

Approved for 115.9esr

Comment 30

[Pulsebot]

https://hg.mozilla.org/releases/mozilla-esr115/rev/a64b95dfc24a

Comment 31

[Dianna Smith [:diannaS]]

Backed out of esr115 for causing xpcshell failures
https://hg.mozilla.org/releases/mozilla-esr115/rev/04cbfcf89188fddd5b11141dfe653385648c3e0d

As this is a result of not having bug 1401466 in esr, kershaw will attempt to provide a test fix before uplifting this

Comment 32

[Kershaw Chang [:kershaw]]

https://treeherder.mozilla.org/jobs?repo=try&revision=c26818648ebd74f72e72105bf9b695fa6c05e77a

Comment 33

[Kershaw Chang [:kershaw]]

(In reply to Dianna Smith [:diannaS] from comment #31)

Backed out of esr115 for causing xpcshell failures
https://hg.mozilla.org/releases/mozilla-esr115/rev/04cbfcf89188fddd5b11141dfe653385648c3e0d

As this is a result of not having bug 1401466 in esr, kershaw will attempt to provide a test fix before uplifting this

The test failure is fixed. Could you try to uplift this again?
Thanks.

Comment 34

[Pulsebot]

https://hg.mozilla.org/releases/mozilla-esr115/rev/cf90e0fe4b07