1321466 - XHR/fetch failed over HTTPS-proxy

Status: VERIFIED FIXED

(Core :: Networking: HTTP, defect, P2)

Description

[Dmitry A. Bakshaev]

User Agent: Mozilla/5.0 (X11; Linux i686) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36

Steps to reproduce:

1. configure firefox to use HTTPS-proxy (https://bugzilla.mozilla.org/show_bug.cgi?id=378637)
2. start firefox -jsconsole https://jsfiddle.net/kaihendry/k36yot0z/
3. click "Just testing" button


Actual results:

1. no result.
2. jsconsole: <a id="errorCode" title="SSL_ERROR_HANDSHAKE_FAILURE_ALERT">SSL_ERROR_HANDSHAKE_FAILURE_ALERT</a>
3. devtools network monitor: "The connection used to fetch this resource was not encrypted"


Expected results:

json-serialized response from "https://httpbin.org/get"

Comment 1

[Dmitry A. Bakshaev]

reproduced on firefox 45.xESR 50.x, seamonkey 2.35-2.38.

chromium --proxy-server=https://myproxy:3129 https://jsfiddle.net/kaihendry/k36yot0z/ works as expected.

direct "open" url https://httpbin.org/get works, only XMLHttpRequest/fetch affected.

also works on non-HTTPS port on same proxy-server.

Comment 2

[Michael Downey]

Cannot reproduce on Fedora 25, Firefox Nightly 53.0a1 -- got expected JSON response from httpbin.org. Can you confirm you're still experiencing this on the latest Firefox Nightly version?

Comment 3

[Dmitry A. Bakshaev]

Attachment: screen-2016-12-06-13-37-26.png

screenshot with devtools network-monitor on
Mozilla/5.0 (X11; Linux i686; rv:53.0) Gecko/20100101 Firefox/53.0

Comment 4

[Dmitry A. Bakshaev]

problem reproduced on:
Mozilla/5.0 (X11; Linux i686; rv:53.0) Gecko/20100101 Firefox/53.0
(firefox-53.0a1.en-US.linux-i686.tar.bz2)

additional details:

proxy-server is squid-3.3.12 configured to request user x509-certificate to access proxy:
https_port myproxy:3129 cert=/etc/ssl/squid/myproxy.pem clientca=/etc/ssl/certs/ourCAbundle.crt

also squid configured to authentificate user using http-md5-digest scheme.

firefox configured to using HTTPS-proxy by "wpad-file", specified manually in firefox preferences or via MCD (Mission Control Desktop) settings.

Comment 5

[roxleitan]

Mozilla/5.0 (X11; Linux x86_64; rv:53.0) Gecko/20100101 Firefox/53.0
Build ID: 20161205030204

Hi Dmitry,

I have tested this issue on Ubuntu 15.10 x64 using the latest Nightly (20161205030204)and Firefox 45.0 esr , with proxy " https_port myproxy:3129 cert=/etc/ssl/squid/myproxy.pem clientca=/etc/ssl/certs/ourCAbundle.crt " and could not reproduce it, got expected JSON response from https://httpbin.org/get.

Could you please provide a screenshot with the Firefox Connection Settings?

Also please check if the issue is reproducible using Firefox with a new profile or in safe mode?

https://support.mozilla.org/en-US/kb/profile-manager-create-and-remove-firefox-profiles

https://support.mozilla.org/en-US/kb/troubleshoot-firefox-issues-using-safe-mode.

Thanks.

Comment 6

[Dmitry A. Bakshaev]

tested again with latest squid (3.5.22)
and Mozilla/5.0 (X11; Linux i686; rv:53.0) Gecko/20100101 Firefox/53.0

firefox --jsconsole --devtools --ProfileManager https://jsfiddle.net/kaihendry/k36yot0z/

1. create new firefox profile
2. install personal certificate (needed to connect to proxy)
3. point firefox "Automatic proxy configuration URL:" to http://myproxy/wpad.dat (for example)
4. wpat.dat contains: function FindProxyForURL(url, host) { return "HTTPS myproxy:3129"; }
5. squid.conf: https_port myproxy:3129 cert=/etc/ssl/squid/myproxy.pem clientca=/etc/ssl/certs/ourCAbundle.crt
6. press 'Just testing' on https://jsfiddle.net/kaihendry/k36yot0z/ page
7. network-monitor: "The connection used to fetch this resource was not encrypted"
8. browser console:
An error occurred during a connection to myproxy:3129.
SSL peer was unable to negotiate an acceptable set of security parameters.
Error code: <a id="errorCode" title="SSL_ERROR_HANDSHAKE_FAILURE_ALERT">SSL_ERROR_HANDSHAKE_FAILURE_ALERT</a>

all works fine if "clientca" options removed from squid configuration.
http://www.squid-cache.org/Versions/v3/3.5/cfgman/https_port.html
"  clientca=	File containing the list of CAs to use when
			requesting a client certificate."

simple test.
1. with "clientca" squid say:
openssl s_client -state -connect myproxy:3129 </dev/null
"...
Acceptable client certificate CA names <-- WE NEED THIS
...
Client Certificate Types: RSA sign, DSA sign, ECDSA sign"

2. without "clientca" sqiud not request client certs:
openssl s_client -state -connect myproxy:3129 </dev/null
"...
No client certificate CA names sent"

3. also no problem on non-HTTPS proxy-port (3128).
 
"proxy over HTTPS" not same as "ssl proxy" or "Use this proxy server for all protocols" through "HTTP proxy" settings in firefox.

Comment 7

[Dmitry A. Bakshaev]

Attachment: firefox proxy settings screenshot

Comment 8

[Dmitry A. Bakshaev]

firefox --safe-mode https://jsfiddle.net/kaihendry/k36yot0z/
not displayed correctly.

Comment 9

[Dragana Damjanovic [:dragana]]

I will take a look.

Comment 10

[Dragana Damjanovic [:dragana]]

Can you make http log:
https://developer.mozilla.org/en-US/docs/Mozilla/Debugging/HTTP_logging

only change the line:
export MOZ_LOG=timestamp,rotate:200,nsHttp:5,nsSocketTransport:5,nsStreamPump:5,nsHostResolver:5
to be:
export MOZ_LOG=timestamp,rotate:200,nsHttp:5,nsSocketTransport:5,nsStreamPump:5,nsHostResolver:5,proxy:5

And try to reproduce your problem. Also fetch https://httpbin.org/get directly not using xhr or fetch. 

Which version of firefox are using? for the older ones MOZ_LOG should be change to NSPR_LOG_MODULES and MOZ_LOG_FILE with NSPR_LOG_FILE

Comment 11

[Dmitry A. Bakshaev]

test1:
1 run
MOZ_LOG=timestamp,rotate:200,nsHttp:5,nsSocketTransport:5,nsStreamPump:5,nsHostResolver:5,proxy:5 MOZ_LOG_FILE=test1.log ./firefox --jsconsole --devtools --ProfileManager https://jsfiddle.net/kaihendry/k36yot0z/
2 press "Just testing"
3 quit

test2:
1 MOZ_LOG=timestamp,rotate:200,nsHttp:5,nsSocketTransport:5,nsStreamPump:5,nsHostResolver:5,proxy:5 MOZ_LOG_FILE=test2.log ./firefox --jsconsole --devtools --ProfileManager https://httpbin.org/get
2 quit

Comment 12

[Dmitry A. Bakshaev]

Attachment: test1.log

Comment 13

[Dmitry A. Bakshaev]

Attachment: test2.log

Comment 14

[Dmitry A. Bakshaev]

tested on:
[App]
Vendor=Mozilla
Name=Firefox
RemotingName=firefox
CodeName=Nightly
Version=53.0a1
BuildID=20170115030210
SourceRepository=https://hg.mozilla.org/mozilla-central
SourceStamp=5ce3882eec21be3a70e4afc050959ca2f76bfa76
ID={ec8030f7-c20a-464f-9b0e-13a3a9e97384}

Comment 15

[Matthew Francis]

I have also experienced what I believe to be this issue.
HTTPS proxies seem to have been hardly used in anger, and there isn't much coherent information available on how to set one up. The following from-scratch configuration is intended to allow the bug to be reproduced quickly using Squid.

Note: This is not an example of how to set up an HTTPS proxy securely. Here be dragons, etc.
- It doesn't set up a full CA configuration / directory structure for OpenSSL, so the generated certificates are probably a bit on the minimal side
- All password protection of certificates is deliberately omitted
- Further configuration of Squid and the network environment would be required to make this even remotely suitable for exposure to the internet

Tested on a clean base install of Fedora 25, plus Squid (Fedora package) and Firefox nightly (BuildID=20170120225344)


1. Generate the required certificates (as an unprivileged user):


# Generate CA certificate
openssl req -new -newkey rsa:2048 -nodes -x509 -keyout myca.pem -out myca.pem -days 1000 -subj "/C=GB/ST=Nowhere/L=Anywhere/O=Nonesuch CA/SN=Nonesuch CA"

# Generate client certificate
openssl req -new -newkey rsa:2048 -nodes -keyout client.key -out client.csr -subj "/C=GB/ST=Nowhere/L=Anywhere/SN=Joe Bloggs"
openssl x509 -req -in client.csr -CA myca.pem -CAkey myca.pem -out client.crt -days 1000 -set_serial 01
openssl pkcs12 -export -clcerts -in client.crt -inkey client.key -out client.p12 -passout pass:

# Generate server certificate
openssl req -new -newkey rsa:2048 -nodes -keyout serverkey.pem -out servercert.csr -days 1000 -subj "/C=GB/ST=Nowhere/L=Anywhere/O=Nonesuch CA/CN=localhost"
openssl x509 -req -in servercert.csr -CA myca.pem -CAkey myca.pem -out servercert.pem -days 1000 -set_serial 02



2. Set up Squid (as root):


mkdir /etc/squid/mycert
cp servercert.pem serverkey.pem myca.pem /etc/squid/mycert/
echo "https_port 8080 cert=/etc/squid/mycert/servercert.pem key=/etc/squid/mycert/serverkey.pem cafile=/etc/squid/mycert/myca.pem clientca=/etc/squid/mycert/myca.pem sslcontext=id" >> /etc/squid/squid.conf
service squid start



3. Configure Firefox:

 > Smoke test 1: check https://localhost:8080/ is inaccessible (SSL error)

 > Install the certificates in Firefox (Preferences -> Advanced -> Certificates -> View Certificates)
 - Install the CA certificate "myca.pem" (The "Authorities" tab; select "Trust this CA to identify web sites")
 - Install the client certificate "client.p12" (The "Your Certificates" tab; blank password)

 > Smoke test 2: check https://localhost:8080/ is now accessible and yields a Squid error page (you will be prompted to select the client certificate installed above)

 > In Preferences -> Advanced -> Network -> Connection -> Settings, select "Automatic proxy configuration URL", and set the value to:
     data:text/plain,function FindProxyForURL(u,h) { return "HTTPS localhost:8080" }


 > Smoke test 3: load a remote site and check that it appears in /var/log/squid/access.log
 (At this point you should no longer be able to access https://localhost:8080/ directly)


 > Actual test 1 (fail):
 Go to https://jsfiddle.net/kaihendry/k36yot0z/ as above, and click the button

 > Actual test 2 (pass):
 Set the network connection back to "No proxy", and retry the above test site

Comment 16

[Patrick McManus [:mcmanus]]

Is the summary of this bug "trouble authenticating to HTTPS proxy with Client Certificate"?

I wouldn't be surprised if client certs weren't currently supported.

Comment 17

[Dmitry A. Bakshaev]

HTTPS proxy with Client Certificate works as expected, only XHR/fetch on firefox affected.

Comment 18

[Dmitry A. Bakshaev]

retested with same results on:
[App]
Vendor=Mozilla
Name=Firefox
RemotingName=firefox
Version=53.0
BuildID=20170307144748
SourceRepository=https://hg.mozilla.org/releases/mozilla-beta
SourceStamp=2b89485cf946cdfdd69e5d11aa6af2563fa7520d
ID={ec8030f7-c20a-464f-9b0e-13a3a9e97384}

Comment 19

[Dragana Damjanovic [:dragana]]

This one is failing:

2017-01-15 18:23:06.101675 UTC - [Socket Thread]: V/nsHttp nsHttpConnection::EnsureNPNComplete 83461100 [TSA....httpbin.org:443 (https:myproxy:3129)[:]] negotiated to ''
2017-01-15 18:23:06.101681 UTC - [Socket Thread]: V/nsHttp nsHttpConnection::EnsureNPNComplete setting complete to true
2017-01-15 18:23:06.101683 UTC - [Socket Thread]: V/nsHttp   writing CONNECT request stream
2017-01-15 18:23:06.101687 UTC - [Socket Thread]: D/nsSocketTransport nsSocketOutputStream::Write [this=8360d9e0 count=193]
2017-01-15 18:23:06.101689 UTC - [Socket Thread]: D/nsSocketTransport   calling PR_Write [count=193]
2017-01-15 18:23:06.103730 UTC - [Socket Thread]: D/nsSocketTransport   PR_Write returned [n=-1]
2017-01-15 18:23:06.103739 UTC - [Socket Thread]: D/nsSocketTransport JIMB: ReleaseFD_Locked: mFDref = 2
2017-01-15 18:23:06.103742 UTC - [Socket Thread]: D/nsSocketTransport ErrorAccordingToNSPR [in=-12227 out=805a2fc3]

This one works fine:
2017-01-15 18:23:39.825963 UTC - [Socket Thread]: V/nsHttp nsHttpConnection::EnsureNPNComplete 89084400 [TS.....httpbin.org:443 (https:myproxy:3129)[:]] negotiated to ''
2017-01-15 18:23:39.825968 UTC - [Socket Thread]: V/nsHttp nsHttpConnection::EnsureNPNComplete setting complete to true
2017-01-15 18:23:39.825971 UTC - [Socket Thread]: V/nsHttp   writing CONNECT request stream
2017-01-15 18:23:39.825974 UTC - [Socket Thread]: D/nsSocketTransport nsSocketOutputStream::Write [this=88a80de0 count=193]
2017-01-15 18:23:39.825976 UTC - [Socket Thread]: D/nsSocketTransport   calling PR_Write [count=193]
2017-01-15 18:23:39.826247 UTC - [Socket Thread]: D/nsSocketTransport   PR_Write returned [n=193]
2017-01-15 18:23:39.826251 UTC - [Socket Thread]: D/nsSocketTransport JIMB: ReleaseFD_Locked: mFDref = 2


I do not see any difference between this two, only that one is anonymous and the other not. But I do not know how that can produce this error. Keeler, can you take a look?

Comment 20

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

If I set MOZ_LOG to just "proxy:5" and then send a simple xhr, I see a ton of this:

[ProxyResolution]: D/proxy Use proxy from PAC: HTTPS localhost:8080
[Main Thread]: D/proxy pac thread callback HTTPS localhost:8080

along with:

1502131498.613      0 ::1 NONE/000 0 NONE error:transaction-end-before-headers - HIER_NONE/- -

in the squid log. I think this is an issue in the proxy auto-config code.

(Note that in a debug build, I had to comment out the MOZ_ASSERT(false) at nsNSSIOLayer.cpp:1181)

Comment 21

[Dmitry A. Bakshaev]

I tried 3 different method to configure HTTPS-proxy:
1 auto-config url: wpad.dat with function FindProxyForURL ...
2 auto-config url: data:text/plain,function FindProxyForURL ...
3 using FoxyProxy mentioned https://bugzilla.mozilla.org/show_bug.cgi?id=378637

all with same results: xhr not working.

squid log:
2017/08/09 23:59:27 kid1| Error negotiating SSL connection on FD 37: error:140890C7:SSL routines:ssl3_get_client_certificate:peer did not return a certificate (1/-1)

seems like a problem on the firefox side?

if "clientca=..." removed from "https_port" all works fine,
but squid does not require client certificate from browser - it's not what i wanted.

Comment 22

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

hmmm - I don't think we send client certificates with XHRs, so maybe there's a bug where we don't send one to the proxy to authenticate?

Comment 23

[Felix Uellendall]

(In reply to Dmitry A. Bakshaev from comment #17)
> HTTPS proxy with Client Certificate works as expected, only XHR/fetch on
> firefox affected.

Same for me. But I also want to note here that it does not work in IE 11, but in Chrome.
So in Firefox it is a "SSL_ERROR_HANDSHAKE_FAILURE_ALERT". In IE 11 it might be the same issue?! In IE 11 I get a "XMLHttpRequest: Network Error 0x2ee4" (https://developer.microsoft.com/en-us/microsoft-edge/platform/issues/245775/).

Comment 24

[Firefox Bug Husbandry Bot]

Bulk change to priority: https://bugzilla.mozilla.org/show_bug.cgi?id=1399258

Comment 25

[Shian-Yow Wu [:swu]]

Ni? Gary who probably have some ideas about the issue.

Comment 26

[Gary Chen [:xeonchen]]

(In reply to David Keeler [:keeler] (use needinfo?) from comment #22)
> hmmm - I don't think we send client certificates with XHRs, so maybe there's
> a bug where we don't send one to the proxy to authenticate?

Probably Patrick knows more about this?

Comment 27

[Dragana Damjanovic [:dragana]]

I think this is the place where we disable client certs for anonymous connection, like xhr.

We could probably enable this for proxy authentication even if a channel is anonymous. 

What do you think?

Comment 28

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

That's probably a reasonable solution.

Comment 29

[Patrick McManus [:mcmanus]]

dragana's suggestion seems fine.. but I actually didn't think we did client certs for proxy authentication.. maybe I'm just wrong about that

Comment 30

[Dragana Damjanovic [:dragana]]

(In reply to Patrick McManus [:mcmanus] from comment #29)
> dragana's suggestion seems fine.. but I actually didn't think we did client
> certs for proxy authentication.. maybe I'm just wrong about that

Client certs for proxy authentication are working for non anonymous connection - see comment 11. The second test works and it is connecting to a proxy using client certs.

Comment 31

[Junior [inactive]]

Attachment: Bug 1321466 - provide client cert when using proxy with SSL

Comment 32

[Junior [inactive]]

Will ask for a review after verification

Comment 33

[Junior [inactive]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=bd9cd48405173e5056c039f0cd48ffe54d554776

Comment 34

[Junior [inactive]]

Per comment 15, I can reproduce this issue.

However, the console shows
"Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://httpbin.org/get. (Reason: CORS request did not succeed)." which indicates there's another issue we need to solve (either test code or browser code)

Comment 35

[Junior [inactive]]

The error shown in console triggered from a unrelated PR error SSL_ERROR_HANDSHAKE_FAILURE_ALERT, which indicates attachment 9025187 [details] not working

Comment 36

[Junior [inactive]]

 https://treeherder.mozilla.org/#/jobs?repo=try&revision=a72e37096ea9328195656c5fef221df74041ddea

Comment 37

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/d834c0fe16e5
provide client cert when using proxy with SSL r=keeler

Comment 38

[Razvan Maries]

https://hg.mozilla.org/mozilla-central/rev/d834c0fe16e5

Comment 39

[Dmitry A. Bakshaev]

CodeName=Firefox Nightly
Version=65.0a1
BuildID=20181115100051
SourceRepository=https://hg.mozilla.org/mozilla-central
SourceStamp=dca9c72df68bb59692118595dea65b78cde5b371
ID={ec8030f7-c20a-464f-9b0e-13a3a9e97384}

everything works fine
thank you all

Comment 40

[Gabi Cheta [:Gabi] Desktop QA]

Verified as fixed with Firefox 65.0, no "SSL_ERROR_HANDSHAKE_FAILURE_ALERT" error in browser console with proxy " https_port myproxy:3129 cert=/etc/ssl/squid/myproxy.pem clientca=/etc/ssl/certs/ourCAbundle.crt.