75865 - HTTPS doesn't work through proxy with the new PSM 2.0

Status: VERIFIED FIXED

(Core :: Networking, defect, P3)

Description

[Tom Mraz]

I've entered this as new bug because the bug 31174 shouldn't be reopened again.

I'm behind a Squid proxy and the Mozilla browser doesn't correctly issue CONNECT
request for https sites behind the proxy.

Comment 1

[Tom Mraz]

Attachment: Screenshot of the proxy error message

Comment 2

[Chris Hiner]

This seems to be caused by the same thing as bug 74326.  It sends "CONNECT /" to
the proxy instead of "CONNECT actualhost.example.com:443".

Comment 3

[Chris Hiner]

*** This bug has been confirmed by popular vote. ***

Comment 4

[Chris Hiner]

I just tried it again using build 2001041610 on Linux, and it does the same
thing.  I was wrong in my last comment, it sends "GET / HTTP/1.1" to the proxy,
instead of the proper CONNECT.

Comment 5

[Niko Pavlicek]

*** Bug 76266 has been marked as a duplicate of this bug. ***

Comment 6

[Niko Pavlicek]

i can confirm this, here my comments from the dupe bug 76266:

build id: 2001041604; win32-installer

When I try to access a secure site, the throbber begins to "throb", the status
line reports "Resolving xyz.abc" and then - after under one second - Mozilla
stops and says "Document: Done".
It doesn't hang or crash, just stops connecting to the page.
I use a ssl-proxy (no problems with NS4.x using the same proxy).

Reproducable with all https-sites I tried (freemail.de, amazon.de,
puretec.de, mail.yahoo.com).

-- So I don't get an error message, but I think this is because my proxy doesn't
sends a one for this problem.

Comment 7

[Niko Pavlicek]

so it is os -> all!

Comment 8

[John Unruh]

Milestone 2.0

Comment 9

[Niko Pavlicek]

priority P3? milestone 2.0?

Many users won't be able to use mozilla as their full-time browser if they can't
surf secure sites! So this is 

1) nsDogFood or at least nsCatFood
2) priority -> P1
3) target milestone -> M18 to indicate the pressure

It's worth the work - especially if it's only changing the "CONNECT /" command
to a "GET/ HTTP/1.1" commmand when the user connects through a proxy!

Comment 10

[Niko Pavlicek]

oh, another reason for the priority and milesone changes and the keywords:

this bug has got 4 votes in only 6 days!

Comment 11

[Bob Lord]

PSM 2.0 is the version of PSM in beta.  It isn't far off in the future.  Please
see the schedule here:
http://www.mozilla.org/projects/security/pki/psm/plan_20.html

Comment 12

[Niko Pavlicek]

oh, sorry - my failure: I didn't realize that PSM has its own milestones and
expected the version is the Mozilla version, since there are M16 - M20 in the
list. I was a bit confused...
but IMO it is still nsCatFood ;-)

Comment 13

[olaf]

this is becoming mostfreq - just found two more dups just from querying for "proxy".

Comment 14

[Fabian Guisset]

*** Bug 77026 has been marked as a duplicate of this bug. ***

Comment 15

[Fabian Guisset]

*** Bug 74326 has been marked as a duplicate of this bug. ***

Comment 16

[Fabian Guisset]

*** Bug 74963 has been marked as a duplicate of this bug. ***

Comment 17

[Ben Guthrie]

add cc

Comment 18

[John Unruh]

*** Bug 74864 has been marked as a duplicate of this bug. ***

Comment 19

[quan_tony]

After bug 31174, it is frustrating to see this problem appear once again.  Is 
testing Mozilla with proxies part of regular QA for the product?

Comment 20

[Nelson Bolyard (seldom reads bugmail)]

Hmmm.  I just grepped through psm and manager sources on the trunk
for the string SSL_SetSockPeerID and didn't find it.  That calls 
exists explicitly to help support proxy tunneling.  If it's not 
being called ...

Maybe this function _is_ being called in code on some branch?
If I could get read access to the raw rcs files, it would be 
easy to check.

Comment 21

[Asa Dotzler [:asa]]

benc, any chance you can help us track this down in time for 0.9?  

Comment 22

[Brian Ryner (not reading)]

This was definitely working at one point, because I tested the proxy code before
checking it in.  I'll try to figure out what's going on.

Comment 23

[Brian Ryner (not reading)]

CC'ing darin and gagan

What seems to be happening here is that the socket's reuse count gets
initialized to 1 (in nsHTTPHandler::RequestTransport), but then
nsHTTPRequest::formBuffer sees this and decides that means it has already
completed the SSL proxy connect.  So, this step never happens, and things don't
work.

So as far as I can tell, this is really nothing to do with PSM.

Comment 24

[Brendan Eich [:brendan]]

Reassigning -- darin, can you help for 0.9?  Thanks.

/be

Comment 25

[Darin Fisher]

ok... i think i can come up with a patch to fix this.

Comment 26

[Nelson Bolyard (seldom reads bugmail)]

There may be more than one problem here under the broad heading of 
"doesn't work through proxy".  

I think SSL session restart is negatively impacted if the application
is using a tunnel but doesn't call SSL_SetSockPeerID().

Comment 27

[Darin Fisher]

Attachment: should fix the problem

Comment 28

[Darin Fisher]

the patch i submitted should fix the problem.  back when i landed changes to
cleanup socket lifetime referencing, i didn't realize that the SSL proxy logic
was using the socket reuse count.  my changes required the HTTP handler to
increment the reuse connection before giving it out to be used for a HTTP 
transaction.  then if the socket was not to be reused, the handler would decrement
the reuse count.  this was done to ensure that the socket stayed alive for the
period between AsyncWrite completion and the start of AsyncRead.

so, because of this change, it is necessary for the SSL proxy logic to check if
the reuse count is greater than 1 not 0.

i haven't checked this patch b/c i don't have a PSM2 build, but if someone
who does could kindly test it, i'd appreciate it.  i'm 99% sure it will fix
the problem.

Comment 29

[Brian Ryner (not reading)]

Darin, this does fix the necko problem, thanks.

As nelsonb points out, we also need to be calling SSL_SetSockPeerID for PSM2 to
function correctly with proxy servers.  I'm attaching a patch that does this.

Comment 30

[Brian Ryner (not reading)]

Attachment: PSM patch to use SSL_SetSockPeerID

Comment 31

[Brian Ryner (not reading)]

Aside from the obvious addition of SSL_SetSockPeerID, this patch also makes sure
to call NSS functions on the SSL layer instead of PSM's new IO layer.  The old
code had made some assumptions about what NSS did that are not always valid.

Comment 32

[Javier Delgadillo]

r=javi for PSM changes

Comment 33

[Brendan Eich [:brendan]]

darin: how about a comment talking about the reuse count, maybe citing this bug,
definitely mentioning SSL proxying?

Anyone had time to test both patches and find them good for all cases?

/be

Comment 34

[Darin Fisher]

Attachment: adds comments to HTTP patch

Comment 35

[benc]

darin: The last day's conversations sound like (to me) this is a networking, not
PSM bug. If so, let me know, I can take QA of this bug, load the right build and
verify.

asa: It seems like connect breaks a lot where it "de-volves" to a GET request.
Identification of this proxy failures probably needs some work, to make problems
more rapid and transparent.

Comment 36

[rpotts (gone)]

sr=rpotts for HTTP patch...

Comment 37

[John Unruh]

*** Bug 76492 has been marked as a duplicate of this bug. ***

Comment 38

[neeti]

*** Bug 77839 has been marked as a duplicate of this bug. ***

Comment 39

[rpotts (gone)]

sr=rpotts for PSM patch

thanks, Brian for enlightening me about what's really going on there... :-)

Comment 40

[Mike Shaver (:shaver -- probably not reading bugmail closely)]

These patches (32189 and 32225) don't work for me, when applied to the 0.9 branch.

With an SSL proxy selected in my prefs, Mozilla sends a request like:

GET / HTTP/1.1
Host: sourceforge.net
User-Agent: Mozilla/5.0 (X11; U; Linux 2.2.16-22 i686; en-US; rv:0.8.1+)
Gecko/20010427
Accept: */*
Accept-Language: en
Accept-Encoding: gzip,deflate,compress,identity
Accept-Charset: ISO-8859-1, utf-8; q=0.667, *; q=0.667
Keep-Alive: 300
Connection: keep-alive

for https://sourceforge.net

Obviously Not Correct.

Comment 41

[Darin Fisher]

you needed to apply the patches: 32225 and 32307 ... not 32189

Comment 42

[Brian Ryner (not reading)]

This works fine for me, and Asa has given approval to check into the 0.9 branch.

Comment 43

[Brian Ryner (not reading)]

Checked in on both the trunk and branch.

Comment 44

[John Unruh]

Worksforme with the 4/30 WinNT Netscape 6 build.

Comment 45

[vectro]

Looks good with 2001043008 Linux (latest trunk build).

Comment 46

[Ben Guthrie]

Windows NT Version 4.0 ( Build 1381: Service pack 4)

Mozilla Build ID: 2001043004

Just worked for me on Amazon.com via the proxy at w@rk

Comment 47

[Asa Dotzler [:asa]]

I tested connecting to https://www.sourceforge.net and logging in via SSL
through benc's proxy server and it worked!  Thanks guys.

Tested with trunk win32 build 2001043004 on win2K with a new profile configured
to make ssl connections through a proxy server.

Comment 48

[Woodrow Hill]

Works for me on hotmail.com and accessing https://www.springmail.com.
Verra fast, too. Thanks to everyone!

Comment 49

[John Unruh]

Works on Mac too. Verified.